Capabilities of AI Transformers#
Background#
Whether GPT, ChatGPT, DALL-E, Whisper, Satablity AI or whatever significant you see in the AI worlds nowdays it is because of Transformer Architecture. Transformers are a type of neural network architecture that have several properties that make them effective for modeling data with long-range dependencies. They generally feature a combination of multi-headed attention mechanisms, residual connections, layer normalization, feedforward connections, and positional embeddings.
Precursors of Transformers were RNN, LSTM, and GRU architecture. Transformers are based on the 2017 research paper “Attention is All You Need”
Initially, Transformers were used for NLP-related tasks. Slowly researchers started exploring the power of the Transformer Architectures and as of 2023 these are used for hundreds of tasks in different AI domains of technologies like:
- Text Models (NLP, NLU, NLG)
- Vision Models (Computer Vision)
- Audio Models (Audio Processing, Classification, Audio Generation)
- Reinforcement (RL) Models
- Time-series Models
- Multimodal: OCR (extract information from scanned documents), video classification, visual QA, table data question answering
- Graph Models
Starting the journey in 2017, as of now (2023) we have approx 200 Transformer based architectures proposed by various researchers for various purposes. Using these architecture and various benchmark datasets thousands of models have been created which give SOTA performance on various tasks. Based on your need you choose which architecture can help you meet your project objective. There are high chances you will get some pre-trained models which you can use without training (Zero-shot) or small finetuning (one-shot or few-shot) efforts. For that you need to explore Huggingface and PaperWithCode
This articles list all the major Transformer related researcher paper, their object, and capabilities.
Note : Name starting with * are not Transformers, most of them are pretransformer age architectures.
Capabilities of AI Transformers#
| Sno | Transformer | Objective | Summary | NLP Tasks | CV Tasks |
|---|---|---|---|---|---|
| 1 | *AlexNet | Image Classification | A deep convolutional neural network architecture for image classification tasks. | - | Image Classification, Object Detection |
| 2 | *VGG16 | Visual Geometry Group Network (16 layers) | A deep CNN model with 16 convolutional layers developed by the Visual Geometry Group at Oxford University. | - | Image Classification, Object Detection |
| 3 | *VGG19 | Visual Geometry Group Network (19 layers) | A deep CNN model with 19 convolutional layers, an extended version of VGG16. | - | Image Classification, Object Detection |
| 4 | *ResNet | Residual Networks | A deep CNN architecture that introduces residual connections to alleviate the vanishing gradient problem. | - | Image Classification, Object Detection |
| 5 | *InceptionResNet | Combination of Inception and ResNet | A hybrid CNN model that combines the strengths of the Inception and ResNet architectures. | - | Image Classification, Object Detection |
| 6 | *ConvNeXt | Improved Convolutional Neural Network | A convolutional neural network architecture that aims to capture richer spatial relationships in images. | - | Image Classification, Object Detection |
| 7 | *DenseNet | Dense Connections in Convolutional Networks | A densely connected convolutional neural network architecture that encourages feature reuse and reduces the number of parameters. | - | Image Classification, Object Detection |
| 8 | *MobileNetV1 | Mobile-oriented CNN Architecture | A lightweight convolutional neural network architecture designed for mobile and embedded devices. | - | Image Classification, Object Detection |
| 9 | *Xception | Extreme Inception | A deep CNN architecture that replaces the standard Inception modules with depthwise separable convolutions. | - | Image Classification, Object Detection |
| 10 | EncoderDecoder | Sequence-to-sequence modeling | A transformer-based model architecture that combines encoder and decoder for sequence-to-sequence tasks such as machine translation. | Machine Translation, Text Summarization | - |
| 11 | *MobileNetV2 | Improved MobileNet Architecture | An enhanced version of MobileNet with improved performance and efficiency. | - | Image Classification, Object Detection |
| 12 | Data2Vec | Embedding data tables | A transformer-based model for embedding and encoding structured data tables. | Tabular Data Embedding, Data Table Encoding | - |
| 13 | GPT | Language modeling and text generation | A transformer-based model trained on a large corpus to generate coherent and contextually relevant text. | Text Generation, Text Completion, Language Modeling | - |
| 14 | BERT | Pre-training and fine-tuning on various NLP tasks | A transformer-based model widely used for pre-training and fine-tuning on NLP tasks. | Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 15 | MarianMT | Multilingual Neural Machine Translation | A multilingual neural machine translation model based on the Marian framework. | Machine Translation | - |
| 16 | BiT | Vision transformer for image classification | A vision transformer model pre-trained on large-scale datasets for image classification tasks. | - | Image Classification, Object Detection, Semantic Segmentation |
| 17 | Transformer-XL | Transformer model with extended context | A transformer model architecture that extends the context window, enabling longer-range dependencies. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 18 | XLM | Cross-lingual Language Model | A transformer-based model for cross-lingual language understanding and machine translation. | Cross-lingual Language Understanding, Machine Translation | - |
| 19 | CTRL | Text generation with control codes | A transformer-based model that allows fine-grained control over generated text using control codes. | Text Generation, Controlled Text Generation | - |
| 20 | GPT-2 | Language modeling and text generation | A transformer-based model similar to GPT but with a smaller architecture, trained on a large corpus to generate coherent and contextually relevant text. | Text Generation, Text Completion, Language Modeling | - |
| 21 | Funnel Transformer | Improving the efficiency and effectiveness of transformers | A transformer-based model architecture that reduces the computational cost of transformers while maintaining their effectiveness. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 22 | *EfficientNet B0 | Efficient and Scalable CNN | A family of convolutional neural network architectures that achieve high accuracy with fewer parameters and computations. | - | Image Classification, Object Detection |
| 23 | ALBERT | Improve the efficiency of BERT | A lite version of BERT that uses parameter reduction techniques to achieve faster training and lower memory consumption. | Classification, Translation, Named Entity Recognition (NER) | - |
| 24 | EfficientNet | Efficient convolutional neural network architecture | A convolutional neural network architecture that achieves state-of-the-art performance with significantly fewer parameters. | - | Image Classification, Object Detection, Semantic Segmentation |
| 25 | MobileNetV3 | Efficient Mobile Neural Network for Computer Vision | A lightweight and efficient neural network architecture designed for computer vision tasks on mobile devices. | Image Classification, Object Detection, Semantic Segmentation | - |
| 26 | Nezha | Neural Encoder for Zero-shot Transfer Learning | A transformer-based model that enables zero-shot transfer learning by learning a shared semantic space. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 27 | BART | Text generation and summarization | A denoising autoencoder model that can be used for text generation and summarization tasks. | Text Generation, Summarization | - |
| 28 | ERNIE | Enhanced representation through knowledge integration | A transformer-based model that enhances representation learning by integrating external knowledge sources. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 29 | ErnieM | Enhanced representation through multitask learning | A multitask learning framework that enhances representation learning by jointly training multiple downstream NLP tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 30 | FlauBERT | French language representation learning | A transformer-based model specifically trained for French language representation learning tasks. | French Language Processing, Text Classification | - |
| 31 | LXMERT | Vision and Language Multimodal Transformer | A multimodal transformer model that combines vision and language information for various tasks. | Visual Question Answering (VQA), Visual Dialog, Image Captioning, Visual Grounding | - |
| 32 | Pegasus | Pre-training with Extracted Gap Sentences for Abstractive Summarization | A transformer-based model trained for abstractive text summarization tasks. | Text Summarization | - |
| 33 | XLNet | Generalized Autoregressive Pretraining | A transformer-based model that leverages permutation-based training to learn bidirectional context. | Language Modeling, Text Classification | - |
| 34 | BioGpt | Processing biomedical text | A variant of the GPT model specifically designed for processing biomedical text. | Biomedical Text Processing, Named Entity Recognition (NER), Clinical Text Understanding | - |
| 35 | Hubert | Automatic speech recognition with transformers | A transformer-based model designed for automatic speech recognition tasks. | Automatic Speech Recognition | - |
| 36 | REALM | Retrieval-Augmented Language Model | A language model augmented with a dense retrieval mechanism to improve performance on text retrieval tasks. | Information Retrieval, Text Classification, Question Answering (QA) | - |
| 37 | SpeechToTextTransformer | Transformer for Speech-to-Text Conversion | A transformer-based model designed specifically for speech-to-text conversion tasks. | Speech-to-Text Conversion | - |
| 38 | XLM-V | Cross-lingual Language Understanding | A transformer-based model for cross-lingual language understanding, leveraging multilingual embeddings. | Cross-lingual Language Understanding | - |
| 39 | RoBERTa | Robustly optimized BERT variant | An optimized variant of BERT (Bidirectional Encoder Representations from Transformers) for various NLP tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 40 | GPT Neo | Efficient and scalable variant of GPT | A transformer-based model architecture that provides an efficient and scalable variant of GPT for various natural language processing tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 41 | CamemBERT | French language processing and text classification | A transformer-based model specifically trained for French language processing and text classification tasks. | French Language Processing, Text Classification | - |
| 42 | DialoGPT | Conversational AI chatbot | A transformer-based model trained for generating human-like conversational responses. | Conversational AI, Chatbot | - |
| 43 | DistilBERT | Distilled version of BERT | A smaller and faster version of BERT with a similar performance on various NLP tasks. | Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 44 | LiLT | Language learning from transliterated text | A transformer-based model for language learning that utilizes transliterated text as training data. | Language Learning | - |
| 45 | LUKE | Language Understanding with Knowledge-based Entities | A model that integrates knowledge-based entities into transformer-based language understanding tasks. | Named Entity Recognition (NER), Relation Extraction, Knowledge Graph Completion | - |
| 46 | MobileBERT | Efficient BERT for Mobile and Edge Devices | A compact and efficient version of BERT designed for deployment on mobile and edge devices. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 47 | MT5 | Multilingual Text-to-Text Transfer Transformer | A transformer-based model capable of multilingual text-to-text transfer learning across various NLP tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 48 | RAG | Retrieval-Augmented Generation | A model that combines retrieval and generation methods for open-domain question answering. | Open-Domain Question Answering | - |
| 49 | ConvBERT | Text classification and named entity recognition (NER) | A transformer-based model for text classification and named entity recognition (NER) tasks. | Classification, Named Entity Recognition (NER), Sentiment Analysis | - |
| 50 | Megatron-GPT2 | High-performance GPT-2-based language model | A high-performance GPT-2-based language model developed using the Megatron framework. | Text Generation, Text | - |
| 51 | PhoBERT | Pretrained language model for Vietnamese | A pretrained language model specifically designed for the Vietnamese language. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 52 | RoBERTa-PreLayerNorm | RoBERTa with PreLayerNorm | A variant of RoBERTa with the PreLayerNorm (PLN) technique, which improves training stability and efficiency. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 53 | BERTweet | Pre-trained BERT models for processing tweets | BERT models specifically trained on Twitter data for tweet processing tasks. | Classification, Named Entity Recognition (NER), Sentiment Analysis | - |
| 54 | mBART | Multilingual Denoising Autoencoder | A multilingual denoising autoencoder based on the BART framework, capable of generating text in multiple languages. | Text Generation, Text Completion, Multilingual Language Modeling | - |
| 55 | Megatron-BERT | High-performance BERT-based language model | A high-performance BERT-based language model developed using the Megatron framework. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 56 | SpeechToTextTransformer2 | Transformer model for Speech-to-Text Conversion | Another transformer-based model for speech-to-text conversion, providing an alternative approach. | Speech-to-Text Conversion | - |
| 57 | BERT For Sequence Generation | Text generation using BERT-based models | Fine-tuned BERT models for sequence generation tasks, such as text generation or summarization. | Text Generation, Summarization | - |
| 58 | ConvNeXT | Language modeling and text generation | A transformer-based model for language modeling and text generation tasks. | Language Modeling, Text Generation | - |
| 59 | ELECTRA | Pre-training method for language representation learning | A pre-training method that replaces masked language modeling with a generator-discriminator setup for better language representation. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 60 | Longformer | Long-range sequence modeling with transformers | A transformer-based model architecture that extends the standard transformer to handle long-range dependencies. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 61 | RegNet | Regularized Convolutional Neural Network | A convolutional neural network architecture with regularization techniques for efficient and scalable training. | - | Image Classification, Object Detection, Semantic Segmentation |
| 62 | SqueezeBERT | Lightweight BERT model with Squeeze-and-Excitation | A lightweight variant of BERT with Squeeze-and-Excitation (SE) blocks for efficient training and inference. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 63 | LayoutLM | Text and layout understanding for document analysis | A transformer-based model that combines text and layout information for document understanding tasks. | Document Understanding, OCR, Named Entity Recognition (NER) | - |
| 64 | MPNet | Megatron Pretrained Network | A model pretrained using the Megatron framework, designed for various NLP tasks with high performance. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 65 | VisualBERT | Integrating Visual Information with BERT | A BERT-based model that incorporates visual information for multimodal understanding. | - | Vision-Language Tasks, Image Captioning, Visual Question Answering (VQA) |
| 66 | Conditional DETR | Object detection and instance segmentation | A transformer-based model for object detection and instance segmentation tasks. | - | Object Detection, Instance Segmentation |
| 67 | GPTBigCode | Code generation for programming languages | A transformer-based model trained on a large corpus of code to generate code snippets or complete programs for various programming languages. | Code Generation, Programming Language Processing | - |
| 68 | M-CTC-T | Music Transcription with Transformer | A transformer-based model designed for music transcription, converting audio into musical notation. | Music Transcription | - |
| 69 | Pix2Struct | Image-to-Structure Translation | A transformer-based model for translating images into structured representations. | - | Image-to-Structure Translation |
| 70 | ProphetNet | Pretrained Sequence-to-Sequence Model | A sequence-to-sequence model pretrained for various NLP tasks, based on the transformer architecture. | Text Generation, Text Completion, Machine Translation, Summarization | - |
| 71 | SEW | Simple and Efficient Word-level language model | A word-level language model that is simple and efficient, designed for various NLP tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 72 | T5 | Text-to-Text Transfer Transformer | A text-to-text transfer transformer model that can be fine-tuned for various NLP tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 73 | DeBERTa | Improving the effectiveness of BERT | A transformer-based model that enhances BERT by addressing its limitations and improving performance on various NLP tasks. | Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 74 | Informer | Time series forecasting with transformers | A transformer-based model for time series forecasting tasks, capturing long-term dependencies in the data. | Time Series Forecasting | - |
| 75 | LED | Language model for efficient decoding | A transformer-based language model designed for efficient decoding, suitable for constrained environments. | Text Generation, Text Completion, Language Modeling | - |
| 76 | SwitchTransformers | Transformers with Dynamic Routing | A library that provides implementations of various transformer models with dynamic routing capabilities. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | Image Classification, Object Detection, Semantic Segmentation |
| 77 | Whisper | Unsupervised Representation Learning | A transformer-based model for unsupervised representation learning on audio data. | Speech Representation Learning | - |
| 78 | XLM-ProphetNet | Cross-lingual Language Generation | A transformer-based model for cross-lingual language generation, extending the ProphetNet architecture. | Cross-lingual Language Generation | - |
| 79 | XLM-RoBERTa | Cross-lingual Language Representation | A cross-lingual variant of RoBERTa, providing multilingual representation learning. | Cross-lingual Language Representation | - |
| 80 | Deformable DETR | Object detection and instance segmentation with deformable attention | A transformer-based model for object detection and instance segmentation tasks, incorporating deformable attention mechanisms. | - | Object Detection, Instance Segmentation |
| 81 | FNet | Image generation with Fourier features | A transformer-based model that generates images using Fourier features instead of traditional positional encodings. | - | Image Generation |
| 82 | GPTSAN-japanese | Japanese language variant of GPT for sentiment analysis | A version of GPT specifically designed and trained for sentiment analysis tasks in the Japanese language. | Japanese Language | |
| 83 | SEW-D | Deep version of Simple and Efficient Word-level language model | A deep variant of SEW for improved performance on NLP tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 84 | CPM | Chinese language processing and text generation | A transformer-based model specifically designed for Chinese language processing and text generation tasks. | Chinese Language Processing, Text Generation | - |
| 85 | GIT | Generating informative text from structured data | A transformer-based model that generates informative text, such as explanations or summaries, from structured data inputs. | Data-to-Text Generation, Structured Data Processing | - |
| 86 | LayoutXLM | Multilingual document understanding with transformers | A transformer-based model for multilingual document understanding, incorporating text and layout information. | Multilingual Document Understanding, OCR, Named Entity Recognition (NER) | - |
| 87 | DETR | Object detection and instance segmentation | A transformer-based model for object detection and instance segmentation tasks. | - | Object Detection, Instance Segmentation |
| 88 | GPT NeoX | Further improved version of GPT Neo | An advanced version of GPT Neo that incorporates additional enhancements and optimizations for natural language processing tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 89 | RemBERT | Transformer model for code | A transformer-based model specifically designed for code-related tasks, such as code generation and understanding. | Code Generation, Code Understanding | - |
| 90 | RoCBert | Robustly optimized Chinese BERT variant | A Chinese language variant of RoBERTa, optimized for various NLP tasks in Chinese text. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 91 | TAPAS | Table Parsing via Transformer | A transformer-based model designed for table parsing, enabling natural language queries over tabular data. | Table Parsing, Question Answering (QA) over Tabular Data | - |
| 92 | UPerNet | Unified Perceptual Parsing Network | A unified perceptual parsing network based on the transformer model, designed for image segmentation tasks. | Semantic Segmentation, Image Parsing | - |
| 93 | Vision Transformer (ViT) | Transformer-based model for image classification | A transformer-based model designed for image classification tasks, replacing convolutional layers with self-attention. | - | Image Classification, Object Detection, Semantic Segmentation |
| 94 | Wav2Vec2 | Self-supervised Audio Representation Learning | A transformer-based model for self-supervised audio representation learning, capturing phonetic information. | Speech Recognition, Speech Representation Learning | - |
| 95 | PLBart | Pre-trained Language model for BART | A pre-trained variant of BART (Bidirectional and AutoRegressive Transformers) for various NLP tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 96 | DiT | Vision transformer for image classification | A transformer-based model for image classification tasks that applies vision transformers to process image data. | - | Image Classification, Object Detection, Semantic Segmentation |
| 97 | DPR | Dense Passage Retrieval | A transformer-based model for dense passage retrieval, enabling efficient and accurate retrieval of relevant passages. | Passage Retrieval, Document Ranking | - |
| 98 | GLPN | Learning global-local patterns in natural language processing | A transformer-based model that captures both global and local patterns in text for various natural language processing tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 99 | LeViT | Vision transformer with less computations | A vision transformer model that reduces computational requirements by using fewer computations. | - | Image Classification, Object Detection, Semantic Segmentation |
| 100 | NAT | Neural Architecture Transformer | A transformer-based model that learns to design neural architectures for various tasks. | Neural Architecture Search, AutoML | - |
| 101 | TAPEX | Transformer model for text and program execution | A transformer-based model capable of executing programs described in natural language text. | Text-to-Program Execution, Natural Language Processing | - |
| 102 | VideoMAE | Video Motion Analysis Encoder | A transformer-based model for video motion analysis tasks, encoding motion information in videos. | - | Video Motion Analysis, Action Recognition, Video Understanding |
| 103 | Wav2Vec2-Conformer | Conformer-based variant of Wav2Vec2 | A variant of Wav2Vec2 that incorporates Conformer architecture, improving its performance on speech-related tasks. | Speech Recognition, Speech Representation Learning | - |
| 104 | CLIP | Image-text matching and zero-shot learning | A transformer-based model that learns to match images and text, enabling zero-shot learning capabilities. | - | Image-Text Matching, Zero-Shot Learning |
| 105 | XLS-R | Cross-lingual Speech Recognition | A transformer-based model for cross-lingual speech recognition, trained on multilingual speech data. | Cross-lingual Speech Recognition | - |
| 106 | Audio Spectrogram Transformer | Processing audio spectrograms | A transformer model specifically designed for processing audio spectrograms. | Automatic Speech Recognition (ASR), Sound Classification | - |
| 107 | M2M100 | Multilingual Multimodal Transformer | A transformer-based model capable of multilingual and multimodal tasks, trained on 100 different languages. | Machine Translation, Multilingual Text Classification, Multimodal Tasks | - |
| 108 | MEGA | Multilingual Language Generation with Transformers | A transformer-based model for multilingual language generation tasks, capable of producing text in multiple languages. | Text Generation, Text Completion, Multilingual Language Modeling | - |
| 109 | BEiT | Vision transformer for image classification | Combines concepts from CNNs and transformers for image classification tasks. | - | Image Classification, Object Detection, Semantic Segmentation |
| 110 | BigBird-Pegasus | Text generation and summarization | A variant of the Pegasus model that incorporates the BigBird sparse attention mechanism. | Text Generation, Summarization | - |
| 111 | BigBird-RoBERTa | Classification and named entity recognition | A variant of the RoBERTa model that incorporates the BigBird sparse attention mechanism. | Classification, Named Entity Recognition (NER) | - |
| 112 | CLIPSeg | Image segmentation | A transformer-based model for image segmentation tasks. | - | Image Segmentation |
| 113 | DPT | Object detection and instance segmentation with deformable attention | A transformer-based model for object detection and instance segmentation tasks, incorporating deformable attention mechanisms. | - | Object Detection, Instance Segmentation |
| 114 | Perceiver IO | Perceiver with Input/output processing | A transformer model architecture that handles input and output processing jointly, enabling cross-modal tasks. | Multimodal Tasks | - |
| 115 | Reformer | Memory-efficient Transformer | A transformer model variant designed to be more memory-efficient by using reversible layers. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 116 | RoFormer | Robustly optimized Transformer variant for images | A transformer-based model specifically designed for image-related tasks, leveraging self-attention mechanisms. | - | Image Classification, Object Detection, Semantic Segmentation |
| 117 | Swin Transformer | Shifted Window Transformer | A transformer model that uses shifted windows to capture long-range dependencies in images. | - | Image Classification, Object Detection |
| 118 | TrOCR | Transformer-based OCR model | A transformer-based model designed for Optical Character Recognition (OCR) tasks, converting images to text. | Optical Character Recognition (OCR) | - |
| 119 | Wav2Vec2Phoneme | Phoneme-level variants of Wav2Vec2 | Phoneme-level variants of Wav2Vec2 designed for speech recognition tasks at the phoneme level. | Phoneme-level Speech Recognition | - |
| 120 | X-CLIP | Cross-modal Learning with CLIP | A transformer-based model for cross-modal learning, incorporating the CLIP framework. | - | Vision-Language Tasks, Cross-modal Learning |
| 121 | XLSR-Wav2Vec2 | Cross-lingual Speech Representation | A variant of Wav2Vec2 trained for cross-lingual speech representation learning. | Cross-lingual Speech Representation | - |
| 122 | Blenderbot | Conversational AI chatbot | A chatbot model designed for multi-turn conversations that combines language and dialogue understanding. | - | - |
| 123 | BlenderbotSmall | Conversational AI chatbot | A smaller version of Blenderbot, designed for multi-turn conversations with language and dialogue understanding capabilities. | - | - |
| 124 | BLIP | Image classification and image captioning | A transformer-based model for image classification and image captioning tasks. | - | Image Classification, Image Captioning |
| 125 | ByT5 | Text translation, classification, and question answering | A transformer-based model trained on T5 architecture, suitable for text translation, classification, and question answering tasks. | Translation, Text Classification, Question Answering (QA) | - |
| 126 | CvT | Cross Vision and Transformer | A transformer-based model that combines vision and language understanding, enabling cross-modal tasks in computer vision. | - | Image-Text Matching, Vision-Language Tasks |
| 127 | DeBERTa-v2 | Improved version of DeBERTa | An updated version of DeBERTa with improved performance and compatibility for various NLP tasks. | Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 128 | DeiT | Vision transformer for image classification | A vision transformer model designed for image classification tasks. | - | Image Classification, Object Detection, Semantic Segmentation |
| 129 | GroupViT | Vision transformer with group-based operations | A vision transformer model that incorporates group-based operations to enhance its representation capacity. | - | Image Classification, Object Detection, Semantic Segmentation |
| 130 | LayoutLMv2 | Improved version of LayoutLM for document analysis | An enhanced version of LayoutLM with improved performance and additional capabilities for document analysis. | Document Understanding, OCR, Named Entity Recognition (NER) | - |
| 131 | MaskFormer | Masked Language Modeling with Transformers | A transformer-based model architecture for masked language modeling tasks, such as pretraining BERT. | Language Modeling, Pretraining BERT | - |
| 132 | SegFormer | Segmentation Transformer for computer vision | A transformer-based model designed for image segmentation tasks in computer vision. | Semantic Segmentation, Object Detection | - |
| 133 | Time Series Transformer | Transformer model for time series data | A transformer-based model designed specifically for time series data analysis and forecasting tasks. | Time Series Forecasting, Anomaly Detection, Sequence Modeling | - |
| 134 | TimeSformer | Time Series Transformer for video analysis | A transformer-based model for video analysis and action recognition tasks, leveraging temporal information. | - | Video Action Recognition, Temporal Modeling |
| 135 | Trajectory Transformer | Transformer model for trajectory forecasting | A transformer-based model designed for trajectory forecasting tasks, such as predicting object movement. | Trajectory Forecasting, Object Movement Prediction | - |
| 136 | UniSpeech | Unified Speech Recognition and Synthesis Transformer | A unified transformer-based model for both speech recognition and speech synthesis tasks. | Speech Recognition, Text-to-Speech Synthesis | - |
| 137 | UniSpeechSat | Self-supervised pre-training for UniSpeech | A self-supervised pre-training method for UniSpeech, improving its performance on speech-related tasks. | Speech Recognition, Text-to-Speech Synthesis | - |
| 138 | ALIGN | Joint representation learning for textual and tabular data | Enables joint representation learning by aligning textual and tabular data. | Text-Tabular Alignment, Joint Representation Learning | - |
| 139 | BORT | Language modeling and reinforcement learning | A transformer-based model for language modeling and reinforcement learning tasks. | Language Modeling, Text Generation | - |
| 140 | DePlot | Data visualization | A transformer-based model that generates interactive and informative visualizations from data. | Data Visualization | - |
| 141 | DETA | Document extraction and text analysis | A transformer-based model for document extraction, information retrieval, and text analysis tasks. | Document Extraction, Information Retrieval, Text Analysis | - |
| 142 | DiNAT | Network traffic anomaly detection | A transformer-based model for network traffic anomaly detection, specifically designed for cybersecurity applications. | Network Traffic Analysis, Anomaly Detection | - |
| 143 | Jukebox | Music generation with transformers | A transformer-based model architecture for generating music with various styles and genres. | Music Generation | - |
| 144 | mBART-50 | Compact version of mBART for resource-constrained | A compact version of mBART with reduced parameters and computational requirements. | Text Generation, Text Completion, Multilingual Language Modeling | - |
| 145 | Nyströmformer | Approximating Full Transformers with Nyström | A transformer variant that approximates full self-attention using the Nyström method for efficiency. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 146 | ViT Hybrid | Hybrid Architecture of Vision Transformer | A hybrid architecture that combines vision transformer with convolutional neural networks for image understanding. | - | Image Classification, Object Detection, Semantic Segmentation |
| 147 | X-MOD | Cross-modal Language Modeling | A transformer-based model for cross-modal language modeling, integrating vision and text. | - | Vision-Language Tasks, Cross-modal Language Modeling |
| 148 | BARTpho | Text-to-speech synthesis | A variant of BART model for text-to-speech synthesis tasks. | Text-to-Speech Synthesis | - |
| 149 | BridgeTower | Language modeling and text generation | A transformer-based model for language modeling and text generation tasks. | Text Generation, Language Modeling | - |
| 150 | CodeGen | Code generation | A transformer-based model for generating code. | Code Generation | - |
| 151 | GPT-J | Japanese language variant of GPT-2 | A version of GPT-2 specifically designed and trained for Japanese language understanding and generation tasks. | Japanese Language Processing, Text Generation | - |
| 152 | LLaMA | Label-agnostic learning with transformers | A transformer-based model that learns to perform tasks without explicit labels, leveraging self-supervision. | Self-Supervised Learning, Representation Learning, Clustering | - |
| 153 | MarkupLM | Transformer for document structure understanding | A transformer-based model for understanding document structure and semantic relationships in text. | Document Structure Understanding, Semantic Analysis | - |
| 154 | PoolFormer | Pooling-based Vision Transformer | A vision transformer model that incorporates pooling operations for handling images of varying sizes. | - | Image Classification, Object Detection, Semantic Segmentation |
| 155 | QDQBert | Query-Doc Bidirectional Transformer | A transformer model specifically designed for query-document ranking and retrieval tasks. | Information Retrieval, Question Answering, Document Ranking | - |
| 156 | ViLT | Vision-and-Language Transformer | A transformer-based model that combines vision and language understanding for multimodal tasks. | - | Vision-Language Tasks, Image Captioning, Visual Question Answering (VQA) |
| 157 | BARThez | Text generation and summarization | A variant of BART model trained specifically for the French language. | Text Generation, Summarization | - |
| 158 | Donut | Anomaly detection in time series data | A transformer-based model for detecting anomalies in time series data, suitable for various applications such as monitoring systems. | - | Anomaly Detection, Time Series Analysis |
| 159 | ImageGPT | Image generation with transformers | A transformer-based model architecture for generating images based on text prompts. | - | Image Generation |
| 160 | OPT | Optimization Pretraining Transformer | A transformer model pre-trained for optimization tasks, such as combinatorial optimization and planning. | Combinatorial Optimization, Planning | - |
| 161 | Splinter | Speech and Language Integrated Transformer | A transformer-based model designed for integrating speech and language tasks. | Speech-to-Text Conversion, Speech Recognition, Natural Language Processing | - |
| 162 | XGLM | Cross-lingual Language Modeling | A transformer-based model for cross-lingual language modeling, learning representations across languages. | Cross-lingual Language Modeling | - |
| 163 | YOSO | You Only Speak Once | A transformer-based model for low-resource machine translation, using only monolingual data. | Low-resource Machine Translation | - |
| 164 | EfficientFormer | Efficient transformer architecture for sequence modeling | A transformer-based model architecture designed to improve efficiency and performance for sequence modeling tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Machine Translation | - |
| 165 | ESM | Protein structure prediction | A transformer-based model for predicting the 3D structure of proteins from their amino acid sequences. | Protein Structure Prediction, Bioinformatics | - |
| 166 | Mask2Former | Transformer-based masked image inpainting | A transformer-based model for masked image inpainting, reconstructing missing parts of an image. | - | Image Inpainting |
| 167 | MGP-STR | Music Generation with Pre-trained Model | A pre-trained model for generating music, leveraging a transformer-based architecture. | Music Generation | - |
| 168 | NLLB | Natural Language Logic Board | A model that combines natural language understanding and symbolic logic reasoning for language understanding. | Natural Language Understanding, Logic Reasoning | - |
| 169 | T5v1.1 | Version 1.1 of the Text-to-Text Transfer Transformer | An updated version of the T5 model with improvements and enhancements for better performance. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 170 | TVLT | Tiny Vision-Language Transformer | A compact vision-language transformer model designed for efficient processing of vision and language inputs. | - | Vision-Language Tasks, Image Captioning, Visual Question Answering (VQA) |
| 171 | WavLM | Language Modeling for Speech | A transformer-based model for language modeling on speech data. | Speech Language Modeling | - |
| 172 | XLM-RoBERTa-XL | Cross-lingual Language Representation | A larger variant of XLM-RoBERTa for cross-lingual language representation learning. | Cross-lingual Language Representation | - |
| 173 | Chinese-CLIP | Chinese language processing and image-text matching | A transformer-based model designed for Chinese language processing and image-text matching tasks. | Chinese Language Processing, Image-Text Matching | - |
| 174 | CLAP | Image-text representation learning | A transformer-based model for learning joint image-text representations. | - | Image-Text Representation Learning |
| 175 | Decision Transformer | Decision-making tasks | A transformer-based model designed for decision-making tasks that require complex reasoning and inference. | Decision-Making, Reasoning, Inference | - |
| 176 | BLIP-2 | Image classification | An updated version of BLIP, specializing in image classification tasks. | - | Image Classification |
| 177 | CANINE | Document classification | A transformer-based model for document classification tasks. | Document Classification | - |
| 178 | Graphormer | Graph representation learning with transformers | A transformer-based model architecture specifically designed for graph representation learning. | Graph Representation Learning, Node Classification, Graph Classification, Graph Generation | - |
| 179 | I-BERT | Incremental learning with transformers | A transformer-based model architecture that supports incremental learning, allowing continual model updates. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 180 | MatCha | Matching Challenge Transformer | A transformer-based model for solving matching challenge tasks, such as natural language inference. | Natural Language Inference, Textual Entailment | - |
| 181 | mLUKE | Multilingual Language Understanding with Knowledge | A multilingual model that incorporates knowledge-based entities for language understanding tasks. | Named Entity Recognition (NER), Relation Extraction, Knowledge Graph Completion | - |
| 182 | MobileViT | Vision Transformer for Mobile and Edge Devices | A mobile-friendly version of Vision Transformer, optimized for efficient deployment on mobile and edge devices. | - | Image Classification, Object Detection, Semantic Segmentation |
| 183 | OWL-ViT | Object-Wide Learning Vision Transformer | A vision transformer model designed for object detection and recognition tasks in computer vision. | Object Detection, Object Recognition | - |
| 184 | SpeechT5 | T5-based model for Speech-to-Text | A transformer-based model trained for speech-to-text conversion tasks using the T5 architecture. | Speech-to-Text Conversion | - |
| 185 | Swin Transformer V2 | Advanced version of Swin Transformer | An advanced version of the Swin Transformer model, incorporating improvements for better performance in vision tasks. | - | Image Classification, Object Detection, Semantic Segmentation |
| 186 | ViTMAE | Vision Transformer for Multi-label Image Classification | A vision transformer model designed specifically for multi-label image classification tasks. | - | Multi-label Image Classification |
| 187 | BLOOM | Language modeling and text generation | A transformer-based model designed for language modeling and text generation tasks. | Text Generation, Language Modeling | - |
| 188 | ConvNeXTV2 | Language modeling and text generation | An improved version of ConvNeXT for language modeling and text generation tasks. | Language Modeling, Text Generation | - |
| 189 | CPM-Ant | Chinese language processing and text generation | An enhanced version of CPM with better performance and compatibility for Chinese language processing and text generation tasks. | Chinese Language Processing, Text Generation | - |
| 190 | GPT-Sw3 | Swedish language variant of GPT | A version of GPT specifically designed and trained for Swedish language understanding and generation tasks. | Swedish Language Processing, Text Generation | - |
| 191 | LongT5 | Text-to-Text Transfer Transformer | A transformer-based model for text-to-text transfer learning, capable of performing various NLP tasks. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 192 | OneFormer | Transformer for Text-to-Text Transfer Learning | A transformer-based model designed for text-to-text transfer learning tasks across multiple languages. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Question Answering (QA) | - |
| 193 | Table Transformer | Transformer model for table-related tasks | A transformer-based model specifically designed for table-related tasks, such as table understanding and extraction. | Table Understanding, Table Extraction | - |
| 194 | VAN | Vision-Adaptive Transformer for Video Analysis | A transformer model designed specifically for video analysis tasks, adapting to the dynamic visual context. | - | Video Classification, Action Recognition, Video Understanding |
| 195 | AltCLIP | Predicting the relationship between two images | A transformer-based model that learns to predict the relationship between two images. | - | Image-Text Matching, Vision-Language Tasks |
| 196 | MVP | Multimodal Variational Pretraining | A multimodal pretraining framework that combines text and image modalities for various downstream tasks. | Multimodal Tasks | - |
| 197 | NLLB-MOE | Natural Language Logic Board with MOE | An enhanced version of NLLB that incorporates Mixture of Experts (MOE) for improved performance. | Natural Language Understanding, Logic Reasoning | - |
| 198 | PEGASUS-X | Large-Scale Pre-training for Abstractive Summarization | A variant of Pegasus with larger model capacity, trained on a large-scale corpus for abstractive summarization. | Text Summarization | - |
| 199 | Swin2SR | Swin Transformer for Super-Resolution | A variant of the Swin Transformer model specifically designed for super-resolution tasks in computer vision. | - | Super-Resolution Image Reconstruction |
| 200 | UL2 | Unsupervised Language Learning | A transformer-based model designed for unsupervised language learning tasks, leveraging self-supervised learning techniques. | Language Modeling, Text Representation Learning | - |
| 201 | ViTMSN | Vision Transformer with Masked Spatial Neurons | A vision transformer model with masked spatial neurons, enabling better spatial representation learning. | - | Image Classification, Object Detection, Semantic Segmentation |
| 202 | YOLOS | You Only Learn One Sentence | A transformer-based model that learns sentence representations for zero-shot classification. | Zero-shot Text Classification | - |
| 203 | FLAN-T5 | Fast and lightweight adapter-based transformers for T5 | A transformer-based model architecture that enables efficient and lightweight adaptation of T5 models. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Machine Translation | - |
| 204 | GPT NeoX Japanese | Japanese language variant of GPT NeoX | A version of GPT NeoX specifically designed and trained for Japanese language understanding and generation tasks. | Japanese Language Processing, Text Generation | - |
| 205 | LayoutLMv3 | Further improved version of LayoutLM for documents | An advanced version of LayoutLM that incorporates additional enhancements and optimizations. | Document Understanding, OCR, Named Entity Recognition (NER) | - |
| 206 | FLAN-UL2 | Fast and lightweight adapter-based transformers for UL2 | A transformer-based model architecture that enables efficient and lightweight adaptation of UL2 models. | Text Classification, Named Entity Recognition (NER), Sentiment Analysis, Machine Translation | - |
| 207 | FLAVA | Fluency and acceptability evaluation for machine translation | A transformer-based model that evaluates the fluency and acceptability of machine translations. | Machine Translation Evaluation | - |
Conclusion#
The purpose of this article is to give you a general understanding of the capabilities of the Transformer architecture. It is now up to you to decide which architecture is most suitable to your needs based on the task you have in front of you. Afterwards, you can use hugginface or tfhub to see if there are already models that have been trained using these architectures. The chances are that you will be able to complete your work using zero-shot transfer learning are high.
Comments: