Complete Technical Paper Content — Multimodal Fusion Ai
1. Executive Abstract, Problem Formulation & Background
Multimodal Deep Learning 🎆 🎆 🎆 Announcing the multimodal deep learning repository that contains implementation of various deep learning-based models to solve different multimodal problems such as multimodal representation learning, multimodal fusion for downstream tasks e.g., multimodal sentiment analysis. For those enquiring about how to extract visual and audio features, please check this out: https://github.com/soujanyaporia/MUStARD Models Multimodal-Infomax (PyTorch) MISA (PyTorch) BBFN (PyTorch) Hfusion (Keras) contextual-attention-based-LSTM (Tensorflow) bc-LSTM (Keras) Contextual-Multimodal-Fusion (Keras) Tensor Fusion Network (PyTorch) Low-rank-Multimodal-Fusion (PyTorch) Datasets MELD MUStARD M2H2 Models Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis , accepted to EMNLP 2021 . 🔥 If you would be interested in other multimodal works in our DeCLaRe Lab, welcome to visit the clustered repository Introduction Multimodal-informax (MMIM) synthesizes fusion results from multi-modality input through a two-level mutual information (MI) maximization. We use BA (Barber-Agakov) lower bound and contrastive predictive coding as the target function to be maximized. To facilitate the computation, we design an entropy estimation module with associated history data memory to facilitate the computation of BA lower bound and the training process. Usage Download the CMU-MOSI and CMU-MOSEI dataset from Google Drive or Baidu Disk . Please them under the folder Multimodal-Infomax/datasets Set up the environment (need conda prerequisite) conda env create -f environment.yml conda activate MMIM Start training python main.py --dataset mosi --contrast Citation Please cite our paper if you find our work useful for your research: @article { han2021improving , title = { Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis } , author = { Han, Wei and Chen, Hui and Poria, Soujanya } , journal = { Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) } , year = { 2021 } } Contact Should you have any question, feel free to contact me through henryhan88888@gmail.com MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis Code for the ACM MM 2020 paper MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis Setup the environment We work with a conda environment. conda env create -f environment.yml conda activate misa-code Data Download Install CMU Multimodal SDK . Ensure, you can perform from mmsdk import mmdatasdk . Option 1: Download pre-computed splits and place the contents inside datasets folder. Option 2: Re-create splits by downloading data from MMSDK. For this, simply run the code as detailed next. Running the code cd src Set word_emb_path in config.py to glove file . Set sdk_dir to the path of CMU-MultimodalSDK. python train.py --data mosi . Replace mosi with mosei or ur_funny for other datasets. Citation If this paper is useful for your research, please cite us at: @article{hazarika2020misa, title={MISA: Modality-Invariant and-Specific Representations for Multimodal Sentiment Analysis}, author={Hazarika, Devamanyu and Zimmermann, Roger and Poria, Soujanya}, journal={arXiv preprint arXiv:2005.03545}, year={2020} } Contact For any questions, please email at hazarika@comp.nus.edu.sg Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis This repository contains official implementation of the paper: Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis (ICMI 2021) Model Architecture Overview of our Bi-Bimodal Fusion Network (BBFN). It learns two text-related pairs of representations, text-acoustic and text-visual by enforcing each pair of modalities to complement mutually. Finally, the four (two pairs) head representations are concatenated to generate the final prediction. A single complementation layer: two identical pipelines (left and right) propagate the main modality and fuse that with complementary modality with regularization and gated control. Results Results on the test set of CMU-MOSI and CMU-MOSEI dataset. Notation: △ indicates results in the corresponding line are excerpted from previous papers; † means the results are reproduced with publicly visible source code and applicable hyperparameter setting; ‡ shows the results have experienced paired t-test with 𝑝 < 0.05 and demonstrate significant improvement over MISA, the state-of-the-art model. Usage Set up conda environemnt conda env create -f environment.yml conda activate BBFN Install CMU Multimodal SDK Set sdk_dir in src/config.py to the path of CMU-MultimodalSDK Train the model cd src python main.py --dataset <dataset_name> --data_path <path_to_dataset> We provide a script scripts/run.sh for your reference. Citation Please cite our paper if you find our work useful for your research: @article { han2021bi , title = { Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis } , author = { Han, Wei and Chen, Hui and Gelbukh, Alexander and Zadeh, Amir and Morency, Louis-philippe and Poria, Soujanya } , journal = { ICMI 2021 } , year = { 2021 } } Contact Should you have any question, feel free to contact me through henryhan88888@gmail.com Hfusion Codes for the paper Multimodal sentiment analysis using hierarchical fusion with context modeling How to run python3 hfusion.py Requirements Keras >= 2.0, Tensorflow >= 1.7, Numpy, Scikit-learn Citation Majumder, N., Hazarika, D., Gelbukh, A., Cambria, E. and Poria, S., 2018. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowledge-Based Systems, 161, pp.124-133. Attention-based multimodal fusion for sentiment analysis Attention-based multimodal fusion for sentiment analysis Code for the paper Context-Dependent Sentiment Analysis in User-Generated Videos (ACL 2017). Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis (ICDM 2017). Preprocessing Edit: the create_data.py is obsolete. The pre-processed datasets have already been provided in the dataset/ folder in the repo. Use them directly. As data is typically present in utterance format, we combine all the utterances belonging to a video using the following code python create_data.py Note: This will create speaker independent train and test splits In dataset/mosei, extract the zip into a folder named 'raw'. Also, extract 'unimodal_mosei_3way.pickle.zip' Running the model Sample command: With attention-based fusion: python run.py --unimodal True --fusion True python run.py --unimodal False --fusion True Without attention-based and with concatenation-based fusion: python run.py --unimodal True --fusion False python run.py --unimodal False --fusion False Utterance level attention: python run.py --unimodal False --fusion True --attention_2 True python run.py --unimodal False --fusion True --attention_2 True Note: Keeping the unimodal flag as True (default False) shall train all unimodal lstms first (level 1 of the network mentioned in the paper) Setting --fusion True applies only to multimodal network. Datasets: We provide results on the MOSI , MOSEI and IEMOCAP datasets. Please cite the creators. We are adding more datasets, stay tuned. Use --data [mosi|mosei|iemocap] and --classes [2|3|6] in the above commands to test different configurations on different datasets. mosi: 2 classes mosei: 3 classes iemocap: 6 classes Example: python run.py --unimodal False --fusion True --attention_2 True --data mosei --classes 3 Dataset details MOSI: 2 classes: Positive/Negative Raw Features: (Pickle files) Audio: dataset/mosi/raw/audio_2way.pickle Text: dataset/mosi/raw/text_2way.pickle Video: dataset/mosi/raw/video_2way.pickle Each file contains: train_data, train_label, test_data, test_label, maxlen, train_length, test_length train_data - np.array of dim (62, 63, feature_dim) train_label - np.array of dim (62, 63, 2) test_data - np.array of dim (31, 63, feature_dim) test_label - np.array of dim (31, 63, 2) maxlen - max utterance length int of value 63 train_length - utterance length of each video in train data. test_length - utterance length of each video in test data. Train/Test split: 62/31 videos. Each video has utterances. The videos are padded to 63 utterances. IEMOCAP: 6 classes: happy/sad/neutral/angry/excited/frustrated Raw Features: dataset/iemocap/raw/IEMOCAP_features_raw.pkl (Pickle files) The file contains: videoIDs[vid] = List of utterance IDs in this video in the order of occurance videoSpeakers[vid] = List of speaker turns. e.g. [M, M, F, M, F]. here M = Male, F = Female videoText[vid] = List of textual features for each utterance in video vid. videoAudio[vid] = List of audio features for each utterance in video vid. videoVisual[vid] = List of visual features for each utterance in video vid. videoLabels[vid] = List of label indices for each utterance in video vid. videoSentence[vid] = List of sentences for each utterance in video vid. trainVid = List of videos (videos IDs) in train set. testVid = List of videos (videos IDs) in test set. Refer to the file dataset/iemocap/raw/loadIEMOCAP.py for more information. We use this data to create a speaker independent train and test splits in the format. (videos x utterances x features) Train/Test split: 120/31 videos. Each video has utterances. The videos are padded to 110 utterances. MOSEI: 3 classes: positive/negative/neutral Raw Features: (Pickle files) Audio: dataset/mosei/raw/audio_3way.pickle Text: dataset/mosei/raw/text_3way.pickle Video: dataset/mosei/raw/video_3way.pickle The file contains: train_data, train_label, test_data, test_label, maxlen, train_length, test_length train_data - np.array of dim (2250, 98, feature_dim) train_label - np.array of dim (62, 63, 2) test_data - np.array of dim (31, 63, feature_dim) test_label - np.array of dim (31, 63, 2) maxlen - max utterance length int of value 98 train_length - utterance length of each video in train data. test_length - utterance length of each video in test data. Train/Test split: 2250/678 videos. Each video has utterances. The videos are padded to 98 utterances. Citation If using this code, please cite our work using : @inproceedings{soujanyaacl17, title={Context-dependent sentiment analysis in user-generated videos}, author={Poria, Soujanya and Cambria, Erik and Hazarika, Devamanyu and Mazumder, Navonil and Zadeh, Amir and Morency, Louis-Philippe}, booktitle={Association for Computational Linguistics}, year={2017} } @inproceedings{poriaicdm17, author={S. Poria and E. Cambria and D. Hazarika and N. Mazumder and A. Zadeh and L. P. Morency}, booktitle={2017 IEEE International Conference on Data Mining (ICDM)}, title={Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis}, year={2017}, pages={1033-1038}, keywords={data mining;feature extraction;image classification;image fusion;learning (artificial intelligence);sentiment analysis;attention-based networks;context learning;contextual information;contextual multimodal sentiment;dynamic feature fusion;multilevel multiple attentions;multimodal sentiment analysis;recurrent model;utterances;videos;Context modeling;Feature extraction;Fuses;Sentiment analysis;Social network services;Videos;Visualization}, doi={10.1109/ICDM.2017.134}, month={Nov},} Credits Soujanya Poria Gangeshwar Krishnamurthy ( gangeshwark@gmail.com ; Github: @gangeshwark) Context-Dependent Sentiment Analysis in User Generated Videos Code for the paper Context-Dependent Sentiment Analysis in User-Generated Videos (ACL 2017). Requirements Code is written in Python (2.7) and requires Keras (2.0.6) with Theano backend. Description In this paper, we propose a LSTM-based model that enables utterances to capture contextual information from their surroundings in the same video, thus aiding the classification process in multimodal sentiment analysis. This repository contains the code for the mentioned paper. Each contextual LSTM (Figure 2 in the paper) is implemented as shown in above figure. For more details, please refer to the paper. Note: Unlike the paper, we haven't used an SVM on the penultimate layer. This is in effort to keep the whole network differentiable at some performance cost. Dataset We provide results on the MOSI dataset Please cite the creators Preprocessing As data is typically present in utterance format, we combine all the utterances belonging to a video using the following code python create_data.py Note: This will create speaker independent train and test splits Running sc-lstm Sample command: python lstm.py --unimodal True python lstm.py --unimodal False Note: Keeping the unimodal flag as True (default False) shall train all unimodal lstms first (level 1 of the network mentioned in the paper) Citation If using this code, please cite our work using : @inproceedings{soujanyaacl17, title={Context-dependent sentiment analysis in user-generated videos}, author={Poria, Soujanya and Cambria, Erik and Hazarika, Devamanyu and Mazumder, Navonil and Zadeh, Amir and Morency, Louis-Philippe}, booktitle={Association for Computational Linguistics}, year={2017} } Credits Devamanyu Hazarika, Soujanya Poria Contextual Inter-modal Attention for Multimodal Sentiment Analysis Code for the paper Contextual Inter-modal Attention for Multi-modal Sentiment Analysis (EMNLP 2018). Dataset We provide results on the MOSI dataset . Please cite the creators. Requirements: Python 3.5 Keras (Tensorflow backend) 2.2.4 Scikit-learn 0.20.0 Experiments python create_data.py python trimodal_attention_models.py Citation If you use this code in your research, please cite our work using: @inproceedings{ghosal2018contextual, title={Contextual Inter-modal Attention for Multi-modal Sentiment Analysis}, author={Ghosal, Deepanway and Akhtar, Md Shad and Chauhan, Dushyant and Poria, Soujanya and Ekbal, Asif and Bhattacharyya, Pushpak}, booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing}, pages={3454--3466}, year={2018} } Credits Some of the functionalities in this repo are borrowed from https://github.com/soujanyaporia/contextual-utterance-level-multimodal-sentiment-analysis Authors Deepanway Ghosal , Soujanya Poria Tensor Fusion Network (TFN) IMPORTANT NOTICE The CMU-MultimodalSDK on which this repo depend has drastically changed its API since this code is written. Hence the code in this repo cannot be run off-the-shelf anymore. However, the code for the model itself can still be of reference. Tensor Fusion Networks This is a PyTorch implementation of: Zadeh, Amir, et al. "Tensor fusion network for multimodal sentiment analysis." EMNLP 2017 Oral. It requires PyTorch and the CMU Multimodal Data SDK ( https://github.com/A2Zadeh/CMU-MultimodalDataSDK ) to function properly. The training data (CMU-MOSI dataset) will be automatically downloaded if you run the script for the first time. The model is defined in model.py , and the training script is train.py . Here's a list of commandline arguments for train.py : --dataset: default is 'MOSI', currently don't really support other datasets. Just ignore this option --epochs: max number of epochs, default is 50 --batch_size: batch size, default is 32 --patience: specifies the early stopping condition, similar to that in Keras, default 20 --cuda: whether or not to use GPU, default False --model_path: a string that specifies the location for storing trained models, default='models' --max_len: max sequence length when preprocessing data, default=20 In a nutshell, you can train the model using the following command: python train.py --epochs 100 --patience 10 The script starts with a randomly selected set of hyper-parameters. If you want to tune it, you can change them yourself in the script. Citation If you use this code in your research, please cite our work using: @inproceedings{tensoremnlp17, title={Tensor Fusion Network for Multimodal Sentiment Analysis}, author={Zadeh, Amir and Chen, Minghai and Poria, Soujanya and Cambria, Erik and Morency, Louis-Philippe}, booktitle={Empirical Methods in Natural Language Processing, EMNLP}, year={2017} } Low rank Multimodal Fusion This is the repository for "Efficient Low-rank Multimodal Fusion with Modality-Specific Factors", Liu and Shen, et. al. ACL 2018. Dependencies Python 2.7 (now experimentally has Python 3.6+ support) torch=0.3.1 sklearn numpy You can install the libraries via python -m pip install -r requirements.txt . Data for Experiments The processed data for the experiments (CMU-MOSI, IEMOCAP, POM) can be downloaded here: https://drive.google.com/open?id=1CixSaw3dpHESNG0CaCJV6KutdlANP_cr To run the code, you should download the pickled datasets and put them in the data directory. Note that there might be NaN values in acoustic features, you could replace them with 0s. Training Your Model To run the code for experiments (grid search), use the scripts train_xxx.py . They have some commandline arguments as listed here: `--run_id`: an user-specified unique ID to ensure that saved results/models don't override each other. `--epochs`: the number of maximum epochs in training. Since early-stopping is used to prevent overfitting, in actual training the number of epochs could be less than what you specify here. `--patience`: if the model performance does not improve in `--patience` many validation evaluations consecutively, the training will early-stop. `output_dim`: output dimension of the model. Default value in each script should work. `signiture`: an optional string that's added to the output file name. Intended to use as some sort of comment. `cuda`: whether or not to use GPU in training. If not specified, will use CPU. `data_path`: the path to the data directory. Defaults to './data', but if you prefer storing the data else where you can change this. `model_path`: the path to the directory where models will be saved. `output_path`: the path to the directory where the grid search results will be saved. `max_len`: the maximum length of training data sequences. Longer/shorter sequences will be truncated/padded. `emotion`: (exclusive for IEMOCAP) specifies which emotion category you want to train the model to predict. Can be 'happy', 'sad', 'angry', 'neutral'. An example would be python train_mosi.py --run_id 19260817 --epochs 50 --patience 20 --output_dim 1 --signiture test_run_big_model Hyperparameters Some hyper parameters for reproducing the results in the paper are in the hyperparams.txt file. Citation @misc{liu2018efficient, title={Efficient Low-rank Multimodal Fusion with Modality-Specific Factors}, author={Zhun Liu and Ying Shen and Varun Bharadhwaj Lakshminarasimhan and Paul Pu Liang and Amir Zadeh and Louis-Philippe Morency}, year={2018}, eprint={1806.00064}, archivePrefix={arXiv}, primaryClass={cs.AI} } Dataset MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversation Note 🔥 🔥 🔥 For updated baselines please visit this link: conv-emotion Leaderboard Updates 10/10/2020: New paper and SOTA in Emotion Recognition in Conversations on the MELD dataset. Refer to the directory COSMIC for the code. Read the paper -- COSMIC: COmmonSense knowledge for eMotion Identification in Conversations . 22/05/2019: MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversation has been accepted as a full paper at ACL 2019. The updated paper can be found here - https://arxiv.org/pdf/1810.02508.pdf 22/05/2019: Dyadic MELD has been released. It can be used to test dyadic conversational models. 15/11/2018: The problem in the train.tar.gz has been fixed. Research Works using MELD Zhang, Yazhou, Qiuchi Li, Dawei Song, Peng Zhang, and Panpan Wang. "Quantum-Inspired Interactive Networks for Conversational Sentiment Analysis." IJCAI 2019. Zhang, Dong, Liangqing Wu, Changlong Sun, Shoushan Li, Qiaoming Zhu, and Guodong Zhou. "Modeling both Context-and Speaker-Sensitive Dependence for Emotion Detection in Multi-speaker Conversations." IJCAI 2019. Ghosal, Deepanway, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. "DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation." EMNLP 2019. Introduction Multimodal EmotionLines Dataset (MELD) has been created by enhancing and extending EmotionLines dataset. MELD contains the same dialogue instances available in EmotionLines, but it also encompasses audio and visual modality along with text. MELD has more than 1400 dialogues and 13000 utterances from Friends TV series. Multiple speakers participated in the dialogues. Each utterance in a dialogue has been labeled by any of these seven emotions -- Anger, Disgust, Sadness, Joy, Neutral, Surprise and Fear. MELD also has sentiment (positive, negative and neutral) annotation for each utterance. Example Dialogue Dataset Statistics Statistics Train Dev Test # of modality {a,v,t} {a,v,t} {a,v,t} # of unique words 10,643 2,384 4,361 Avg. utterance length 8.03 7.99 8.28 Max. utterance length 69 37 45 Avg. # of emotions per dialogue 3.30 3.35 3.24 # of dialogues 1039 114 280 # of utterances 9989 1109 2610 # of speakers 260 47 100 # of emotion shift 4003 427 1003 Avg. duration of an utterance 3.59s 3.59s 3.58s Please visit https://affective-meld.github.io for more details. Dataset Distribution Train Dev Test Anger 1109 153 345 Disgust 271 22 68 Fear 268 40 50 Joy 1743 163 402 Neutral 4710 470 1256 Sadness 683 111 208 Surprise 1205 150 281 Purpose Multimodal data analysis exploits information from multiple-parallel data channels for decision making. With the rapid growth of AI, multimodal emotion recognition has gained a major research interest, primarily due to its potential applications in many challenging tasks, such as dialogue generation, multimodal interaction etc. A conversational emotion recognition system can be used to generate appropriate responses by analysing user emotions. Although there are numerous works carried out on multimodal emotion recognition, only a very few actually focus on understanding emotions in conversations. However, their work is limited only to dyadic conversation understanding and thus not scalable to emotion recognition in multi-party conversations having more than two participants. EmotionLines can be used as a resource for emotion recognition for text only, as it does not include data from other modalities such as visual and audio. At the same time, it should be noted that there is no multimodal multi-party conversational dataset available for emotion recognition research. In this work, we have extended, improved, and further developed EmotionLines dataset for the multimodal scenario. Emotion recognition in sequential turns has several challenges and context understanding is one of them. The emotion change and emotion flow in the sequence of turns in a dialogue make accurate context modelling a difficult task. In this dataset, as we have access to the multimodal data sources for each dialogue, we hypothesise that it will improve the context modelling thus benefiting the overall emotion recognition performance. This dataset can also be used to develop a multimodal affective dialogue system. IEMOCAP, SEMAINE are multimodal conversational datasets which contain emotion label for each utterance. However, these datasets are dyadic in nature, which justifies the importance of our Multimodal-EmotionLines dataset. The other publicly available multimodal emotion and sentiment recognition datasets are MOSEI, MOSI, MOUD. However, none of those datasets is conversational. Dataset Creation The first step deals with finding the timestamp of every utterance in each of the dialogues present in the EmotionLines dataset. To accomplish this, we crawled through the subtitle files of all the episodes which contains the beginning and the end timestamp of the utterances. This process enabled us to obtain season ID, episode ID, and timestamp of each utterance in the episode. We put two constraints whilst obtaining the timestamps: (a) timestamps of the utterances in a dialogue must be in increasing order, (b) all the utterances in a dialogue have to belong to the same episode and scene. Constraining with these two conditions revealed that in EmotionLines, a few dialogues consist of multiple natural dialogues. We filtered out those cases from the dataset. Because of this error correction step, in our case, we have the different number of dialogues as compare to the EmotionLines. After obtaining the timestamp of each utterance, we extracted their corresponding audio-visual clips from the source episode. Separately, we also took out the audio content from those video clips. Finally, the dataset contains visual, audio, and textual modality for each dialogue. Paper The paper explaining this dataset can be found - https://arxiv.org/pdf/1810.02508.pdf Download the data Please visit - http://web.eecs.umich.edu/~mihalcea/downloads/MELD.Raw.tar.gz to download the raw data. Data are stored in .mp4 format and can be found in XXX.tar.gz files. Annotations can be found in https://github.com/declare-lab/MELD/tree/master/data/MELD . Description of the .csv files Column Specification Column Name Description Sr No. Serial numbers of the utterances mainly for referencing the utterances in case of different versions or multiple copies with different subsets Utterance Individual utterances from EmotionLines as a string. Speaker Name of the speaker associated with the utterance. Emotion The emotion (neutral, joy, sadness, anger, surprise, fear, disgust) expressed by the speaker in the utterance. Sentiment The sentiment (positive, neutral, negative) expressed by the speaker in the utterance. Dialogue_ID The index of the dialogue starting from 0. Utterance_ID The index of the particular utterance in the dialogue starting from 0. Season The season no. of Friends TV Show to which a particular utterance belongs. Episode The episode no. of Friends TV Show in a particular season to which the utterance belongs. StartTime The starting time of the utterance in the given episode in the format 'hh:mm:ss,ms'. EndTime The ending time of the utterance in the given episode in the format 'hh:mm:ss,ms'. The files /data/MELD/train_sent_emo.csv - contains the utterances in the training set along with Sentiment and Emotion labels. /data/MELD/dev_sent_emo.csv - contains the utterances in the dev set along with Sentiment and Emotion labels. /data/MELD/test_sent_emo.csv - contains the utterances in the test set along with Sentiment and Emotion labels. /data/MELD_Dyadic/train_sent_emo_dya.csv - contains the utterances in the training set of the dyadic variant of MELD along with Sentiment and Emotion labels. For getting the video clip corresponding to a particular utterance refer to the columns 'Old_Dialogue_ID' and 'Old_Utterance_ID'. /data/MELD_Dyadic/dev_sent_emo_dya.csv - contains the utterances in the dev set of the dyadic variant along with Sentiment and Emotion labels. For getting the video clip corresponding to a particular utterance refer to the columns 'Old_Dialogue_ID' and 'Old_Utterance_ID'. /data/MELD_Dyadic/test_sent_emo_dya.csv - contains the utterances in the test set of the dyadic variant along with Sentiment and Emotion labels. For getting the video clip corresponding to a particular utterance refer to the columns 'Old_Dialogue_ID' and 'Old_Utterance_ID'. Description of Pickle Files There are 13 pickle files comprising of the data and features used for training the baseline models. Following is a brief description of each of the pickle files. Data pickle files: data_emotion.p, data_sentiment.p - These are the primary data files which contain 5 different elements stored as a list. data : It consists of a dictionary with the following key/value pairs. text : original sentence. split : train/val/test - denotes the which split the tuple belongs to. y : label of the sentence. dialog : ID of the dialog the utterance belongs to. utterance : utterance number of the dialog ID. num_words : number of words in the utterance. W: glove embedding matrix vocab: the vocabulary of the dataset word_idx_map: mapping of each word from vocab to its index in W. max_sentence_length: maximum number of tokens in an utterance in the dataset. label_index: mapping of each label (emotion or sentiment) to its assigned index, eg. label_index['neutral']=0 import pickle data , W , vocab , word_idx_map , max_sentence_length , label_index = pickle . load ( open ( filepath , 'rb' )) text_glove_average_emotion.pkl, text_glove_average_sentiment.pkl - It consists of 300 dimensional textual feature vectors of each utterance initialized as the average of the Glove embeddings of all tokens per utterance. It is a list comprising of 3 dictionaries for train, val and the test set with each dictionary indexed in the format dia_utt , where dia is the dialogue id and utt is the utterance id. For eg. train_text_avg_emb['0_0'].shape = (300, ) import pickle train_text_avg_emb , val_text_avg_emb , test_text_avg_emb = pickle . load ( open ( filepath , 'rb' )) audio_embeddings_feature_selection_emotion.pkl,audio_embeddings_feature_selection_sentiment.pkl - It consists of 1611/1422 dimensional audio feature vectors of each utterance trained for emotion/sentiment classification. These features are originally extracted from openSMILE and then followed by L2-based feature selection using SVM. It is a list comprising of 3 dictionaries for train, val and the test set with each dictionary indexed in the format dia_utt , where dia is the dialogue id and utt is the utterance id. For eg. train_audio_emb['0_0'].shape = (1611, ) or (1422, ) import pickle train_audio_emb , val_audio_emb , test_audio_emb = pickle . load ( open ( filepath , 'rb' )) Model output pickle files: text_glove_CNN_emotion.pkl, text_glove_CNN_sentiment.pkl - It consists of 100 dimensional textual features obtained after training on a CNN-based network for emotion/sentiment calssification. It is a list comprising of 3 dictionaries for train, val and the test set with each dictionary indexed in the format dia_utt , where dia is the dialogue id and utt is the utterance id. For eg. train_text_CNN_emb['0_0'].shape = (100, ) import pickle train_text_CNN_emb , val_text_CNN_emb , test_text_CNN_emb = pickle . load ( open ( filepath , 'rb' )) text_emotion.pkl, text_sentiment.pkl - These files contain the contextual feature representations as produced by the uni-modal bcLSTM model. It consists of 600 dimensional textual feature vector for each utterance for emotion/sentiment classification stored as a dictionary indexed with dialogue id. It is a list comprising of 3 dictionaries for train, val and the test set. For eg. train_text_emb['0'].shape = (33, 600), where 33 is the maximum number of utterances in a dialogue. Dialogues with less utterances are padded with zero-vectors. import pickle train_text_emb , val_text_emb , test_text_emb = pickle . load ( open ( filepath , 'rb' )) audio_emotion.pkl, audio_sentiment.pkl - These files contain the contextual feature representations as produced by the uni-modal bcLSTM model. It consists of 300/600 dimensional audio feature vector for each utterance for emotion/sentiment classification stored as a dictionary indexed with dialogue id. It is a list comprising of 3 dictionaries for train, val and the test set. For eg. train_audio_emb['0'].shape = (33, 300) or (33, 600), where 33 is the maximum number of utterances in a dialogue. Dialogues with less utterances are padded with zero-vectors. import pickle train_audio_emb , val_audio_emb , test_audio_emb = pickle . load ( open ( filepath , 'rb' )) bimodal_sentiment.pkl - This file contains the contextual feature representations as produced by the bi-imodal bcLSTM model. It consists of 600 dimensional bimodal (text, audio) feature vector for each utterance for sentiment classification stored as a dictionary indexed with dialogue id. It is a list comprising of 3 dictionaries for train, val and the test set. For eg. train_bimodal_emb['0'].shape = (33, 600), where 33 is the maximum number of utterances in a dialogue. Dialogues with less utterances are padded with zero-vectors. import pickle train_bimodal_emb , val_bimodal_emb , test_bimodal_emb = pickle . load ( open ( filepath , 'rb' )) Description of Raw Data There are 3 folders (.tar.gz files)-train, dev and test; each of which corresponds to video clips from the utterances in the 3 .csv files. In any folder, each video clip in the raw data corresponds to one utterance in the corresponding .csv file. The video clips are named in the format: diaX1_uttX2.mp4, where X1 is the Dialogue_ID and X2 is the Utterance_ID as provided in the corresponding .csv file, denoting the particular utterance. For example, consider the video clip dia6_utt1.mp4 in train.tar.gz . The corresponding utterance for this video clip will be in the file train_sent_emp.csv with Dialogue_ID=6 and Utterance_ID=1 , which is 'You liked it? You really liked it?' Reading the Data There are 2 python scripts provided in './utils/': read_meld.py - displays the path of the video file corresponding to an utterance in the .csv file from MELD. read_emorynlp - displays the path of the video file corresponding to an utterance in the .csv file from Multimodal EmoryNLP Emotion Detection dataset. Labelling For experimentation, all the labels are represented as one-hot encodings, the indices for which are as follows: Emotion - {'neutral': 0, 'surprise': 1, 'fear': 2, 'sadness': 3, 'joy': 4, 'disgust': 5, 'anger': 6}. Therefore, the label corresponding to the emotion 'joy' would be [0., 0., 0., 0., 1., 0., 0.] Sentiment - {'neutral': 0, 'positive': 1, 'negative': 2}. Therefore, the label corresponding to the sentiment 'positive' would be [0., 1., 0.] Class Weights For the baseline on emotion classification, the following class weights were used. The indexing is the same as mentioned above. Class Weights: [4.0, 15.0, 15.0, 3.0, 1.0, 6.0, 3.0]. Run the baseline Please follow these steps to run the baseline - Download the features from here . Copy these features into ./data/pickles/ To train/test the baseline model, run the file: baseline/baseline.py as follows: python baseline.py -classify [Sentiment|Emotion] -modality [text|audio|bimodal] [-train|-test] example command to train text unimodal for sentiment classification: python baseline.py -classify Sentiment -modality text -train use python baseline.py -h to get help text for the parameters. For pre-trained models, download the model weights from here and place the pickle files inside ./data/models/ . Citation Please cite the following papers if you find this dataset useful in your research S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, R. Mihalcea. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversation. ACL 2019. Chen, S.Y., Hsu, C.C., Kuo, C.C. and Ku, L.W. EmotionLines: An Emotion Corpus of Multi-Party Conversations. arXiv preprint arXiv:1802.08379 (2018). Multimodal EmoryNLP Emotion Recognition Dataset Description Multimodal EmoryNLP Emotion Detection Dataset has been created by enhancing and extending EmoryNLP Emotion Detection dataset. It contains the same dialogue instances available in EmoryNLP Emotion Detection dataset, but it also encompasses audio and visual modality along with text. There are more than 800 dialogues and 9000 utterances from Friends TV series exist in the multimodal EmoryNLP dataset. Multiple speakers participated in the dialogues. Each utterance in a dialogue has been labeled by any of these seven emotions -- Neutral, Joyful, Peaceful, Powerful, Scared, Mad and Sad. The annotations are borrowed from the original dataset. Dataset Statistics Statistics Train Dev Test # of modality {a,v,t} {a,v,t} {a,v,t} # of unique words 9,744 2,123 2,345 Avg. utterance length 7.86 6.97 7.79 Max. utterance length 78 60 61 Avg. # of emotions per scene 4.10 4.00 4.40 # of dialogues 659 89 79 # of utterances 7551 954 984 # of speakers 250 46 48 # of emotion shift 4596 575 653 Avg. duration of an utterance 5.55s 5.46s 5.27s Dataset Distribution Train Dev Test Joyful 1677 205 217 Mad 785 97 86 Neutral 2485 322 288 Peaceful 638 82 111 Powerful 551 70 96 Sad 474 51 70 Scared 941 127 116 Data Video clips of this dataset can be download from this link . The annotation files can be found in https://github.com/SenticNet/MELD/tree/master/data/emorynlp . There are 3 .csv files. Each entry in the first column of these csv files contain an utterance whose corresponding video clip can be found here . Each utterance and its video clip is indexed by the season no., episode no., scene id and utterance id. For example, sea1_ep2_sc6_utt3.mp4 implies the clip corresponds to the utterance with season no. 1, episode no. 2, scene_id 6 and utterance_id 3. A scene is simply a dialogue. This indexing is consistent with the original dataset. The .csv files and the video files are divided into the train, validation and test set in accordance with the original dataset. Annotations have been directly borrowed from the original EmoryNLP dataset (Zahiri et al. (2018)). Description of the .csv files Column Specification Column Name Description Utterance Individual utterances from EmoryNLP as a string. Speaker Name of the speaker associated with the utterance. Emotion The emotion (Neutral, Joyful, Peaceful, Powerful, Scared, Mad and Sad) expressed by the speaker in the utterance. Scene_ID The index of the dialogue starting from 0. Utterance_ID The index of the particular utterance in the dialogue starting from 0. Season The season no. of Friends TV Show to which a particular utterance belongs. Episode The episode no. of Friends TV Show in a particular season to which the utterance belongs. StartTime The starting time of the utterance in the given episode in the format 'hh:mm:ss,ms'. EndTime The ending time of the utterance in the given episode in the format 'hh:mm:ss,ms'. Note : There are a few utterances for which we were not able to find the start and end time due to some inconsistencies in the subtitles. Such utterances have been omitted from the dataset. However, we encourage the users to find the corresponding utterances from the original dataset and generate video clips for the same. Citation Please cite the following papers if you find this dataset useful in your research S. Zahiri and J. D. Choi. Emotion Detection on TV Show Transcripts with Sequence-based Convolutional Neural Networks. In The AAAI Workshop on Affective Content Analysis, AFFCON'18, 2018. S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, R. Mihalcea. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversation. ACL 2019. MUStARD: Multimodal Sarcasm Detection Dataset This repository contains the dataset and code for our ACL 2019 paper: Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper) We release the MUStARD dataset which is a multimodal video corpus for research in automated sarcasm discovery. The dataset is compiled from popular TV shows including Friends , The Golden Girls , The Big Bang Theory , and Sarcasmaholics Anonymous . MUStARD consists of audiovisual utterances annotated with sarcasm labels. Each utterance is accompanied by its context, which provides additional information on the scenario where the utterance occurs. Example Instance Example sarcastic utterance from the dataset along with its context and transcript. Raw Videos We provide a Google Drive folder with the raw video clips , including both the utterances and their respective context Data Format The annotations and transcripts of the audiovisual clips are available at data/sarcasm_data.json . Each instance in the JSON file is allotted one identifier (e.g. "1_60") which is a dictionary of the following items: Key Value utterance The text of the target utterance to classify. speaker Speaker of the target utterance. context List of utterances (in chronological order) preceding the target utterance. context_speakers Respective speakers of the context utterances. sarcasm Binary label for sarcasm tag. Example format in JSON: { "1_60" : { "utterance" : " It's just a privilege to watch your mind at work. " , "speaker" : " SHELDON " , "context" : [ " I never would have identified the fingerprints of string theory in the aftermath of the Big Bang. " , " My apologies. What's your plan? " ], "context_speakers" : [ " LEONARD " , " SHELDON " ], "sarcasm" : true } } Citation Please cite the following paper if you find this dataset useful in your research: @inproceedings { mustard , title = " Towards Multimodal Sarcasm Detection (An \_Obviously\_ Perfect Paper) " , author = " Castro, Santiago and Hazarika, Devamanyu and P{\'e}rez-Rosas, Ver{\'o}nica and Zimmermann, Roger and Mihalcea, Rada and Poria, Soujanya " , booktitle = " Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) " , month = " 7 " , year = " 2019 " , address = " Florence, Italy " , publisher = " Association for Computational Linguistics " , } Run the code Setup an environment with Conda: conda env create -f environment.yml conda activate mustard python -c " import nltk; nltk.download('punkt') " Download Common Crawl pretrained GloVe word vectors of size 300d, 840B tokens somewhere. Download the pre-extracted visual features to the data/ folder (so data/features/ contains the folders context_final/ and utterances_final/ with the features) or extract the visual features yourself. Download the pre-extracted BERT features and place the two files directly under the folder data/ (so they are data/bert-output.jsonl and data/bert-output-context.jsonl ), or [extract the BERT features in another environment with Python 2 and TensorFlow 1.11.0 following "Using BERT to extract fixed feature vectors (like ELMo)" from BERT's repo and running: # Download BERT-base uncased in some dir: wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip # Then put the location in this var: BERT_BASE_DIR=... python extract_features.py \ --input_file=data/bert-input.txt \ --output_file=data/bert-output.jsonl \ --vocab_file= ${BERT_BASE_DIR} /vocab.txt \ --bert_config_file= ${BERT_BASE_DIR} /bert_config.json \ --init_checkpoint= ${BERT_BASE_DIR} /bert_model.ckpt \ --layers=-1,-2,-3,-4 \ --max_seq_length=128 \ --batch_size=8 Check the options in python train_svm.py -h to select a run configuration (or modify config.py ) and then run it: python train_svm.py # add the flags you want Evaluation: We evaluate using weighted F-score metric in a 5-fold cross validation scheme. The fold indices are available at data/split_incides.p . Refer to our baseline scripts for more details. M2H2: A Multimodal Multiparty Hindi Dataset For Humor Recognition in Conversations ⚡ ⚡ ⚡ Baseline Codes will be released soon! 🔥🔥🔥 Read the paper The M2H2 dataset is compiled from a famous TV show "Shrimaan Shrimati Phir Se" (Total of 4.46 hours in length) and annotated them manually. We make groups of these samples (utterances) based on their context into scenes. Each utterance in each scene consists of a label indicating humor of that utterance i.e., humor or non-humor. Besides, each utterance is also annotated with its speaker and listener information. In multiparty conversation, listener identification poses a great challenge. In our dataset, we define the listener as the party in the conversation to whom the speaker is replying. Each utterance in each scene is coupled with its context utterances, which are preceding turns by the speakers participating in the conversation. It also contains multi-party conversations that are more challenging to classify than dyadic variants. Data Format Text Data 🔥🔥🔥 The Raw-Text/Ep-NUMBER.tsv acts as a master annotation file which does not only contain the textual data but also contains other metadata as described below. It also contains the manually annotated labels of the utterances. Using the Episode id and scene id, one can map the utterances in the Raw-Text folder to the corresponding audio and visual segments in Raw-Audio and Raw-Visual . This should result in multimodal data. The Label column in the TSV files e.g., Raw-Text/Ep-NUMBER.tsv contains the desired manually annotated labels for each utterance. The text data are stored in TSV format. Each of the file is named as Raw-Text/Ep-NUMBER.tsv . Here the NUMBER is episode number which one should use to map with the corresponding audio and visual segments. The text data contains the following fields: Scenes: The scene id. It will match the corresponding audio and visual segments. SI. No.: Utterance number. Start_time: Start time of the utterance in the video. End_time: End time of the utterance in the video. Utterance: The spoken utterance. Label: The annotated label of the utterance. This can either be humor or non-humor. Speaker: The format is "Speaker,listener". It has the form of "Speaker_name,utterance_id" e.g., "Dilruba,u3" which means the speaker is Dilruba and he is responding to utterance no. 3. This is particularly useful to resolve coreferences in a multiparty conversation. Audio Data Every episode has a dedicated folder e.g., Raw-Audio/22/ contains all the annotated audio samples for Episode no. 22. For every episode, each scene has a dedicated folder e.g., Raw-Audio/22/Scene_1 contains all the annotated audio samples for Episode no. 22 Scene 1. Visual Data Every episode has a dedicated folder e.g., Raw-Visual/22/ contains all the annotated visual samples for Episode no. 22. For every episode, each scene has a dedicated folder e.g., Raw-Visual/22/Scene_1 contains all the annotated visual samples for Episode no. 22 Scene 1. Baselines ⚡ ⚡ ⚡ Baseline Codes will be released soon! Citation Dushyant Singh Chauhan, Gopendra Vikram Singh, Navonil Majumder, Amir Zadeh,, Asif Ekbal, Pushpak Bhattacharyya, Louis-philippe Morency, and Soujanya Poria. 2021. M2H2: A Multimodal MultipartyHindi Dataset For Humor Recognition in Conversations. In ICMI ’21: 23rd ACM International Conference on Multimodal Interaction , Montreal, Canada. ACM, New York, NY, USA, 5 pages.
2. Related Work, Literature Survey & Algorithmic Foundations
Awesome Multimodal Modeling A Comprehensive Survey & Curated List of Multimodal Modeling From Traditional Fusion to Native & Unified Architectures Overview · Traditional · MLLMs · UMMs · NMMs · Closed Source Models · Resources 📢 News [2026-10-19] Updated the curated model and design-analysis coverage; added a dedicated Benchmarks & Evaluation section and matching website collection, with explicit date and release-status qualifications. [2026-06-06] 🚀 We released two new open-source model collections : Unified Multimodal Model Zoo (UMM Zoo) – a curated collection of unified multimodal models for understanding, generation, reasoning, and editing. Native Multimodal Model Zoo (NMM Zoo) – a curated collection of native multimodal models that natively integrate perception, reasoning, and generation. [2026-04-13] ⭐ The repository has already gained over 100 stars in just one day! Thank you all for the incredible support. We will keep updating this list with more cutting-edge models and resources. Your continued stars and PRs are warmly welcomed! [2026-04-12] 🎉 We are excited to launch Awesome Multimodal Modeling — a curated reading list organized by architectural paradigms. A comprehensive survey paper is coming soon! Stay tuned. Table of Contents Browse the list Awesome Multimodal Modeling 📢 News Table of Contents About This List At a Glance Curation Principles 1. Introduction & Definitions 1.1 Multimodal Model Evolution Stages Traditional Multimodal Models Multimodal Large Language Models (MLLMs) Unified Multimodal Models (UMMs) Native Multimodal Models (NMMs) NMM — Early Fusion NMM — Late Fusion 1.2 Scope & Taxonomy 1.3 Architecture Diagrams 2. Traditional Multimodal Models 2.1 Multimodal Representations & Alignment Multimodal Representations Multimodal Fusion Multimodal Alignment 2.2 Multimodal Pretraining 3. Multimodal Large Language Models (MLLMs) 3.1 Taxonomy Based on Vision Adapter MLP/Others Projector Q-Former Cross-Attention Hybrid Adaptor Interface Unspecified 3.2 Omni MLLMs 4. Unified Multimodal Models (UMMs) 4.1 Taxonomy by Generation Paradigm Diffusion-Based UMMs Autoregressive (AR) UMMs Pixel Encoding Semantic Encoding Learnable Query Encoding Hybrid Encoding (Pseudo) Hybrid Encoding (Joint) Hybrid (AR + Diffusion) UMMs Pixel Encoding Hybrid Encoding 4.2 Any-to-Any / Omni UMMs 4.3 Related Generators (U+G Unverified) 5. Native Multimodal Models (NMMs) 5.1 Design Analyses & Scaling Laws 5.2 Early Fusion NMMs 5.3 Encoder-Based NMMs 5.4 Any-to-Any / Omni NMMs 5.5 Training-Native Models 6. Closed-Source Multimodal Models Year 2026 Year 2025 Year 2024 Year 2023 7. Resources 7.1 Related Awesome Lists 7.2 Slides & Survey Papers 7.3 Code Repositories & Tools 7.4 Benchmarks & Evaluation Unified Understanding & Generation Interleaved & Any-to-Any Evaluation General Multimodal & Robustness Image Generation Evaluation How to Contribute Validation Rules Entry Format Citation License About This List In this section: At a Glance · Curation Principles This repository provides a structured, community-maintained survey of multimodal models, covering the full evolutionary arc from early fusion methods to today's natively-trained omni-models. We emphasize precise architectural definitions and classification, especially for the often-conflated categories of Unified Multimodal Models (UMMs) and Native Multimodal Models (NMMs). Scope : Primary focus on image + text modalities; audio/video/3D are annotated where present. Omni/any-to-any models are marked with Omni . At a Glance Dimension Coverage Primary scope Image + text multimodal models, with explicit annotations for video, audio, and omni extensions Core taxonomy Traditional multimodal models, MLLMs, UMMs, and NMMs with explicit architecture/training qualifications Key distinction U+G unification for UMMs; architectural and optimization coupling for NMMs, with overlapping membership allowed What makes this repo different Architecture-first categorization, fusion-aware definitions, and curated links to adjacent awesome lists Intended audience Researchers, students, and engineers building or surveying multimodal systems Curation Principles Coverage audit: the additions and release-status updates in this revision were checked against primary sources through 2026-10-19 . This is a curated selection, not an exhaustive index of every paper published by that date. Principle Rule Source quality Prefer official conference proceedings, OpenReview, ACL Anthology, CVF Open Access, arXiv, and official project pages Classification policy Category assignment is based on this repository's architecture-first definitions , which may differ from authors' own branding Venue policy Prefer the published venue; label author-reported acceptances where proceedings have not been checked. Keep first-paper dates separate from conference years Date policy First public: YYYY-MM-DD records the earliest verified exact-date public announcement; retain month precision where only a month is known. Distinguish paper submission, API availability, weights release and revision dates. Website year filters use the venue/release year; chronological sorting uses the explicit first-public date where recorded, otherwise the dated venue/release Availability policy Distinguish code, downloadable weights, API previews and future promises; record model-weight licenses separately from code licenses Scope discipline Models, benchmarks, datasets, and analysis papers are tracked separately to avoid mixing artifacts Inclusion bar We prioritize landmark papers, broadly adopted benchmarks, open implementations, or papers that clarify important taxonomy boundaries Classification note: for ambiguous models sitting between MLLM , UMM , and NMM , this list records the category that best matches the training recipe and architectural coupling , not just the paper title. Architecture-native and training-native qualifications distinguish fusion structure from training history. Back to Top 1. Introduction & Definitions In this section: 1.1 Multimodal Model Evolution Stages · 1.2 Scope & Taxonomy · 1.3 Architecture Diagrams 1.1 Multimodal Model Evolution Stages Subtopics: Traditional Multimodal Models · Multimodal Large Language Models (MLLMs) · Unified Multimodal Models (UMMs) · Native Multimodal Models (NMMs) We use the following precise, architecture-first definitions throughout this list. Understanding these distinctions is critical for correctly classifying modern models. Traditional Multimodal Models Pre-2023 mainstream era Representation alignment or fusion of modality-specific processing streams, without a large language model backbone. Covers cross-modal retrieval, classification and captioning; alignment-only dual encoders and models with joint fusion layers are distinguished by their architectures. Examples: CLIP, ALIGN, ViLBERT, BLIP. Multimodal Large Language Models (MLLMs) Pretrained-backbone multimodal language models Combine a pretrained visual backbone or visual abstractor (e.g., ViT/CLIP/SigLIP, Q-Former, cross-attention adapter) with a pretrained LLM through a connector. The defining property is inheritance from strong pretrained unimodal backbones rather than joint multimodal pretraining from scratch. These models are primarily text-output understanding/reasoning systems , even when auxiliary generators are attached externally. Key characteristics: ✅ Pretrained visual encoder / abstractor ✅ Pretrained LLM backbone ✅ Connector layer or cross-attention bridge ❌ No end-to-end multimodal pretraining from scratch ❌ No native image generation inside the same backbone Examples: LLaVA , Qwen-VL , InternVL , MiniCPM-V , CogVLM Unified Multimodal Models (UMMs) Single framework for Understanding + Generation (U+G) A single framework that handles both multimodal understanding and visual generation. UMMs may reuse pretrained components or modular tokenizers; the defining feature is U+G unification , not whether the model is trained from scratch. Key characteristics: ✅ Unified understanding + generation ✅ Shared model interface or shared backbone for U+G ⚠️ May use pretrained components ⚠️ May use decoupled encoders / modular tokenizers ⚠️ If a model is also natively trained from scratch, its architectural details belong primarily in NMMs (§5) Examples: Show-o , Janus , OpenUni , BAGEL , BLIP3-o Native Multimodal Models (NMMs) Architectural integration and multimodal optimization NMMs are described along two dimensions: architectural nativity captures how early and persistently modalities share core computation; optimization nativity captures when multimodal objectives enter training and which core parameters they optimize. Entries that establish only one dimension are qualified as architecture-native or training-native . Training from scratch provides strong optimization evidence, but is not assumed for every entry. UMM and NMM membership can overlap. Key characteristics: ✅ Fusion structure and training history are recorded separately ✅ Pretrained components and initialization are annotated per model ✅ Architecture-native entries may inherit pretrained backbones ✅ Input: text tokens + image patches/tokens ✅ Output: text (understanding focus; generation optional) NMMs are described by fusion topology and input architecture , with initialization recorded separately. §5.2 records supported early-fusion architectures; §5.3 groups encoder-based systems without inferring fusion depth from encoder presence. Training-native entries without a confirmed architectural grouping are listed separately in §5.5 : NMM — Early Fusion Modality tokens or embeddings enter shared core computation before the first shared backbone block . Separate tokenizers, input encoders, modality-specific experts, or output decoders can remain; their presence does not by itself imply late fusion. Shared backbone computation over multimodal states Discrete tokens, continuous latents, or pixel patches Modality interaction from the first shared block Initialization and trainable components annotated separately Examples: Chameleon , Emu3 , Transfusion ; Show-o , Show-o2 , OneCat , and Llama 4 carry architecture-native qualifications NMM — Late Fusion Cross-modal interaction begins only after modality-specific backbone layers, rather than at the first shared block. This is a claim about where modalities interact , independent of whether the components were pretrained or initialized from scratch. Identify the modality-specific stages and the first interaction layer Record initialization and multimodal optimization separately A dedicated vision encoder alone does not establish this topology; encoder-based systems with unverified fusion depth are grouped in §5.3 1.2 Scope & Taxonomy Multimodal Models ├── 2. Traditional Multimodal Models │ ├── 2.1 Multimodal Representations & Alignment │ │ ├── Multimodal Representations │ │ ├── Multimodal Fusion │ │ └── Multimodal Alignment │ └── 2.2 Multimodal Pretraining ├── 3. Multimodal Large Language Models (MLLMs) │ ├── 3.1 Taxonomy Based on Vision Adapter │ └── 3.2 Omni MLLMs ├── 4. Unified Multimodal Models (UMMs) │ ├── 4.1 Taxonomy by Generation Paradigm │ │ ├── Diffusion-Based UMMs │ │ ├── Autoregressive (AR) UMMs │ │ │ ├── Pixel Encoding │ │ │ ├── Semantic Encoding │ │ │ ├── Learnable Query Encoding │ │ │ ├── Hybrid Encoding (Pseudo) │ │ │ └── Hybrid Encoding (Joint) │ │ └── Hybrid (AR + Diffusion) UMMs │ │ ├── Pixel Encoding │ │ └── Hybrid Encoding │ ├── 4.2 Any-to-Any / Omni UMMs │ └── 4.3 Related Generators (U+G Unverified) └── 5. Native Multimodal Models (NMMs) ├── 5.1 Design Analyses & Scaling Laws ├── 5.2 Early Fusion NMMs ├── 5.3 Encoder-Based NMMs ├── 5.4 Any-to-Any / Omni NMMs └── 5.5 Training-Native Models 1.3 Architecture Diagrams The NMM diagrams below distinguish fusion topology from input architecture. The early-fusion diagram illustrates a from-scratch variant; initialization and input interfaces are specified per model in §5. Encoder-based entries do not inherit a strict late-fusion label from these schematic examples. ┌─────────────────────────────────────────────────────────────────┐ │ TRADITIONAL MULTIMODAL MODEL │ │ │ │ [Image] ──► [CNN/ViT Encoder] ──┐ │ │ ├──► [Fusion] ──► [Output] │ │ [Text] ──► [LSTM/BERT] ──┘ │ └─────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────┐ │ MLLM — PRETRAINED BACKBONES + VISUAL INTERFACE │ │ │ │ [Image] ──► [Pretrained ViT/CLIP] ──► [Projector/Q-Former] │ │ │ │ │ ▼ │ │ [Text] ──────────────────────────► [Pretrained LLM] ──► [Text]│ └─────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────┐ │ UMM — UNIFIED UNDERSTANDING + GENERATION │ │ │ │ [Image/Text Input] ──► [Shared/Modular Tokenizer] │ │ │ │ │ ▼ │ │ [Unified Transformer] │ │ │ │ │ │ ▼ ▼ │ │ [Text Output] [Image Output] │ └─────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────┐ │ NMM — EARLY FUSION (Trained from Scratch) │ │ │ │ [Text tokens] ──┐ │ │ └──► [Single Decoder Transformer] ──► [Text] │ │ [Image patches ──► Linear Patchify] ──┘ │ │ (raw pixels, minimal preprocessing) │ │ Multimodal interaction from Layer 1 │ └─────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────┐ │ NMM — ENCODER-BASED INPUT ARCHITECTURE │ │ │ │ [Image] ──► [Vision Encoder] ──► [Visual Interface] │ │ │ │ │ ▼ │ │ [Text] ──────────────────────► [Backbone] ──► [Text] │ │ Initialization and first interaction layer recorded per model │ └─────────────────────────────────────────────────────────────────┘ Back to Top 2. Traditional Multimodal Models In this section: 2.1 Multimodal Representations & Alignment · 2.2 Multimodal Pretraining Pre-chat-MLLM and non-native multimodal systems that established the basic vocabulary of alignment, fusion, retrieval, captioning, and multimodal pretraining. 2.1 Multimodal Representations & Alignment Subtopics: Multimodal Representations · Multimodal Fusion · Multimodal Alignment Multimodal Representations Paper Venue Links Notes Task Identifiability Results for Multimodal Contrastive Learning ICLR 2023 Paper Theoretical identifiability analysis of contrastive multimodal learning representation learning Unpaired Vision-Language Pre-training via Cross-Modal CutMix ICML 2022 Paper Introduces CutMix-style augmentation for unpaired VLP vision-language pretraining Balanced Multimodal Learning via On-the-fly Gradient Modulation CVPR 2022 Paper Balances modality learning via dynamic gradient reweighting multimodal optimization FLAVA: A Foundational Language And Vision Alignment Model arXiv 2021 Paper Unified architecture for vision-language understanding and generation foundation multimodal model UniT: Multimodal Multitask Learning with a Unified Transformer arXiv 2021 Paper Single transformer for multiple multimodal tasks multimodal multitask learning Perceiver: General Perception with Iterative Attention ICML 2021 Paper General-purpose architecture for high-dimensional multimodal inputs general multimodal architecture Learning Transferable Visual Models From Natural Language Supervision ICML 2021 Paper Contrastive vision-language pretraining at scale vision-language contrastive learning VinVL: Revisiting Visual Representations in Vision-Language Models arXiv 2021 Paper Improved visual features for VL tasks vision-language representation improvement 12-in-1: Multi-Task Vision and Language Representation Learning CVPR 2020 Paper Unified multi-task learning across 12 VL tasks multi-task learning Learning Video Representations using Contrastive Bidirectional Transformer arXiv 2019 Paper Contrastive transformer for video representation learning video contrastive learning OmniNet: A Unified Architecture for Multi-modal Multi-task Learning arXiv 2019 Paper Unified encoder-decoder for multimodal tasks unified multimodal architecture Learning Representations by Maximizing Mutual Information Across Views arXiv 2019 Paper InfoMax principle for cross-view representation learning self-supervised learning ViCo: Word Embeddings from Visual Co-occurrences ICCV 2019 Paper Learning word embeddings from visual context vision-language embeddings Learning Factorized Multimodal Representations ICLR 2019 Paper Factorized latent space for multimodal data representation disentanglement Deep Fragment Embeddings for Bidirectional Image Sentence Mapping NeurIPS 2014 Paper Fragment-level image-sentence alignment vision-language alignment DeViSE: A Deep Visual-Semantic Embedding Model NeurIPS 2013 Paper Early deep vision-to-language embedding model vision-language embedding Multimodal Deep Learning ICML 2011 Paper Foundational multimodal deep learning framework multimodal deep learning Multimodal Fusion Paper Venue Links Notes Task Robust Contrastive Learning against Noisy Views arXiv 2022 Paper Robust contrastive learning under noisy multi-view inputs contrastive learning Attention Bottlenecks for Multimodal Fusion NeurIPS 2021 Paper Introduces bottleneck attention mechanism for efficient multimodal fusion multimodal fusion VMLoc: Variational Fusion For Learning-Based Multimodal Camera Localization AAAI 2021 Paper Variational multimodal fusion for camera localization tasks multimodal localization Trusted Multi-View Classification ICLR 2021 Paper Confidence-aware weighting for multi-view classification multi-view classification Deep-HOSeq: Deep Higher-Order Sequence Fusion for Multimodal Sentiment Analysis ICDM 2020 Paper Higher-order sequence fusion for multimodal sentiment analysis multimodal sentiment analysis What Makes Training Multi-Modal Classification Networks Hard? CVPR 2020 Paper Analyzes optimization challenges in multimodal classification theoretical/empirical analysis DeepCU: Integrating Both Common and Unique Latent Information for Multimodal Sentiment Analysis IJCAI 2019 Paper Separates shared and private latent representations for fusion multimodal sentiment analysis XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification IEEE TNNLS 2019 Paper Cross-modal feature exchange network for audio-visual tasks audio-visual classification MFAS: Multimodal Fusion Architecture Search CVPR 2019 Paper Neural architecture search for optimal multimodal fusion design architecture search The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision ICLR 2019 Paper Neuro-symbolic model combining perception and reasoning neuro-symbolic learning Efficient Low-rank Multimodal Fusion with Modality-Specific Factors ACL 2018 Paper Low-rank factorization for efficient multimodal fusion efficient fusion Memory Fusion Network for Multi-view Sequential Learning AAAI 2018 Paper Memory-based fusion across temporal multimodal sequences sequential multimodal learning Tensor Fusion Network for Multimodal Sentiment Analysis EMNLP 2017 Paper Tensor-based full interaction modeling across modalities multimodal sentiment analysis Multimodal Alignment Paper Venue Links Notes Task LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment arXiv 2023 Paper Code Language-centered alignment across image, video, audio, depth, thermal, and IMU modalities multimodal alignment CLIP ICML 2021 Paper 400M+ image-text pairs; dual-encoder (Vision Transformer + Text Transformer); contrastive alignment of separate image/text embeddings; no joint image-text backbone zero-shot image classification, retrieval CoMIR: Contrastive Multimodal Image Representation for Registration NeurIPS 2020 Paper Contrastive learning for multimodal image registration alignment multimodal alignment Multimodal Transformer for Unaligned Multimodal Language Sequences ACL 2019 Paper Transformer-based alignment for unaligned multimodal sequences sequence alignment Temporal Cycle-Consistency Learning CVPR 2019 Paper Uses cycle-consistency for temporal cross-modal alignment temporal alignment Deep Canonical Correlation Analysis ICML 2013 Paper Deep learning extension of CCA for cross-view representation alignment representation alignment 2.2 Multimodal Pretraining Paper Venue Links Notes Task Align before Fuse: Vision and Language Representation Learning with Momentum Distillation NeurIPS 2021 Spotlight Paper Momentum distillation for aligning vision-language representations before fusion vision-language pretraining Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling CVPR 2021 Paper Sparse frame sampling for efficient video-language pretraining video-language pretraining UniT: Multimodal Multitask Learning with a Unified Transformer arXiv 2021 Paper Unified transformer for multitask multimodal learning unified multimodal pretraining Large-Scale Adversarial Training for Vision-and-Language Representation Learning NeurIPS 2020 Paper Adversarial training improves robustness of vision-language representations robust multimodal pretraining Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision EMNLP 2020 Paper Grounds language tokens in visual context via voken supervision vision-grounded language modeling Integrating Multimodal Information in Large Pretrained Transformers ACL 2020 Paper Injects multimodal signals into large pretrained transformer architectures multimodal transformer pretraining VL-BERT: Pre-training of Generic Visual-Linguistic Representations arXiv 2019 Paper Joint vision-language BERT-style pretraining vision-language pretraining VisualBERT: A Simple and Performant Baseline for Vision and Language arXiv 2019 Paper Early unified transformer for vision-language understanding vision-language pretraining ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks NeurIPS 2019 Paper Two-stream transformer for cross-modal vision-language learning vision-language pretraining Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training arXiv 2019 Paper Cross-modal encoder for universal vision-language representations vision-language pretraining LXMERT: Learning Cross-Modality Encoder Representations from Transformers EMNLP 2019 Paper Cross-modality transformer encoder for vision-language reasoning vision-language pretraining VideoBERT: A Joint Model for Video and Language Representation Learning ICCV 2019 Paper Joint discrete token modeling for video and language video-language pretraining Back to Top 3. Multimodal Large Language Models (MLLMs) In this section: 3.1 Taxonomy Based on Vision Adapter · 3.2 Omni MLLMs Models that connect a pretrained visual encoder / abstractor to a pretrained LLM . Primarily text-output understanding and reasoning systems, defined by inherited pretrained unimodal backbones rather than multimodal pretraining from scratch. 3.1 Taxonomy Based on Vision Adapter These groups describe the vision-to-language interface , rather than attention variants inside the language backbone. MLP/Others includes linear projections, spatial merging and pixel shuffle; Q-Former covers query-transformer connectors; Cross-Attention refers to bridges attending to separate visual features. Hybrid Adaptor covers combinations such as learned pooling plus projection, or projection plus visual experts inside the language backbone. MLP/Others Projector Paper Venue Links Notes Task WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsing arXiv 2026-09-17 Paper Code HF Paper first submitted: 2026-09-17. Fine-tunes Qwen3-VL-2B/4B-Instruct, retaining their visual encoder and merger/projector; page images and instructions produce structured Markdown, LaTeX formulas and HTML tables. Coverage-driven and residual-error-driven training target degraded documents. Code and 2B/4B checkpoints available as of 2026-10-19. document parsing, OCR, table/formula recognition, robustness VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding arXiv 2026-09-14 Paper Paper first submitted: 2026-09-14. Qwen2.5-VL-7B initialization retains its vision encoder and projector; SFT and trajectory-level DAPO learn adaptive viewing speed, revisiting and note taking for long-video answers. Text/video input produces text and viewing actions; code and checkpoint release not established in the checked primary source as of 2026-10-19. long-video understanding, active exploration, temporal reasoning LFM2.5-VL-3B Model release 2026-08-12 Announcement HF Implementation First public: 2026-08-12. Pretrained LFM2.5-2.6B with SigLIP2 NaFlex 400M, pixel-unshuffle and a two-layer MLP projector; text/image/multi-image inputs produce text, grounding coordinates and tool calls. Weights and inference implementations available under the Liquid AI model license; aimed at low-latency edge use. image understanding, multi-image reasoning, OCR, grounding, tool use, edge deployment PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Training arXiv 2026-06-02 Paper Code HF First public: 2026-05-28 (weights); paper first submitted 2026-06-02. 0.9B document VLM retaining the 1.5 native-resolution encoder, adaptive MLP connector and ERNIE language backbone; region refinement and progressive post-training improve recognition. Document-image/text inputs produce structured text, tables, formulas and spotting coordinates. Code and weights available. document parsing, multilingual OCR, tables, formulas, text spotting PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsing arXiv 2026-01-29 Paper Code HF First public: 2026-01-29 (paper and weights). 0.9B VLM combines a native-resolution visual encoder, adaptive MLP connector and ERNIE-4.5-0.3B; document-image/text inputs produce structured text and localized recognition. Adds seal recognition, text spotting and robustness training; full page parsing uses the accompanying layout pipeline. Code and Apache-2.0 checkpoints available. document parsing, multilingual OCR, seal recognition, text spotting GLM-4.6V / GLM-4.6V-Flash Model release 2025-12-08 Announcement Code HF Flash First public: 2025-12-08. 106B and 9B vision-language variants with 128K trained context; visual patch merging and gated MLP projection connect the vision path to the language backbone. Text/image/video inputs produce text and tool calls, with visual observations passed directly to tools. Code and MIT-licensed weights available; native tool use does not establish from-scratch multimodal training. image/video understanding, document understanding, visual tool use SmolVLM2 Model release 2025-02-20 Announcement HF Paper First public: 2025-02-20. Video-trained 256M, 500M and 2.2B family extending the SigLIP/SmolLM projector design; text/image/multi-image/video inputs produce text. Uses video/image training mixtures and efficient visual compression for device deployment. Checkpoints and inference/fine-tuning support available under Apache 2.0; the shared family report was submitted 2025-04-07. image/video understanding, multi-image reasoning, efficiency, on-device deployment Idefics3 arXiv 2024 Paper HF SigLIP vision encoder with pixel-shuffle spatial compression and projection to a Llama 3 backbone; replaces Idefics2’s Perceiver resampler. visual understanding, document understanding MiniMax-01: Scaling Foundation Models with Lightning Attention arXiv 2025 Paper MiniMax-VL-01 uses a 303M ViT, a two-layer MLP visual projector and MiniMax-Text-01; Lightning/Softmax hybrid attention describes the language backbone. visual understanding VideoJev: System One Model for Video Understanding and Interaction arXiv 2026 Paper Code 4B / 35B-A3B; initialized from Qwen3.5-4B / Qwen3.6-35B-A3B, retaining their pretrained ViT, visual projection and language backbone; candidate-scoring heads unify temporal grounding and multiple-choice QA. video QA, temporal grounding Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning ICML 2026 Paper Code Qwen2.5-VL-7B (also Qwen3-VL-4B); inherits the pretrained Qwen ViT, visual projector and language backbone; interleaves reasoning with VideoCrop tool calls and learns from SFT plus trajectory-guided RL. long-video reasoning, visual tool use VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding arXiv 2026 Paper Code 4B; I3D-ViT initialized from image-pretrained MoonViT, with a lightweight projector and Qwen3-4B text backbone; spatiotemporal visual compression supports generalist offline and proactive streaming understanding. video understanding, temporal grounding, proactive streaming TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs arXiv 2026 Paper Code 2B / 4B / 8B; inherits the pretrained Qwen3-VL ViT, visual projector and language backbones; verified SFT and temporal-grounding GRPO adapt them to generalist interval localization. video temporal grounding OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction arXiv 2026 Paper Code 4B; initialized from Qwen3-VL-4B-Instruct with its pretrained vision encoder frozen and the multimodal projector and language model fine-tuned; hierarchical caption memory and control states support proactive streaming interaction. streaming video understanding, memory, proactive response Ling-3.0-flash-VL Model release 2026 HF Open-weight VLM (MIT) extending Ling-3.0-flash with a ViT and a two-layer MLP projector; supports image and video inputs with text outputs. BF16 and quantized checkpoints belong to the same model entry image/video understanding, document understanding, visual reasoning, GUI agents LLaDA-UI Model release 2026 Code HF SigLIP-initialized native-resolution ViT connects to LLaDA2.0-mini-base through spatial 4-to-1 feature grouping and a two-layer MLP; block-wise diffusion generates text reasoning and GUI actions. Code and weights are available; checkpoint license is unspecified in the model card as of 2026-09-15 screenshot understanding, GUI grounding, mobile/desktop/web agents Muse Glimmer-30B Model release 2026 HF Implementation Open-weight 30B model (Apache 2.0), distilled from Muse Spark; a dedicated Perception Encoder connects to the language backbone through an MLP adapter and linear projection; text and image inputs, text outputs visual understanding, visual reasoning, coding, agentic tool use DiffusionGemma Google DeepMind, 2026 Model Card HF Implementation Open-weight Gemma 4-based model (Apache 2.0); normalized visual features enter the language model through a linear projection, while an encoder-decoder architecture uses block-wise discrete diffusion for text generation. Image/video understanding with text output; no native image generation visual understanding, video understanding, reasoning, efficient text generation Mistral Medium 3.5 Mistral AI, 2026 Announcement HF Implementation Open-weight 128B dense VLM under a Modified MIT license; a from-scratch vision encoder connects through a patch merger and MLP projector to the language backbone. Supports text/image inputs and text outputs; whole-model from-scratch multimodal training is not established visual understanding, reasoning, coding, long-horizon agents Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model arXiv 2026 Paper Project Model 4B codec-native streaming VLM combining a from-scratch Mage-ViT visual encoder, two-layer MLP projector, and pretrained Qwen3-4B-Instruct-2507 decoder; codec-guided token selection reduces visual tokens by over 75% image/video understanding, long video, proactive streaming StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design arXiv 2026 Paper 0.9B on-device UI VLM with UI-aware layered visual encoding and a progressive dimensionality projection connector; quantized deployment is validated on Snapdragon 8 Gen5 UI understanding, OCR, grounding, on-device deployment MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe arXiv 2025 Paper Code Model Efficient 8B MLLM built with Qwen3-8B, SigLIP2-400M and a unified 3D-Resampler for compact image-video encoding visual understanding, document/OCR, video understanding, edge deployment Moondream 3.1 Model Release 2026 Project Model Efficient sparse MoE VLM with 9B total and 2B active parameters; supports visual reasoning, query, caption, detection, pointing and segmentation visual understanding, grounding, segmentation, edge deployment LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training arXiv 2025 Paper Code HF 4B / 8B; 85M mid-training data + 22M instruction data; ViT–MLP–LLM architecture with pretrained RICE-ViT, a two-layer MLP projector, and Qwen3 backbone; supports image, multi-image, and video visual understanding, video SAIL-VL2 Technical Report arXiv 2025 Paper Open-suite 2B/8B vision-language foundation model with SAIL-ViT, progressive multimodal training, SFT-RL thinking fusion, and strong image/video reasoning across 106 datasets visual understanding, video, reasoning Kwai Keye-VL 2.0 Technical Report arXiv 2026 Paper achieves state-of-the-art performance among models of similar scale, particularly excelling in fine-grained temporal localization video understanding Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders arXiv 2026 Paper LLM-initialized vision encoder (non-CLIP); text-to-vision weight reuse, generative-aligned visual features, optimized for dense perception. visual understanding Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision arXiv 2026 Paper SigLIP2 vision encoder with 2×2 Spatial Merge and a two-layer MLP projector to Youtu-LLM; unified autoregressive supervision over visual and text targets supports fine-grained understanding and dense prediction. visual understanding, grounding, segmentation, depth estimation STEP3-VL-10B Technical Report arXiv 2026 Paper 10B-scale foundation multimodal; unified unfrozen pre-training + PaCoRe test-time scaling, frontier-level reasoning with compact footprint. visual understanding GLM-OCR arXiv 2026 Paper GLM-OCR is an efficient 0.9B-parameter compact multimodal model designed for real-world document understanding. OCR, structured extraction Kimi K2.5 arXiv 2026 Paper joint text-vision pretraining, Agent Swarm framework; coding, vision, reasoning, agentic tasks; reduces latency by up to 4.5x visual agentic intelligence, agentic, reasoning Kwai Keye-VL 1.5 Technical Report arXiv 2025 Paper Adaptive Slow-Fast encoding; 8B parameter scale with 128K long-context; SOTA video reasoning & human-preference aligned. visual understanding olmOCR / olmOCR-2 arXiv 2025 Paper Efficient low-VRAM OCR model based on Qwen2.5-VL fine-tune; excels at preserving semantic structure and markdown output OCR, structured extraction PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model arXiv 2025-10-16 Paper Code HF Paper first submitted: 2025-10-16. Original 0.9B document VLM with a NaViT-style dynamic-resolution encoder, adaptive MLP connector and ERNIE-4.5-0.3B; document-image/text inputs produce multilingual text, table, formula and chart representations. Code and weights available; later 1.5 and 1.6 releases are listed separately. OCR, multilingual document parsing DeepSeek-OCR arXiv 2025 Paper HF Lightweight ~3B MoE vision model optimized for high-volume OCR, document digitization, charts and formulas; efficient inference OCR, document Kimi-VL arXiv 2025 Paper HF Projector + MoE backbone; long video/PDF/GUI, agentic capabilities, chain-of-thought vision reasoning visual understanding, agentic, video Seed1.5-VL Technical Report arXiv 2025 Paper 20B MoE + 532M ViT; native-resolution vision-language foundation model; efficient asymmetric architecture. visual understanding Qwen3-VL arXiv 2025 Paper HF Frontier-grade vision/OCR (32+ languages), video analysis, agentic capabilities, strong multimodal reasoning; includes large MoE variants (e.g., 235B) visual understanding, video, omni SmolVLM: Redefining small and efficient multimodal models arXiv 2025-04-07 Paper Announcement HF 256M First public: 2024-11-26 (original release); family paper submitted 2025-04-07. Compact image-first SigLIP/SmolLM family using pixel-shuffle compression and projection; text/image/multi-image inputs produce text, with sampled-frame video explored in the original release. Code and Apache-2.0 checkpoints available; SmolVLM2 separately records dedicated video training and release. visual understanding, efficiency, on-device deployment LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning arXiv 2025 Paper Diffusion LLM as LLM backbone; vision encoder: SigLIP visual understanding jina-vlm arXiv 2025 Paper HF SigLIP2 + Qwen backbone with custom projector; optimized for semantic VQA, diagrams, scans and document semantics visual understanding, VQA, document Phi-4-Multimodal arXiv 2025 Paper HF Small-parameter (LoRA + projectors) multimodal; vision + speech support, efficient on-device deployment visual understanding, on-device Molmo / PixMo CVPR 2025 Paper Code Strong open-data/open-weight VLM pipeline visual understanding FastVLM: Efficient Vision Encoding for Vision Language Models CVPR 2025 Paper efficient multimodal visual encoding for on-device deployment visual understanding, on-device Qwen2.5-VL: Technical Report arXiv 2025 Paper HF Stronger document, grounding, and video capabilities visual understanding General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model arXiv 2024 Paper HF/Code Specialized end-to-end OCR model with grounding (boxes + points); strong on scientific papers, slides, and mixed visual-text docs OCR, grounding LLaVA-OneVision: Easy Visual Task Transfer arXiv 2024 Paper Code Single model for image, multi-image, and video transfer visual understanding MiniCPM-V: A GPT-4V Level MLLM on Your Phone arXiv 2024 Paper Code On-device efficient MLLM visual understanding NVILA: Efficient Frontier Visual Language Models CVPR 2025 Paper Efficient general purpose multimodal LLM; spatial and temporal "Scale then compress" design; vision encoder: SigLIP visual understanding GLM-4V arXiv 2024 Paper ViT-based vision encoder (EVA-02-CLIP-L); high-resolution input support (up to 1120x1120) via image tiling and a visual interface to the language model, optimized for document and OCR tasks visual understanding xGen-MM (BLIP-3) arXiv 2024 Paper Open training recipe, datasets, and safety-tuned variants visual understanding DeepSeek-VL2: Mixture-of-Experts Vision-Language Models arXiv 2024 Paper Code MoE VLM with dynamic tiling and efficient inference visual understanding Pixtral arXiv 2024 Paper HF 12B open-weight model with strong instruction following, image+text understanding; competitive with larger open VLMs visual understanding Qwen2-VL arXiv 2024 Paper HF Dynamic resolution; native video visual understanding Cambrian-1: A Fully Open, Vision-Centric Exploration NeurIPS 2024 Paper Code Spatial Vision Aggregator visual understanding PaliGemma: A Versatile 3B VLM for Transfer arXiv 2024 Paper HF SigLIP encoder + Gemma backbone; strong transfer model visual understanding InternLM-XComposer2 arXiv 2024 Paper Code Compositional visual grounding visual understanding Phi-3-Vision arXiv 2024 Paper HF Small but capable visual understanding LLaVA-HR: High Resolution MLLMs CVPR 2024 Paper Mixture-of-Resolution Adaptation visual understanding InternVL2 Model release 2024 HF Instruction-tuned InternVL family release with strong multilingual and OCR capabilities visual understanding InternVL: Scaling up Vision Foundation Models CVPR 2024 Paper Code Progressively aligned ViT + LLM visual understanding MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training arXiv 2024 Paper Large-scale proprietary recipe study for multimodal LLM pretraining visual understanding Ovis: Structural Embedding Alignment for Multimodal Large Language Model arXiv 2024 Paper Code Structural embedding alignment between visual tokens and LLM token space visual understanding TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document arXiv 2024 Paper Code OCR-free document MLLM emphasizing text-heavy images document understanding, OCR MoE-LLaVA: Mixture of Experts for Large Vision-Language Models arXiv 2024 Paper Code Sparse MoE extension of LLaVA-style visual instruction tuning visual understanding MobileVLM: A Fast, Strong and Open Vision Language Assistant for Mobile Devices arXiv 2023 Paper Code Lightweight VLM for mobile deployment efficient visual assistant Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models arXiv 2023 Paper Code Expands visual vocabulary for dense OCR/document-style perception document understanding, OCR LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models arXiv 2023 Paper Code Compresses each frame/image into compact context tokens for efficient video MLLMs video understanding Video-LLaVA: Learning United Visual Representation by Alignment Before Projection arXiv 2023 Paper Code Unified image-video representation before projection into LLM image/video understanding GLaMM: Pixel Grounding Large Multimodal Model arXiv 2023 Paper Code Pixel-level grounding and phrase-region reasoning grounded visual understanding Ferret: Refer and Ground Anything Anywhere at Any Granularity arXiv 2023 Paper Code Referring and grounding across points, boxes, and free-form regions grounded visual dialogue LLaVA-1.5: Improved Baselines with Visual Instruction Tuning arXiv 2023 Paper Code Strong simple baseline with CLIP visual encoder, MLP projector, and instruction tuning visual instruction tuning ImageBind-LLM: Multi-modality Instruction Tuning arXiv 2023 Paper Code Connects ImageBind-aligned modalities to an LLM for multi-modality instruction following multi-modal instruction tuning PointLLM: Empowering Large Language Models to Understand Point Clouds arXiv 2023 Paper Code Extends LLM-based multimodal understanding to 3D point clouds 3D understanding LISA: Reasoning Segmentation via Large Language Model arXiv 2023 Paper Code Couples MLLM reasoning with segmentation mask output reasoning segmentation GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest arXiv 2023 Paper Code Region-of-interest instruction tuning for fine-grained visual reasoning region-level understanding 3D-LLM: Injecting the 3D World into Large Language Models arXiv 2023 Paper Code Projects 3D scene features into LLMs for 3D reasoning and dialogue 3D understanding Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic arXiv 2023 Paper Code Referential dialogue with natural-language coordinates and grounding grounded visual dialogue LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day arXiv 2023 Paper Code Biomedical adaptation of LLaVA-style visual instruction tuning biomedical visual assistant DetGPT: Detect What You Need via Reasoning arXiv 2023 Paper Code Uses LLM reasoning to orchestrate detection tools and visual grounding detection reasoning VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks arXiv 2023 Paper Code Treats vision-centric tasks as open-ended decoding with LLMs vision-centric decoding MultiModal-GPT: A Vision and Language Model for Dialogue with Humans arXiv 2023 Paper Code Instruction-tuned image-text dialogue model visual dialogue LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model arXiv 2023 Paper Code Parameter-efficient visual instruction tuning via adapters visual instruction tuning LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention arXiv 2023 Paper Code Early adapter-based multimodal instruction tuning on LLaMA visual instruction tuning LLaVA arXiv 2023 Paper Code 7B / 13B+ CLIP vision encoder (frozen/pretrained) + linear projection to LLM (Vicuna/LLaMA); common modular pretrained-backbone baseline visual instruction tuning, VQA, image captioning Q-Former Paper Venue Links Notes Task M-MiniGPT4: Multilingual VLLM Alignment via Translated Data arXiv 2026 Paper Q-Former based (inherits from MiniGPT-4 / BLIP-2) vision-language understanding Video Q-Former: Multimodal Large Language Model with Spatio-Temporal Querying Transformer OpenReview Paper Spatio-temporal Q-Former (learnable queries for video spatial-temporal feature extraction) video understanding HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding arXiv 2025 Paper Hierarchical Q-Former (multi-level learnable queries with memory bank for long video) long video understanding Towards Efficient Visual-Language Alignment of the Q-Former arXiv 2024 Paper PEFT-tuned Q-Former (parameter-efficient fine-tuning on InstructBLIP-style Q-Former) visual reasoning Matryoshka Query Transformer (MQT) for Large Vision-Language Models NeurIPS 2024 Paper Matryoshka Query Transformer (elastic learnable queries, variable token count) vision-language understanding Semantically Grounded QFormer for Efficient Vision Language Understanding arXiv 2023 Paper Improved Grounded QFormer (direct latent conditioning, bypass input projection) vision-language understanding Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding arXiv 2023 Paper Code BLIP-2/Q-Former-style visual and audio query transformers for video dialogue audio-video understanding InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning arXiv 2023 Paper Code Instruction-aware Q-Former trained over diverse vision-language tasks visual instruction tuning VideoChat: Chat-Centric Video Understanding arXiv 2023 Paper Code Video-centric MLLM for temporal dialogue and understanding video understanding mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality arXiv 2023 Paper Code Modular visual abstractor connected to a pretrained LLM visual instruction tuning MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models arXiv 2023 Paper Code Uses BLIP-2 visual encoder/Q-Former and aligns visual features to Vicuna visual dialogue BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models ICML 2023 Paper Code Foundational Q-Former architecture bridging frozen vision encoders and frozen LLMs vision-language pretraining Cross-Attention Paper Venue Links Notes Task CASA: Cross-Attention over Self-Attention arXiv 2025 Paper Efficient cross-attention via self-attention reformulation; competitive with token insertion on image benchmarks, strong for long video efficient vision-language fusion, video captioning LLaMA 3.2 Vision arXiv 2024 Paper HF Adapter-based vision addition to Llama 3.2; strong OCR, document VQA, 128K context visual understanding, document Qwen-VL: A Versatile Vision-Language Model arXiv 2023 Paper HF High-res, multi-lang, bounding box visual understanding Kosmos-2: Grounding Multimodal Large Language Models to the World arXiv 2023 Paper Code Adds grounded phrase-region modeling to multimodal language modeling grounded visual understanding Kosmos-1: Language Is Not All You Need: Aligning Perception with Language Models arXiv 2023 Paper Code Perception-language model aligning images and language for multimodal reasoning multimodal reasoning Flamingo: a Visual Language Model for Few-Shot Learning NeurIPS 2022 Paper Perceiver Resampler + gated cross-attention layers for few-shot multimodal prompting few-shot visual understanding IDEFICS — Hugging Face 80B Flamingo-inspired model; Perceiver resampling and gated cross-attention connect visual features to the language model open-source multimodal understanding Hybrid Adaptor Paper Venue Links Notes Task MiniCPM-V 4.7 Model release 2026-10-12 Code 35B-A3B 1B Architecture First public: 2026-10-12. 35B-A3B MoE and 1B VLMs combining SigLIP, a window-attention merger, MLP merger and Qwen3.5 language backbone; canvas M-RoPE and mixed 4x/16x compression support image/multi-image/video inputs with text outputs. Code and Apache-2.0 weights available; Lookahead Expert Scheduling supports on-device MoE inference; DSpark heads released 2026-10-15. visual understanding, document/OCR, video understanding, on-device deployment MolmoPoint: Better Pointing for VLMs with Grounding Tokens arXiv 2026-03-30 Paper Code HF Paper first submitted: 2026-03-30. Extends Molmo2 with grounding tokens that select image patches/subpatch locations instead of spelling coordinates; retains the attention-pooling plus MLP visual input connector. Image/video/text inputs produce text, points and tracks; includes GUI/video specializations. Code, checkpoints and grounding data available as of 2026-10-19. Output pointing attention is distinct from the visual input bridge. visual grounding, pointing, tracking, GUI grounding, video understanding Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding arXiv 2026-01-15 Paper Code HF Paper first submitted: 2026-01-15. 4B/8B and OLMo-based 7B family with pretrained SigLIP2 and language backbones, spatial multi-head attention pooling and a shared MLP connector. Text/image/multi-image/video inputs produce text, pointing coordinates and object tracks. Open training code, checkpoints and video/multi-image data available; packing and message-tree training support multiple QA contexts. image/video understanding, multi-image reasoning, pointing, tracking mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding arXiv 2024-09-05 Paper Code HF Paper first submitted: 2024-09-05. Adds a global-feature-guided DocCompressor after the convolution/projection H-Reducer, retaining Modality Adaptive Modules in the language backbone; compresses a high-resolution page to 324 visual tokens. Document-image/multi-page/video-frame and text inputs produce textual answers. Code and checkpoints available; three-stage training covers single-page and multi-page documents. document understanding, multi-page document QA, document video CogVLM: Visual Expert for Pretrained Language Models arXiv 2023 Paper Code Pretrained ViT + two-layer MLP adapter + pretrained Vicuna; visual tokens use separate QKV and FFN experts within shared causal self-attention for deep fusion. visual understanding MiniCPM-V 4.6 Model release 2026 HF Architecture Open-weight VLM (Apache 2.0) combining a SigLIP vision encoder, window-attention merger, MLP merger, and Qwen3.5 language backbone; supports 4x and 16x visual downsampling for efficient image/video understanding visual understanding, document/OCR, video understanding, on-device deployment EXAONE 4.5 Technical Report arXiv 2026 Paper Code Integrates a dedicated visual encoder with the EXAONE 4.0 framework for multimodal pretraining, with strong document understanding and Korean contextual reasoning visual understanding, document Phoenix-VL 1.5 Medium Technical Report arXiv 2026 Paper 123B multilingual multimodal model continued-pretrained from Mistral Medium 3.1 on localized multimodal and long-context corpora visual understanding, multilingual DeepSeek-OCR-2 arXiv 2026 Paper HF Optimized for high-volume OCR, document digitization, charts and formulas; efficient inference OCR, document Ovis2.5 arXiv 2025 Paper Following VET architecture; excellent document understanding and fine-grained quantization visual understanding, document Ovis2 arXiv 2025 HF Embedding table / projector architecture; excellent document understanding and fine-grained quantization visual understanding, document mPLUG-Owl3 arXiv 2024 Paper Code Long visual sequences visual understanding NVLM 1.0: Open Frontier-Class Multimodal LLMs arXiv 2024 Paper HF Hybrid multimodal design with strong OCR and reasoning visual understanding Idefics2 arXiv 2024 Paper HF SigLIP vision encoder → learned Perceiver pooling → MLP projection; the resulting visual sequence is concatenated with text embeddings in a pretrained Mistral backbone. visual understanding, document understanding mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding arXiv 2024-03-19 Paper Code Paper first submitted: 2024-03-19. OCR-free document understanding with convolution/projection H-Reducer and Modality Adaptive Modules separating visual/text key-value projections inside the pretrained language backbone. Image/text inputs produce textual answers and structured document parsing; unified structure learning uses DocStruct4M and DocReason25K. Code, models and data available; DocOwl2 is a separate entry. document understanding, OCR, layout understanding Interface Unspecified These understanding-model references have confirmed multimodal capabilities, but inspected primary sources do not establish the visual-to-language interface well enough to assign an adapter category. Placement is provisional: it neither establishes pretrained-language-model initialization nor excludes NMM membership. Initialization, fusion depth and training nativity remain independent evidence questions. Paper Venue Links Notes Task Muse Spark Open Model release 2026-10-15 Announcement HF Open-weight 816B-total / 42B-active MoE under the custom Muse Model License, with a 1.8B Perception Encoder. Text/image/audio/video input, text output. Initialization, multimodal training-entry stage, exact connector, and fusion depth are undisclosed; native branding alone does not establish them. Released separately from proprietary Muse Spark 1.3. image/video/audio understanding, multimodal reasoning, coding, agents Nemotron 3.5 Super VL Model release 2026-10-15 Announcement Report HF Code First public: 2026-10-14 (announcement); weights released 2026-10-15. 120B total/12B active, pretrained Nemotron language backbone with a C-RADIO vision encoder and VL connector; precise connector design not established here. Text/image/video inputs produce text and tool calls; BF16/NVFP4 checkpoints and recipes available. Mamba/Transformer hybrid describes the language backbone; Lightning is text-only. image/video understanding, document understanding, visual tool use, agents Infinity-Parser2 Technical Report arXiv 2026-07-08 Paper Code Pro Flash First public: 2026-05-11 (models); paper first submitted 2026-07-08. Flash and Pro fine-tune Qwen3.5-2B and Qwen3.5-35B-A3B with 5M bilingual document examples and joint RL across eight tasks; the visual connector is not independently verified here. Document-image/text inputs produce Markdown, HTML, LaTeX and SMILES. Code, checkpoints and Infinity-Doc2-5M available. document parsing, OCR, layout grounding, table/formula/chart recognition 3.2 Omni MLLMs Paper Venue Links Notes Task Adaptor TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming arXiv 2026 Paper Code 4B/9B; pretrained Qwen3.5 language and vision backbone with a grafted AuT audio encoder and lightweight MLP aligner; Per-vGrid aligns audio-video tokens; up to 256K context; text output image/video/audio understanding, live-commerce QA MLP Projector MiMo-V2.5 Model release 2026 HF Open-weight MoE model (MIT) with a dedicated MiMo ViT and a MiMo-Audio-initialized audio encoder; text pretraining is followed by visual/audio MLP projector warmup and multimodal pretraining. Supports image, video, and audio understanding with text outputs image/video/audio understanding, multimodal reasoning, long-context agents MLP Projector MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction arXiv 2026 Paper Code Model 9B edge-oriented omni model using Omni-Flow for simultaneous visual/audio perception and speech response; supports proactive full-duplex interaction with less than 12GB memory vision-language understanding, audio understanding, speech generation, full-duplex live interaction Hybrid Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence arXiv 2026 Paper Code Efficiency-optimized omni-modal backbone using Hybrid Mamba2-Transformer MoE; supports massive multi-modal contexts (10k+ tokens) for long-video reasoning and agentic GUI navigation on edge devices omni-modal understanding & reasoning Hybrid OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention arXiv 2026 Paper Reinforced audio-visual reasoning framework with query intention grounding and modality attention fusion audio-visual reasoning Hybrid Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data arXiv 2025 Paper Code MoE-based scaling for omnimodal understanding and generation omni-modal understanding & generation MLP Projector Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models arXiv 2025 Paper Code Unified audio-visual speech recognition using LLMs audio-visual speech recognition Hybrid LongCat-Flash-Omni Technical Report arXiv 2025 Paper Code Long-context omni-modal model supporting text and audio generation long-context omni-modal Hybrid OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM arXiv 2025 Paper Code Architecture and data enhancements for omni-modal understanding omni-modal understanding Hybrid InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue arXiv 2025 Paper Code Unified model for audio-visual multi-turn dialogue audio-visual dialogue Hybrid OneLLM: One Framework to Align All Modalities with Language CVPR 2024 Paper Unified framework aligning eight modalities to language through modality tokenizers and lightweight projectors all-in-one LLM Hybrid MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition NeurIPS 2025 Paper Mixture of Matryoshka experts for efficient audio-visual speech recognition audio-visual speech recognition Hybrid Qwen3-Omni Technical Report arXiv 2025 Paper Code Omni-modal model with text and audio capabilities (Alibaba/Qwen series) omni-modal Native Qwen2.5-Omni Technical Report arXiv 2025 Paper Code Omni-modal technical report with text and audio support (Alibaba/Qwen series) omni-modal Hybrid MiniCPM-o 2.6: A GPT-4o Level MLLM for Vision, Speech, and Multimodal Live Streaming on Your Phone 2025 Paper Code On-device GPT-4o level MLLM for vision, speech and multimodal live streaming (OpenBMB) on-device multimodal live streaming Hybrid Baichuan-Omni Technical Report arXiv 2024 Paper Code Technical report for Baichuan-Omni (Baichuan Inc.) omni-modal Hybrid Baichuan-Omni-1.5 Technical Report arXiv 2025 Paper Code Technical report for Baichuan-Omni 1.5 (Baichuan Inc.) omni-modal Hybrid VITA: Towards Open-Source Interactive Omni Multimodal LLM arXiv 2024 Paper Code Open-source interactive omni multimodal LLM interactive omni multimodal Hybrid VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction NeurIPS 2025 Paper Code Real-time vision and speech interaction model real-time multimodal interaction Hybrid Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities NeurIPS 2024 Paper Code Open-source GPT-4o style model with vision, speech and duplex capabilities vision-speech duplex Hybrid Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment arXiv 2025 Paper Code Progressive modality alignment for omni-modal language model omni-modal alignment MLP Projector MIO: A Foundation Model on Multimodal Tokens arXiv 2024 Paper Code Foundation model based on multimodal tokens multimodal tokens Native EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions CVPR 2025 Paper Code Multimodal model supporting seeing, hearing and emotional speech emotional multimodal Hybrid Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model arXiv 2025 Paper Code Simultaneous multimodal interactions with language-vision-speech model simultaneous multimodal Hybrid ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding arXiv 2025 Paper Code Native multimodal LLM focused on 3D generation and understanding 3D multimodal Native Pengi: An Audio Language Model for Audio Tasks arXiv 2023 Paper Code Audio-language model using audio representations with a frozen language model for audio captioning, QA, and retrieval-style tasks audio-language understanding Hybrid LTU: Listen, Think, and Understand arXiv 2023 Paper Code Audio-oriented multimodal instruction-following model for open-ended audio understanding audio-language understanding Hybrid Back to Top 4. Unified Multimodal Models (UMMs) In this section: 4.1 Taxonomy by Generation Paradigm · 4.2 Any-to-Any / Omni UMMs · 4.3 Related Generators (U+G Unverified) Models that unify multimodal understanding and visual generation within one framework. The defining property is U+G unification , not necessarily training from scratch. Boundary with NMMs: if a unified model's central contribution is native end-to-end multimodal pretraining from scratch , we document its architectural details primarily in §5 NMMs and keep §4 focused on the unified U+G perspective. Overview of representative paradigms and architectures of Unified Multimodal Models (UMMs). Source: https://github.com/AIDC-AI/Awesome-Unified-Multimodal-Models 4.1 Taxonomy by Generation Paradigm Subtopics: Diffusion-Based UMMs · Autoregressive (AR) UMMs · Hybrid (AR + Diffusion) UMMs This taxonomy first identifies how visual content is sampled , then records whether the described system also provides an autoregressive understanding/text-output path: Autoregressive (AR): visual content tokens or embeddings are predicted autoregressively (including masked or next-scale variants). A separately trained image reconstruction decoder may use diffusion; the visual-content prediction stage remains AR. Diffusion-Based: visual content is sampled through diffusion, masked denoising, or flow matching, and the described system has no separately established AR understanding/text-output path. An MLLM used only to encode conditions, or a prompt rewriter used only to prepare generation inputs, does not establish that output path. Hybrid (AR + Diffusion): the described system provides both an AR language-output path for understanding/reasoning and diffusion/flow-based visual sampling. The paths may share a backbone or use a query, hidden-state, or prompt connector. The entry must establish both paths; the presence of an AR-capable condition encoder alone is insufficient. For example, Qwen-Image uses its Qwen-VL component as a condition encoder for MMDiT image sampling and is placed in Diffusion-Based. Qwen-Image-2.0 uses the same generation paradigm, but its U+G membership is unverified; it is retained separately as a related generator (§4.3) . UniWorld (Fig. 3 and Table 1) explicitly retains an AR VLM understanding branch, evaluates image-to-text understanding, and pairs it with a flow-matching image generator, so it is placed in Hybrid. Merely inheriting a pretrained VLM does not prove that the resulting system exposes its standalone outputs. Pixel, semantic, learnable-query, and hybrid encoding describe a separate representation dimension. Subsections retain these distinctions within a generation paradigm; entries using diffusion reconstruction decoders state that component explicitly. Diffusion-Based UMMs Model Venue Links Paradigm Notes Task Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation arXiv 2026-10-14 Paper Code Model Discrete masked diffusion; codebook-depth AR head Paper first submitted: 2026-10-14; official inference-code announcement is dated 2026-08 (day unspecified). Pretrained LLaDA2.0-mini masked-diffusion MoE with pretrained AToken eight-codebook visual tokens. Spatial positions are denoised in parallel; a local AR head predicts only the codebook depth within each position. This is not an AR visual-content sequence or whole-model scratch training. As of 2026-10-19: inference/SFT code and NVIDIA/Ascend checkpoints released. image/video/3D understanding, text generation, image generation and editing Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces (MF-1) arXiv 2026-09-30 Paper Code Model Continuous flow matching for text and vision First public: 2026-09-30. Models continuous text/image embeddings with a shared flow field, chunk-causal attention and modality-specific FFNs. Sequential chunk continuation does not establish a separate AR next-token text-output objective. Reuses modality representations/decoders; whole-model scratch initialization is not established. As of 2026-10-19: code and 1.6B pretraining/SFT checkpoints released. visual understanding, text/image generation, mixed-sequence continuation Qwen-Image Technical Report arXiv 2025 Paper Code MMDiT diffusion Frozen Qwen2.5-VL conditions an MMDiT image generator using semantic and VAE representations; supports text rendering, editing and image-form dense perception tasks. image generation, editing, dense visual perception LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model arXiv 2026-04-22 Paper Code Model FP8 Unified discrete masked diffusion First public: 2026-04-22. LLaDA2.0-based MoE predicts masked text/visual tokens; SigLIP-VQ produces discrete semantic tokens and a distilled diffusion reconstruction decoder produces images. As of 2026-10-19: inference code/checkpoints released 2026-04-23; FP8 released 2026-05-06, Diffusers/ComfyUI support 2026-05-12, SGLang-Omni support 2026-05-29. visual understanding, text generation, image generation and editing, interleaved reasoning Dual Diffusion arXiv 2025 Paper Code Dual Diffusion Unified image generation + understanding via bidirectional diffusion visual understanding, visual generation UniDisc arXiv 2025 Paper Code Unified Discrete Diffusion Discrete diffusion for multimodal U+G visual understanding, visual generation MMaDA arXiv 2025 Paper Code Multimodal Large Diffusion LM Diffusion LM for unified understanding/generation visual understanding, visual generation FUDOKI arXiv 2025 Paper Discrete Flow-based Unified Kinetic-optimal velocities for U+G visual understanding, visual generation Muddit arXiv 2025 Paper Code Unified Discrete Diffusion Liberating generation beyond T2I visual understanding, visual generation Lavida-O arXiv 2025 Paper Code Elastic Large Masked Diffusion Elastic masked diffusion for U+G visual understanding, visual generation UniModel arXiv 2025 Paper Visual-Only MMDiT Framework Visual-only unified multimodal U+G visual understanding, visual generation Autoregressive (AR) UMMs Pixel Encoding Model Venue Links Modalities Notes Task LWM arXiv 2024 Paper video + language World model on million-length video and language with blockwise ring attention visual understanding, visual generation Chameleon arXiv 2024 Paper Code image + text Mixed-modal early-fusion foundation models; token-by-token generation visual understanding, visual generation ANOLE arXiv 2024 Paper Code image + text Open autoregressive native LMM for interleaved image-text generation visual understanding, visual generation MMAR arXiv 2024 Paper image + text Lossless multi-modal auto-regressive probabilistic modeling visual understanding, visual generation Orthus arXiv 2024 Paper Code image + text Autoregressive interleaved image-text generation with modality-specific heads visual understanding, visual generation SynerGen-VL arXiv 2024 Paper image + text Synergistic image understanding and generation with vision experts and token folding visual understanding, visual generation Liquid arXiv 2024 Paper Code image + text Language models are scalable and unified multi-modal generators visual understanding, visual generation UGen arXiv 2025 Paper image + text Unified autoregressive multimodal model with progressive vocabulary learning visual understanding, visual generation Harmon arXiv 2025 Paper Code image + text Shared MAR encoder for semantic + fine-grained harmony; SOTA GenEval visual understanding, visual generation TokLIP arXiv 2025 Paper Code image + text Marry visual tokens to CLIP for U+G visual understanding, visual generation Selftok arXiv 2025 Paper Code image + text Discrete visual tokens for AR / Diffusion / Reasoning visual understanding, visual generation OneCat arXiv 2025 Paper Code image + text Pure decoder-only unified U+G visual understanding, visual generation Uni-X arXiv 2025 Paper Code image + text Two-end-separated architecture mitigating modality conflict visual understanding, visual generation Emu3 arXiv 2024 Paper Code image + video + text Early-fusion native autoregressive model; see NMM / Early Fusion for architectural classification visual understanding, visual generation Semantic Encoding Title Venue Links Focus Task Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer arXiv 2025 Paper Code Unified continuous tokenizer for joint understanding and generation visual understanding, visual generation Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents arXiv 2025 Paper Code Masked autoregression predicts patch-level CLIP image embeddings; a latent-ControlNet-conditioned diffusion decoder reconstructs the final image. visual understanding, visual generation X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again arXiv 2025 Paper Code RL-enhanced discrete autoregressive unified modeling visual understanding, visual generation UniCode²: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation arXiv 2025 Paper Cascaded large-scale codebooks for unified modeling visual understanding, visual generation Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations arXiv 2025 Paper Code Text-aligned discrete semantic representations visual understanding, visual generation UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation arXiv 2025 Paper Code Y-shaped architecture for modality alignment visual understanding, visual generation Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation arXiv 2025 Paper Auto-regressive foundation model visual understanding, visual generation DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies arXiv 2025 Paper Dual visual vocabularies visual understanding, visual generation UniTok: A Unified Tokenizer for Visual Generation and Understanding arXiv 2025 Paper Code Unified tokenizer visual understanding, visual generation QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation arXiv 2025 Paper Code Text-aligned visual tokenization visual understanding, visual generation MetaMorph: Multimodal Understanding and Generation via Instruction Tuning arXiv 2024 Paper Instruction tuning for unified multimodal visual understanding, visual generation ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance arXiv 2024 Paper Self-enhancing unified see-and-draw visual understanding, visual generation PUMA: Empowering Unified MLLM with Multi-granular Visual Generation arXiv 2024 Paper Code Multi-granular visual generation visual understanding, visual generation VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation ICLR 2025 Paper Code Unified foundation model visual understanding, visual generation VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation arXiv 2023 Paper Generative pre-trained transformer visual understanding, visual generation Generative Multimodal Models are In-Context Learners (Emu2) CVPR 2024 Paper In-context learning generative multimodal model visual understanding, visual generation LaVIT: Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization ICLR 2024 Paper Code Dynamic discrete visual tokenization visual understanding, visual generation Emu: Generative Pretraining in Multimodality ICLR 2024 Paper Generative pretraining in multimodality visual understanding, visual generation Learnable Query Encoding Title Venue Links Focus Task Nexus-Gen: A Unified Model for Image Understanding, Generation, and Editing arXiv 2025 Paper Code Prefilled autoregression predicts visual embeddings in a shared space; a FLUX diffusion decoder converts them into images for generation and editing. visual understanding, visual generation, editing Making LLaMA SEE and Draw with SEED Tokenizer ICLR 2024 Paper Code SEED tokenizer enabling LLaMA for scalable multimodal autoregression (see and draw) visual understanding, visual generation Planting a SEED of Vision in Large Language Model arXiv 2023 Paper Code SEED image tokenizer with 1D causal dependency and high-level semantics for LLM vision visual understanding, visual generation Hybrid Encoding (Pseudo) Title Venue Links Focus Task Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generation arXiv 2025 Paper Code Unified autoregressive modeling with decoupled encoding for image understanding, generation and editing visual understanding, visual generation UniFluid: Unified Autoregressive Visual Generation and Understanding with Continuous Tokens arXiv 2025 Paper Unified autoregressive framework using continuous visual tokens visual understanding, visual generation OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models arXiv 2025 Paper Code Efficient linear-time unified multimodal model based on Mamba (state space models) multimodal understanding, visual generation Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling arXiv 2025 Paper Code Scaled-up version of Janus with improved training strategy, more data and larger model size multimodal understanding, visual generation Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation arXiv 2024 Paper Code Decoupling visual encoding to enable unified understanding and generation in an autoregressive framework multimodal understanding, visual generation Hybrid Encoding (Joint) Title Venue Links Focus Task Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation arXiv 2026-07-28 Paper First public: 2026-07-28. A frozen pretrained VLM vision encoder supplies continuous understanding features and quantized generation codes; an LLM predicts visual codes autoregressively and a reconstruction decoder produces images. Reuses pretrained VLM components; studies compact models including a 0.5B language backbone. As of 2026-10-19: paper public; official code/weight release not confirmed by inspected primary sources. image understanding, image generation UniAR: Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification ICML 2026 Paper Shared context-visual tokenizer bridging understanding and generation; AR visual-token prediction with diffusion-based decoding for high-fidelity generation and editing visual understanding, visual generation, editing AToken: A Unified Tokenizer for Vision arXiv 2025 Paper Code AToken unified visual tokenizer achieving high-fidelity reconstruction and semantic understanding for images, videos and 3D visual understanding, visual generation UniWeTok: An Unified Binary Tokenizer with Codebook Size 2128 for Unified Multimodal Large Language Model arXiv 2026 Paper UniWeTok unified binary tokenizer with 2^{128} codebook, pre-post distillation and generative-aware prior for MLLMs visual understanding, visual generation Towards Scalable Pre-training of Visual Tokenizers for Generation arXiv 2025 Paper Code VTP unified visual tokenizer pre-training framework with joint image-text contrastive, self-supervised and reconstruction losses visual understanding, visual generation The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding arXiv 2025 Paper Code Prism Hypothesis and unified autoencoding (UAE) harmonizing semantic and pixel representations across modalities visual understanding, visual generation UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding CVPRW 2025 Paper Code Unified visual encoding combining discrete and continuous representations for autoregressive multimodal models multimodal understanding and generation VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning arXiv 2025 Paper Code Enhanced visual autoregressive unified model with iterative instruction tuning and DPO reinforcement learning visual understanding, generation and editing ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement arXiv 2025 Paper Code Dual visual tokenization and diffusion refinement for unified multimodal large language model multimodal understanding and generation SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation arXiv 2025 Paper Semantic-guided hierarchical codebook for unified image tokenization supporting understanding and generation multimodal understanding and generation VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model arXiv 2025 Paper Code Visual autoregressive framework unifying understanding and generation in a single MLLM visual understanding and generation TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation CVPR 2025 Paper Code Unified image tokenizer with dual-codebook architecture bridging understanding and generation multimodal understanding and generation MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding arXiv 2024 Paper Semantic discrete encoding for unified vision-language model enabling efficient multimodal understanding and generation multimodal understanding and generation Hybrid (AR + Diffusion) UMMs Pixel Encoding Paper Venue Links Notes Task PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation arXiv 2026-09-29 Paper Project Code Model First public: 2026-09-29. Encoder-free MoT directly projects raw image patches/video tubelets into shared multimodal attention; AR text modeling and pixel-space flow matching provide understanding and visual generation. Architecture-native, initialized with Qwen3-8B rather than all components from scratch. As of 2026-10-19: code and checkpoints released; checkpoint use is limited to non-commercial research/evaluation under the NVIDIA One-Way Noncommercial License. image/video understanding, image/video generation Tuna: Taming Unified Visual Representations for Native Unified Multimodal Models arXiv 2025 Paper Code Native unified multimodal model with cascaded VAE + representation encoder for unified continuous visual representations multimodal understanding and generation LMFusion: Adapting Pretrained Language Models for Multimodal Generation arXiv 2024 Paper Adapting pretrained LLMs (Llama) for multimodal generation by adding parallel diffusion modules while keeping autoregressive text modeling multimodal understanding and generation MonoFormer: One Transformer for Both Diffusion and Autoregression arXiv 2024 Paper Code Single shared transformer backbone that handles both autoregressive modeling and diffusion for unified multimodal tasks visual understanding and generation Show-o: One Single Transformer to Unify Multimodal Understanding and Generation ICLR 2025 Paper Code Unified transformer combining autoregressive and discrete diffusion modeling to flexibly handle mixed-modality inputs/outputs multimodal understanding and generation Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model ICLR 2025 Paper 7B-scale report; single multimodal model combining next-token text prediction with image diffusion over mixed discrete/continuous sequences visual understanding, visual generation Hybrid Encoding Paper Venue Links Notes Task Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation arXiv 2026-10-12 Paper First public: 2026-10-12. MoT combines AR language/scene-state outputs with flow-matching image synthesis. Persistent visual anchors, scene-aware memory retrieval and self-rollout stabilize long interleaved trajectories; reports 256K multimodal context and 50+ generated images. Whole-model scratch initialization is not established. As of 2026-10-19: paper public; official code/weight release not confirmed by inspected primary sources. visual understanding, long-horizon interleaved text/image generation Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States arXiv 2026-09-03 Paper Project Code Model First public: 2026-09-03. Geometry-aligned visual encoding and an AR language model produce camera/scene understanding; learnable queries connect language hidden states to a diffusion generator with appearance/depth and camera conditions. Native 3D state terminology does not establish whole-model scratch training. As of 2026-10-19: code, weights and Puffin-16M data released. spatial understanding, camera-controlled image generation, 3D reconstruction Object-Uni: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation arXiv 2026-08-24 Paper First public: 2026-08-24. Adapts pretrained Ming-Lite-Uni with PoseAnchor object tokens and orientation heads for AR spatial reasoning; multi-scale queries and CNOCS conditions guide a diffusion transformer. Most pretrained backbone parameters are frozen during understanding adaptation. As of 2026-10-19: paper public; official code/weight release not confirmed by inspected primary sources. object pose perception, spatial reasoning, pose-controlled image and novel-view generation MedUAG: Unified Understanding and Generation for Medical Multimodal Models arXiv 2026-08-19 Paper First public: 2026-08-19. Initializes from BAGEL-7B-MoT with semantic ViT and VAE latent paths, AR text outputs and flow-matching image synthesis. Domain alignment and instruction tuning cover 14 medical imaging modalities; end-to-end adaptation does not mean training from scratch. As of 2026-10-19: paper public; official code/weight release not confirmed by inspected primary sources. medical visual QA/reporting, image synthesis, translation, reconstruction and prediction Rosetta: Composable Native Multimodal Pretraining NeurIPS 2026 Paper Project Code Model First public: 2026-07-01. Composable experts and momentum-anchored gradient projection preserve prior capabilities as modalities are added; shared attention combines AR understanding with flow-matching visual generation. Appendix C specifies Qwen3-0.6B sparse upcycling, frozen Qwen3-VL ViT and FLUX.2 VAE, so native does not mean all components from scratch. As of 2026-10-19: code and checkpoints released; NeurIPS venue is author-reported. language/visual understanding, image generation, composable multimodal training ILLUME-X: Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation ECCV 2026 (author-reported) Paper Code Model First public: 2026-06-29. ViT semantic and VAE latent paths feed a unified transformer with causal text/control tokens and bidirectional visual segments; text cross-entropy and rectified-flow image losses support free-form interleaving. Distinct from ILLUME/ILLUME+. As of 2026-10-19: inference code and checkpoints released; training data marked coming soon. visual understanding, free-form interleaved text/image generation and editing Lance: Unified Multimodal Modeling by Multi-Task Synergy arXiv 2026-05-18 Paper Project Code Model First public: 2026-05-18. 3B active parameters; shared interleaved context with specialized AR understanding and flow-matching generation pathways. Paper §3.2 explicitly initializes the dual-expert architecture from Qwen2.5-VL and reuses its text embeddings/ViT plus a Wan2.2 3D causal VAE; whole-model scratch training is not supported by these details. As of 2026-10-19: inference code/weights released 2026-05-18; fine-tuning code released 2026-06-17. image/video understanding, generation and editing Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training CVPR 2026 (Oral) Paper Project First public: 2026-03-26. MoT planner/visualizer uses a pretrained Qwen2.5-VL AR planner for understanding answers and dense prompts, plus a diffusion transformer visualizer trained from scratch. Decoupled training avoids requiring real interleaved training data; only the visualizer has the stated scratch initialization. As of 2026-10-19: paper/project demos public; official code/weight release not confirmed by inspected primary sources. Oral venue is author-reported. visual understanding, interleaved text/image generation ChatUMM: Robust Context Tracking for Conversational Interleaved Generation arXiv 2026-02-06 Paper Project First public: 2026-02-06. BAGEL-style MoT uses shared attention, task-specific FFNs, semantic ViT tokens and VAE image latents; AR text/control prediction and flow matching produce context-dependent interleaved replies. Whole-model scratch initialization is not established. As of 2026-10-19: paper revised 2026-06-01 and project examples public; official code/weight release not confirmed by inspected primary sources. conversational visual understanding, history-dependent image editing, interleaved text/image generation Vision as Unified Multimodal Generation (SenseNova-Vision) arXiv 2026-07-07 Paper Code Model Collection First public: 2026-07-07. Fine-tunes pretrained BAGEL-7B-MoT to express computer-vision tasks through AR text, diffusion-generated images or mixed responses without task-specific prediction heads. As of 2026-10-19: inference code, model weights and corpus public; checkpoint license is CC BY-NC 4.0 (non-commercial), distinct from repository code licensing. detection, OCR, keypoints, segmentation, depth, normals, point maps, camera pose S1-Omni-Image: A Unified Model for Scientific Image Understanding, Generation, and Editing arXiv 2026 Paper Builds on S1-VL-32B and injects reasoning hidden states into an image generation module for scientific image understanding, generation and editing scientific image understanding, generation and editing Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE arXiv 2026-05-04 Paper Code First public: 2026-02-15 (inference code); paper first submitted 2026-05-04. AR multimodal understanding coupled to a DiT-MoE generation backbone (25B total / 3B active in DiT), supporting image/video generation and editing, including few-step video editing. As of 2026-10-19: paper and inference code public; official repository still lists Mamoda2.5 weights under internal review. Released Mamoda2 checkpoints are a separate version. multimodal understanding, image/video generation, image/video editing InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing arXiv 2026 Paper Code 4B unified multimodal model integrating a strong MLLM with an MMDiT-based visual generation head for understanding, reasoning, generation and editing multimodal understanding, reasoning, generation and editing Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing arXiv 2026 Paper Code Connects pretrained MLLMs with video diffusion models through lightweight adapters for unified video generation and editing video generation and editing EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture arXiv 2025 Paper Code Efficient unified architecture with autoencoders, channel-wise concatenation, shared-decoupled networks and MoE for understanding, generation and editing multimodal understanding, generation and editing HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation arXiv 2025 Paper Asymmetric H-shaped architecture bridging heterogeneous experts with symmetric dense mid-layer connections for unified multimodal modeling multimodal understanding and generation LightFusion: A Light-weighted, Double Fusion Framework for Unified Multimodal Understanding and Generation arXiv 2025 Paper Code Light-weighted double fusion framework that efficiently integrates pretrained vision-language and diffusion models multimodal understanding and generation Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Model arXiv 2025 Paper Qwen2.5-VL connects through MetaQueries to an SD3.5-Medium DiT; unified understanding, diffusion-based generation and editing with online RL. visual understanding, visual generation, editing TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuning arXiv 2025 Paper A pretrained diffusion generator receives conditions from multiple MLLM layers through ladder-side connections. visual understanding, visual generation UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing arXiv 2025 Paper A continuous CLIP-based tokenizer supports reconstruction; MLLM hidden states and learnable queries jointly condition a diffusion transformer for generation and editing. visual understanding, visual generation, editing Ovis-U1 Technical Report arXiv 2025 Paper Code Ovis understanding backbone coupled to an MMDiT visual decoder and bidirectional token refiner; diffusion samples images from LLM conditions. visual understanding, visual generation OmniGen2: Exploration to Advanced Multimodal Generation arXiv 2025 Paper Code Autoregressive MLLM with a separate diffusion transformer; text hidden states and VAE features condition rectified-flow image generation and editing. visual generation Show-o2: Improved Native Unified Multimodal Models NeurIPS 2025 Paper Code Autoregressive language modeling and flow matching for image/video generation share unified visual representations in a 3D causal VAE space. multimodal understanding and generation UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation arXiv 2025 Paper Code Retains a frozen autoregressive Qwen2.5-VL understanding/text-output branch (Fig. 3; Table 1); SigLIP2 reference features and an MLP connector condition a FLUX generator trained with flow matching. visual understanding, visual generation OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation arXiv 2025 Paper Code Learnable queries and a lightweight transformer connector bridge an autoregressive MLLM to a diffusion image generator. visual understanding, visual generation BAGEL: Emerging Properties in Unified Multimodal Pretraining arXiv 2025 Paper Code Open-source foundational decoder-only model pretrained on trillions of interleaved multimodal tokens supporting native understanding and generation multimodal understanding and generation MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO arXiv 2025 Paper Code Autoregressive multimodal reasoning conditions a diffusion decoder trained with rectified-flow objectives; RGPO improves reasoning-guided generation. multimodal understanding, reasoning, image generation, editing BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset arXiv 2025 Paper Autoregressive multimodal understanding with a diffusion transformer that generates CLIP image features for image synthesis. visual understanding, visual generation Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation arXiv 2025 Paper Causal interleaved multi-modal generation framework with deep-fusion, dual vision encoders and multi-modal classifier-free guidance interleaved multimodal generation Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction arXiv 2025 Paper A fixed autoregressive MLLM connects through multi-scale learnable tokens to a trainable diffusion model for image generation and editing. visual understanding, visual generation Transfer between Modalities with MetaQueries arXiv 2025 Paper Code Learnable MetaQueries connect autoregressive MLLM latents to a diffusion decoder; the MLLM can remain frozen while diffusion learns image generation. visual understanding, visual generation JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation arXiv 2024 Paper Code Minimalist framework harmonizing autoregressive LLMs with rectified flow for efficient unified understanding and generation multimodal understanding and generation SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation arXiv 2024 Paper Code Learnable image queries reconstruct ViT features; an SDXL-initialized visual decoder samples images and supports high-precision manipulation. visual understanding, visual generation Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models arXiv 2024 Paper Code Multimodal understanding with an autoregressive language model; generated text prompts drive a pretrained SDXL image generator. visual understanding, visual generation MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer arXiv 2024 Paper Code Autoregressive text modeling with a diffusion image decoder, connected through resampling and a multi-scale multi-image feature synchronizer. visual understanding, visual generation DreamLLM: Synergistic Multimodal Comprehension and Creation ICLR 2024 Paper Autoregressive text and interleaved structure modeling; learnable dream queries condition a Stable Diffusion image decoder. visual understanding, visual generation 4.2 Any-to-Any / Omni UMMs Models that extend unified understanding and visual generation to additional modalities such as audio, video, and speech. “Omni” does not imply that every input modality can also be generated; supported outputs are specified per entry. Models restricted to text/speech output, including Qwen3-Omni and LongCat-Flash-Omni, belong in Omni MLLMs . Model Paper Links Notes Task Kling-Omni Technical Report arXiv 2025 Paper Unified Diffusion Transformer (DiT) framework with Prompt Enhancer for high-fidelity video generation and reasoning-based editing Multi-modal visual language (MVL) for unified generation and understanding Ming-flash-omni 2.0 Model release 2026-02-11 HF Open-weight 100B-total / 6B-active MoE model (MIT) based on Ling-2.0; unifies image, text, video, and audio inputs with image, text, and audio outputs, including image editing and controllable speech/audio/music synthesis multimodal understanding, image generation/editing, audio generation, streaming video conversation Ming-Flash-Omni arXiv 2025 Paper Code Sparse unified MoE architecture (100B total, 6.1B active) for efficient multimodal perception and generation any-to-any multimodal perception and generation Ming-Omni arXiv 2025 Paper Code Unified multimodal architecture for perception + generation (images, text, audio, video) any-to-any multimodal tasks M2-Omni arXiv 2025 Paper Extends Omni-MLLM with broader modality support and competitive performance to GPT-4o any-to-any multimodal modeling Spider arXiv 2024 Paper Code Any-to-many multimodal LLM with flexible output heads for arbitrary modality combinations multimodal understanding and generation MIO arXiv 2024 Paper Token-level unified multimodal foundation model on discrete multimodal tokens any-to-any multimodal token modeling X-VILA arXiv 2024 Paper Cross-modality alignment for LLM-based multimodal systems (image/video/audio) multimodal understanding AnyGPT arXiv 2024 Paper Code Discrete token modeling for unified multimodal generation any-to-any multimodal generation OmniFlow CVPR 2025 Paper Uses multi-modal rectified flows for any-to-any generation across modalities any-to-any generation across modalities Video-LaVIT ICML 2024 Paper Code Decoupled visual-motion tokenization for video-language modeling video understanding and generation Unified-IO 2 CVPR 2024 Paper Code Scales autoregressive multimodal models across modalities any-to-any multimodal tasks (vision, language, audio, action) NExT-GPT arXiv 2023 Paper Code Any-to-any; encoder+LLM+diffusion decoders visual understanding, visual generation, omni 4.3 Related Generators (U+G Unverified) These boundary references support architecture comparisons and are not verified UMM members . The linked report establishes generation/editing but does not establish a separate image-understanding task; move an entry into the UMM taxonomy only when that evidence is available. Reference rows count toward the website’s catalog-entry totals, not a count of verified UMMs. Model Venue Links Paradigm Notes Task Qwen-Image-2.0 Technical Report arXiv 2026 Paper MMDiT diffusion Frozen Qwen3-VL conditions an MMDiT image generator with a high-compression VAE. The report establishes generation and editing; separate image-understanding task evidence is unverified, so this is a boundary reference rather than a verified UMM. image generation, editing Back to Top 5. Native Multimodal Models (NMMs) In this section: 5.1 Design Analyses & Scaling Laws · 5.2 Early Fusion NMMs · 5.3 Encoder-Based NMMs · 5.4 Any-to-Any / Omni NMMs · 5.5 Training-Native Models Two dimensions of nativity: architectural integration and multimodal optimization. Architecture-native entries may inherit pretrained backbones; training-native entries are distinguished from claims about fusion depth. From-scratch initialization is stated only where supported. Models may also retain their UMM classification. What recent arXiv work emphasizes: native multimodality is increasingly defined by end-to-end multimodal pretraining, tokenizer/representation co-design, and scaling strategies that explicitly address the asymmetry between vision and language. 5.1 Design Analyses & Scaling Laws Recent arXiv papers sharpen the definition of NMMs and identify the main bottlenecks in native multimodal pretraining. Paper Venue Links Insights Can Visual Generation Expand What Unified Multimodal Models Know? arXiv 2026-10-15 Paper Knowledge Transfer Probing supplies knowledge through image-generation supervision and tests access through text-only QA across seven categories and three UMMs. Transfer is uneven; textual and interleaved alignment improve access, without establishing a universal benefit from generation. More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training arXiv 2026-10-14 Paper Controlled from-scratch raw-patch MoE pretraining compares visual content and multimodal mixtures, including transfer to text-only spatial tasks. Shared decoder experiments span 71M–3B active parameters and 325B input tokens; this is a pretraining study rather than a separately released frontier model. Scaling Laws for Multimodal Data Mixtures NeurIPS 2026 Spotlight (author-reported); arXiv 2026-10-10 Paper Studies text/vision/audio mixtures through 268 MoE runs (457M–8.3B parameters, up to 150B tokens). Finds modality-specific compute optima and asymmetric interference within the tested mixtures; results and extrapolations retain this experimental scope. Multimodal Co-Generation With Continuous Bitstream Diffusion (CoBit) arXiv 2026-10-09 Paper Code A common bitstream-diffusion objective is tested with separately trained image–text, speech–text, and protein sequence–structure models. Denoisers train from scratch, while some experiments use frozen pretrained codecs. The image experiment uses MNIST-Sum; this is not one general-purpose omni checkpoint. How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining arXiv 2026-09-28 Paper Matched encoder-based and encoder-free pretraining compares 11 MoE decoder scales (1.1B–44B total; 71M–2.4B active parameters). Raw patches enter the shared decoder directly in the encoder-free branch. The projected crossover near 10²² FLOPs is an extrapolation beyond measured runs. Uncovering Understanding–Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System arXiv 2026-09-01 Paper Controlled pixel-in/pixel-out experiments study representation, task mixtures, routing, and planner/executor synergy. The backbone initializes from Qwen3-1.7B, so native describes architectural integration rather than whole-model training from scratch. Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes arXiv 2026-08-05 Paper Project Compares when vision enters pretraining, joint versus sequential schedules, representations, and shared versus modality-specific computation; scales to 13.5B MoE and 2T tokens. Default visual representations use a frozen pretrained SigLIP2, so the study does not establish all-components-from-scratch training. Scaling Native Multimodal Pre-Training From Scratch arXiv 2026 Paper Derives compute and allocation laws for transformer-based VLMs trained from scratch, showing distinct language and multimodal scaling behavior, data-mixture-sensitive multimodal allocation, and positive cross-modal transfer Toward Native Multimodal Modeling: A Roadmap arXiv 2026 Paper The end-to-end pipeline from industrial perspectives from architectural coordination, massive data curation, to full-stack training recipes, inference & deployment, and the comprehensive evaluation for truly native modeling Beyond Language Modeling: An Exploration of Multimodal Pretraining ICML 2026; arXiv 2026-03-03 Paper Proceedings Controlled Transfusion-style multimodal pretraining studies unified visual representations, understanding/generation and world-modeling transfer, MoE, and vision–language scaling asymmetry. NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints arXiv 2025 Paper Code End-to-end native MLLM scaling shows positive correlation between visual encoder and LLM size under data constraints; optimal meta-architecture balances cost and performance Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training arXiv 2025 Paper reveals that LLMs develop latent visual priors during text-only pre-training, where reasoning-centric data (code and math) builds transferable visual reasoning skills while broad corpora foster perception, enabling models to 'see' before ever processing an image. Scaling Laws for Native Multimodal Models arXiv 2025 Paper Early-fusion NMMs match or outperform late-fusion at low compute; early-fusion needs fewer params; MoE with modality-agnostic routing boosts sparse NMM scaling The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models arXiv 2024 Paper Native models often funnel image-to-text communication through a single post-image token 5.2 Early Fusion NMMs Modality tokens or embeddings enter shared backbone computation before its first block. Separate input encoders, tokenizers, experts, and decoders are compatible with early fusion. Architecture-native entries are explicitly qualified where pretrained initialization prevents an equally strong from-scratch claim. Recent scaling-law evidence suggests early-fusion NMMs are often stronger at lower parameter counts and simpler to deploy when paired with sufficiently strong visual representations. Model Paper Links Training Scale Notes Task NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference arXiv 2026-08-31 Paper HF 260M / 800M; 16K context Open-weight bidirectional single-tower encoder (Apache 2.0), trained from random initialization on text and raw image patches without a separate vision tower. Multimodal examples participate from backbone pretraining; masked text prediction supports representations and retrieval, not image generation or general chat. multimodal representations, visual document retrieval, multilingual understanding DeepSeek-V4.1-Flash Model release 2026-09-10 Announcement HF 552B backbone; 45T multimodal pretraining tokens; 1M context Open-weight model (MIT) trained from scratch on multimodal data; DeepSeek-ViT and a two-layer MLP produce image embeddings jointly processed with text from the start of language-model pretraining. Shared computation uses a Causal Encoder-Decoder architecture image/text understanding, visual reasoning, coding, long-context agents Qwen3.8-Flash-Next Model release 2026-08-26 Code HF Implementation 125B backbone / 6B active; 51B n-gram embeddings + 4B MTP Open-weight experimental architecture preview under Qwen Community License 1.0; architecture-native early fusion inserts image/video embeddings into the shared language-backbone input. Combines Gated DeltaNet, Qwen Sparse Attention, and gated residuals; all-components-from-scratch initialization is unverified image/video understanding, visual reasoning, coding, long-horizon agents Qwen3.8-27B Model release 2026-08-14 Code HF 27B dense; 262K native context, extensible to 1M Open-weight architecture-native VLM built on Qwen3.5's early-fusion vision-language architecture; image/video embeddings and text share the language backbone. Training stage is pretraining plus post-training; all-components-from-scratch initialization is unverified image/video understanding, document understanding, visual reasoning, agentic work Inkling-Small Model release 2026-07-30 Announcement Model Card HF 276B total / 12B active Open-weight MoE model with a hierarchical image-patch encoder and discrete audio encoding; text, image, and audio representations enter a shared hidden space and are jointly processed by one decoder. Outputs text; fusion classification does not assert all-components-from-scratch initialization text/image/audio understanding, visual and audio reasoning, coding, agentic tool use Inkling Thinking Machines Lab, 2026 Announcement Model Card HF 975B total / 41B active; 45T multimodal pretraining tokens; 1M context Open-weight MoE model trained from scratch with encoder-free vision and audio inputs; images use 40×40 patches with a lightweight four-layer hMLP, while audio uses dMel spectrograms, jointly processed with text tokens text/image/audio understanding, visual and audio reasoning, speech transcription, coding, agentic tool use Gemma 4 12B Unified Model release 2026-06-03; report 2026-07-02 Paper Announcement HF 12B Open-weight encoder-free architecture (Apache 2.0): lightweight linear layers project raw image patches and audio waveforms into the shared decoder’s embedding space. Listed separately from encoder-based Gemma 4 variants; Unified refers to input architecture, with text-only output. The shared technical report first appeared 2026-07-02. image/video/audio understanding, multimodal reasoning, on-device agents HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers arXiv 2026 Paper 7B dense model Native unified multimodal model with a holistic image-video ViT tokenizer, covering image/video understanding, image/video generation, and image editing image/video understanding, image/video generation, editing NEO-OV arXiv 2026 Paper — Native one-vision scaling from pixels to words, extending the NEO line toward stronger native visual primitives vision-language understanding SenseNova-U1.5-8B-MoT Model release 2026-08-20 Paper Code HF — NEO-unify-based encoder-free MoT model with improved patchify layers, native 4K generation, text rendering, and image editing; grouped with U1 by architecture, with from-scratch initialization unverified visual understanding, image generation, editing, interleaved generation SenseNova-U1 arXiv 2026 Paper Code — NEO-Unify-based native unified multimodal model handling understanding, generation, and reasoning in one model Unified Understanding & Generation HiDream-O1-Image arXiv 2026 Paper — Natively unified image generative foundation model with pixel-level unified transformer image generation, unified modeling Tuna-2 arXiv 2026 Paper — Native unified multimodal model that discards traditional vision encoders in favor of direct pixel embeddings for end-to-end understanding and generation Unified Understanding & Generation NEO arXiv 2025 Paper — Native vision-language primitives at scale; paired with reusable components for cost-effective native VLM development vision-language understanding NEO-Unify Blog 2025 Blog — Native unified extension of NEO for understanding, generation, and reasoning Unified Understanding & Generation Emu3.5 Nature 2026 Paper Code Large-scale (trillion+ tokens) Native world model; next-state prediction on interleaved video/text; Discrete Diffusion Adaptation for efficiency interleaved generation, world modeling, any-to-image LongCat-Next arXiv 2026 Paper — Early fusion: DiNA represents text, vision, and audio as discrete tokens processed by a shared modality-agnostic autoregressive backbone; Omni capabilities text/image/audio understanding and generation Llama4 arXiv 2026 Paper Blog Scout/Maverick: 17B active / ~109B–400B total; Behemoth: ~2T total Architecture-native early fusion of text and vision tokens in a shared MoE backbone; separately trained vision encoder, so early fusion does not imply all components were trained from scratch vision-language understanding Emu3 arXiv 2024 Paper Code 8B Early-fusion autoregressive backbone jointly models text, image, and video tokens; also indexed under UMM / AR / Pixel Encoding visual understanding, visual generation Chameleon arXiv 2024 Paper Code — Early-fusion backbone trained from scratch on interleaved image-text tokens; separate visual tokenizer; also listed under UMM / AR / Pixel Encoding visual understanding, visual generation Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model ICLR 2025 Paper 7B Early-fusion shared Transformer jointly models text tokens and continuous image latents with next-token prediction and diffusion; also listed under Hybrid UMMs visual understanding, visual generation Show-o: One Single Transformer to Unify Multimodal Understanding and Generation ICLR 2025 Paper Code — Architecture-native early fusion with a pretrained language-model initialization; shared Transformer combines autoregressive text modeling and masked image-token diffusion; UMM entry multimodal understanding and generation Show-o2: Improved Native Unified Multimodal Models NeurIPS 2025 Paper Code — Architecture-native early fusion with autoregressive text modeling and visual flow matching; uses a pretrained language backbone, not an all-components-from-scratch model; UMM entry multimodal understanding and generation OneCat arXiv 2025 Paper Code — Architecture-native early fusion with a Qwen2.5-initialized decoder and modality-specific experts; unified autoregressive understanding and generation; UMM entry visual understanding, visual generation, editing 5.3 Encoder-Based NMMs Encoder-based systems discussed in the native multimodal literature. A vision encoder may be pretrained or trained from scratch; its presence alone establishes neither strict late fusion nor whole-model training from scratch. The notes qualify the available training evidence independently, and leave the first cross-modal interaction layer unspecified unless supported by a source. Model Paper Links Training Scale Notes Task Mistral Large 4 API preview 2026-10-06 Announcement Docs 1.05T total / 52B active; 1.6B vision encoder; 1M context Officially described as a natively multimodal model trained from scratch, with text/image understanding and text output. The exact encoder-to-backbone fusion depth is undisclosed. API public preview is available; weights are promised by end-October 2026 and were not released as of 2026-10-19. visual understanding, grounding, document reasoning, coding, agents Kimi K3: Open Frontier Intelligence arXiv 2026 Paper HF 2.8T total / 104B active; 1M context Joint language-vision training from the outset; MoonViT-V2 is trained from scratch and connected to the language backbone through pixel-shuffle downsampling and an MLP projector image/video understanding, visual reasoning, long-horizon agents Kimi K2.6 Moonshot AI 2026 Blog MoE Architecture: 32B active / 1T total parameters; supports 256K context Native multimodal MoE with MLA (Multi-head Latent Attention) and MoonViT encoder. Encoder initialization and strict fusion depth are not established in this entry; native training terminology does not imply all components were trained from scratch. Unified multimodal understanding, long-horizon coding, and agent swarms GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents arXiv 2026 Paper — Native multimodal foundation model featuring CogViT vision encoder, Multimodal Multi-Token Prediction (MMTP), and joint RL for agentic GUI/Design2Code tasks. Encoder initialization and strict fusion depth are not established in this entry; native training terminology does not imply all components were trained from scratch. multimodal agentic reasoning LLaVA-OneVision-2 arXiv 2026 Code HF 8B Native multimodal training with OneVision-Encoder and Qwen3 text backbone; codec-aligned visual encoding for image, long-video, spatial, document/OCR/chart understanding. Encoder initialization and strict fusion depth are not established in this entry; native training terminology does not imply all components were trained from scratch. visual understanding, video, spatial reasoning Qwen3.5 - Blog — Discrete Native Any-resolution Visual Transformer. Encoder initialization and strict fusion depth are not established in this entry; native training terminology does not imply all components were trained from scratch. vision-language understanding Gemma 4 (encoder-based variants) Model release 2026-04-02; report 2026-07-02 Paper Announcement Model Card — First public: 2026-04-02. Technical report first submitted 2026-07-02. Open-weight encoder-based Gemma 4 variants (Apache 2.0) with multimodal pretraining and text output; the public evidence cited here does not establish exact encoder initialization or strict fusion depth. The encoder-free 12B Unified variant is listed separately above. vision-language understanding 5.4 Any-to-Any / Omni NMMs The latest arXiv-native multimodal papers increasingly blur the boundaries between omni understanding, any-to-any generation, world modeling, and RL-enhanced post-training. Model Paper Links Training Scale Notes Task HyperCLOVA X 8B Omni arXiv 2026 Paper HF 8B Any-to-any omnimodal model unifying text, audio, and vision through a shared next-token prediction interface over interleaved multimodal sequences text/audio/vision understanding and generation MiniMax-M3 MiniMax 2026 Blog Code ~428B total / ~23B active; 1M context Native multimodal model with MiniMax Sparse Attention (MSA), trained for text, image, and video understanding, long-context coding, agentic workflows, and computer-use tasks native multimodal understanding, video, coding, agentic computer use The Design Space of Tri-Modal Masked Diffusion Models NeurIPS 2026; arXiv 2026-02-25 Paper Apple Research 3B; 6.4T tokens From-scratch tri-modal masked-diffusion pretraining over text, image–text, and audio–text data; analyzes scaling, modality mixing, noise schedules, batch size, and inference. Apple’s October NeurIPS listing updates the venue, not the first-paper date. text generation, text-to-image, text-to-speech Qwen3.5-Omni Qwen Blog 2026 Blog — Discrete native any-resolution visual transformer with omni-modal extension vision-language understanding, omni ERNIE 5.0 Technical Report arXiv 2026 (Late fusion) Paper — Natively autoregressive foundation model designed for unified multimodal understanding and generation across text, image, video, and audio vision-language understanding, omni 5.5 Training-Native Models These entries record a native multimodal training route separately from fusion topology. This qualification does not claim that every component is trained from scratch; no Early, Mid, or Late Fusion label is assigned without architectural evidence. Model Paper Links Training Scale Notes Task InternVL3 arXiv 2025 Paper — Training-native route: native multimodal pretraining within a ViT–MLP–LLM architecture; pretrained components are retained. This training qualification does not establish strict late or mid fusion; fusion depth is left unspecified here. vision-language understanding InternVL3.5 arXiv 2025 Paper — Training-native route: native multimodal pretraining within a ViT–MLP–LLM architecture; pretrained components are retained. This training qualification does not establish strict late or mid fusion; fusion depth is left unspecified here. vision-language understanding Back to Top 6. Closed-Source Multimodal Models Year 2026 Model Venue Links Notes Task Claude Haiku 5.5 Anthropic; release 2026-10-07 Announcement Models Efficient model for focused, high-volume work; proprietary Claude/API release supporting text/image inputs and text output. A distinct 5.5-family release rather than a rename of Claude 5 or Fable/Mythos 5.1. visual understanding, reasoning, coding, computer use, agents GPT-6.1 Sol OpenAI; API release 2026-09-29 Model Changelog Proprietary reasoning model accepting text/image inputs and producing text, available through the Responses and Chat Completions APIs. Supports visual reasoning, computer use and coding; image_generation is an external tool capability, not evidence of a native image-generation backbone. visual understanding, reasoning, coding, computer use, agents Claude Sonnet 5.5 Anthropic; release 2026-09-28 Announcement Models Balanced model for coding and agentic work; proprietary Claude/API release supporting text/image inputs and text output. A distinct 5.5-family release rather than a rename of Claude 5 or Fable/Mythos 5.1. visual understanding, reasoning, coding, computer use, agents Sarvam Vision 2.1 Model release 2026-09-24 Announcement Digitize API Extract API First public: 2026-09-24. Proprietary document VLM with semantic-layout and pointer reading-order harnesses; document-page/text inputs produce structured text, tables and key-value fields, including Indic handwriting. Digitize and Extract APIs available; open checkpoints not established as of 2026-10-19. Backbone initialization and vision-to-language interface are undisclosed; benchmark claims are author-reported. multilingual document understanding, Indic OCR, handwriting, structured extraction GPT-6 Sol / GPT-6 Luna OpenAI; API release 2026-09-22 Sol Luna Changelog Proprietary reasoning models accepting text/image inputs and producing text through the Responses and Chat Completions APIs. Sol targets complex coding and agent workflows; Luna targets efficient focused tasks. Retained separately from GPT-6.1 Sol, GPT-6 Astra, and the GPT-5.6 family. visual understanding, reasoning, coding, computer use, agents Claude Opus 5.5 Anthropic; release 2026-09-22 Announcement Models Flagship model for complex coding, computer use, and professional work; proprietary Claude/API release supporting text/image inputs and text output. A distinct 5.5-family release rather than a rename of Claude 5 or Fable/Mythos 5.1. visual understanding, reasoning, coding, computer use, agents Gemini 3.8 Live / Live Extended Thinking Google DeepMind; release 2026-09-15 Announcement Live Avatar Proprietary live-dialogue models combine audio dialogue, near-real-time visual grounding, and asynchronous tools; Extended Thinking reasons while speaking. Available through Gemini Live/API services. The 2026-09-24 Live Avatar extension adds streamed speech/video responses in Gemini Enterprise; its announcement does not establish one shared native video-generation backbone. audio-visual understanding, speech generation, live dialogue, tool use GPT-6 Astra OpenAI, 2026 Announcement System Card September 2026 proprietary model with text/image input and text output, supporting visual reasoning, coding, computer use, and long-horizon agentic work visual understanding, multimodal reasoning, coding, computer use, agents Claude Fable 5.1 / Mythos 5.1 Anthropic, 2026 Announcement Models September 2026 release; Fable 5.1 and Mythos 5.1 are the same underlying model with different safeguards. Fable is generally available, while Mythos is restricted to trusted access programs; grouped as one entry visual understanding, reasoning, coding, computer use, long-horizon agents Muse Spark 1.3 Meta; September 2026 release Announcement Muse Spark Open Proprietary Muse Spark update for multimodal perception, coding, and agentic work through Muse Code and Meta Model API. Meta separately released Muse Spark Open weights on 2026-10-15; this does not establish that the proprietary 1.3 API model has identical weights or license. Status rechecked 2026-10-19. visual understanding, multimodal reasoning, coding, agents Gemini 3.8 Flash Google DeepMind, 2026 Model Card September 2026 Gemini Flash model supporting text, image, audio, and video inputs with text output; up to 1M input context, with configurable reasoning effort multimodal understanding, video/audio understanding, reasoning, coding, agents Grok 4.6 xAI, 2026 Announcement Model Docs August 2026 proprietary model with text/image input and text output; extends the Grok family with visual work, coding, and long-running agent workflows visual understanding, multimodal reasoning, coding, agents Claude 5 Family (Opus 5 / Sonnet 5) Anthropic, 2026 Opus 5 Sonnet 5 Models Opus 5 (July 2026) and Sonnet 5 (June 2026) support text/image input and text output, with vision, reasoning, tool use, and agentic coding; retained separately from the Fable/Mythos family visual understanding, reasoning, coding, computer use, agents Seed2.1 Pro / Turbo ByteDance Seed, 2026 Announcement Models June 2026 proprietary Seed2.1 family; Pro and Turbo provide different model sizes for visual/video understanding, reasoning, software engineering, and cross-tool agent workflows visual/video understanding, multimodal reasoning, coding, agents GPT-5.6 Sol / Terra / Luna OpenAI, 2026 Announcement July 2026 general-availability release of the GPT-5.6 family: Sol is the flagship model, Terra balances capability and cost, and Luna prioritizes speed and affordability. Multimodal Reasoning + Computer Use + Agentic Work GPT-5.4 / GPT-5.5 OpenAI Blog GPT-5.4 GPT-5.5 2026 GPT-5-series updates with improved reasoning, multimodal capability, and deployment efficiency. Omni-Modal + Professional/Agentic Workflows Claude 4.6 Family (Opus 4.6 / Sonnet 4.6) Anthropic Blog Opus 4.6 Sonnet 4.6 Claude 4.6 proprietary model family with vision, coding, tool use, and computer-use workflows. Multimodal + Agentic/Coding/Computer-Use Gemini 3.1 Pro / Gemini 3.5 Google Blog Gemini 3.1 Pro Gemini 3.5 2026 Gemini 3-series updates with stronger multimodal reasoning, long-context, and action-oriented capabilities. Frontier Multimodal (text/image/audio/video + reasoning) Year 2025 Model Venue Links Notes Task Gemini 3 / Gemini 3 Pro Google Blog Gemini 3 Native multimodal Gemini 3 generation with advanced reasoning, coding, and long-context capabilities. Frontier Multimodal + Reasoning GPT-5 / GPT-4.5 OpenAI Blog GPT-5 GPT-4.5 Proprietary OpenAI GPT-series updates with multimodal reasoning and tool-use support. Omni-Modal + Advanced Reasoning/Agentic Claude 4 Family (Opus 4 / Sonnet 4) Anthropic Blog Claude 4 Claude 4 model family for coding, agentic tasks, vision, and extended tool use. Vision + Advanced Reasoning/Agentic Workflows Grok 3 / Grok 4 / Grok 4.1 xAI Announcement Grok 3 Grok 4 Grok 4.1 Proprietary xAI models with multimodal input, real-time platform integration, and reasoning-focused releases. Multimodal Reasoning + Real-Time Integration Gemini 2.0 / 2.5 (Pro / Flash) Google Blog Gemini 2.0 Gemini 2.5 Pro Gemini 2.5 Updates Native multimodal Gemini 2-series models with long context, video/audio/image understanding, and agentic features. Advanced Native Multimodal + Agentic Mistral Medium 3 Mistral AI Announcement Proprietary Mistral model offering text and vision capabilities through API/platform deployments. General Multimodal Tasks Year 2024 Model Venue Links Notes Task Gemini 1.5 (Pro / Flash) Google DeepMind Blog Gemini 1.5 Announcement Released February 2024. Massive context (>1M tokens), strong long-context multimodal (video, audio, images). Proprietary. Long-Context Multimodal (video/audio/image/text) Claude 3 Family (Opus / Sonnet / Haiku) Anthropic Blog Claude 3 Family Released March 2024. Strong native vision for images, charts, diagrams, and documents. Proprietary API + Claude.ai. Vision-Language + Reasoning Claude 3.5 Sonnet Anthropic Blog Announcement Stronger proprietary Claude vision/reasoning model used for images, documents, charts, and coding workflows. Vision-Language + Reasoning GPT-4o / GPT-4o mini OpenAI Blog GPT-4o GPT-4o mini GPT-4o introduced real-time omni interactions; GPT-4o mini provided a lower-cost multimodal model. Real-Time Omni-Modal (text/vision/audio) Grok-1.5V / Grok-2 Vision xAI Announcement Grok-1.5V Grok-2 Vision-capable Grok models for image understanding, diagram reasoning, and X platform integration. Vision-Language Amazon Nova (Pro / Lite / Canvas / Reel) Amazon Announcement AWS Blog Amazon Bedrock model family covering multimodal understanding and image/video generation variants. Multimodal Understanding + Generation Pixtral Large Mistral AI Announcement Proprietary frontier multimodal Mistral model for image-text understanding through Le Chat/API deployments. Vision-Language Year 2023 Model Venue Links Notes Task GPT-4V (Vision) OpenAI Announcement GPT-4V System Card Released September 2023. First widely available multimodal GPT-4 variant. Image + text input, text output. API/ChatGPT access only. Vision-Language (image understanding, VQA, OCR, document analysis, captioning) Gemini 1.0 (Ultra / Pro / Nano) Google DeepMind Blog Gemini Announcement Released December 2023. Native multimodal from training (text + image + audio + video). Proprietary API + Gemini chatbot. Native Multimodal Understanding (text/image/audio/video) 7. Resources In this section: 7.1 Related Awesome Lists · 7.2 Slides & Survey Papers · 7.3 Code Repositories & Tools · 7.4 Benchmarks & Evaluation 7.1 Related Awesome Lists Repository Focus Author awesome-multimodal-ml General multimodal ML pliang279 Awesome-Multimodal-Large-Language-Models MLLMs + evaluation BradyFU Awesome-Multimodal-Research Broad multimodal research Eurus-Holmes Awesome-Unified-Multimodal-Models UMMs ShowLab Awesome-Multimodal-Large-Language-Models MLLMs yfzhang114 awesome-foundation-and-multimodal-models Foundation + multimodal SkalskiP Awesome-Multimodality General multimodality Yutong-Zhou-cv Awesome-Unified-Multimodal Unified models Purshow Awesome-Unified-Multimodal Unified models AIDC-AI 7.2 Slides & Survey Papers Type Resource Notes Slides Native LMM Slides Ziwei Liu (NTU); concise framing for native multimodal models Survey A Survey on Multimodal Large Language Models Broad survey of MLLM architectures, data, and evaluation Report The Dawn of LMMs: Preliminary Explorations with GPT-4V Early capability analysis around GPT-4V Survey Multimodal Foundation Models: From Specialists to General-Purpose Assistants Broader foundation-model view across multimodal systems 7.3 Code Repositories & Tools Tool Description Link TorchUMM Unified evaluation, analysis and post-training toolkit for heterogeneous unified multimodal model architectures, tasks and datasets Paper Code LMMs-Eval Unified evaluation harness for multimodal models Code LAVIS Library for Language-Vision Intelligence (Salesforce) Code OpenFlamingo Open reproduction of DeepMind Flamingo Code xtuner Efficient fine-tuning for multimodal LLMs Code LLaMA-Factory Multimodal instruction tuning framework Code MMEngine Foundation for perception research (OpenMMLab) Code DeepSpeed-VisualChat Scalable multimodal chat training Code 7.4 Benchmarks & Evaluation Evaluation artifacts are catalogued separately from model architectures. Dataset coverage, agentic baselines and model-native input/output capabilities are distinct. The companion website exposes these rows in its Evaluation collection. Unified Understanding & Generation Paper Venue Links Type Notes Task Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation ICML 2026 Paper Code Project Benchmark First public: 2026-06-25. Provides joint and decoupled tracks for internal consistency, understanding-guided generation, generation-guided understanding and mutual enhancement; 2,169 task samples plus Unison-Judge. As of 2026-10-19: code, benchmark and judge public. joint understanding/generation, synergy evaluation UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? arXiv 2026-03-03 Paper Benchmark First public: 2026-03-03. Seven regimes and 30 subtasks test when visual generation benefits understanding, including implicit/explicit transformations and multi-step reasoning. Improvements are task-dependent; generating before answering is not uniformly beneficial. As of 2026-10-19: paper public; code/data release not confirmed by inspected sources. generation-to-understanding evaluation UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark arXiv 2025-12-19 Paper Project Benchmark First public: 2025-12-19. Assesses understanding, generation and editing in one process, with decoupled tracks for diagnosis; covers 13 domains and over 200 concepts. As of 2026-10-19: paper and project public; resource downloads require checking the project. understanding, generation, editing evaluation MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models ICLR 2026 Paper Proceedings Project Benchmark First public: 2025-04-04. Standardizes understanding/generation tasks drawn from 12 datasets and introduces five mixed-modality tasks for joint reasoning. Also called MME-U in the proceedings. As of 2026-10-19: official project links evaluation code and data. understanding, generation, mixed-modality reasoning evaluation Interleaved & Any-to-Any Evaluation Paper Venue Links Type Notes Task LongMIG-Bench: Long-Horizon Interleaved Generation Evaluation arXiv 2026-10-12 Paper Benchmark First public: 2026-10-12. Benchmark introduced with Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation; evaluates text-to-multi-image and image-to-multi-image trajectories, character/scene consistency and long-range continuity. As of 2026-10-19: paper public; standalone benchmark download not confirmed. long-horizon interleaved generation evaluation IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation arXiv 2026-06-08 Paper Benchmark First public: 2026-06-08. Multi-turn interleaved image-text evaluation with static spatial, temporal causal and hybrid tasks; 3,113 samples and 12,034 turns expose understanding/generation errors and exposure bias. As of 2026-10-19: paper public; code/data release not confirmed by inspected sources. multi-turn understanding/generation, exposure-bias evaluation UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark CVPR 2026 Paper Project Code Data Benchmark First public: 2026-03-05. Contains 31K instances across 30 domains and seven modalities; measures semantic/generation quality, response structure and interleaved coherence. UniMA is its agentic baseline, not evidence of a single backbone generating every modality. CVPR venue is author-reported. As of 2026-10-19: code and CC BY 4.0 data public; official repository records data release 2026-04-18 and evaluation scripts 2026-05-21. any-to-any interleaved multimodal evaluation General Multimodal & Robustness Paper Venue Links Type Notes Task FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models NeurIPS 2026 (author-reported); arXiv 2026-10-15 Paper Evaluation method PaRMA audits pathway-conditioned robustness using pixel perturbations and rendered-text insertion across 18 open-weight VLMs; FusionAudit instantiates diagnostic-guided attacks. An auditing method, not a foundation model; native categories and conclusions follow the authors’ tested taxonomy. robustness auditing, pixel attacks, rendered-text attacks OmniGAIA: Towards Native Omni-Modal AI Agents arXiv 2026-02-26 Paper Code Benchmark First public: 2026-02-26. Benchmark for omni-modal agents with multi-hop queries across video, audio and images; includes the separate OmniAtlas tool-using agent baseline. This is an evaluation artifact, not a visual-adapter category. omni-modal agent evaluation MultiBench: Multiscale Benchmarks for Multimodal Representation Learning NeurIPS 2021 Paper Benchmark Benchmark suite for multimodal representation learning and cross-modal robustness across tasks and modalities; retained as a general evaluation reference rather than a model architecture. multimodal representation evaluation Image Generation Evaluation Paper Venue Links Type Notes Task ImageBench Live benchmark Site Methodology Benchmark Live text-to-image evaluation using VLM judges, with generated images published for inspection. Ranking/model counts change over time; consult the linked methodology and current results. This evaluates image generation and does not establish U+G membership. text-to-image evaluation Back to Top How to Contribute In this section: Validation Rules · Entry Format We welcome contributions! Please follow these guidelines: Validation Rules For NMM submissions: Record architectural integration and multimodal training history separately Identify pretrained backbones, encoders, tokenizers, and the trainable parameters at each stage Qualify architecture-native or training-native entries; claim from-scratch training only with supporting evidence Assign fusion depth only with architectural evidence. Use Encoder-Based (§5.3) when the input architecture is established but strict fusion depth is unknown; use Training-Native (§5.5) when only the training route is established, leaving fusion unspecified For UMM submissions: Confirm the model handles both image understanding AND image generation Note whether pretrained components are used (annotate accordingly) For MLLM submissions: Identify the pretrained vision and language components and the visual-to-language interface for adapter-based classification Use Interface Unspecified only as a provisional understanding-model reference when those details are unavailable; do not infer initialization or native training For evaluation submissions: Use §7.4 and state whether the artifact is a benchmark or an evaluation method Describe evaluated tasks separately from any model or agent baseline Distinguish paper, code and data availability, licenses and dates Entry Format Match the column headers of the destination table. For a typical five-column model table: | Paper | Venue | Links | Notes | Task | | --- | --- | --- | --- | --- | | <!-- entry:REPLACE_WITH_UNIQUE_ID --> Model Name | arXiv 2026 | [ Paper ] ( https://arxiv.org/abs/PAPER_ID ) [ Code ] ( https://github.com/ORG/REPO ) [ HF ] ( https://huggingface.co/ORG/MODEL ) | Model size; architecture and training qualifications | visual understanding | For a six-column NMM table: | Model | Paper | Links | Training Scale | Notes | Task | | --- | --- | --- | --- | --- | --- | | <!-- entry:REPLACE_WITH_UNIQUE_ID --> Model Name | arXiv 2026 | [ Paper ] ( https://arxiv.org/abs/PAPER_ID ) [ Code ] ( https://github.com/ORG/REPO ) | 7B | Architecture, initialization and trainable components | visual understanding, image generation | Every catalog row has an immutable hidden entry ID in its first cell. For a new row, replace REPLACE_WITH_UNIQUE_ID with a fresh value generated by: node -e " console.log(require('node:crypto').randomUUID()) " Keep the existing ID when correcting a title, venue, link, or category. A cross-listed row gets its own ID; copied templates must receive a new one. Missing or duplicated IDs fail validation. When merging duplicate rows or removing an incorrect cross-listing, add an explicit old-ID → surviving-ID alias in data/entry-identities.json so existing saved entries and detail URLs remain usable. Submit a PR with: The paper/model entry in the correct section A one-line justification for the chosen category Links to paper, code, and/or weights Back to Top Citation If this list is useful in your research, please consider citing: @misc { awesome-multimodal-modeling-2026 , title = { Awesome Multimodal Modeling: From Traditional to Native & Unified } , author = { {OpenEnvision Lab} } , year = { 2026 } , url = { https://github.com/OpenEnvision/Awesome-Multimodal-Modeling } , note = { GitHub repository } } Back to Top ⭐ Star Growth of Awesome-Multimodal-Modeling License This list is released under the CC0 1.0 Universal license. Maintained by the community for the multimodal research community. Back to Top Companion Website The companion research library is designed for openenvision.github.io/Awesome-Multimodal-Modeling/ , under the OpenEnvision organization site. 📄 Survey paper — Coming soon. A survey of multimodal foundation models is in preparation. 💻 GitHub repository . Explore the list, suggest entries, and contribute. The website derives its catalog and category memberships from this README. It supports search, nested category filters, year and resource filters, sorting, pagination, entry details, shareable filter URLs, and a browser-local reading list. The dedicated Evaluation collection derives from §7.4; other resource tables retain their README format. Counts represent entries, including cross-listed models, analysis papers, and explicitly unverified boundary references, rather than unique or verified models. Files ready to publish index.html lives in the repository root, beside README.md . It is a complete static page, with its styles, scripts, icons, diagrams, and catalog in assets/site/ . The legacy website/index.html entry redirects to the root page; edit the root index.html and assets/site/ when maintaining the website. Awesome-Multimodal-Modeling/ ├── index.html # Website entry point, ready to serve ├── README.md # Source catalog and project documentation ├── .nojekyll # Serve static files directly on GitHub Pages ├── robots.txt ├── sitemap.xml ├── assets/ │ ├── site/ # Website styles, scripts, icons, diagrams, and catalog.json │ └── ... # Existing README illustrations ├── scripts/ # Catalog synchronization and local preview tools └── .github/workflows/pages.yml Publish index.html together with assets/ ; keep their relative paths intact. No server, package installation, or API key is required to host the prepared files. Publish from the repository root Commit the root index.html , README.md , assets/ , .nojekyll , robots.txt , and sitemap.xml to main . Open Settings → Pages → Build and deployment . Choose Deploy from a branch , select main and /(root) , then save. The site will be available at openenvision.github.io/Awesome-Multimodal-Modeling/ . The custom workflow detects branch publishing and skips its own deployment. Keep the website synchronized After editing the model list, run: npm test npm run build Node.js 22 or later is sufficient. The build refreshes the generated catalog blocks inside the root index.html , writes assets/site/catalog.json , and exports a clean copy to dist/ . Commit the refreshed root files when publishing from a branch. Page content outside the marked catalog blocks can be edited directly in the root index.html . For automatic catalog synchronization on every push, choose GitHub Actions as the Pages source and keep the included workflow. It checks pull requests, rebuilds the catalog from README changes, and deploys successful main builds. A first deployment can also be started with Run workflow . The generated dist/ directory is only used as the Actions deployment artifact and is excluded from version control. See GitHub's publishing-source documentation . Local preview Build the prepared files before starting production preview: npm run build npm run preview Open http://127.0.0.1:4173/Awesome-Multimodal-Modeling/ . This serves the prepared files directly from the repository root, including the .nojekyll marker generated by the build. Use npm run dev for an in-memory preview that watches README.md , index.html , and assets/ ; reload the browser after edits. The project uses relative asset URLs. It needs no custom domain, CNAME file, or change to the OpenEnvision organization's main website. The brand mark comes from the OpenEnvision site . The GitHub icon comes from GitHub Octicons ; its MIT license is included in assets/site/OCTICONS-LICENSE.txt . Model membership, release annotations, and availability qualifications remain governed by the README.
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
Multi-Modal-Fusion Early Fusion, Late Fusion, and Hybrid Fusion on Emotion Recognition Physiological Data Multimodal Fusion Techniques for Machine Learning October 2024 / Lugano / Switzerland This repository contains various multimodal fusion methods for combining data from different modalities (Eye Tracking, GSR, ECG). The repository explores Early Stage , Late Stage , and Hybrid Fusion approaches to improve classification performance on fused datasets. Overview Multimodal fusion aims to combine data from multiple sources to make more accurate predictions by utilizing the strengths of each modality. This project implements fusion techniques using concatenation , autoencoders , CNN , LSTM , PCA , SVM , and XGBoost models. It also explores voting mechanisms and attention-weighted approaches in hybrid models. Early Stage Fusion In early stage fusion, the features from each modality are concatenated at the feature level before classification. Here are the implementations: Auto Encoder Early Stage Fusion : Combines features from all modalities using autoencoders to reduce dimensionality before classification. CNN Early Stage Fusion : Applies Convolutional Neural Networks (CNN) for feature extraction and then concatenates these features for classification. Concatenation Early Stage Iterative Imputation Fusion : Concatenates the features from each modality with iterative imputation for missing values. Concatenation Early Stage KNN Imputation Fusion : KNN imputation is used for handling missing data before concatenation. Concatenation Early Stage with NaNs Fusion : Directly concatenates the features from all modalities without imputing NaNs. GCCA Early Stage KNN Imputation Fusion : Applies Generalized Canonical Correlation Analysis (GCCA) with KNN for imputation before concatenation. LSTM Early Stage Fusion : Combines features using LSTM networks for sequential data and then concatenates for classification. Late Stage Fusion In late stage fusion, predictions from different models (each trained on one modality) are combined to form the final classification: Auto Encoder Late Stage Fusion : Autoencoders are applied to each modality, and the final decision is made by combining predictions. CNN Late Stage Fusion : CNN is applied to each modality separately, and the final classification is achieved by combining predictions. Majority Voting Late Stage Fusion : Combines the predictions of classifiers for each modality through majority voting. Weighted Averaging Late Stage Fusion : Combines classifiers' predictions through weighted averaging for final classification. Hybrid Fusion Hybrid fusion uses a combination of early and late stage techniques, often with specialized mechanisms like attention or PCA: Attention-Weighted XGBoost Hybrid Fusion : Attention mechanism is applied to each modality, and XGBoost is used to classify the attention-weighted features. CNN and LSTM Hybrid Fusion : Combines CNNs for feature extraction and LSTMs for sequence modeling, followed by classification. PCA and LDA Hybrid Fusion : Principal Component Analysis (PCA) is used for dimensionality reduction, followed by Linear Discriminant Analysis (LDA) for classification. PCA and SVM Hybrid Fusion : PCA is used for dimensionality reduction, followed by a Support Vector Machine (SVM) classifier for final classification. Data The dataset includes features extracted from three modalities: Eye Tracking ( EyeTracking.csv ) GSR ( GSR.csv ) ECG ( ECG.csv ) Download the dataset from: https://www.kaggle.com/datasets/lumaatabbaa/vr-eyes-emotions-dataset-vreed
4. Neural Network Backbone, Layer Specifications & Loss Formulations
salesforce / LAVIS Star 11.3k LAVIS - A One-stop Library for Language-Vision Intelligence deep-learning salesforce image-captioning deep-learning-library vision-framework vision-and-language multimodal-deep-learning multimodal-datasets vision-language-transformer vision-language-pretraining visual-question-anwsering Updated Sep 18, 2026 Jupyter Notebook
5. Training Protocol, Convergence Behavior & Experimental Evaluation
Artificial-Intelligence-Fusion-Resources Journal IEEE Transactions on Multimedia (TMM) IEEE Multimedia Magazine (MM) Information Fusion (IF) Coference ACM International Conference on Multimedia (ACM MM) IEEE International Conference on Multimedia & Expo (ICME) Group Multimodal Communication and Machine Learning Laboratory (MultiComp Lab) Louis-Philippe Morency Carnegie Mellon University [ Homepage ] Resource awesome-multimodal-ml [ Github ] Awesome-Multimodal-Research [ Github ] Review Multimodal Machine Learning A Survey and Taxonomy. Tadas Baltrušaitis; Chaitanya Ahuja; Louis-Philippe Morency. TPAMI, 2019. [ PDF ] Deep multimodal learning A survey on recent advances and trends. D Ramachandram, GW Taylor. TIP, 2017. [ PDF ] Multimodal intelligence: Representation learning, information fusion, and applications. C Zhang, Z Yang, X He, L Deng. IEEE Journal of Selected Topics in Signal Processing, 2020. [ PDF ] A survey on multimodal large language models. S Yin, C Fu, S Zhao, K Li, X Sun, T Xu, et al. arXiv, 2023. [ PDF ] Multimodal foundation models: From specialists to general-purpose assistants. C Li, Z Gan, Z Yang, J Yang, L Li, L Wang, J Gao. Foundations and Trends® in Computer Graphics and Vision, 2024. [ PDF ] Multimodality representation learning: A survey on evolution, pretraining and its applications. MA Manzoor, S Albarri, Z Xian, Z Meng, P Nakov, S Liang. ACM Transactions on Multimedia Computing, Communications and Applications, 2023. [ PDF ] Vlp: A survey on vision-language pre-training. FL Chen, DZ Zhang, ML Han, XY Chen, J Shi, S Xu, B Xu. Machine Intelligence Research, 2023. [ PDF ] Vision-language models for vision tasks: A survey. J Zhang, J Huang, S Jin, S Lu. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. [ PDF ] Mm-llms: Recent advances in multimodal large language models. D Zhang, Y Yu, C Li, J Dong, D Su, C Chu, D Yu. arXiv:2401.13601, 2024. [ PDF ] Representation DeViSE: A Deep Visual-Semantic Embedding Model. A Frome, G Corrado, J Shlens, S Bengio, J Dean, et al. NIPS, 2013. [ PDF ] Learning joint embedding with multimodal cues for cross-modal video-text retrieval. NC Mithun, J Li, F Metze, et al. ICMR, 2018. [ PDF ] Deep multimodal representation learning: A survey. W Guo, J Wang, S Wang. IEEE Access, 2019. [ PDF ] Multibench: Multiscale benchmarks for multimodal representation learning. PP Liang, Y Lyu, X Fan, Z Wu, Y Cheng, J Wu, et al. NeurlPS, 2021. Translation Show and tell: A neural image caption generator. O Vinyals, A Toshev, S Bengio, et al. CVPR, 2015. [ PDF ] Show, attend and tell: Neural image caption generation with visual attention. K Xu, J Ba, R Kiros, K Cho, A Courville, et al. PMLR, 2018. [ PDF ] A survey on automatic image caption generation. S Bai, S An. Neurocomputing, 2018. [ PDF ] Alignment Stacked cross attention for image-text matching. KH Lee, X Chen, G Hua, H Hu, et al. CVPR, 2018. [ PDF ] Visual Semantic Reasoning for Image-Text Matching. K Li, Y Zhang, K Li, Y Li, Y Fu. CVPR, 2020. [ PDF ] Oscar Object-Semantics Aligned Pre-training for Vision-Language Tasks. X Li, X Yin, C Li, P Zhang, X Hu, L Zhang, et al. ECCV, 2020. [ PDF ] Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval. D Gao, L Jin, B Chen, M Qiu, P Li, Y Wei, Y Hu, et al. SIGIR, 2020. [ PDF ] Similarity Reasoning and Filtration for Image-Text Matching. H Diao, Y Zhang, L Ma, H Lu. AAAI, 2021. [ PDF ] Fusion Attention-Based Multimodal Fusion for Video Description. C Hori, T Hori, TY Lee, Z Zhang, et al. CVPR, 2017. [ PDF ] Multimodal keyless attention fusion for video classification. X Long, C Gan, G De Melo, X Liu, Y Li, F Li, et al. AAAI, 2018. [ PDF ] MFAS Multimodal Fusion Architecture Search. JM Pérez-Rúa, V Vielzeuf, S Pateux, et al. CVPR, 2019. [ PDF ] Quantum-inspired multimodal fusion for video sentiment analysis. Q Li, D Gkoumas, C Lioma, M Melucci. Information Fusion, 2020. [ PDF ] A survey on machine learning for data fusion. T Meng, X Jing, Z Yan, W Pedrycz. Information Fusion, 2020. [ PDF ] Attention bottlenecks for multimodal fusion. A Nagrani, S Yang, A Arnab, A Jansen, et al. NIPS, 2021. [ PDF ] Dynamic Multimodal Fusion. Z Xue, R Marculescu. ArXiv, 2022. [ PDF ] Co-learning Visualbert: A simple and performant baseline for vision and language. LH Li, M Yatskar, D Yin, CJ Hsieh, et al. ArXiv, 2019. Lxmert: Learning cross-modality encoder representations from transformers. H Tan, M Bansal. EMNLP, 2019. [ PDF ] Videobert: A joint model for video and language representation learning. C Sun, A Myers, C Vondrick, et al. ICCV, 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. J Lu, D Batra, D Parikh, S Lee. NIPS, 2019. [ PDF ] Univl: A unified video and language pre-training model for multimodal understanding and generation. H Luo, L Ji, B Shi, H Huang, N Duan, T Li, J Li, et al. ArXiv, 2020. [ PDF ] Actbert: Learning global-local video-text representations. L Zhu, Y Yang. CVPR, 2020. What Makes Training Multi-modal Classification Networks Hard. W Wang, D Tran, M Feiszli. CVPR, 2020. [ PDF ] Hero: Hierarchical encoder for video+ language omni-representation pre-training. L Li, YC Chen, Y Cheng, Z Gan, L Yu, J Liu. EMNLP, 2020. Vl-bert: Pre-training of generic visual-linguistic representations. W Su, X Zhu, Y Cao, B Li, L Lu, F Wei, J Dai. ICLR, 2020. Foundations of multimodal co-learning. A Zadeh, PP Liang, LP Morency. Information Fusion, 2020. [ PDF ] Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. W Li, C Gao, G Niu, X Xiao, H Liu, J Liu, H Wu, et al. ArXiv, 2021. [ PDF ] M6: A chinese multimodal pretrainer. J Lin, R Men, A Yang, C Zhou, M Ding, Y Zhang, et al. ArXiv, 2021. Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions. A Rahate, R Walambe, S Ramanna, et al. ArXiv, 2021. [ PDF ] Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text. Q Li, B Gong, Y Cui, D Kondratyuk, X Du, et al. ArXiv, 2021. VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts. W Wang, H Bao, L Dong, F Wei. ArXiv, 2021. WenLan: Bridging vision and language by large-scale multi-modal pre-training. Y Huo, M Zhang, G Liu, H Lu, Y Gao, G Yang, et al. ArXiv, 2021. WenLan 2.0: Make AI Imagine via a Multimodal Foundation Model. N Fei, Z Lu, Y Gao, G Yang, Y Huo, J Wen, H Lu, R Song, X Gao, T Xiang, et al. ArXiv, 2021. Learning transferable visual models from natural language supervision. A Radford, JW Kim, C Hallacy, et al. ICML, 2021. Vilt: Vision-and-language transformer without convolution or region supervision. W Kim, B Son, I Kim. ICML, 2021. Zero-shot text-to-image generation. A Ramesh, M Pavlov, G Goh, S Gray, et al. ICML, 2021. Multimodal few-shot learning with frozen language models. M Tsimpoukelli, J Menick, S Cabi, et al. NIPS, 2021. Wukong: 100 Million Large-scale Chinese Cross-modal Pre-training Dataset and A Foundation Framework. J Gu, X Meng, G Lu, L Hou, M Niu, H Xu, et al. ArXiv, 2022. Applications Understanding Image Deep Collaborative Embedding for Social Image Understanding. Z Li, J Tang, T Mei. TPAMI, 2019. [ PDF ] Layoutlm: Pre-training of text and layout for document image understanding. Y Xu, M Li, L Cui, S Huang, F Wei, M Zhou. KDD, 2020. Beyond visual semantics: Exploring the role of scene text in image understanding. AU Dey, SK Ghosh, E Valveny, G Harit. Pattern Recognition Letters, 2021. Video Eco Efficient convolutional network for online video understanding. M Zolfaghari, K Singh, T Brox. ECCV, 2018. [ PDF ] TSM Temporal Shift Module for Efficient Video Understanding. J Lin, C Gan, S Han. CVPR, 2019. [ PDF ] Assemblenet: Searching for multi-stream neural connectivity in video architectures. MS Ryoo, AJ Piergiovanni, M Tan, et al. ICLR, 2020. Is Space-Time Attention All You Need for Video Understanding? G Bertasius, H Wang, et al. ArXiv, 2021. MDMMT Multidomain Multimodal Transformer for Video Retrieval. M Dzabraev, M Kalashnikov, et al. CVPR, 2021. [ PDF ]
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)