Machine Learning For Predictive Quality Control

Industrial ML · ML Project

Python · Data Preprocessing · Model Development · Evaluation

Project focus: predictive analytics using machine sensors, process variables and maintenance records.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 1

A Survey on Multimodal Large Language Models Shukang Yin*, Chaoyou Fu*†, Sirui Zhao*, Ke Li, Xing Sun, Tong Xu, and Enhong Chen, Fellow, IEEE

Abstract—Recently, Multimodal Large Language Model (MLLM) represented by GPT-4V has been a new rising research hotspot, which uses powerful Large Language Models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of MLLM, such as writing stories based on images and OCR-free math reasoning, are rare in traditional multimodal methods, suggesting a potential path to artificial general intelligence. To this end, both academia and industry have endeavored to develop MLLMs that

can compete with or even better than GPT-4V, pushing the limit of research at a surprising speed. In this paper, we aim to trace and summarize the recent progress of MLLMs. First of all, we present the basic formulation of MLLM and delineate its related concepts, arXiv:2306.13549v4 [cs.CV] 29 Nov 2024

including architecture, training strategy and data, as well as evaluation. Then, we introduce research topics about how MLLMs can be extended to support more granularity, modalities, languages, and scenarios. We continue with multimodal hallucination and extended techniques, including Multimodal ICL (M-ICL), Multimodal CoT (M-CoT), and LLM-Aided Visual Reasoning (LAVR). To conclude the paper, we discuss existing challenges and point out promising research directions. In light of the fact that the era of MLLM has only just

begun, we will keep updating this survey and hope it can inspire more research. An associated GitHub link collecting the latest papers is available at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.

Index Terms—Multimodal Large Language Model, Vision Language Model, Large Language Model.

1 I NTRODUCTION manifests two representative traits compared with the tradi- tional counterparts: (1) MLLM is based on LLM with billion- R ECENT years have seen the remarkable progress of LLMs [1], [2], [3], [4], [5]. By scaling up data size and model size, these LLMs raise extraordinary emergent abil- scale parameters, which is not available in previous models. (2) MLLM uses new training paradigms to unleash its full

ities, typically including instruction following [5], [6], In- potential, such as using multimodal instruction tuning [19], Context Learning (ICL) [7], and Chain of Thought (CoT) [8]. [20] to encourage the model to follow new instructions. Although LLMs have demonstrated surprising zero/few- Armed with the two traits, MLLM exhibits new capabilities, shot reasoning performance on most Natural Language such as writing website code based on images [21], under-

Processing (NLP) tasks, they are inherently “blind” to vision standing the deep meaning of a meme [22], and OCR-free since they can only understand discrete text. Concurrently, math reasoning [23]. Large Vision Models (LVMs) can see clearly [9], [10], [11], Ever since the release of GPT-4 [3], there has been a [12], but commonly lag in reasoning. research frenzy over MLLMs because of the amazing mul-

In light of this complementarity, LLM and LVM run timodal examples it shows. Rapid development is fueled towards each other, leading to the new field of Multimodal by efforts from both academia and industry. Preliminary Large Language Model (MLLM). Formally, it refers to the research on MLLMs focuses on text content generation LLM-based model with the ability to receive, reason, and grounded in text prompts and image [20], [24]/video [25],

output with multimodal information. Prior to MLLM, there [26]/audio [27]. Subsequent works have expanded the capa- have been a lot of works devoted to multimodality, which bilities or the usage scenarios, including: (1) Better granular- can be divided into discriminative [13], [14], [15] and gen- ity support. Finer control on user prompts is developed to erative [16], [17], [18] paradigms. CLIP [13], as a represen- support specific regions through boxes [28] or a certain ob-

tative of the former, projects visual and textual information ject through a click [29]. (2) Enhanced support on input and into a unified representation space, building a bridge for output modalities [30], [31], such as image, video, audio, downstream multimodal tasks. In contrast, OFA [16] is a and point cloud. Besides input, projects like NExT-GPT [32] representative of the latter, which unifies multimodal tasks further support output in different modalities. (3) Improved

in a sequence-to-sequence manner. MLLM can be classified language support. Efforts have been made to extend the as the latter according to the sequence operation, but it success of MLLMs to other languages (e.g. Chinese) with relatively limited training corpus [33], [34]. (4) Extension • †Chaoyou Fu is the project leader. to more realms and usage scenarios. Some studies transfer • *Shukang Yin, Chaoyou Fu, and Sirui Zhao contribute equally. the strong capabilities of MLLMs to other domains such as

• Shukang Yin, Sirui Zhao, Tong Xu, and Enhong Chen are with the medical image understanding [35], [36], [37] and document Department of Data Science, University of Science and Technology of parsing [38], [39], [40]. Moreover, multimodal agents are China, No.96, JinZhai Road Baohe District, Hefei, Anhui, 230026, China. E-mail: sirui@mail.ustc.edu.cn, cheneh@ustc.edu.cn developed to assist in real-world interaction, e.g. embodied

• Chaoyou Fu, Ke Li, and Xing Sun are with the Tencent YouTu Lab, agents [41], [42] and GUI agents [43], [44], [45]. An MLLM Shanghai 200233, China. E-mail: bradyfu24@gmail.com timeline is illustrated in Fig. 1. Corresponding author: Chaoyou Fu, Sirui Zhao, and Enhong Chen. In view of such rapid progress and the promising results

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2

Fuyu-8B V* SPHINX MoE-LLaVA Qwen-VL-Max Publicly Available/Unavailable MobileVLM Vary Monkey TextMonkey Mobile-Agent MM1 ImageBind-LLM DreamLLM 1-3 MMICL Xcomposer NExT-GPT 10-12 2024 Gemini AnyMAL Woodpecker Video-LLaMA 3D-LLM GPT-4V Qwen-VL

8-9 Video-LLaVA Kosmos-2 Lynx GPT4RoI PointLLM ASM LLaMA-VID VisCPM LISA Pengi Chameleon LanguageBIND

DetGPT VisionLLM Otter LLaVA-Med LLaVA-1.5 6-7 LaVIN MultiModal-GPT Shikra MotionGPT CogVLM

Emu Ferret PaLM-E LLaMA-Adapter 4-5 GLaMM Kosmos-1 LLaVA VideoChat VIMA Flamingo MiniGPT-4 InstructBLIP 1-3 BLIP-2 HuggingGPT LTU EmbodiedGPT 2023 2022 MM-REACT ViperGPT GPT4Tools mPLUG-Owl

released GitHub page, which is updated daily.

of this field, we write this survey to provide researchers to humans, modality encoders such as image/audio en- with a grasp of the basic idea, main method, and current coders are human eyes/ears that receive and pre-process progress of MLLMs. Note that we mainly focus on visual optical/acoustic signals, while LLMs are like human brains and language modalities, but also include works involving that understand and reason with the processed signals. In

other modalities like video and audio. Specifically, we cover between, the modality interface serves to align different the most important aspects of MLLMs with corresponding modalities. Some MLLMs also include a generator to output summaries and open a GitHub page that would be updated other modalities apart from text. A diagram of the architec- in real time. To the best of our knowledge, this is the first ture is plotted in Fig. 2. In this section, we introduce each

survey on MLLM. module in sequence. The following parts of the survey are structured as such: the survey starts with a comprehensive review of the essential aspects of MLLMs, including (1) Mainstream archi- 2.1 Modality encoder tectures (§2); (2) A full recipe of training strategy and data (§3); (3) Common practices of performance evaluation (§4). The encoders compress raw information, such as images

Then, we delve into a deeper discussion on some important or audio, into a more compact representation. Rather than topics about MLLMs, each focusing on a main problem: (1) training from scratch, a common approach is to use a pre- What aspects can be further improved or extended (§5)? trained encoder that has been aligned to other modalities. (2) How to relieve the multimodal hallucination issue (§6)? For example, CLIP [13] incorporates a visual encoder se-

The survey continues with the introduction of three key mantically aligned with the text through large-scale pre- techniques (§7), each specialized in a specific scenario: M- training on image-text pairs. Therefore, it is easier to use ICL (§7.1) is an effective technique commonly used at the such initially pre-aligned encoders to align with LLMs inference stage to boost few-shot performance. Another im- through alignment pre-training (see §3.1).

portant technique is M-CoT (§7.2), which is typically used in The series of commonly used image encoders are sum- complex reasoning tasks. Afterward, we delineate a general marized in Table 1. Apart from vanilla CLIP image en- idea to develop LLM-based systems to solve composite coders [13], some works also explore using other variants. reasoning tasks or to address common user queries (§7.3). For example, MiniGPT-4 [21] adopts an EVA-CLIP [47],

Finally, we finish our survey with a summary and potential [48] (ViT-G/14) encoder, which is trained with improved research directions. training techniques. In contrast, Osprey [29] introduces a convolution-based ConvNext-L encoder [46] to utilize higher resolution and multi-level features. Some works also 2 A RCHITECTURE explore encoder-free architecture. For instance, the image

A typical MLLM can be abstracted into three modules, i.e. patches of Fuyu-8b [49] are directly projected before sending a pre-trained modality encoder, a pre-trained LLM, and a to LLMs. Thus, the model naturally supports flexible image modality interface to connect them. Drawing an analogy resolution input.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 3

Variants Pretraining Corpus Resolution Samples (B) Parameter Size (M) OpenCLIP-ConvNext-L [46] LAION-2B 320 29 197.4 CLIP-ViT-L/14 [13] OpenAI’s WIT 224/336 13 304.0 EVA-CLIP-ViT-G/14 [47] LAION-2B,COYO-700M 224 11 1000.0 OpenCLIP-ViT-G/14 [46] LAION-2B 224 34 1012.7

OpenCLIP-ViT-bigG/14 [46] LAION-2B 224 34 1844.9

Text training data composition are of less importance compared Image with input resolution, found by empirical studies [52]. Text Similar encoders are also available for other modali- Audio ties. For example, Pengi [27] uses CLAP [54] model as LLM the audio encoder. ImageBind-LLM [30] uses the Image-

Video Bind [55] encoder, which supports encoding image, text, Modality Connector Generator audio, depth, thermal, and Inertial Measurement Unit (IMU) Encoder data. Equipped with the strong encoder, ImageBind-LLM … … can respond to the input of multiple modalities.

LLM MH-Attn 2.2 Pre-trained LLM K Q Q K V Q-Former K Instead of training an LLM from scratch, it is more effi- MLP V V cient and practical to start with a pre-trained one. Through tremendous pre-training on web corpus, LLMs have been embedded with rich world knowledge, and demonstrate Learnable Queries strong generalization and reasoning capabilities. We summarize the commonly used and publicly avail-

able LLMs in Table 2. Notably, most LLMs fall in the causal includes an encoder, a connector, and a LLM. An optional decoder category, following GPT-3 [7]. Among them, Flan- generator can be attached to the LLM to generate more T5 [56] series are relatively early LLMs used in works modalities besides text. The encoder takes in images, audios like BLIP-2 [59] and InstructBLIP [60]. LLaMA series [5], or videos and outputs features, which are processed by the

[57] and Vicuna family [4] are representative open-sourced connector so that the LLM can better understand. There are LLMs that have attracted much academic attention. Since broadly three types of connectors: projection-based, query- the two LLMs are predominantly pre-trained on English based, and fusion-based connectors. The former two types corpus, they are limited in multi-language support, such adopt token-level fusion, processing features into tokens to

as Chinese. In contrast, Qwen [58] is a bilingual LLM that be sent along with text tokens, while the last type enables a supports Chinese and English well. feature-level fusion inside the LLM. It should be noted that scaling up the parameter size of LLMs also brings additional gains, similar to the case of increasing input resolution. Specifically, Liu et al. [50], When choosing encoders, one often considers factors [61] find that simply scaling up LLM from 7B to 13B brings

like resolution, parameter size, and pretraining corpus. comprehensive improvement on various benchmarks. Fur- Notably, many works have empirically verified that us- thermore, when using a 34B LLM, the model shows emer- ing higher resolution can achieve remarkable performance gent zero-shot Chinese capability, given that only English gains [34], [50], [51], [52]. The approaches for scaling up multimodal data are used during training. Lu et al. [62] see

input resolution can be categorized into direct scaling and a similar phenomenon by scaling up LLMs from 13B to 35B patch-division methods. The direct scaling way inputs im- and 65B/70B, where the larger model size brings consistent ages of higher resolutions to the encoder, which often gains on benchmarks specifically designed for MLLMs. involves further tuning the encoder [34] or replacing a There are also works that use smaller LLMs to facilitate

pre-trained encoder with higher resolution [50]. Similarly, deployment on mobile devices. For example, MobileVLM CogAgent [44] uses a dual-encoder mechanism, where two series [63], [64] use downscaled LLaMA [5] (termed as encoders process high and low-resolution images, respec- MobileLLaMA 1.4B/2.7B), enabling efficient inference on tively. High-resolution features are injected into the low- mobile processors.

resolution branch through cross-attention. Patch-division Recently, explorations of Mixture of Experts (MoE) archi- methods cut a high-resolution image into patches and reuse tecture for LLMs have garnered rising attention [65], [66], the low-resolution encoder. For example, Monkey [51] and [67]. Compared with dense models, the sparse architecture SPHINX [53] divide a large image into smaller patches enables scaling up total parameter size without increasing

and send sub-images together with a downsampled high- computational cost, by selective activation of the parame- resolution image to the image encoder, where the sub- ters. Empirically, MM1 [52] and MoE-LLaVA [68] find that images and the low-resolution image capture local and MoE implementation achieves better performance than the global features, respectively. In contrast, parameter size and dense counterpart on almost all the benchmarks.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 4

German, respectively. Model Release Date Pretrain Data Scale Parameter Size (B) Language Support Architecture Flan-T5-XL/XXL [56] Oct-2022 - 3/ 11 en, fr, de Encoder-Decoder LLaMA [5] Feb-2023 1.4T tokens 7/ 13/ 33/ 65 en Causal Decoder Vicuna [4] Mar-2023 1.4T tokens 7/ 13/ 33 en Causal Decoder

LLaMA-2 [57] Jul-2023 2T tokens 7/ 13/ 70 en Causal Decoder Qwen [58] Sep-2023 3T tokens 1.8 / 7/ 14/ 72 en, zh Causal Decoder

2.3 Modality interface are first embedded with visual knowledge and then con- catenated with text features as prefixes. Since LLMs can only perceive text, bridging the gap be- In terms of parameter size, learnable interfaces generally tween natural language and other modalities is necessary. comprise a small portion compared with encoders and However, it would be costly to train a large multimodal LLMs. Take Qwen-VL [34] as an example, the parameter

model in an end-to-end manner. A more practical way is size of the Q-Former is about 0.08B, accounting for less to introduce a learnable connector between the pre-trained than 1% of the whole parameters, while the encoder and visual encoder and LLM. The other approach is to translate the LLM account for about 19.8% (1.9B) and 80.2% (7.7B), images into languages with the help of expert models, and respectively. then send the language to LLM. Expert Model. Apart from the learnable interface, using

Learnable Connector. It is responsible for bridging the expert models, such as an image captioning model, is also gap between different modalities. Specifically, the module a feasible way to bridge the modality gap [77], [78], [79], projects information into the space that LLM can understand [80]. The basic idea is to convert multimodal inputs into efficiently. Based on how multimodal information is fused, languages without training. In this way, LLMs can under-

there are broadly two ways to implement such interfaces, stand multimodality by the converted languages. For ex- i.e. token-level and feature-level fusion. ample, VideoChat-Text [25] uses pre-trained vision models For token-level fusion, features output from encoders are to extract visual information such as actions and enriches transformed into tokens and concatenated with text tokens the descriptions using a speech recognition model. Though

before being sent into LLMs. A common and feasible solu- using expert models is straightforward, it may not be as tion is to leverage a group of learnable query tokens to ex- flexible as adopting a learnable interface. The conversion of tract information in a query-based manner [69], which first foreign modalities into text would cause information loss. has been implemented in BLIP-2 [59], and subsequently in- For example, transforming videos into textual descriptions

herited by a variety of work [26], [60], [70]. Such Q-Former- distorts spatial-temporal relationships [25]. style approaches compress visual tokens into a smaller num- ber of representation vectors. In contrast, some methods simply use a MLP-based interface to bridge the modality 3 T RAINING S TRATEGY AND DATA gap [20], [37], [71], [72]. For example, LLaVA series adopts A full-fledged MLLM undergoes three stages of training,

one/two linear MLP [20], [50] to project visual tokens and i.e. pre-training, instruction-tuning, and alignment tuning. align the feature dimension with word embeddings. Each phase of training requires different types of data and On a related note, MM1 [52] has ablated on design fulfills different objectives. In this section, we discuss train- choices on the connector and found that for token-level ing objectives, as well as data collection and characteristics

fusion, the type of modality adapter is far less important for each training stage. than the number of visual tokens and input resolution. Nevertheless, Zeng et al. [73] compare the performance 3.1 Pre-training of token and feature-level fusion, and empirically reveal that the token-level fusion variant performs better in terms 3.1.1 Training Detail of VQA benchmarks. Regarding the performance gap, the As the first training stage, pre-training mainly aims to align

authors suggest that cross-attention models might require different modalities and learn multimodal world knowl- a more complicated hyper-parameter searching process to edge. Pre-training stage generally entails large-scale text- achieve comparable performance. paired data, e.g. caption data. Typically, the caption pairs de- As another line, feature-level fusion inserts extra mod- scribe images/audio/videos in natural language sentences.

ules that enable deep interaction and fusion between text Here, we consider a common scenario where MLLMs are features and visual features. For example, Flamingo [74] trained to align vision with text. As illustrated in Table 3, inserts extra cross-attention layers between frozen Trans- given an image, the model is trained to predict autore- former layers of LLMs, thereby augmenting language fea- gressively the caption of the image, following a standard

tures with external visual cues. Similarly, CogVLM [75] cross-entropy loss. A common approach for pre-training plugs in a visual expert module in each Transformer layer is to keep pre-trained modules (e.g. visual encoders and to enable dual interaction and fusion between vision and LLMs) frozen and train a learnable interface [20], [35], [72]. language features. For better performance, the QKV weight The idea is to align different modalities without losing

matrix of the introduced module is initialized from the pre-trained knowledge. Some methods [34], [81], [82] also pre-trained LLM. Similarly, LLaMA-Adapter [76] introduces unfreeze more modules (e.g. visual encoder) to enable more learnable prompts into Transformer layers. These prompts trainable parameters for alignment. It should be noted that

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 5

Input: <image> Response: {caption} Dataset Samples Date Coarse-grained Image-Text data. {<image>} is the placeholder for the visual tokens, and CC-12M [85] 12.4M 2020 SBU Captions [86] 1M 2011 {caption} is the caption for the image. Note that only the LAION-5B [87] 5.9B Mar-2022 part marked in red is used for loss calculation. LAION-2B [87] 2.3B Mar-2022

LAION-COCO [88] 600M Sep-2022 COYO-700M [90] 747M Aug-2022 the training scheme is closely related to the data quality. Fine-grained Image-Text For short and noisy caption data, a lower resolution (e.g. ShareGPT4V-PT [83] 1.2M Nov-2023 224) can be adopted to speed up the training process, while LVIS-Instruct4V [91] 111K Nov-2023 for longer and cleaner data, it is better to utilize higher ALLaVA [92] 709K Feb-2024

resolutions (e.g. 448 or higher) to mitigate hallucinations. Be- Video-Text sides, ShareGPT4V [83] finds that with high-quality caption MSR-VTT [93] 200K 2016 data in the pretraining stage, unlocking the vision encode promotes better alignment. Audio-Text WavCaps [94] 24K Mar-2023 3.1.2 Data Pretraining data mainly serve two purposes, i.e. (1) aligning

different modalities and (2) providing world knowledge. LAION. This series are large web-scale datasets, with im- The pretraining corpora can be divided into coarse-grained ages scrawled from the internet and associated alt-text as and fine-grained data according to granularities, which we captions. To filter the image-text pairs, the following steps will introduce sequentially. We summarize commonly used are performed: (1) Text with short lengths or images with too

pretraining datasets in Table 4. small or too big sizes are dropped. (2) Image deduplication Coarse-grained caption data share some typical traits in based on URL. (3) Extract CLIP [13] embeddings for images common: (1) The data volume is large since samples are and text, and use the embeddings to drop possibly illegal generally sourced from the internet. (2) Because of the web- content and image-text pairs with low cosine similarity

scrawled nature, the captions are usually short and noisy between embeddings. Here we offer a brief summary of since they originate from the alt-text of the web images. some typical variants: These data can be cleaned and filtered via automatic tools, • LAION-5B [87]: It is a research-purpose dataset of 5.85B for example, using CLIP [13] model to filter out image- image-text pairs. The dataset is multilingual with a 2B

text pairs whose similarities are lower than a pre-defined English subset. threshold. In what follows, we introduce some representa- • LAION-COCO [88]: It contains 600M images extracted tive coarse-grained datasets. from the English subset of LAION-5B. The captions are CC. CC-3M [84] is a web-scale caption dataset of 3.3M synthetic, using BLIP [89] to generate various image cap-

image-caption pairs, where the raw descriptions are derived tions and using CLIP [13] to pick the best fit for the image. from alt-text associated with images. The authors design a COYO-700M [90]. It contains 747M image-text pairs, which complicated pipeline to clean data: (1) For images, those are extracted from CommonCrawl. For data filtering, the with inappropriate content or aspect ratio are filtered. (2) authors design the following strategies: (1) For images,

For text, NLP tools are used to obtain text annotations, with those with inappropriate size, content, format, or aspect samples filtered according to the designed heuristics. (3) For ratio are filtered. Moreover, the images are filtered based image-text pairs, images are assigned labels via classifiers. on the pHash value to remove images overlapped with If text annotations do not overlap with image labels, the public datasets such as ImageNet and MS-COCO. (2) For

corresponding samples are dropped. text, only English text with satisfactory length, noun forms, CC-12M [85] is a following work of CC-3M and contains and appropriate words are saved. Whitespace before and 12.4M image-caption pairs. Compared with the previous after the sentence will be removed, and consecutive whites- work, CC-12M relaxes and simplifies the data-collection pace characters will be replaced with a single whitespace.

pipeline, thus collecting more data. Moreover, text appearing more than 10 times (e.g. “image SBU Captions [86]. It is a captioned photo dataset con- for”) will be dropped. (3) For image-text pairs, duplicated taining 1M image-text pairs, with images and descriptions samples are removed based on (image pHash, text) tuple. sourced from Flickr. Specifically, an initial set of images is Recently, more works [83], [91], [92] have explored

acquired by querying the Flickr website with a large number generating high-quality fine-grained data through prompt- of query terms. The descriptions attached to the images ing strong MLLMs (e.g. GPT-4V). Compared with coarse- thus serve as captions. Then, to ensure that descriptions grained data, these data generally contain longer and more are relevant to the images, the retained images fulfill these accurate descriptions of the images, thus enabling finer-

requirements: (1) Descriptions of the images are of satisfac- grained alignment between image and text modalities. tory length, decided by observation. (2) Descriptions of the However, since this approach generally requires calling images contain at least 2 words in the predefined term lists commercial-use MLLMs, the cost is higher, and the data vol- and a propositional word (e.g. “on”, “under”) that generally ume is relatively smaller. Notably, ShareGPT4V [83] strikes a

suggests spatial relationships. balance by first training a captioner with GPT-4V-generated

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 6

(A) Pretrain–finetune (BERT, T5) like the image caption task [100]. The output is the answer Pretrained Finetune on Inference LM task A on task A (C) Instruction tuning (FLAN) to the instruction conditioned on the input. The instruction • Typically requires many task-specific examples Pretrained

LM Instruction-tune on many tasks: Inference on task A template is flexible and subject to manual designs [20], [25], • One specialized model B, C, D, … for each task Model learns to perform Inference on [98], as exemplified in Table 5. Note that the instruction many tasks via natural unseen task (B) Prompting (GPT-3) language instructions template can also be generalized to the case of multi-round

Pretrained via few-shot prompting or prompt engineering Inference conversations [20], [37], [71], [98]. LM on task A Formally, a multimodal instruction sample can be de- image is from [19]. sent the instruction, the multimodal input, and the ground truth response, respectively. The MLLM predicts an answer given the instruction and the multimodal input:

100K data, then scaling up the data volume to 1.2M using the pre-trained captioner. A = f (I, M; θ) (1)

Here, A denotes the predicted answer, and θ are the pa- 3.2 Instruction-tuning rameters of the model. The training objective is typically the 3.2.1 Introduction original auto-regressive objective used to train LLMs [20], Instruction refers to the description of tasks. Intuitively, [37], [71], [101], based on which the MLLM is encouraged to

instruction tuning aims to teach models to better under- predict the next token of the response. The objective can be stand the instructions from users and fulfill the demanded expressed as: tasks. Tuning in this way, LLMs can generalize to unseen N tasks by following new instructions, thus boosting zero-shot X

L(θ) = − log p(Ri |I, R<i ; θ) (2) performance. This simple yet effective idea has sparked the i=1 success of subsequent NLP works, such as ChatGPT [2], InstructGPT [95], FLAN [19], [56], and OPT-IML [96]. where N is the length of the ground-truth response. The comparisons between instruction tuning and related

typical learning paradigms are illustrated in Fig. 3. The 3.2.3 Data Collection supervised fine-tuning approach usually requires a large Since instruction data are more flexible in formats and amount of task-specific data to train a task-specific model. varied in task formulations, it is usually trickier and more

The prompting approach reduces the reliance on large-scale costly to collect data samples. In this section, we summarize data and can fulfill a specialized task via prompt engi- three typical ways to harvest instruction data at scale, i.e. neering. In such a case, though the few-shot performance data adaptation, self-instruction, and data mixture.

has been improved, the zero-shot performance is still quite Data Adaptation. Task-specific datasets are rich sources of average [7]. Differently, instruction tuning learns how to high-quality data. Hence, abundant works [60], [70], [76], generalize to unseen tasks rather than fitting specific tasks [82], [101], [102], [103], [104] have utilized existing high-

like the two counterparts. Moreover, instruction tuning is quality datasets to construct instruction-formatted datasets. highly related to multi-task prompting [97]. Take the transformation of VQA datasets for an example, In this section, we delineate the format of instruction the original sample is an input-out pair where the input

samples, the training objectives, typical ways to gather in- comprises an image and a natural language question, and struction data, and corresponding commonly used datasets. the output is the textual answer to the question conditioned on the image. The input-output pairs of these datasets could 3.2.2 Training Detail naturally comprise the multimodal input and response of

A multimodal instruction sample often includes an optional the instruction sample (see §3.2.2). The instructions, i.e. the instruction and an input-output pair. The instruction is descriptions of the tasks, can either derive from manual typically a natural language sentence describing the task, design or from semi-automatic generation aided by GPT.

such as, “Describe the image in detail.” The input can be an Specifically, some works [21], [35], [60], [70], [102], [105] image-text pair like the VQA task [99] or only an image hand-craft a pool of candidate instructions and sample one of them during training. We offer an example of instruction templates for the VQA datasets as shown in Table 6. The

Below is an instruction that describes a task. Write a response other works manually design some seed instructions and that appropriately completes the request use these to prompt GPT to generate more [25], [82], [98]. Instruction: <instruction> Note that since the answers of existing VQA and caption

Input: {<image>, <text>} datasets are usually concise, directly using these datasets for Response: <output> instruction tuning may limit the output length of MLLMs. There are two common strategies to tackle this problem. The instruction data. <instruction> is a textual description of the ChatBridge [104] explicitly declares short and brief for short-

task. {<image>, <text>} and <output> are input and output answer data, as well as a sentence and single sentence for from the data sample. Note that <text> in the input may be conventional coarse-grained caption data. The second one is missed for some datasets, such as image caption datasets to extend the length of existing answers [105]. For example,

merely have <image>. The example is adapted from [98]. M3 IT [105] proposes to rephrase the original answer by

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 7

• <Image> {Question} • <Image> Question: {Question} • <Image> {Question} A short answer to the question is • <Image> Q: {Question} A: • <Image> Question: {Question} Short answer: • <Image> Given the image, answer the following question with no more than three words. {Question} • <Image> Based on the image, respond to this question with a short answer: {Question}. Answer: • <Image> Use the provided image to answer the question: {Question} Provide your answer as short as possible:

• <Image> What is the answer to the following question? "{Question}" • <Image> The question "{Question}" can be answered using the image. A short answer is

in the original VQA datasets, respectively.

Video, A: Audio. For data composition, M-T and S-T denote multi-turn and single-turn, respectively. Dataset Sample Modality Source Composition LLaVA-Instruct 158K I+T→T MS-COCO 23K caption + 58K M-T QA + 77K reasoning LVIS-Instruct 220K I+T→T LVIS 110K caption + 110K M-T QA ALLaVA 1.4M I+T→T VFlan, LAION 709K caption + 709K S-T QA

Video-ChatGPT 100K V+T→T ActivityNet 7K description + 4K M-T QA VideoChat 11K V+T → T WebVid description + summarization + creation Clotho-Detail 3.9K A+T→T Clotho caption

prompting ChatGPT with the original question, answer, and and randomly shuffle) and sequential instruction tuning contextual information of the image (e.g. caption and OCR). (text data followed by multimodal data). Self-Instruction. Although existing multi-task datasets can 3.2.4 Data Quality contribute a rich source of data, they usually do not meet human needs well in real-world scenarios, such as multiple Recent research has revealed that the data quality of

rounds of conversations. To tackle this issue, some works instruction-tuning samples is no less important than quan- collect samples through self-instruction [106], which utilizes tity. Lynx [73] finds that models pre-trained on large-scale LLMs to generate textual instruction-following data using a but noisy image-text pairs do not perform as well as mod- few hand-annotated samples. Specifically, some instruction- els pre-trained with smaller but cleaner datasets. Similarly,

following samples are hand-crafted as demonstrations, af- Wei et al. [108] finds that less instruction-tuning data with ter which ChatGPT/GPT-4 is prompted to generate more higher quality can achieve better performance. For data instruction samples with the demonstrations as guidance. filtering, the work proposes some metrics to evaluate data LLaVA [20] extends the approach to the multimodal field quality and, correspondingly, a method to automatically

by translating images into text of captions and bound- filter out inferior vision-language data. Here we discuss two ing boxes, and prompting text-only GPT-4 to generate important aspects regarding data quality. new data with the guidance of requirements and demon- Prompt Diversity. The diversity of instructions has been strations. In this way, a multimodal instruction dataset found to be critical for model performance. Lynx [73] em-

is constructed, called LLaVA-Instruct-150k. Following this pirically verifies that diverse prompts help improve model idea, subsequent works such as MiniGPT-4 [21], Chat- performance and generalization ability. Bridge [104], GPT4Tools [107], and DetGPT [72] develop Task Coverage. In terms of tasks involved in training data, different datasets catering for different needs. Recently, with Du et al. [109] perform an empirical study and find that

the release of the more powerful multimodal model GPT- the visual reasoning task is superior to captioning and 4V, many works have adopted GPT-4V to generate data of QA tasks for boosting model performance. Moreover, the higher quality, as exemplified by LVIS-Instruct4V [91] and study suggests that enhancing the complexity of instruc- ALLaVA [92]. We summarize the popular datasets gener- tions might be more beneficial than increasing task diversity

ated through self-instruction in Table 7. and incorporating fine-grained spatial annotations.

Data Mixture. Apart from the multimodal instruction 3.3 Alignment tuning data, language-only user-assistant conversation data can also be used to improve conversational proficiencies 3.3.1 Introduction and instruction-following abilities [81], [98], [101], [103]. Alignment tuning is more often used in scenarios where LaVIN [101] directly constructs a minibatch by randomly models need to be aligned with specific human preferences,

sampling from both language-only and multimodal data. e.g. response with fewer hallucinations (see §6). Currently, MultiInstruct [102] probes different strategies for training Reinforcement Learning with Human Feedback (RLHF) and with a fusion of single modal and multimodal data, includ- Direct Preference Optimization (DPO) are two main tech- ing mixed instruction tuning (combine both types of data niques for alignment tuning. In this section, we introduce

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 8

the main ideas of the two techniques in sequence and TABLE 8: A summary of datasets for alignment-tuning. For offer some examples of how they are utilized in addressing input/output modalities, I: Image, T: Text. practical problems, and finally, give a compilation of the Dataset Sample Modality Source related datasets. LLaVA-RLHF [112] 10K I+T→T Human

RLHF-V [114] 5.7K I+T→T Human VLFeedback [115] 380K I+T→T GPT-4V 3.3.2 Training Detail RLHF [110], [111]. This technique aims to utilize rein- forcement learning algorithms to align LLMs with human response and uses the obtained data to perform dense DPO. preferences, with human annotations as supervision in the Silkie [115] instead collects preference data via prompting training loop. As exemplified in InstructGPT [95], RLHF GPT-4V and distills the preference supervision into an

incorporates three key steps: instruction-tuned model through DPO. 1) Supervised fine-tuning. This step aims to fine-tune a pre-trained model to present the preliminary desired 3.3.3 Data output behavior. The fine-tuned model in the RLHF The gist of data collection for alignment-tuning is to collect setting is called a policy model. Note that this step might feedback for model responses, i.e. to decide which response

be skipped since the supervised policy model π SFT can be is better. It is generally more expensive to collect such data, initialized from an instruction-tuned model (see §3.2). and the amount of data used for this phase is typically 2) Reward modeling. A reward model is trained using pref- even less than that used in previous stages. In this part, we erence pairs in this step. Given a multimodal prompt introduce some datasets and summarize them in Table 8.

(e.g. image and text) x and a response pair (yw , yl ), the LLaVA-RLHF [112]. It contains 10K preference pairs col- reward model rθ learns to give a higher reward to the lected from human feedback in terms of honesty and help- preferred response yw , and vice versa for yl , according to fulness. The dataset mainly serves to reduce hallucinations the following objective: in model responses.

L(θ) = −E(x,yw ,yl )∼D [log(σ(rθ (x, yw ) − rθ (x, yl )] (3) RLHF-V [114]. It has 5.7K fine-grained human feedback data collected by segment-level hallucination corrections. where D = {(x, yw , yl )} is the comparison dataset VLFeedback [115]. It utilizes AI to provide feedback on labeled by human annotators. In practice, the reward model responses. The dataset contains more than 380K model rθ shares a similar structure with the policy model. comparison pairs scored by GPT-4V in terms of helpfulness,

3) Reinforcement learning. In this step, the Proximal Policy faithfulness, and ethical concerns. Optimization (PPO) algorithm is adopted to optimize the RL policy model πϕRL . A per-token KL penalty is often added to the training objective to avoid deviating too far 4 E VALUATION from the original policy [95], resulting in the objective: Evaluation is an essential part of developing MLLMs since h it provides feedback for model optimization and helps to

L(ϕ) = −Ex∼D,y∼πϕRL (y|x) rθ (x, y) compare the performance of different models. Compared i (4) with evaluation methods of traditional multimodal mod- − β · DKL πϕRL (y|x)||π REF (y|x) els, the evaluation of MLLMs exhibits several new traits: (1) Since MLLMs are generally versatile, it is important where β is the coefficient for the KL penalty term. Typ- to evaluate MLLMs comprehensively. (2) MLLMs exhibit

ically, both the RL policy πϕRL and the reference model many emergent capabilities that require special attention π REF are initialized from the supervised model π SFT . (e.g. OCR-free math reasoning) and thus require new eval- The obtained RL policy model is expected to align with uation schemes. The evaluation of MLLMs can be broadly human preferences through this tuning process. categorized into two types according to the question genres,

Researchers have explored using the RLHF techniques including closed-set and open-set. for better multimodal alignment. For example, LLaVA- RLHF [112] collects human preference data and tunes a 4.1 Closed-set model with fewer hallucinations based on LLaVA [20]. DPO [113]. It learns from human preference labels utilizing Closed-set questions refer to a type of question where the a simple binary classification loss. Compared with the PPO- possible answer options are predefined and limited to a

based RLHF algorithm, DPO is exempt from learning an finite set. The evaluation is usually performed on task- explicit reward model, thus simplifying the whole pipeline specific datasets. In this case, the responses can be naturally to two steps, i.e. human preference data collection and judged by benchmark metrics [20], [60], [70], [76], [101], preference learning. The learning objective is as follows: [102], [103], [104]. For example, InstructBLIP [60] reports

the accuracy on ScienceQA [116], as well as the CIDEr h πϕRL (yw |x) score [117] on NoCaps [118] and Flickr30K [119]. The evalu- L(ϕ) = −E(x,yw ,yl )∼D log σ β log REF π (yw |x) ation settings are typically zero-shot [60], [102], [104], [105] RL (5) or finetuning [20], [35], [60], [70], [76], [101], [103], [105]. The πϕ (yl |x) i − β log REF first setting often selects a wide range of datasets covering

π (yl |x) different general tasks and splits them into held-in and RLHF-V [114] collects fine-grained (segment-level) prefer- held-out datasets. After tuning on the former, zero-shot ence data pairs by correcting hallucinations in the model performance is evaluated on the latter with unseen datasets

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 9

or even unseen tasks. In contrast, the second setting is often [134] exploit a more advanced GPT-4V model to assess observed in the evaluation of domain-specific tasks. For the performance of MLLMs. For example, Woodpecker [77] example, LLaVA [20] and LLaMA-Adapter [76] report fine- adopts GPT-4V to judge the response quality of model tuned performance on ScienceQA [116]. LLaVA-Med [35] answers based on the image. The evaluation is expected to

reports results on biomedical VQA [120], [121], [122]. be more accurate than using text-only GPT-4 since GPT-4V The above evaluation methods are usually limited to a has direct access to the image. small range of selected tasks or datasets, lacking a compre- A supplementary approach is to compare the different hensive quantitative comparison. To this end, some efforts capabilities of MLLMs through case studies. For instance,

have endeavored to develop new benchmarks specially some studies evaluate two typical advanced commercial-use designed for MLLMs [123], [124], [125], [126], [127], [128], models, GPT-4V and Gemini. Yang et al. [135] perform in- [129]. For example, Fu et al. [123] construct a comprehen- depth qualitative analysis on GPT-4V by crafting a series of sive evaluation benchmark MME that includes a total of samples across various domains and tasks, spanning from

14 perception and cognition tasks. All instruction-answer preliminary skills, such as caption and object counting, to pairs in MME are manually designed to avoid data leakage. complex tasks that require world knowledge and reasoning, MMBench [124] is a benchmark specifically designed for such as joke understanding and indoor navigation as an evaluating multiple dimensions of model capabilities, using embodied agent. Wen et al. [136] make a more focused eval-

ChatGPT to match open responses with pre-defined choices. uation of GPT-4V by designing samples targeting automatic Video-ChatGPT [130] and Video-Bench [131] focus on video driving scenarios. Fu et al. [137] carry out a comprehensive domains and propose specialized benchmarks as well as evaluation on Gemini-Pro by comparing the model against evaluation tools for assessment. There are also evaluation GPT-4V. The results suggest that GPT-4V and Gemini exhibit

strategies designed to evaluate a specific aspect of the comparable visual reasoning abilities in spite of different model [102], as exemplified by POPE [132] for assessment response styles. of hallucination degree. 5 E XTENSIONS 4.2 Open-set Recent studies have made significant strides in extending In contrast to the closed-set questions, the responses to the capabilities of MLLMs, spanning from more potent

open-set questions can be more flexible, where MLLMs foundational abilities to broader coverage of scenarios. We usually play a chatbot role. Because the content of the trace the principal development of MLLMs in this regard. chat can be arbitrary, it would be trickier to judge than Granularity Support. To facilitate better interaction between the closed-ended output. The criterion can be classified agents and users, researchers have developed MLLMs with

into manual scoring, GPT scoring, and case study. Manual finer support of granularities in terms of model inputs and scoring requires humans to assess the generated responses. outputs. On the input side, models that support finer control This kind of approach often involves hand-crafted ques- from user prompts are developed progressively, evolving tions that are designed to assess specific dimensions. For from image to region [28], [138], [139] and even pixels [29],

example, mPLUG-Owl [81] collects a visually related eval- [140], [141]. Specifically, Shikra [28] supports region-level uation set to judge capabilities like natural image under- input and understanding. Users may interact with the assis- standing, diagram, and flowchart understanding. Similarly, tant more flexibly by referring to specific regions, which are GPT4Tools [107] builds two sets for the finetuning and zero- represented in bounding boxes of natural language forms.

shot performance, respectively, and evaluates the responses Ferret [141] takes a step further and supports more flexible in terms of thought, action, arguments, and the whole. referring by devising a hybrid representation scheme. The Since manual assessment is labor intensive, some re- model supports different forms of prompts, including point, searchers have explored rating with GPT, namely GPT scor- box, and sketch. Similarly, Osprey [29] supports point input

ing. This approach is often used to evaluate performance by utilizing a segmentation model [9]. Aided by the excep- on multimodal dialogue. LLaVA [20] proposes to score the tional capabilities of the pre-trained segmentation model, responses via text-only GPT-4 in terms of different aspects, Osprey enables specifying a single entity or part of it with a such as helpfulness and accuracy. Specifically, 30 images single click. On the output side, grounding capabilities are

are sampled from the COCO [133] validation set, each improved in line with the development of input support. associated with a short question, a detailed question, and Shikra [28] supports response grounded in the image with a complex reasoning question via self-instruction on GPT-4. box annotations, resulting in higher precision and finer The answers generated by both the model and GPT-4 are referring experience. LISA [142] further supports mask-

sent to GPT-4 for comparison. Subsequent works follow this level understanding and reasoning, which makes pixel-level idea and prompt ChatGPT [81] or GPT-4 [35], [70], [101], grounding possible. [104], [105] to rate results [35], [70], [81], [101], [104] or judge Modality Support. Increased support for modalities is a which one is better [103]. tendency for MLLM studies. On the one hand, researchers

A main issue of applying text-only GPT-4 as an evaluator have explored adapting MLLMs to support the input of is that the judge is only based on image-related text content, more multimodal content, such as 3D point cloud [41], such as captions or bounding box coordinates, without [143], [144], [145]. On the other hand, MLLMs are also accessing the image [35]. Thus, it may be questionable to set extended to generate responses of more modalities, such

GPT-4 as the performance upper bound in this case. With as image [32], [146], [147], [148], audio [32], [147], [149], the release of the vision interface of GPT, some works [77], [150], and video [32], [151]. For example, NExT-GPT [32]

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 10

proposes a framework that supports inputs and outputs of the image content [77]. As a fundamental and important mixed modalities, specifically, combinations of text, image, problem, the issue has received increased attention. In this audio, and video, with the help of diffusion models [152], section, we briefly introduce some related concepts and [153] attached to the MLLM. The framework applies an research development.

encoder-decoder architecture and puts LLM as a pivot for understanding and reasoning. Language Support. Current models are predominantly 6.1 Preliminaries unilingual, probably due to the fact that high-quality non- Current research on multimodal hallucinations can be fur- English training corpus is scarce. Some works have been de- ther categorized into three types [159]: voted to developing multilingual models so that a broader

1) Existence Hallucination is the most basic form, meaning range of users can be covered. VisCPM [33] transfers model that models incorrectly claim the existence of certain capabilities to the multilingual setting by designing a multi- objects in the image. stage training scheme. Specifically, the scheme takes English 2) Attribute Hallucination means describing the attributes as a pivotal language, with abundant training corpus. Uti- of certain objects in a wrong way, e.g. failure to identify

lizing a pre-trained bilingual LLM, the multimodal capa- a dog’s color correctly. It is typically associated with ex- bilities are transferred to Chinese by adding some trans- istence hallucination since descriptions of the attributes lated samples during instruction tuning. Taking a similar should be grounded in objects present in the image. approach, Qwen-VL [34] is developed from the bilingual 3) Relationship Hallucination is a more complex type and is

LLM Qwen [58] and supports both Chinese and English. also based on the existence of objects. It refers to false During pre-training, Chinese data is mixed into the training descriptions of relationships between objects, such as corpus to preserve the bilingual capabilities of the model, relative positions and interactions. taking up 22.7% of the whole data volume. Scenario/Task Extension. Apart from developing common In what follows, we first introduce some specific eval-

general-purpose assistants, some studies have focused on uation methods (§6.2), which are useful to gauge the per- more specific scenarios where practical conditions should formance of methods for mitigating hallucinations (§6.3). be considered, while others extend MLLMs to downstream Then, we will discuss in detail the current methods for tasks with specific expertise. reducing hallucinations, according to the main categories

A typical tendency is to adapt MLLMs to more specific each method falls into. real-life scenarios. MobileVLM [63] explores developing small-size variants of MLLMs for resource-limited scenarios. 6.2 Evaluation Methods Some designs and techniques are utilized for deployment on mobile devices, such as LLMs of smaller size and quantiza- CHAIR [160] is an early metric that evaluates hallucina- tion techniques to speed up computation. Other works de- tion levels in open-ended captions. The metric measures

velop agents that interact with real-world [41], [154], [155], the proportion of sentences with hallucinated objects or e.g. user-friendly assistants specially designed for Graphical hallucinated objects in all the objects mentioned. In con- User Interface (GUI), as exemplified by CogAgent [44], trast, POPE [132] is a method that evaluates closed-set AppAgent [43], and Mobile-Agent [45]. These assistants choices. Specifically, multiple prompts with binary choices

excel in planning and guiding through each step to fulfill a are formulated, each querying if a specific object exists task specified by users, acting as helpful agents for human- in the image. The method also covers more challenging machine interaction. Another line is to augment MLLMs settings to evaluate the robustness of MLLMs, with data with specific skills for solving tasks in different domains, e.g. statistics taken into consideration. The final evaluation uses

document understanding [38], [39], [156], [157] and medi- a simple watchword mechanism, i.e. by detecting keywords cal domains [35], [36], [37]. For document understanding, “yes/no”, to convert open-ended responses into closed- mPLUG-DocOwl [38] utilizes various forms of document- set binary choices. With a similar evaluation approach, level data for tuning, resulting in an enhanced model in MME [123] provides a more comprehensive evaluation,

OCR-free document understanding. TextMonkey [39] incor- covering aspects of existence, count, position and color, as porates multiple tasks related to document understanding exemplified in [77]. to improve model performance. Apart from conventional Different from previous approaches that use match- document image and scene text datasets, position-related ing mechanisms to detect and decide hallucinations,

tasks are added to reduce hallucinations and help mod- HaELM [161] proposes using text-only LLMs as a judge to els learn to ground responses in the visual information. automatically decide whether MLLMs’ captions are correct MLLMs can also be extended to medical domains by in- against reference captions. In light of the fact that text-only stilling knowledge of the medical domain. For example, LLMs can only access limited image context and require

LLaVA-Med [158] injects medical knowledge into vanilla reference annotations, Woodpecker [77] uses GPT-4V to di- LLaVA [20] and develops an assistant specialized in medical rectly assess model responses grounded in the image. Faith- image understanding and question answering. Score [162] is a more fine-grained metric based on a routine that breaks down descriptive sub-sentences and evaluates each sub-sentence separately. Based on previous studies,

6 M ULTIMODAL H ALLUCINATION AMBER [163] is an LLM-free benchmark that encompasses Multimodal hallucination refers to the phenomenon of both discriminative tasks and generative tasks and involves responses generated by MLLMs being inconsistent with three types of possible hallucinations (see §6.1).

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 11

6.3 Mitigation Methods <BOS> Below are some examples and an instruction that According to high-level ideas, the current methods can be describes a task. Write a response that appropriately completes roughly divided into three categories: pre-correction, in- the request process-correction, and post-correction. ### Instruction: {instruction} Pre-correction. An intuitive and straightforward solution ### Image: <image>

### Response: {response} for hallucination is to collect specialized data (e.g. negative data) and use the data for fine-tuning, thus resulting in ### Image: <image> ### Response: {response} models with fewer hallucinated responses. LRV-Instruction [164] introduces a visual instruction tun- ing dataset. Apart from common positive instructions, the ### Image: <image> ### Response: <EOS> dataset incorporates delicately designed negative instruc-

tions at different semantic levels to encourage responses faithful to the image content. LLaVA-RLHF [112] collects TABLE 9: A simplified example of the template to structure human-preference pairs and finetunes models with rein- an M-ICL query, adapted from [98]. For illustration, we list forcement learning techniques, leading to models more two in-context examples and a query divided by a dashed aligned with less hallucinated answers. line. {instruction} and {response} are texts from the data

In-process-correction. Another line is to make improve- sample. <image> is a placeholder to represent the multi- ments in architectural design or feature representation. modal input (an image in this case). <BOS> and <EOS> are These works try to explore the reasons for hallucinations tokens denoting the start and the end of the input to the and design corresponding remedies to mitigate them in the LLM, respectively.

generation process. HallE-Switch [159] performs an empirical analysis of ditional supervised learning paradigms that learn implicit possible factors of object existence hallucinations and hy- patterns from abundant data, the crux of ICL is to learn from pothesizes that existence hallucinations derive from objects analogy [168]. Specifically, in the ICL setting, LLMs learn not grounded by visual encoders, and they are actually from a few examples along with an optional instruction and

inferred based on knowledge embedded in the LLM. Based extrapolate to new questions, thereby solving complex and on the assumption, a continuous controlling factor and unseen tasks in a few-shot manner [22], [169], [170]. (2) ICL corresponding training scheme are introduced to control the is usually implemented in a training-free manner [168] and extent of imagination in model output during inference. VCD [165] suggests that object hallucinations derive thus can be flexibly integrated into different frameworks at

from two primary causes, i.e. statistical bias in training the inference stage. A closely related technique to ICL is corpus and strong language prior embedded in LLMs. The instruction-tuning (see §3.2), which is shown empirically to authors take notice of the phenomenon that when injecting enhance the ICL ability [19]. noise into the image, MLLMs tend to lean towards language In the context of MLLM, ICL has been extended to more

prior rather than the image content for response genera- modalities, leading to Multimodal ICL (M-ICL). Building tion, leading to hallucinations. Correspondingly, this work upon the setting in (§3.2), at inference time, M-ICL can be designs an amplify-then-contrast decoding scheme to offset implemented by adding a demonstration set, i.e. a set of the false bias. in-context samples, to the original sample. In this case, the

HACL [166] investigates the embedding space of vision template can be extended as illustrated in Table 9. Note and language. Based on the observation, a contrastive learn- that we list two in-context examples for illustration, but ing scheme is devised to pull paired cross-modal repre- the number and the ordering of examples can be flexibly sentation closer while pushing away non-hallucinated and adjusted. In fact, models are commonly sensitive to the

hallucinated text representation. arrangement of demonstrations [168], [171]. Post-correction. Different from previous paradigms, post- correction mitigates hallucinations in a post-remedy way 7.1.1 Improvement on ICL capabilities and corrects hallucinations after output generation. Wood- Recently, a growing amount of work has focused on en- pecker [77] is a training-free general framework for hal- hancing ICL performance under various scenarios. In this

lucination correction. Specifically, the method incorporates section, we trace the development of this field and summa- expert models to supplement contextual information of the rize some relevant works. image and crafts a pipeline to correct hallucinations step by MIMIC-IT [172] combines in-context learning with in- step. The method is interpretable in that intermediate results struction tuning by building an instruction dataset for-

of each step can be checked, and objects are grounded in the matted with multimodal context. The model instruction image. The other method LURE [167] trains a specialized tuned on the introduced dataset shows improved few-shot revisor to mask objects with high uncertainty in the descrip- performance in the caption task. Emu [173] extends the ti

Continued Discussion and Extended Analysis