Train Of Four Monitor

Abstract

Vision-and-Language Pre-training (VLP) improves model performance for downstream tasks that require image and text inputs. Current VLP approaches differ on (i) model architecture (especially image embedders), (ii) loss functions, and (iii) masking policies. Image embedders are either deep models like ResNet or linear projections that directly feed image-pixels into the transformer. Typically, in addition to the Masked Language Modeling (MLM) loss, alignment-based objectives are used for cross-modality interaction, and RoI feature regression and classification tasks for Masked Image- Region Modeling (MIRM). Both alignment and MIRM objectives mostly do not have ground truth.

Alignment-based objectives require pairings of image and text and heuristic objective functions. MIRM relies on object detectors. Masking policies either do not take advantage of multi-modality or are strictly coupled with alignments generated by other models. In this paper, we present Masked Language and Image Modeling (MLIM) for VLP. MLIM uses two loss functions: Masked Language Modeling (MLM) loss and image reconstruction (RECON) loss. We propose Modality Aware Masking (MAM) to boost cross-modality interaction and take advantage of MLM and RECON losses that separately capture text and image reconstruction quality. Using MLM + RECON tasks coupled with MAM, we present a simplified VLP methodology and show that it has better downstream task performance on a proprietary e-commerce multi-modal dataset.

Keywords Vision and language models, BERT, transformers, multimodal models, pre-training

Introduction

Since the emergence of transformer architectures (Vaswani et al., 2017), and their first successful use in Natural Language Processing (NLP) tasks (Devlin et al., 2018), they have been used for vision tasks (Dosovitskiy et al., 2021; Wang et al., 2021), and finally in vision-and-language tasks (Su et al., 2020; Lu et al., 2019; Li et al., 2020). For vision-and-language pre-training, researchers have proposed i) diverse architectures, ii) loss functions, and iii) masking policies. Model architectures employ different techniques to convert images into a suitable input sequence to the transformer. Loss functions are designed for image-region modeling and cross-modality interaction via alignment functions between text and images. Masking strategies aim to avoid disrupting this alignment. Each of these techniques has advanced the state of the art and provided valuable insights. However, it is unclear how to integrate all of them to ∗These authors contributed equally.

Arxiv:2109.12178V1 [Cs.Cv] 24 Sep 2021

MLIM: Vision-and-Language Model Pre-training with Masked Language and Image ModelingA PREPRINT achieve a level of simplicity and accuracy for vision-and-language pre-training that is comparable to text transformer pre-training using masked language modeling (MLM) loss and text-token masking.

Image embedders convert an image to an embedding sequence. Image embedding sequences can be concatenated with text token embedding sequences and forwarded to the transformer. Most VLP models use object detectors as image embedders. These object detectors are deep convolutional models trained on Visual Genome (Krishna et al., 2016) or ImageNet classification (Russakovsky et al., 2015) datasets. Object detectors are limited to their original detection categories and are heavyweight models. While end-to-end training can adapt these models, computational complexity or end-to-end depth of the VLP model (object detector + transformer) can be an issue for training or inference. Visual transformers (ViLT) remove deep image-embedders and directly feed pixel-level inputs to the transformer via a linear projection on image patches defined by a grid (Kim et al., 2021). While ViLT significantly reduces computational complexity, it leaves image feature learning to the transformer layers, limiting interaction between text and image embeddings to later layers of the transformer. We propose using shallow CNN models as embedders to learn high-level features for low-level pixel-data and enable better cross-modality interaction throughout the transformer layers.

In addition to MLM loss for language modeling, Masked Image-Region Modeling (MIRM) uses various loss functions for image modeling. UNITER proposes loss functions that use RoI-features and predicted object-classes obtained from the object detector as target features and target labels for masked image regions (Chen et al., 2020). Another category of loss functions is alignment-based functions to encourage cross-modality interaction. Some alignment-based objectives aim to match fused representations of images and text (e.g., Image Text Matching (ITM)). Other alignment objectives operate on a finer resolution and aim to align text or image representations at a sequence position level via transport plans. ITM tasks require positive and negative pairs. We believe that training using such pairs is a better fit for downstream tasks and negative/positive definition might hurt generalization while pre-training. In addition, these loss functions do not use ground-truth labels, and instead rely on heuristics. We aim to design a pre-training methodology that is as well-defined as MLM loss on text tokens: ground truth targets with loss functions that are not based on heuristics. We use the image reconstruction (RECON) task for Masked Image Modeling (MIM) with the image itself as the target for this task.

While text token masking is straightforward, masking for images modeling is not. Image regions detected by the object detector are good candidates for masking as they define semantically significant regions in the image. However they do not rely on ground truth, using predictions of the object detector and require heavyweight models. In addition, masking can break cross-modal alignment if it is performed independently on text and image inputs. To avoid these requirements for masking corresponding representations in both modalities, coarse alignments obtained using text generators are utilized (Zhuge et al., 2021). We avoid these problems by not using object detectors or alignment based objectives.

In this paper, we introduce Masked Language and Image Modeling (MLIM) for VLP pe-training. We use Transformers (Vaswani et al., 2017) as the main component in the model. We use a shallow-CNN based image embedder to embed images and perform masking, and a CNN based image decoder to compute RECON loss. We use MLM loss to measure text token prediction performance and RECON loss to measure image reconstruction performance. While MLM loss is only defined on masked text tokens, RECON loss is defined over all pixels of the image. Together, MLM and RECON losses capture text and image reconstruction performance. We do not use any alignment-based loss-function for cross-modality interaction and instead use Modality-Aware Masking (MAM) to achieve this. MAM has three operating modes: (1) heavy image-masking, (2) heavy text-masking, (3) light image-masking and light text-masking.

The first two modes encourage information flow across modalities. Using an image embedder, we aim to bring image embeddings to a common embedding space with text token embeddings. We perform image masking on the image embedder outputs. MAM masks image and text inputs before the transformer, and MLM + RECON losses measure reconstructed text and image quality. We demonstrate the effectiveness of our VLP pre-training methodology on a proprietary e-commerce multi-modal dataset.

Our Contributions Are Summarized As Follows:

• We propose using image reconstruction (RECON) loss and show pre-training with MLM + RECON obviates the need for alignment-based and image-region modeling based loss functions. This simplifies loss function design and does not require image-text pairings to create training instances.

• We use a shallow CNN as an image embedder. Our image embedder is much more lightweight than deep models like ResNet, and image masking friendly. We show that a shallow CNN image embedder can bring image pixels to higher level representations that foster interaction between image and text embeddings.

• We show that MAM takes advantage of the two loss functions capturing text and image reconstruction quality via masking policies that encourage cross-modal information flow.

2

MLIM: Vision-and-Language Model Pre-training with Masked Language and Image ModelingA PREPRINT

Related Work

Recently, transformer based model pre-training has been successfully applied in NLP tasks [elmo, bert gpt2, roberta, albert, etc]. As an extension, multimodal models are proposed to learn representations from image-text pairs. Fine- tuning multimodal models achieve state–of-the-art (SoTA) on downstream tasks. Some of these models use two-stream stream architectures that use two single-modal transformers, and their outputs are fused by a third transformer [vilbert, lxmert]. On the other hand single-stream architectures input image and text embeddings into a transformer. Single- stream models have the potential to enable better cross-modality interaction since attention mechanisms can attend to both image and text. VL-BERT presents promising results for single-stream models. More work on VL tasks are proposed lately: VLP, ViLBERT, VL-BERT, ViLT offer single-stream models for generic understanding tasks such as Visual Question Answering (VQA) and image-text retrieval or generation tasks such as image captioning.

Object detectors are commonly used as image embedders. Image embeddings are either region of interest (RoI) features obtained from the object detector (e.g., (Chen et al., 2020; Tan and Bansal, 2019; Li et al., 2020)) or output feature grids of the CNN backbone (Jiang et al., 2020; Nguyen et al., 2020). While using grid features instead of RoI-features falls behind, it achieves similar performance with heavier CNN models (Huang et al., 2020). ViLT avoids using heavyweight image embedders by directly embedding low-level pixel data with a single-layer projection and achieves similar results with reduced complexity.

Multiple loss functions are proposed for better learning from images. Some of these can be categorized under masked image-region modeling (MIRM) tasks. RoI-feature regression tasks regresses the RoI-feature of the masked region with L2 loss. RoI-classification task learns the labels of masked regions with cross-entropy loss. UNITER (Chen et al., 2020) and LXMERT (Tan and Bansal, 2019) utilize object-detector features and its label-predictions as targets in MIRM losses. Contrastive loss in the form of detecting randomly replaced RoI tags are proposed in Oscar (Li et al., 2020).

Object tags are also generated by the object detector. These MIRM tasks rely on object-detectors and pseudo-targets (i.e., RoI features and class-labels obtained from the detector), which may be wrong or irrelevant in the given context. Alignment-based loss functions are also commonly utilized (Zhuge et al., 2021; Li et al., 2020; Kim et al., 2021) for cross-modality interaction. These alignments either happen at modality level as in widely used Image Text Matching (ITM) task or at finer-grained levels as in Word-Region Alignment (WRA) via Optimal Transport (OT) (Chen et al., 2020). Alignment task requires pairings. While creating negative pairs for ITM is easy, how to retrieve these pairs at train-time and how to find good training pairs remains an issue. Also, WRA does not have ground truth.

Our key differences from prior art is as follows: i) We simplify loss function design by using MLM and RECON losses both of which have ground truths ii) We use MAM to force cross-modal information flow for better cross-modality interaction and take advantage of MLM + RECON iii) We utilize a shallow CNN as image embedder, which is specifically designed for image embedding and masking. Our image embedder can bring low-level pixel data to a common representation space with text-token embeddings for better cross-modality interaction2

Model Overview

The model architecture of MLIM is given in Figure 1. Given an image-text pair, text is tokenized and embedded using transformer’s word embeddings and word-position embeddings. Image is embedded using a shallow CNN model and image-positional embeddings.

Specifically, image embedder is a CNN model with convolutional layers using 2D filters of kernel size 2x2 and stride of 2. Choosing kernel size in horizontal and vertical dimensions to be equal to the stride ensures each input will only contribute to a single filter’s output. Therefore, image embeddings will correspond to non-overlapping pixel regions in the original image-resolution and masking an embedding deletes all information captured from a region in the image.

Hence, prediction of masked region’s pixel data will be left to transformer’s self-attention layers to gather information from existing text and image embeddings. We observed that this helps with text-to-image information flow. Text is tokenized using transformer’s tokenizer (e.g. WordPiece for BERT). Word embeddings are obtained from transformer’s word embeddings. Both image and word embeddings have positional embeddings separately, which are added to the embeddings. Since we have separate positional embeddings for the two modalities, it is redundant to have modality embeddings. Therefore, we do not use any explicit modality embeddings (i.e., “segment embedding” in BERT).

2We speculate low-level pixel embeddings can not directly interact with higher-level text-token embeddings. This might explain why ViLT did not work with BERT as the main transformer but it worked with Vision Transformer (Dosovitskiy et al., 2021) as the main transformer.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.