M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining

Despite the potential of multi-modal pre-training to learn highly discriminative feature representations from complementary data modalities, current progress is being slowed by the lack of large-scale modality-diverse datasets. By leveraging the natural suitability of E-commerce, where different modalities capture complementary semantic information, we contribute a large-scale multi-modal pre-training dataset M5Product. The dataset comprises 5 modalities (image, text, table, video, and audio), covers over 6,000 categories and 5,000 attributes, and is 500 larger than the largest publicly available dataset with a similar number of modalities. Furthermore, M5Product contains incomplete modality pairs and noise while also having a long-tailed distribution, resembling most real-world problems. We further propose Self-harmonized ContrAstive LEarning (SCALE), a novel pretraining framework that integrates the different modalities into a unified model through an adaptive feature fusion mechanism, where the importance of each modality is learned directly from the modality embeddings and impacts the inter-modality contrastive learning and masked tasks within a multi-modal transformer model. We evaluate the current multi-modal pre-training state-of-the-art approaches and benchmark their ability to learn from unlabeled data when faced with the large number of modalities in the M5Product dataset. We conduct extensive experiments on four downstream tasks and demonstrate the superiority of our SCALE model, providing insights into the importance of dataset scale and diversity.

Audio Video Figure 1. Our M5Product dataset contains a large variety of modalities (image, text, table, video and audio) that depict the categories, descriptions, materials, properties and purposes of Ecommerce products, and diverse real-world data samples.

Introduction
Self-supervised learning has been driving the rapid development of fields such as computer vision and natural language processing, as well as research on multi-modal representation learning. In particular, it has been shown both from a theoretical [18] and a practical [16,58] perspective that large scale datasets with diverse modalities can effectively enhance the discrimination of generated features and thus improve the performance in vision-language tasks. However, current advances are severely limited by the lack of such large-scale diverse-modality datasets, with the largest public multi-modal datasets only containing text and image modalities and no category information [41].
Given the prevalence of online shopping in daily life, with its natural occurrence of multi-modal information and diverse categories, multi-modal pre-training on Ecommercial products has received increasing attention and led the developments of next-generation technology for several downstream tasks (e.g., multi-modal retrieval, multimodal classification, and clustering). However, even among the present product datasets (e.g., RPC checkout [48], Dress Retrieval [9] and Product1M [55]), the number of categories is insufficient to robustly verify the performance of downstream tasks.
More importantly, the current research community mostly focuses on two modalities (text and image) in both general multi-modal and E-commerce datasets, while ignoring the importance of additional complementary information from structural data as well as video and audio modalities. Tabular data for instance can provide detailed information about properties and characteristics, such as brand, materials, attributes, and scenarios, while audio and video can convey different perspectives, scales, affordances, selling points, characteristics, and use scenarios that are not obvious from images or text alone. The focus on these two modalities is partly due to the lack of datasets with diverse modalities as well as an under-exploration of approaches to balance the modality importance in these settings. In particular, two key challenges are: 1) Modality Interaction: How to learn common representations from unimodal, bimodal, trimodal, and even multi-modal relationships between different modalities using an elegant approach that scales to a large number of modalities; 2) Modality Noise: How to reduce the influence of modality noise (missing and incompleted modalties) during the training process.
To address the problem of insufficient modality diversity and limited scale, while at the same time providing a challenging real-world scenario, we present a very largescale E-commerce multi-modal product dataset M5Product, which is one of the largest and most diverse multi-modal product dataset to date. Our M5Product dataset contains more than 6 million multi-modal samples from 6,232 categories and has more complex and diverse modalities than existing datasets. This allows M5Product to be used for more comprehensive evaluation of the practical application and generalization abilities of multi-modal pretraining models and can improve the modality fusion performance, facilitating new directions in multimodal research. Figure 1 shows the five modalities (image, caption, video, audio, and specification (table)) of our dataset.
To further address the modality fusion limitations of existing methods as well as handle modality noise, we propose a generic framework that takes five-modality data as inputs, as shown in Figure 2. The framework consists of a simple and efficient multi-modal five stream pre-training model named Self-harmonized ContrAstive LEarning (SCALE) and is evaluated on several downstream tasks and compared with several recent state-of-the-art vision-language models [7,27,30,38,42,45,56]. SCALE increases modality alignment effectiveness by implementing a self-harmonized strategy that adapts the alignment weights between different modalities in the contrastive learning modules and masked tasks to adaptively integrate complementary modality information. In summary, our contributions are as follows: • We provide the largest five-modality E-commerce dataset M5Product. Through its large scale, diversity, complex real scenarios and number of modalities, M5Product provides a comprehensive environment for evaluating the generalization performance of multi-modal pre-training models.
• Our Self-harmonized Contrastive Learning (SCALE) framework learns adaptive modality interactions, resulting in more effective modality fusion. We compare SCALE to a comprehensive set of baseline methods and demonstrate its superior performance on the M5Product dataset. •

Related Work
Multi-modal pre-training datasets. Most multi-modal pre-training datasets are collected from social websites (e.g., Twitter and Facebook) and are limited to just two modalities collected for specified tasks. These datasets can be divided into four categories according to their modality composition, i.e., audio/text, video/text, image/text, and others. Among these, LJ Speech [19] and SQuAD [25] are classical audio/text datasets and used for voice synthesis and audio Q&A, while most video/text datasets [2,20,24,32,46,47,51,57] are used for video Q&A. However, these datasets commonly only contain a limited number of samples, limiting their applicability to multi-modal pretraining. Image/text datasets [1,4,8,17,22,23,29,34,41,43,48,53], on the other hand, tend to be larger and have been widely used for pretraining multi-modal models. Among these, the CC 3M [41] with more than three million image-text pairs is the most widely used pre-training dataset, and has recently been expanded to CC 12M [5], the largest text-image cross-  [54] and XMedia [36], where CMU-MOSEI mainly focuses on the emotional analysis and XMedia is utilized for cross-modal retrieval. Aside from the abovementioned datasets, there exist several E-commerce datasets. The Dress Retrieval [9], RPC checkout [48] and Product1M [55] are typical E-commerce multi-modal datasets. The Dress Retrieval dataset contains 20,200 samples from 50 clothing categories, RPC checkout offers 30,000 samples of small retail goods on simple backgrounds and Product1M provides 1.18 million samples from 458 cosmetics classes. Compared with these three datasets, our M5Product is not only larger in terms of categories and data scale, but also contains a more diverse set of modalities. A detailed comparison with other multi-modal pre-training datasets is provided in Table 1.
Within E-commerce, fashion-based tasks have been addressed in among others FashionBERT [13], MAAF [11], Kaleido-BERT [59], M6 [28] and CAPTURE [55]. All existing studies in the E-commerce scenarios focus solely on the image and text modalities and none of the approaches can utilize more modalities. Besides, all existing methods default to assigning the same contribution to different modalities when modeling multi-modal interactions.More  Table   specifically, transformer-based approaches combine highlevel features that are extracted from the different inputs via concatenation, where the uni-modal transformers are trained via masked task constraints or via constructing intermodality losses between different modalities. This restricts the models from effectively prioritizing modalities and tends to limit performance improvements as the number of modalities increases.
Our proposed benchmark fills this gap by exploiting all the diverse modalities of the M5Product dataset and provides a strong baseline for multi-modal pre-training research in the field of E-commerce and beyond.

M5Product Dataset
Data Collections. The dataset is crawled from a popular E-commerce website 2 . and the front page of each Ecommerce product is analyzed to collect the multi-modal information consisting of product images, captions, videos, and specifications (table information) 3 . Duplicate data is removed and audio information is extracted from videos via the moviepy 4 tool and saved in mp3 format. For product specifications, we extract 5,679 product properties and 24,398,673 values to construct a table database coarsely labeled by e-commerce merchants. After processing, the dataset contains 6,313,067 samples. Note, being a realworld dataset, our M5Product is, unlike traditional multimodal datasets, not a complete paired dataset and contains samples with only a subset of modalities as well as longtailed distributed ( Figure 3). We summarize the product characteristics that are relayed by the different modalities in our dataset in Table 2, where APP, USA, SPEC, SELL, PROD, MATE and CATE denote Appearance, Usage, Specification, Selling Point, Production, Material and Category Descriptions, respectively. Quantitative analysis. 1) Diversity: The dataset consists of more than 6,000 classes covering various and massive amounts of E-commerce products such as clothes, cosmetics, and instruments. Figure 1 illustrates the diversity of the modalities and categories and we further provide a description of the data format and the collection process in Section E of the supplementary materials. Finally, a quantita- 2 We are authorised by the company to access and obtain the data. We are further authorised to share the dataset and the detailed license is given in Section A of the supplementary material 3 In this work we focus on core data modalities (image, text, video, audio, and table data) only and do not consider extracted feature representations such as OCR and Motion embeddings that are extracted from core modalities as separate modalities.  Figure 2. An illustration of our M5Product benchmark. It consists of a five-modality E-commerce dataset with a more diverse and complex backgrounds collected from the real-world online-shopping website. It also proposes a SCALE model to capture the maximum modality complementary information for four common downstream tasks: 1) multi-modal retrieval, 2) fine-grained retrieval, 3) multimodal classification, and 4) multi-modal clustering. The benchmark verifies the effectiveness of modality diversity in five widely used modalities.  tive analysis of the category and modality distributions can be found in Section F. Note that about 5% of products are unimodal samples e.g. only either contain images, captions, or tabular properties. 2) Quality: We further provide a comparison between our M5Product dataset and some widelyused datasets for multi-modal pre-training in Table 1. A more extensive comparison with other multi-modal datasets can be found in Section H of the supplementary materials. Compared with existing multi-modal datasets, M5Product is the first extremely large public real-world E-commerce product dataset that contains data of more than two modalities. Moreover, our dataset contains a large amount of instances, i.e., more than six million samples from the 6,232 coarse categories. These abundant data will benefit several downstream tasks such as self-learning, weakly-supervised learning, multi-modal retrieval, cross-modal generation and fine-grained recognition. Additional analysis. In the supplementary materials, we provide dataset collection details in Section B and detail how the dataset is split into training and test in Section D and how annotations are obtained in Section C. We further provide a smaller split, referred to as subset, which is used to show the difference in performance for a smaller dataset. Finally, we provide further insights into the composition of the dataset (missing modalities, unimodal data analysis, and data format) in supplementary Section F.

Our Methodology
As shown in Figure 2, our SCALE framework consists of a self-harmonized contrastive learning module and a selfsupervised multi-modal transformer. In this section, we first provide the architectural design of SCALE in Section 4.1 before describing the five masked tasks that enable the selfsupervised learning of SCALE in Section 4.2. Finally, we present the detailed learning process of SCALE and detail how multi-modal alignment is achieved in Section 4.3.

Architectural Design of SCALE
As depicted in Figure 2, SCALE is a typical singlestream transformer architecture. In the bottom part, the Image/Text/Table/Video/Audio embedding layers and transformers aim to extract modality features and generate token features. Specifically, the text and table encoders are standard transformers to encode the caption and table information of products, respectively. The image encoder instead takes proposals extracted by bottom-up-attention [3] as inputs, while ordinal frames sampled from the video are fed into the video encoder. For the audio encoder, SCALE extracts MFCC [33] features from audio. After being processed by the separate modality encoders, the token features of different modalities are concatenated and fed into a Joint Co-Transformer (JCT) module to capture the token relationships between different modalities. Missing Modalities. Zero imputation of missing modalities is leveraged to utilize all available data when training SCALE. We provide experimental evidence that SCALE benefits from the incomplete samples in Section I of the supplementary material.

SCALE by Masked Multi-Modal Tasks
Similar to previous works, we utilize several pretext tasks (PRE) to facilitate self-supervised learning of SCALE in the Joint Co-Transformer module. For modalitywise feature learning from the image and text modalities, we adopt the Masked Region Prediction task (MRP) and the Masked Language Modeling task (MLM), respectively, after the JCT. Utilizing the characteristics of the table, video, and audio modalities, we further propose a Mask Entity Modeling task (MEM), Mask Frame Prediction task (MFP), and Mask Audio Modeling task (MAM) following a similar strategy of predicting masked tokens. In all masked tasks, the ground-truth labels are the features of masked areas. For all masking tasks, 15% of the inputs are masked out and the remaining inputs are used to reconstruct the masked information. Please note that unlike in the MLM task, where 15% of individual words are masked, 15% of the entities (properties, brand names, etc.) are entirely masked out for the MEM task. This drives our model to learn better table representations to recover the masked inputs, which is illustrated in Section 5.3. The loss function of the ith modality is defined as: where t ¬msk denotes the unmasked tokens surrounding the masked token t msk , θ represents the network parameters, and M i and M ¬i are the ith modality and the remaining modalities, respectively.

Self-harmonized Inter-Modality Contrastive Learning
Self-harmonized Inter-Modality Contrastive Learning (SIMCL) is at the core of our proposed SCALE framework. It aims to facilitate the semantic alignment between different modalities via a self-harmonized strategy for adaptive Inter-Modality Contrastive Learning (IMCL). For a minibatch of modality samples D ∈ R B×M ×F , where B, M and F denote the batch size, number of modalities, and embedding dimension, respectively, we first construct the contrastive loss between each modality.

Inter-Modality Contrastive Learning
Inter-Modality Scores where each sample has two modalities (0) and (1) with the remaining N -1 samples from the other modality, resulting in 2(N -1) negative pairs. For a modality pair (d i ), the cross-modal contrastive loss of each modality pair is: where sim is the cosine similarity, τ is the temperature parameter and 1 [k =i] is a binary indicator function, and 1=1 for k = i and 0 otherwise.
In most prior work, only two modalities are considered and Eq. 2 can be used. However, when considering trimodal data or data with even more than three modalities, it is not suitable to directly fit the loss function as it does not account for the difference in complementary information that different modalities contribute. To solve this problem, we define a simple but effective self-harmonized method to model the complementary process of the inter-modal relationships. We introduce a modality alignment score matrix, to encode the relationships among the inter-modal losses L CL and the intra-modal losses L Mi . The alignment score matrix S for each data sample is initialized by a zero matrix and updated as free model parameters. To obtain modality importance scores for each modality combination, we apply the softmax function to S. Finally, the importance scores are multiplied to generate the modality alignment score S as S = S · softmax(S). The learning process is shown in Figure 4 and illustrates that SIMCL takes full advantage of the inter-modal relationships. Given the modality alignment score S, the triangular part S is selected to weight the inter-modal losses L CL and the diagonal part S \ is uti-lized to constrain the intra-modal losses L Mi , resulting in the weighted loss: where logit is the the loss logit.

Experiments
Implementation Details. We use BERT [10] to initialize the text transformer of our proposed SCALE framework, while the remaining transformers are randomly initialized. Both the single-modality encoders and the multi-modal fusion encoders consist of 6 transformer layers each, adding up to a total of 12 transformer layers. The hidden state size of each modality transformer is 768 and the maximum sequence length for the captions and tables are set to 36 and 64, respectively. Using the same setting as in [30] 5 , we utilize Faster R-CNN [40] with a backbone ResNet101 [15] pre-trained on the Visual Genome dataset [23] to extract region features of selected 10 to 36 bounding boxes with high-class detection probability for each image. We train SCALE with a total batch size of 64 for 5 epochs using the Adam optimizer [21] with a warm-up learning rate of 1e-4. Additional implementation details of our model are provided in Section G of the supplementary material. Baselines. We compare SCALE to the following eight alternative pre-training methods that utilize image and text modalities as well as combinations of both: Bert [10] (Text based ), Image based , ViLBERT [30], CLIP [38], VL-BERT [42], VisualBERT [27], UNITER [7] and CAP-TURE [56]. Image based and BERT [10] are 12-layer transformers based on the MLM (Mask Language Modeling) or MRP (Mask Region Prediction) task using image or text modality, providing single-modal baselines for the product retrieval, classification, and clustering tasks. To ensure a fair comparison, the same hidden size of 768 is chosen for all baselines. Evaluation. We consider the following four downstream tasks to evaluate the learned representations: 1) Multimodal retrieval: This task aims to find the most relevant target products using combinations of two or more modalities. A pair is considered a match if both belong to the same category; 2) Fine-grained multi-modal retrieval: Retrieval on an instance level, where only samples of the same product (i.e. color, model, shape, and style) are considered a match 6 ; 3) Multi-modal classification: Product category classification given the multi-modal features extracted from the joint co-transformer of SCALE using a linear classifier; and 4) Multi-modal clustering: Product category clustering using k-Means clustering and the same features as in the classification setting. For product retrieval, we adopt the widely used metrics mean Average Precision (mAP) and Precision (Prec) [14,31,49] to evaluate the retrieval accuracy on the two retrieval tasks. For product classification and clustering, all methods are evaluated using the Classification Precision (Classification accuracy), Normalized Mutual Information (NMI) [52] and Purity metrics. In all experiments, models are trained on the training split. The pretrained model is then applied to extract the modality features of the gallery and test splits for the product retrieval and clustering tasks. For the classification task, we finetune the pre-trained model on the classification subset containing 1,805 categories/classes and utilize the finetuned model to extract the features of the classification test set.

Modality Diversity
To examine the performance of our proposed SCALE framework and to verify the benefits of diverse modalities and dataset scale, we train SCALE with an increasing number of modalities and observe the variations in classification and multi-modal retrieval performance both for the whole M5Product dataset and the subset. More specifically, fused features are extracted from the joint co-transformer (JCT) of our SCALE after finetuning for the classification task and after pre-training and finetuning for the (coarse) multimodal retrieval task. Results in Table 3 show that performance increases across all settings as modalities are added, illustrating the benefit of complementary modality information to learning multi-modality feature representations. It can also be observed that modality gains are larger on the whole dataset, supporting Interesting Observation 1.
We further provide results for an extensive set of modality combinations to verify SCALEs effectiveness in leveraging the diverse modalities of our M5Product dataset. Table 4 provides results for the coarse-and fine-grained multimodal retrieval tasks as well as the classification task after finetuning the model. As in the previous experiment, noticeable improvements can be observed as additional modalities are added. In particular, the addition of the text modality leads to high modality gains, verifying the benefits of including more diverse modalities that can capture different views of the same product. Interestingly, performance on the coarse-grained retrieval task is significantly worse than on the fine-grained retrieval task in most cases, indicating the complexity of the M5Product dataset and the diversity of the products in each category. Semantic Alignment. To additionally demonstrate the importance of modality diversity, we compute the modality correlation, the average cosine similarity between image and text features as obtained by the JCT, for an increasing number of modalities. Figure 5 illustrates that the semantic alignment capability of the pre-training model increases as Table 3. The (pretrain/finetune) performance gains from sequentially adding more modalities using SCALE on the subset (top) and the whole dataset (bottom). The retrieval performances are based on the features extracted from pretrain and finetune stages.   the number of modalities grows.

Multi-modal Downstream Tasks
We evaluate SCALE on the M5Product dataset for the product retrieval, classification, and clustering tasks and compare results to several benchmark approaches in Table 5. For the Image based and BERT [10] models, which only utilize the image and text features, respectively, the extracted features are fed directly into the classification model. For our SCALE approach, we utilize the fused modality features generated by the joint co-transformer, pre-trained on both image and text modalities. Only utilizing the image and text modalities allows us to facilitate a fair comparison to the recent state-of-the-art approaches ViLBERT [30],  CLIP [38], VL-BERT [42], VisualBERT [27], UNITER [7] and CAPTURE [56]. Comparing our SCALE framework to the unimodal models, Image based and Bert [10], we observe that exploiting multi-modal data significantly improves the performance across all tasks. We further observe that SCALE, by leveraging SIMCL, can efficiently fuse the modalities and outperform all the baseline approaches (Interesting Observation 2).

Ablation Studys and Visualization
To explore how SIMCL influences IMCL and the Pretext tasks, we conduct several ablation studies. Table 6 illustrates that improvements of approximately 2% are obtained in the classification task and more than 2% for the coarse-grained retrieval task when including both, highlighting the importance of both the Pretext tasks and the effective modality fusion of SIMCL. We further analyze the effect of the MEM pretext task for the table modality and show the benefit of masking out complete entities over masking out individual tokens (MLM) in Table 7. This benefit can be attributed to the fact that MEM ensures that SCALE learns representations that encode the semantic information of complete entities. Finally, we evaluate the performance of modelling the text and the table modalities using individual modality encoders and compare SCALEs retrieval performance to a baseline where text and table information is concatenated and fed to a single transformer, resembling the process of BERT [10]. By modelling both modalities individually, results in Table 8 illustrate that more information can be preserved and we hypothesize that using a single transformer leads to a loss in table modality information for the benefit of the more expressive text modality. Figure 6 shows t-SNE visualizations of the extracted features for the JCT module of our SCALE model and the alternative approach ViLBert [30] on the M5Product dataset. SCALE not only better distinguishes different classes but also improves class compactness compared to the ViLBert model. Further, the attention attribution for different modalities are shown in Figure 7 and verify that the visual features generated by SCALE are object-oriented and semantically interpretable.

Limitations and future work
The experimental evaluation showed that SCALE is able to learn efficient representations from a large number of modalities for retrieval, classification, and clustering. However, more evaluation of the generative capabilities of the models representations is lacking and tasks such as image and caption generation could be promising directions to explore. We further provide some of SCALEs failure cases in supplementary Section J. Potential negative societal impact. As a result of the strict ethical considerations used in the data collection process, where among others personally identifiable information has been removed, M5Product does not pose any ethical risks.

Conclusion and Discussion
To facilitate multi-modal pre-training, we present the M5Product dataset, which is the largest available multimodal E-commerce product dataset, consisting of five core modalities (image, text, table, video, and audio). To further promote multi-modal research in retail and increase seller and buyer engagement, we also propose the novel SCALE multi-modal pre-training framework. By utilizing Self-harmonized Inter-Modality Contrastive Learning (SIMCL), SCALE is able to model and exploit modality relationships effectively and outperforms previous approaches on the M5Product multi-modal retrieval, classification, and clustering tasks. We believe that both the dataset and the proposed framework work will inspire research on scaling multi-modal pre-training beyond the commonly used image and text modalities.

A. Dataset License
Our M5Product dataset is released under CC BY-NC-SA 4.0 license and can freely be used for noncommercial purposes. More detailed information can be found at https://xiaodongsuper.github. io/M5Product_dataset/terms_of_use.html, which also provides dataset details and usage guidance. Note: For anonymity reasons, the link is not included during the review process.

B. Annotation Collection
We resort to crowd-sourcing to obtain human annotations for the product retrieval task. Specifically, we present human annotators with a matching task, where annotators are asked to select the matching image-text pairs for a given query image-text pair. In our crowdsourcing system, each matching task is presented to five different human annotators and a typical example of our interface is shown in Figure 8. The left part of the interface shows the current query data (image and text), while the right side depicts an example from the candidate list. The annotators are then asked to choose from two options: mismatched and uncertain. The default labelling option is matched. The interface also displays the number of examples that have been reviewed and the total amount of examples to review. Each annotation task, can be considered as a binary classification task for the human worker, where he/she has to decide if the pair is a match or not. For each estimated task, the annotators receive a payment of 3 cents RMB.

C. Annotation
The retrieval task annotation for any query sample consists of all the matched instances in the gallery split. To construct a reliable gallery set, we first use a ResNet50 [15] and Bert-Base [10] to extract features and construct the query candidate pool from all the data that is not contained in the training subset. Specifically, we sample an instance from a category that contains more than 2,000 instances and extract the image and text features. We then concatenate the features and compute the cosine similarity to all other instances of the dataset to produce a pre-ranked candidate list in order to minimize the labelling cost. The final size of the candidate shortlist for each query is 500, which is about 0.01% of the whole gallery split. During the crowd-sourced annotation process, human workers review both images and captions in the candidate list to select which samples are matched with the query instance.
Annotation Rules. It is quite challenging to define whether two images contain the same product when critical aspects are not given in their captions and images. In our annotations, we use product images and their captions as the primary materials for gallery construction. Hence, we define several rules to determine the "same product" condition and provide them as instructions to the annotators. Images contain the same product, if: 1. The two images are in different conditions (e.g., backgrounds, angles, etc), but the products in both images are the same.
2. They should have the same color/model/shape/style, or other features that can be distinguished by humans.
3. The caption has the same product name but the product description differs.
4. They share more than one characteristic such as appearances, materials, colors and so on.
To ensure labeling consistency, each annotation pair is labeled by five human workers in the crowd-sourced platform. In the process, we first make a small dataset from our query list as a Gold Problem to evaluate the annotation capability of each human worker. Based on the labeled results ("Matched" or "Not Matched") from human workers and their annotation capability, we utilize the weighted GLAD [50] inference algorithm to determine the final accepted labels.

D. Dataset Split
The M5Product dataset is split into several parts to ensure consistent training and evaluation of the models on the various tasks. The training set contains 4,423,160 samples from 3,593 classes.
Retrieval To evaluate models on the retrieval tasks, the remaining data is split into gallery-c and query-c sets, which are used for the coarse-grained retrieval task, and gallery-fg and query-fg sets, which are used for the fine-grained retrieval task. The difference between the two retrieval tasks lies in the granularity of their annotation. In the fine-grained task, only identical products are considered a match (for example, all IPHONE 11 Black), while in the coarse-grained task, category labels are being used to group products from each category (for example, all phones are considered a match).
To construct the fine-grained sets, we extracted all cosmetics categories and, using the abovementioned annotation procedure, finally obtained 1,991 query-fg samples and 117,858 gallery-fg samples. The query-c and gallery-c sets contain 24,410 and 1,197,905 samples, respectively. Among the samples in the gallery-c set, 249,614 samples are matched with samples in the query-c set, while 948,291 samples do not match. These unmatched samples are added to the gallery-c set to increase the difficulty of the retrieval task. We further report the finetuned retrieval performance  Figure 8. UI for huamn annotation on product retrieval task.
in the paper, which corresponds to retrieval performance after finetuning the model using the classification training set (see next paragaraph).
Finetuning, Classification and Clustering For the classification and clustering tasks, we sampled 1,805 categories from the whole dataset and obtained 18,526 train samples and 4,632 test samples. We first finetune our SCALE using the classification training set and then extract features from the finetuned model to perform the classification and clustering tasks.

E. Data Format
The dataset consists of 6,313,067 products uploaded by 1,000,517 merchants, where merchant information has been removed to ensure anonymity. In the following, we outline the different modalities: Image data Each product has at least five product images, where the first image is the main image that gives the detailed overview of the product, while the rest depict its functionalities or characteristics. We pick all the main images to construct the dataset. Caption/text data are provided by the 1,000,517 merchants. Note that the text description does not always match well with the other modalities. Video data are used to showcase the products' usage and characteristics to customers. In our dataset, these videos are recorded at a speed of 24 frames per second (FPS). To reduce the amount of redundant information that is contained in adjacent frames and the dataset as a whole, we only select one frame per second. Audio data are extracted from the video data. We extract the corresponding audio information of all sampled video frames. Then the audio frames are transformed into spectrograms using Mel-Frequency Cepstral Coefficients (MFCC) [33]. We set the frame size and hop size as 1,024 and 256, respectively.
Tabular data are a special kind of database that records some additional product characteristics such as appearance, purpose and producer. The tabular data is indexed by the product ID and collected from the whole product database. There are 5,679 different types of property information and 24,398,673 unique values.
F. Unimodal and unpair analysis 1) Unimodal analysis: Figure 9 gives the video, text and merchant distributions. From the figure, we can find that the video duration, the text length and the merchant number range from 1 to 60 seconds, 20 to 40 words and 1 to 10 product numbers, respectively. This variation further illustrates the real-world nature of our dataset. 2) Unpair analysis: In the data collection process, there are 82,577 invalid URLs for the image modalities (1.3% of the products), while the number of samples that contain both the Image and Text modalities is 6,230,490. Further taking into account the table modality, the number of complete samples drops by 1.4% to 6,225,598 samples that have all three modalities. Overall, the dataset contains 5,050,078 samples that contain all five modalities. This means that about 20% of the samples are incomplete. This is mostly due to merchants being biased towards specific modalities, which is a common scenario in the real world.

G. Implementation Details
Our models are implemented in Pytorch [35]. To speed up training, we use Nvidia Apex 8 for mixed precision training. All models are trained on 4 Nvidia 3090 and 2080ti GPUs on our workstations. We use Adam [21] to optimize the parameters of our model, with an initial learning rate of 1e-4, and use a linear learning rate decay schedule with a temperature parameter of 0.1.

H. Dataset Comparison
A comprehensive comparison between our M5Product dataset and other widely used multi-modal pre-training datasets is shown in Table 9. From the table, we can observe that our M5Product not only has more diverse modalities but also contains a large amount of data samples from an abundant amount of categories.

I. Missing data verification
Results in Table 10 show the superiority of our methods over the standard approach of ignoring incomplete samples. We compare two variants of our SCALE framework: 1) SCALE (full-modality) and our proposed SCALE. The only difference between the two methods is the input. The input of the former only includes complete samples (all modalities present), while the input of the latter includes the incomplete modality samples. The verification is performed on the subset dataset as mentioned in the main article. Figures 10, 11, and 12. The first column represents the image and text modality of the query sample, while the eight images to its right belong to the matched results from the gallery set. In the matched results, the samples boxed in blue are the correctly matched samples, while the samples boxed in red are mismatched. These retrieval results illustrate that the learned embeddings are discriminative. However, in a few cases, the recalled samples are not matched due to the limited number of category samples in the gallery set or similar descriptions in the text data.

K. More Visualization
Additional attention visualization results are provided in Figures 13 and 14. Similar to the illustrations in the main paper, these visualizations show that SCALE can learn the detailed semantics in the images and the text.

L. Code and Our dataset.
The code is provided in the supplementary and we also include a few full-modality examples of the dataset. Due to space constraints, we have refrained from sharing the whole dataset.
Car silk circle foot mats Changan Ease cs35 new cx20 Yuexiang V5 to Shang XT Ben Ben mini main driver single piece Zhang Xiaoquan kitchen knife home stainless steel slicing knife chef special kitchen knife cutting vegetables and meat free grinding kitchen knives lulu yoga pants female outer wear net red nude sense fitness pants high waist lifting hip running nine points tight original sports pants Old daddy shoes with rainbow mesh shoes women 2020 summer new thick bottom muffin breathable mesh surface inside high sports shoes TOMY Domeka alloy car model male toys TOMICA Ferrari 246GT / F40 / 512BB / 365GT Yun Yun inter Anji white tea 2020 new tea Ming Qian first pick authentic master spring tea premium rare bulk 100g