Multimodal Semantic Mismatch Detection in Social Media Posts

Kehan Wang · Seth Z. Zhao · David Chan · Avideh Zakhor · John Canny

IEEE 24th International Workshop on Multimedia Signal Processing (MMSP) 2022

Download Dataset

We release two dataset, Twitter-1M and Twitter-60k-with-audio, with precomputed features from S3D on video and DeBERTa-v3 on text. Twitter-60k-with-audio also contains DeBERTa-v3 features on the transcript of tweet audios, transcribed using Wav2vec 2.0.

• Twitter-1M contains all features in a zip, and their corresponding tweet-ids. Tweet ids and features are in the same order. Features can be loaded through np.load.
• Twitter-60k-with-audio contains all features and ids in a dict saved in .npy files. To load the dict, use

import numpy as np
dictionary = np.load(f'60k_{mode}_features.npy', allow_pickle=True).flatten()[0]
dictionary.keys() # dict_keys(['text_feats', 'caption_feats', 'video_feats', 'ids'])

Acknowledgement

We would like to graciously acknowledge Google for partially providing cloud computing resouces for this project, and Twitter for Academic Research API.

Citation

Please cite Semantic Mismatch Detector in your publications if it helps your research:

@inproceedings{SMD,
  title     = {Multimodal Semantic Mismatch Detection in Social Media Posts},
  author    = {Kehan Wang and Seth Z. Zhao and David Chan and Avideh Zakhor and John Canny},
  booktitle = {Proceedings of IEEE 24th International Workshop on Multimedia Signal Processing (MMSP)},
  year      = {2022}
}

License

MIT license