Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks

Official implementation of Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks.

TODO

• Add Attack Generation Code.
• Add Eval Code.

Setup

Prepare dataset.

• StanfordCars
  Download StanfordCars dataset

• Aircraft
  Download Aircraft

• OxfordPets
  Download OxfordPets

• Food101
  Download Food101

• Flowers
  Download Flowers

For each dataset, set the right path at dataset_eval.py and dataset_eval_gpt4.py

Model Setup

To setup

• LLaVA/InstructBLIP, simply make sure you have transformers installed.
• MiniGPT4, make sure to clone their repo, follow their instructions, and then set up the path to the config file in utils.py line 236.

Eval.

To evaluate LLaVA/InstructBlip/MiniGPT-4, run:

python dataset_eval.py --model [llava/blip/minigpt4] --method [Method] --dataset [Dataset]

To evaluate GPT-4, first set your api key at utils_models/utils_gpt4.py, and then run:

python dataset_eval_gpt4.py --method [Method] --dataset [Dataset]

Citation

If you find this repository useful please give it a star and cite as follows! :) :

    @article{qraitem2024vision,
    title={Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks},
    author={Qraitem, Maan and Tasnim, Nazia and Saenko, Kate and Plummer, Bryan A},
    journal={arXiv preprint arXiv:2402.00626},
    year={2024}
    }