Finetune with/without flash attention

[chiroiu96]

I am trying to finetune LLaVA on a custom dataset. Flash attention is not supported by my GPU.

1. Do I necessarily need to finetune with flash attention?
2. Will results differ with or without flash attention?

[Lawhori]

No, you do not need flash attention to finetune, in the finetuning script use train.py instead of train_mem.py. It worked for me and i didnt see much difference in results.