Add option use lazy data loading in dataset#285

Merged

khai-meetkai merged 2 commits intomainfrom add_option_to_tokenize_data_in_dataset

Nov 7, 2024

+69-47

Commits on Nov 7, 2024

add option to use lazy data loading or not
khai-meetkai
committed
move resolve_json_refs to prompt_utils to avoid importing fastapi in training
khai-meetkai
committed