You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A tracker params if not passed even in happy case can cause exception.
(.fms-hf-tuning-venv) ➜ fms-hf-tuning git:(fix-bad-value-error) ✗ accelerate launch \
--main_process_port 1234 \
--config_file /home/schrodinger/workspace/ibm/fms-hf-tuning/fixtures/accelerate_fsdp_defaults.yaml \
--num_processes 2 \
-m tuning.sft_trainer \
--model_name_or_path "Maykeye/TinyLLama-v0" \
--training_data_path "/home/schrodinger/bin/twitter_complaints.json" \
--output_dir /home/schrodinger/bin/output/tiny-llama-ft-multigpu \
--num_train_epochs 5 \
--per_device_train_batch_size 4 \
--gradient_accumulation_steps 4 \
--per_device_eval_batch_size 4 \
--learning_rate 0.000001 \
--use_flash_attn no \
--response_template "\n### Label:" \
--dataset_text_field "output" \
--tracker aim \
--aim_repo /home/schrodinger/bin/aimrepo/ \
--experiment aim-test-distributed-main \
--torch_dtype bfloat16
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/accelerate/utils/launch.py:253: FutureWarning: `fsdp_backward_prefetch_policy` is deprecated and will be removed in version 0.27.0 of 🤗 Accelerate. Use `fsdp_backward_prefetch` instead
warnings.warn(
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
max_seq_length 4096 exceeds tokenizer.model_max_length 2048, using tokenizer.model_max_length 2048
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
Generating train split: 10400 examples [00:00, 279328.40 examples/s]
max_seq_length 4096 exceeds tokenizer.model_max_length 2048, using tokenizer.model_max_length 2048
Map: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10400/10400 [00:00<00:00, 19288.10 examples/s]
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/transformers/training_args.py:2007: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.
warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in '__init__': dataset_text_field, max_seq_length. Will not be supported from version '1.0.0'.
Deprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead.
warnings.warn(message, FutureWarning)
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:280: UserWarning: You passed a `max_seq_length` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.
warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:318: UserWarning: You passed a `dataset_text_field` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.
warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/transformers/training_args.py:2007: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.
warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in '__init__': dataset_text_field, max_seq_length. Will not be supported from version '1.0.0'.
Deprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead.
warnings.warn(message, FutureWarning)
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:280: UserWarning: You passed a `max_seq_length` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.
warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:318: UserWarning: You passed a `dataset_text_field` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.
warnings.warn(
Map: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10400/10400 [00:00<00:00, 39494.35 examples/s]
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:408: UserWarning: You passed a tokenizer with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to overflow issues when training a model in half-precision. You might consider adding `tokenizer.padding_side = 'right'` to your code.
warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:408: UserWarning: You passed a tokenizer with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to overflow issues when training a model in half-precision. You might consider adding `tokenizer.padding_side = 'right'` to your code.
warnings.warn(
Exception while saving additional metrics and metadata ValueError('set_params passed to aimstack should be called with a dict of params')
Expected behavior
No error should be thrown.
The text was updated successfully, but these errors were encountered:
Describe the bug
A tracker params if not passed even in happy case can cause exception.
Expected behavior
No error should be thrown.
The text was updated successfully, but these errors were encountered: