Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: tracker params in happy case can cause exception. #289

Closed
dushyantbehl opened this issue Aug 7, 2024 · 0 comments · Fixed by #290
Closed

bug: tracker params in happy case can cause exception. #289

dushyantbehl opened this issue Aug 7, 2024 · 0 comments · Fixed by #290

Comments

@dushyantbehl
Copy link
Contributor

Describe the bug

A tracker params if not passed even in happy case can cause exception.

(.fms-hf-tuning-venv) ➜  fms-hf-tuning git:(fix-bad-value-error) ✗ accelerate launch \
--main_process_port 1234 \
--config_file /home/schrodinger/workspace/ibm/fms-hf-tuning/fixtures/accelerate_fsdp_defaults.yaml \
--num_processes 2 \
-m tuning.sft_trainer \
--model_name_or_path "Maykeye/TinyLLama-v0" \
--training_data_path "/home/schrodinger/bin/twitter_complaints.json" \
--output_dir /home/schrodinger/bin/output/tiny-llama-ft-multigpu \
--num_train_epochs 5 \
--per_device_train_batch_size 4 \
--gradient_accumulation_steps 4 \
--per_device_eval_batch_size 4 \
--learning_rate 0.000001 \
--use_flash_attn no \
--response_template "\n### Label:" \
--dataset_text_field "output" \
--tracker aim \
--aim_repo /home/schrodinger/bin/aimrepo/ \
--experiment aim-test-distributed-main \
--torch_dtype bfloat16
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/accelerate/utils/launch.py:253: FutureWarning: `fsdp_backward_prefetch_policy` is deprecated and will be removed in version 0.27.0 of 🤗 Accelerate. Use `fsdp_backward_prefetch` instead
  warnings.warn(
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
max_seq_length 4096 exceeds tokenizer.model_max_length             2048, using tokenizer.model_max_length 2048
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
Generating train split: 10400 examples [00:00, 279328.40 examples/s]
max_seq_length 4096 exceeds tokenizer.model_max_length             2048, using tokenizer.model_max_length 2048
Map: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10400/10400 [00:00<00:00, 19288.10 examples/s]
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/transformers/training_args.py:2007: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.
  warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in '__init__': dataset_text_field, max_seq_length. Will not be supported from version '1.0.0'.

Deprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead.
  warnings.warn(message, FutureWarning)
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:280: UserWarning: You passed a `max_seq_length` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.
  warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:318: UserWarning: You passed a `dataset_text_field` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.
  warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/transformers/training_args.py:2007: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.
  warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in '__init__': dataset_text_field, max_seq_length. Will not be supported from version '1.0.0'.

Deprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead.
  warnings.warn(message, FutureWarning)
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:280: UserWarning: You passed a `max_seq_length` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.
  warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:318: UserWarning: You passed a `dataset_text_field` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.
  warnings.warn(
Map: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10400/10400 [00:00<00:00, 39494.35 examples/s]
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:408: UserWarning: You passed a tokenizer with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to overflow issues when training a model in half-precision. You might consider adding `tokenizer.padding_side = 'right'` to your code.
  warnings.warn(
/home/schrodinger/.fms-hf-tuning-venv/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:408: UserWarning: You passed a tokenizer with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to overflow issues when training a model in half-precision. You might consider adding `tokenizer.padding_side = 'right'` to your code.
  warnings.warn(
Exception while saving additional metrics and metadata ValueError('set_params passed to aimstack should be called with a dict of params')

Expected behavior

No error should be thrown.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant