Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better logging #262

Merged
merged 8 commits into from
Aug 24, 2021
Merged

Better logging #262

merged 8 commits into from
Aug 24, 2021

Conversation

LysandreJik
Copy link
Member

@LysandreJik LysandreJik commented Aug 5, 2021

This is the second part of the logging overhaul of the huggingface_hub.

This introduces the same logging utility that is defined in transformers and datasets. It aligns closer to datasets' logging system rather than transformers.

Since huggingface_hub is a dependency of both, it may make sense to upstream all logging-related logic in this library.

The third part of the better logging system will be to pass over all modules and define more constant logging with precise level reliability.

cc @lhoestq

Examples:

Logging level ERROR (close to no logging):

from huggingface_hub import Repository
from huggingface_hub import logging

logging.set_verbosity_error()

repository = Repository("new-repo", clone_from="Jikiwa/text-files"
Cloning https://huggingface.co/Jikiwa/text-files into local empty directory.

Process finished with exit code 0

Logging level DEBUG (maximum amount of logs):

from huggingface_hub import Repository
from huggingface_hub import logging

logging.set_verbosity_debug()

repository = Repository("new-repo", clone_from="Jikiwa/text-files"
git version 2.32.0
git-lfs/2.13.3 (GitHub; linux amd64; go 1.16.4)
Cloning https://huggingface.co/Jikiwa/text-files into local empty directory.
Download file file10.txt: 100%|██████████████████████████████████████████████████████████████████████████████| 19.2M/19.2M [00:17<00:00, 1.18MB/s]
Download file file2.txt: 100%|████████████████████████████████████████████████████████████████████████████████| 14.4M/14.4M [00:17<00:00, 883kB/s]
Download file file3.txt: 100%|████████████████████████████████████████████████████████████████████████████████| 14.4M/14.4M [00:17<00:00, 883kB/s]
Download file file1.txt: 100%|████████████████████████████████████████████████████████████████████████████████| 14.4M/14.4M [00:17<00:00, 883kB/s]
Download file file11.txt: 100%|██████████████████████████████████████████████████████████████████████████████| 19.2M/19.2M [00:17<00:00, 1.18MB/s]
Clean file file2.txt: 100%|██████████████████████████████████████████████████████████████████████████████████| 14.4M/14.4M [00:02<00:00, 7.49MB/s]
Clean file file10.txt: 100%|█████████████████████████████████████████████████████████████████████████████████| 14.4M/14.4M [00:02<00:00, 7.49MB/s]
Clean file file8.txt: 100%|██████████████████████████████████████████████████████████████████████████████████| 14.4M/14.4M [00:02<00:00, 7.49MB/s]
Clean file file3.txt: 100%|██████████████████████████████████████████████████████████████████████████████████| 14.4M/14.4M [00:02<00:00, 7.50MB/s]
Clean file file1.txt: 100%|██████████████████████████████████████████████████████████████████████████████████| 14.4M/14.4M [00:02<00:00, 7.50MB/s]
Clean file file11.txt: 100%|█████████████████████████████████████████████████████████████████████████████████| 19.2M/19.2M [00:02<00:00, 7.50MB/s]


Process finished with exit code 0

Base automatically changed from progress-bars to main August 10, 2021 11:10
@LysandreJik LysandreJik merged commit 2e81cf3 into main Aug 24, 2021
@LysandreJik LysandreJik deleted the better-logging branch August 24, 2021 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant