-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CI] [GHA] HuggingFace cache #28481
[CI] [GHA] HuggingFace cache #28481
Changes from 29 commits
52424ae
82a03dd
0b31d9f
51235e9
1a125b3
2e1f773
0c4d7c0
4d0d0a7
eb4b951
c90de13
b4b993f
92823ad
1c9f2a6
2156899
fee163c
f1c383c
aab40fd
42d3572
cccdb79
824d35f
358c68c
4fc43e7
3287f5d
4a864e8
6b803ad
8d37d9a
d4ca97c
a5cd5cd
07898ab
81d262c
d1aaa31
a01ddb5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -26,11 +26,7 @@ | |
os.environ['TFHUB_CACHE_DIR'] = tf_hub_cache_dir | ||
os.environ['HF_HUB_CACHE'] = hf_cache_dir | ||
Comment on lines
26
to
27
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are these two environment variables re-initialized from outside (from GHA)? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, they are set on a per-job basis like in: https://github.com/openvinotoolkit/openvino/pull/28481/files#diff-059da6ae752137afeded3262287ede715ddfb1659b5d31489f723c9f971b008bR71 and https://github.com/openvinotoolkit/openvino/pull/28481/files#diff-58b495d99e8e2631e772171d5dfb9d6c331d35b8f4b4394a78cf053767bda273R63. These two lines are only for the case if the env is empty and they are populated with the tmp directories. |
||
|
||
no_clean_cache_dir = False | ||
hf_hub_cache_dir = tempfile.gettempdir() | ||
if os.environ.get('USE_SYSTEM_CACHE', 'True') == 'False': | ||
no_clean_cache_dir = True | ||
os.environ['HUGGINGFACE_HUB_CACHE'] = hf_hub_cache_dir | ||
clean_hf_cache_dir = bool(os.environ.get('USE_SYSTEM_CACHE')) | ||
mvafin marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# supported_devices : CPU, GPU | ||
test_device = os.environ.get('TEST_DEVICE', 'CPU;GPU').split(';') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we define it in the common job env and get the value from job arguments?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do not do that for any other such variable, e.g., we define the pip cache directory in the reusable jobs so I think it could be done like it is now, when setting the variables.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm concerning about the following case: this job can be run on different runners and in theory the mount point could be different, inside the job we don't know the runners environment, that is why I think it is better to set it in the workflows. In addition to that we will be able to set it once inside the job