WIP: Add cloud testing + refactoring nominal x y replacement #35
Conversation
Codecov Report
@@ Coverage Diff @@
## master #35 +/- ##
==========================================
+ Coverage 33.81% 41.73% +7.91%
==========================================
Files 5 5
Lines 343 357 +14
==========================================
+ Hits 116 149 +33
+ Misses 227 208 -19
Continue to review full report at Codecov.
|
|
@andersy005: This is the PR I was talking about earlier. It seems like the circleci is running but its so slow that it wont even finish a fraction of the test, so that is not feasible at the moment. Could you think of a way to speed the reading with intake-esm or get more bandwidth? For travis it really seems like they dont allow access from within their testing env...see this cryptic error [here])(https://travis-ci.com/github/jbusecke/cmip6_preprocessing/jobs/314580159). |
| table_id="Omon", | ||
| grid_label=grid_label, | ||
| ) | ||
| ddict = cat.to_dataset_dict( |
There was a problem hiding this comment.
Try passing storage_options with token=anon to to_dataset_dict()
ddict = cat.to_dataset_dict(zarr_kwargs={"consolidated": True, "decode_times": False}, \
preprocess=combined_preprocessing, storage_options={'token':'anon'})
I found a similar issue here: googleapis/google-auth-library-python#287. It appears that at some point TravisCI locked access to APIs (including Google's authentication API). Unfortunately the linked issue does not provided any solution (the person who opened the issue ended up migrating to CircleCI). |
| cat = col.search( | ||
| source_id=source_id, | ||
| experiment_id=experiment_id, | ||
| variable_id=variable_id, | ||
| table_id="Omon", | ||
| grid_label=grid_label, | ||
| ) |
There was a problem hiding this comment.
For testing purposes, do you need all member_ids?
There was a problem hiding this comment.
Thats a good point. Ill try to restrict this.
|
It appears that the upstream-dev is failing due to numpy build error. Do we need to test against the master branch of NumPy i.e. would it be okay to just test against the latest stable release of NumPy? |
|
Glad to see Python 3.6 job running successfully. Regarding Python 3.8 and Python 3.7 jobs, I recommend re-running the jobs with SSH and SSH-ing into the container and running one of the hanging tests: pytest -v cmip6_preprocessing/tests/test_preprocessing_cloud.py::test_lat_lon[BCC-ESM1-thetao-historical-gn] |
|
@andersy005 , sorry for the delay on this. I am afraid I am not quite sure how to SSH into the container. Is there some documentation I could follow for this? Many thanks for your help. |
|
Head over to one of the jobs' URL: https://app.circleci.com/pipelines/github/jbusecke/cmip6_preprocessing/26/workflows/1679fc07-8435-4abf-8537-025457a0a94d/jobs/91/steps and click on dropdown
Here' the detailed documentation page: https://circleci.com/docs/2.0/ssh-access-jobs/ |
|
Thank you so much! Ill try that out in a bit. |
|
I think I will give up on making these tests work with CI for this PR. It is not essential for this change, and I would like to move ahead with some other changes. I will for now check these tests only locally. @rabernat mentioned pangeo-bot as a possible solution. I will investigate that possibility further and come back to it. I really appreciete the help on this @andersy005. |
👍 This sounds like a more robust solution.
Anytime! Sorry this didn't work as we had expected. |
Towards fixing #30 and #34