List profile versions from blob storage #575
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BainanXia
reviewed
Jan 14, 2022
Comment on lines
+20
to
+40
| def get_profile_version_cloud(grid_model, kind): | ||
| """Returns available raw profile from blob storage. | ||
|
|
||
| :param str grid_model: grid model. | ||
| :param str kind: *'demand'*, *'hydro'*, *'solar'* or *'wind'*. | ||
| :return: (*list*) -- available profile version. | ||
| """ | ||
| bfs = fs.open_fs("azblob://besciences@profiles").opendir(f"raw/{grid_model}") | ||
| return _get_profile_version(bfs, kind) | ||
|
|
||
|
|
||
| def get_profile_version_local(grid_model, kind): | ||
| """Returns available raw profile from local file. | ||
|
|
||
| :param str grid_model: grid model. | ||
| :param str kind: *'demand'*, *'hydro'*, *'solar'* or *'wind'*. | ||
| :return: (*list*) -- available profile version. | ||
| """ | ||
| profile_dir = fs.path.join(server_setup.LOCAL_DIR, "raw", grid_model) | ||
| lfs = fs.open_fs(profile_dir) | ||
| return _get_profile_version(lfs, kind) |
There was a problem hiding this comment.
Not sure whether there is an easy way to combine these two functions together and make location of the profiles a parameter.
There was a problem hiding this comment.
There might be a simplification we could do, say if we had the local and blob filesystems already created as part of some DataAccess object. This seemed like the simplest approach for now though.
|
Very nice. I have never been a fan of the json file |
Same. I've been looking forward to this PR ever since adding the json file. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request doc
Purpose
Currently the way we do this involves parsing a
version.jsonfile located in blob storage which must match the versions that are actually available. However to list locally available profiles, we check the files directly, which is more reliable than using the json file. This PR makes it so we use the same code for both local and blob storage, with the only difference being which filesystem is provided.What the code is doing
See above. It is a refactor to be more consistent and eliminate the need for the
version.jsonfile.Testing
Ran the
_localand_cloudfunctions interactively. Added new test for the shared logic (which was already in use, but untested).Time estimate
10 min