Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/add syracuse qdr retriever #1

Merged
merged 24 commits into from
Jun 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
5ffbfd3
(HP-1459): update README and gitignore
george42-ctds May 30, 2024
97027ef
(HP-1459): add pyproject.toml, poetry.lock
george42-ctds May 30, 2024
a623c4a
(HP-1459): add qdr_downloads module
george42-ctds May 30, 2024
78d4096
(HP-1459): add unit tests
george42-ctds May 30, 2024
e3b26c2
(HP-1459): add ci.yaml and ci_commands_script.sh
george42-ctds May 30, 2024
b30a114
(HP-1459): add image_build_push and wool
george42-ctds May 30, 2024
9ddd867
(HP-1459): add secrets.baseline and pre-commit-config
george42-ctds May 30, 2024
6062069
(HP-1459): expand documentation for qdr_download
george42-ctds May 31, 2024
5aac914
(HP-1459): use streaming in download
george42-ctds May 31, 2024
eb17006
(HP-1459): clean up tests
george42-ctds May 31, 2024
6d1a0ce
(HP-1459): clean up request_headers test
george42-ctds May 31, 2024
c114766
(HP-1459): clean up tests
george42-ctds Jun 2, 2024
b1ce9e6
(HP-1459): clean up comments
george42-ctds Jun 3, 2024
36201e3
(HP-1459): add content to README
george42-ctds Jun 3, 2024
9919689
Merge branch 'master' into feat/add-syracuse-qdr-retriever
george42-ctds Jun 3, 2024
9246ae2
(HP-1459): remove github action for 'image_build_push'
george42-ctds Jun 3, 2024
a8cc7cc
Merge branch 'master' into feat/add-syracuse-qdr-retriever
mfshao Jun 6, 2024
b34a826
(HP-1459): handle None download_url
george42-ctds Jun 13, 2024
b6e9f36
(HP-1459): allow downloading of single files, unzipped
george42-ctds Jun 13, 2024
1968fe6
(HP-1459): get file_ids one at a time
george42-ctds Jun 14, 2024
c43d128
(HP-1459): remove unused variable
george42-ctds Jun 14, 2024
e73e091
(HP-1459): remove debug statements
george42-ctds Jun 14, 2024
2afcd86
(HP-1459): add instructions for pip install from repo
george42-ctds Jun 17, 2024
061f62f
(HP-1459): move 'pip install' section further down in README
george42-ctds Jun 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 5 additions & 33 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
*.DS_Store
*.vscode
.idea
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
Expand Down Expand Up @@ -27,8 +30,6 @@ share/python-wheels/
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

Expand Down Expand Up @@ -83,31 +84,7 @@ profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
.python-version

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
Expand Down Expand Up @@ -152,9 +129,4 @@ dmypy.json
# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# pytest output
41 changes: 40 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,40 @@
# heal-platform-sdk
# heal-platform-sdk

The HEAL Platform Software Development Kit (SDK) for Python provides classes and functions for handling HEAL specific tasks.

### qdr_downloads

The qdr_downloads module include a retriever function for downloading files from Syracuse QDR.

This is intended to be called by gen3sdk [external download functions](https://github.com/uc-cdis/gen3sdk-python/blob/master/gen3/tools/download/external_file_download.py). It is also possible to write a wrapper script for the qdr download functions.

### Run tests

```
poetry run pytest -vv tests
```

### Pip install for using heal-platform-sdk

Until the `heal-platform-sdk` is available on PyPI, any local pip installs should directly
reference the git repo.

As an example, `pip install` can be called from the command line for getting
the master branch of the `heal-platform-sdk`,

```
pip install -e git+ssh://[email protected]/uc-cdis/heal-platform-sdk.git#egg=heal
```

or a particular branch, eg,

```
pip install -e git+ssh://[email protected]/uc-cdis/heal-platform-sdk.git@feat/add-syracuse-qdr-retriever#egg=heal
```

The specification can also be listed in requirements.txt file
(with, say, a tag specification of 0.1.0)

```
-e git+ssh://[email protected]/uc-cdis/[email protected]#egg=heal
```
4 changes: 4 additions & 0 deletions heal/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from cdislogging import get_logger

LOG_FORMAT = "[%(asctime)s][%(levelname)7s] %(message)s"
logging = get_logger("__name__", format=LOG_FORMAT, log_level="info")
Loading
Loading