v1.1.0 Release #760
Merged
v1.1.0 Release #760
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ead, only pass relevant options required for image ingestion
… writing parquet files outside of the function
…g file that is a subset of the full config options for a pipeline run
[skip ci]
Fixes #541 [skip ci]
Fixes #542 [skip ci]
[skip ci]
[skip ci]
Fix README link and documentation 404 assets [skip ci]
# Conflicts: # vast_pipeline/pipeline/main.py
…so added both to CICD
Unused imports #545
* only read .env if required vars are undefined Fixes #547 [skip ci] * updated CHANGELOG * fixed unused defaults
Missing changelog entry
* Update docker compose syntax * Updated changelog
* Switch virtualenvs-create: false * Force numpy=1.18.1 * Undo numpy version adjustment * Temporarily remove python 3.8 testing * temporarily force numba install * hail mary - let's see if python 3.11 works * Add 3.11 support to pyproject.toml * Undo * Updated test-suite actions * Updated dependencies * Updated lock? * Require datashader>0.14 * weird datashader versioning? * One last attempt at datashader versioning * Updated lock * More fiddling * More dependency fiddling * test-suite fix? * Fix docker compose github action syntax * Limit dask<2022.4.2 * revert to updated test-suite settings * Try testing python 3.10 * Revert experiments * More version fixes * Updated changelog * Update datashader dependency
* Update github actions * Bump to python 3.9 * Updated dependencies * Updated changelog
* Naive attempt * Revert to python <3.11 * Updated changelog
* Add get_memory_usage function * Rewrite memory usage functions * Added memory usage logging to finalise.py * Added logging to forced_extraction.py * Added debug logging to loading.py * Added debug logging to model_generator.py * Added debug logging to main.py * Added debug logging to new_sources.py * Import psutil * Fix truncated variable name * Fixed logging in calculate_n_partitions * Add some logging to partially address #718 * ? * PEP8 * PEP8 * More PEP8 * PEP8 hell * Updated changelog
Co-authored-by: Dougal Dobie <[email protected]>
Update URLs in docs
NOTE: I'm not using f-strings for log messages because this is considered bad practice in logging. f-strings are greedy and will be computed even if the log isn't printed to the screen. While this isn't an issue here since the log message is quick to render, it's good practice to always print log messages in this way in case the message requires an expensive calculation. See: https://bugs.python.org/issue46200 for some discussion of this.
…dictionary to YAML This greatly speeds up the initial configuration step since the expansion of glob expressions for the inputs into thousands of files creates very large YAML objects that strictyaml is slow to validate. The changes here ensure the validation step is only done once when writing the updated (expanded glob) file lists back to _yaml["inputs"].
Each creation of the 'inputs' dictionary in a PipeLine config object can take a very long time since safeyaml is extremely slow to validate long file lists when converting to a python list. The PipelineConfig.validation method recreates the ['inputs'] dictionary many times. Just do it once at the start then use that dictionary throughout.
Just because I was changing things in this method anyway.
Optimise YAML config parsing
Co-authored-by: Dougal Dobie <[email protected]>
Co-authored-by: Dougal Dobie <[email protected]>
Fix noise image statistics
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.