Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update branch #3

Merged
merged 9 commits into from
Mar 8, 2023
Merged

Update branch #3

merged 9 commits into from
Mar 8, 2023

Conversation

heber-urdaneta
Copy link
Collaborator

Description

Development notes

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes

wmoreiraa and others added 9 commits February 9, 2023 13:50
* [kedro-docker] Layers size optimization (kedro-org#92)

* [kedro-docker] Layers size optimization

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Adjust test requirements

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Skip coverage check on tests dir (some do not execute on Windows)

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Update .coveragerc with the setup

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Fix bandit so it does not scan kedro-datasets

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Fixed existence test

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Check why dir is not created

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Kedro starters are fixed now

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Increased no-output-timeout for long spark image build

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Spark image optimized

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Linting

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Switch to slim image always

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Trigger build

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Use textwrap.dedent for nicer indentation

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Revert "Use textwrap.dedent for nicer indentation"

This reverts commit 3a1e3f8.

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Revert "Revert "Use textwrap.dedent for nicer indentation""

This reverts commit d322d35.

Signed-off-by: Mariusz Strzelecki <[email protected]>

* Make tests read more lines (to skip all deprecation warnings)

Signed-off-by: Mariusz Strzelecki <[email protected]>

Signed-off-by: Mariusz Strzelecki <[email protected]>
Signed-off-by: Mariusz Strzelecki <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* Release Kedro-Docker 0.3.1 (kedro-org#94)

* Add release notes for kedro-docker 0.3.1

Signed-off-by: Jannic Holzer <[email protected]>

* Update version in kedro_docker module

Signed-off-by: Jannic Holzer <[email protected]>

Signed-off-by: Jannic Holzer <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* Bump version and update release notes (kedro-org#96)

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* Make the SQLQueryDataSet compatible with mssql.

Signed-off-by: Yassine Alouini <[email protected]>

* Add one test + update RELEASE.md.

Signed-off-by: Yassine Alouini <[email protected]>

* Add missing pyodbc for tests.

Signed-off-by: Yassine Alouini <[email protected]>

* Mock connection as well.

Signed-off-by: Yassine Alouini <[email protected]>

* Add more dates parsing for mssql backend (thanks to [email protected])

Signed-off-by: Yassine Alouini <[email protected]>

* Fix an error in docstring of MetricsDataSet (kedro-org#98)

Signed-off-by: Yassine Alouini <[email protected]>

* Bump relax pyarrow version to work the same way as Pandas (kedro-org#100)

* Bump relax pyarrow version to work the same way as Pandas

We only use PyArrow for `pandas.ParquetDataSet` as such I suggest we keep our versions pinned to the same range as [Pandas does](https://github.com/pandas-dev/pandas/blob/96fc51f5ec678394373e2c779ccff37ddb966e75/pyproject.toml#L100) for the same reason.

As such I suggest we remove the upper bound as we have users requesting later versions in [support channels](https://kedro-org.slack.com/archives/C03RKP2LW64/p1674040509133529)

* Updated release notes

Signed-off-by: Yassine Alouini <[email protected]>

* Add missing type in catalog example.

Signed-off-by: Yassine Alouini <[email protected]>

* Add one more unit tests for adapt_mssql.

Signed-off-by: Yassine Alouini <[email protected]>

* [FIX] Add missing mocker from date test.

Signed-off-by: Yassine Alouini <[email protected]>

* [TEST] Add a wrong input test.

Signed-off-by: Yassine Alouini <[email protected]>

* Add pyodbc dependency.

Signed-off-by: Yassine Alouini <[email protected]>

* [FIX] Remove dict() in tests.

Signed-off-by: Yassine Alouini <[email protected]>

* Change check to check on plugin name (kedro-org#103)

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* Set coverage in pyproject.toml (kedro-org#105)

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* Move coverage settings to pyproject.toml (kedro-org#106)

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* Replace kedro.pipeline with modular_pipeline.pipeline factory (kedro-org#99)

* Add non-spark related test changes
Replace kedro.pipeline.Pipeline with
kedro.pipeline.modular_pipeline.pipeline factory.
This is for symmetry with changes made to the main kedro library.

Signed-off-by: Adam Farley <[email protected]>

Signed-off-by: Yassine Alouini <[email protected]>

* Fix outdated links in Kedro Datasets (kedro-org#111)

* fix links

* fix dill links

Signed-off-by: Yassine Alouini <[email protected]>

* Fix docs formatting and phrasing for some datasets (kedro-org#107)

* Fix docs formatting and phrasing for some datasets

Signed-off-by: Deepyaman Datta <[email protected]>

* Manually fix files not resolved with patch command

Signed-off-by: Deepyaman Datta <[email protected]>

* Apply fix from kedro-org#98

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* Release `kedro-datasets` `version 1.0.2` (kedro-org#112)

* bump version and update release notes

* fix pylint errors

Signed-off-by: Yassine Alouini <[email protected]>

* Bump pytest to 7.2 (kedro-org#113)

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* Prefix Docker plugin name with "Kedro-" in usage message (kedro-org#57)

* Prefix Docker plugin name with "Kedro-" in usage message

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* Keep Kedro-Docker plugin docstring from appearing in `kedro -h` (kedro-org#56)

* Keep Kedro-Docker plugin docstring from appearing in `kedro -h`

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* [kedro-datasets ] Add `Polars.CSVDataSet` (kedro-org#95)

Signed-off-by: wmoreiraa <[email protected]>

Signed-off-by: Yassine Alouini <[email protected]>

* Remove deprecated `test_requires` from `setup.py` in Kedro-Docker (kedro-org#54)

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>

* [FIX] Fix ds to data_set.

Signed-off-by: Yassine Alouini <[email protected]>

---------

Signed-off-by: Mariusz Strzelecki <[email protected]>
Signed-off-by: Mariusz Strzelecki <[email protected]>
Signed-off-by: Yassine Alouini <[email protected]>
Signed-off-by: Jannic Holzer <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Co-authored-by: Mariusz Strzelecki <[email protected]>
Co-authored-by: Jannic <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Co-authored-by: OKA Naoya <[email protected]>
Co-authored-by: Joel <[email protected]>
Co-authored-by: adamfrly <[email protected]>
Co-authored-by: Sajid Alam <[email protected]>
Co-authored-by: Deepyaman Datta <[email protected]>
Co-authored-by: Walber Moreira <[email protected]>
… file path (kedro-org#114)

* Add databricks deployment check and automatic DBFS path addition

Signed-off-by: Jannic Holzer <[email protected]>

* Add newline at end of file

Signed-off-by: Jannic Holzer <[email protected]>

* Remove spurious 'not'

Signed-off-by: Jannic Holzer <[email protected]>

* Move dbfs utility functions from SparkDataSet

Signed-off-by: Jannic Holzer <[email protected]>

* Add edge case logic to _build_dbfs_path

Signed-off-by: Jannic Holzer <[email protected]>

* Add test for dbfs path construction

Signed-off-by: Jannic Holzer <[email protected]>

* Linting

Signed-off-by: Jannic Holzer <[email protected]>

* Remove spurious print statement :)

Signed-off-by: Jannic Holzer <[email protected]>

* Add pylint disable too-many-public-methods

Signed-off-by: Jannic Holzer <[email protected]>

* Move tests into single method to appease linter

Signed-off-by: Jannic Holzer <[email protected]>

* Modify prefix check to /dbfs/

Signed-off-by: Jannic Holzer <[email protected]>

* Modify prefix check to /dbfs/

Signed-off-by: Jannic Holzer <[email protected]>

* Make warning message clearer

Signed-off-by: Jannic Holzer <[email protected]>

* Add release note

Signed-off-by: Jannic Holzer <[email protected]>

* Fix linting

Signed-off-by: Jannic Holzer <[email protected]>

* Update warning message

Signed-off-by: Jannic Holzer <[email protected]>

* Modify log warning level to error

Signed-off-by: Jannic Holzer <[email protected]>

* Modify message back to warning, refer to undefined behaviour

Signed-off-by: Jannic Holzer <[email protected]>

* Modify required prefix to /dbfs/

Signed-off-by: Jannic Holzer <[email protected]>

* Modify doc string

Signed-off-by: Jannic Holzer <[email protected]>

* Modify warning message

Signed-off-by: Jannic Holzer <[email protected]>

* Split tests and add filepath to warning

Signed-off-by: Jannic Holzer <[email protected]>

* Modify f string in logging call

Signed-off-by: Jannic Holzer <[email protected]>

* Fix tests

Signed-off-by: Jannic Holzer <[email protected]>

* Lint

Signed-off-by: Jannic Holzer <[email protected]>

---------

Signed-off-by: Jannic Holzer <[email protected]>
Signed-off-by: heber-urdaneta <[email protected]>
@heber-urdaneta heber-urdaneta merged commit 2716be6 into main Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants