[kedro-datasets ] Add `Polars.CSVDataSet` #95

wmoreiraa · 2023-01-07T16:23:13Z

Signed-off-by: wmoreiraa [email protected]

Description

Introduce python-polars to Kedro Datasets.
https://www.pola.rs/benchmarks.html

Development notes

Add polars.CSVDataSet
Same tests (copy paste with some minor tweaks to pandas testcase)

TODO on this PR:

CSVDataSet

All of those using only the eager I/O.

Checklist

Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the relevant RELEASE.md file
Added tests to cover my changes

wmoreiraa · 2023-01-09T10:45:49Z

I'm figuring it out how to solve the unit test that is failing.

merelcht · 2023-01-16T10:53:19Z

I'm figuring it out how to solve the unit test that is failing.

Right now all unit tests are passing, it's just the DCO check that's failing. You can find instructions here https://github.com/kedro-org/kedro-plugins/pull/95/checks?check_run_id=10658463390 on how to solve it.

wmoreiraa · 2023-01-17T21:39:27Z

Thanks, @merelcht ! My e-mail signoff was wrong, ive commited the fix now. Do you think its valid to finish this PR only with CSVDataSet?

wmoreiraa · 2023-01-18T13:17:14Z

@merelcht , Ive followed the intructions and rewrote my signoff email, but still DCO failing for the same reason. Dont know whats wrong now.

merelcht · 2023-01-18T14:21:35Z

@merelcht , Ive followed the intructions and rewrote my signoff email, but still DCO failing for the same reason. Dont know whats wrong now.

Hmm odd.. sometimes it's very hard to get it working 😅 I can manually approve it when the PR is ready to merge. So don't worry about it.

merelcht · 2023-01-18T14:24:01Z

Do you think its valid to finish this PR only with CSVDataSet?

Yes I think that's fine! Any additional dataset is valuable 🙂 In the future you (or another contributor) can then build more polar datasets.

merelcht

Thanks for the addition @wmoreiraa !

I've added some comments mainly around updating of the docstring. And could you also add this change to the release notes? 🙂

kedro-datasets/kedro_datasets/polars/csv_dataset.py

kedro-datasets/kedro_datasets/polars/ipc_dataset.py

kedro-datasets/kedro_datasets/polars/csv_dataset.py

Signed-off-by: wmoreiraa <[email protected]>

Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: wmoreiraa <[email protected]>

Signed-off-by: wmoreiraa <[email protected]>

* Bump relax pyarrow version to work the same way as Pandas We only use PyArrow for `pandas.ParquetDataSet` as such I suggest we keep our versions pinned to the same range as [Pandas does](https://github.com/pandas-dev/pandas/blob/96fc51f5ec678394373e2c779ccff37ddb966e75/pyproject.toml#L100) for the same reason. As such I suggest we remove the upper bound as we have users requesting later versions in [support channels](https://kedro-org.slack.com/archives/C03RKP2LW64/p1674040509133529) * Updated release notes Signed-off-by: wmoreiraa <[email protected]>

wmoreiraa · 2023-01-23T10:43:07Z

I think everything's fine now? @merelcht
Thx for the comments!

merelcht

Thanks for the updates @wmoreiraa. I left a couple more questions, but most importantly you shouldn't bump the version of kedro-datasets in this PR. We'll do that when we do a release.

kedro-datasets/RELEASE.md

kedro-datasets/kedro_datasets/__init__.py

kedro-datasets/kedro_datasets/polars/csv_dataset.py

Signed-off-by: wmoreiraa <[email protected]>

kedro-datasets/RELEASE.md

merelcht

Thanks for this contribution @wmoreiraa 😄 🎉

Add dataset headers Signed-off by: wmoreiraa <[email protected]> Co-authored-by: Merel Theisen <[email protected]>

wmoreiraa · 2023-01-24T10:59:04Z

Just the first one! I've been impacted by layoffs, and now I'm full time job hunting / available, might as well do some LinkedIn post showcasing some polars + kedro.

merelcht · 2023-01-24T15:36:46Z

Just the first one! I've been impacted by layoffs, and now I'm full time job hunting / available, might as well do some LinkedIn post showcasing some polars + kedro.

I'm so sorry to hear that! Good luck with the job hunting 🍀 Your contributions on Kedro are very much appreciated ❤️

datajoely · 2023-02-02T18:11:11Z

Good luck @wmoreiraa

AhdraMeraliQB

Thanks for the contribution @wmoreiraa! 🌟 Happy to approve with just the minor change noted.

kedro-datasets/kedro_datasets/polars/csv_dataset.py

datajoely · 2023-02-08T13:02:32Z

I love this and want to see it in Kedro as soon as possible.

My only question is if we shouldn't approach this on a file type by file type basis and should in fact approach this the same way we do spark.SparkDataSet and pandas.GenericDataSet?

At the time of writing the followingread_* methods can be imported from polars.io:

read_avro
read_csv
read_csv_batched
read_delta
read_excel
read_ipc
read_json
read_ndjson
read_parquet
read_sql

In terms of write targets the DataFrame class only supports a subset of these:

DataFrame.write_ipc
DataFrame.write_parquet
DataFrame.write_json
DataFrame.write_ndjson
DataFrame.write_avro

Is there any merit in abstracting this into a generic approach?

Changing equals to frame_equal Co-authored-by: Ahdra Merali <[email protected]>

wmoreiraa · 2023-02-08T19:48:32Z

Thank you for the fix! @AhdraMeraliQB .
@datajoely I'm going to be honest with you that I didn't know about the existence of these datasets until now, I'll give it a look at the code, but the direction seems abour right.

merelcht · 2023-02-09T10:00:34Z

I'd suggest getting this merged in first since it's ready now and then look at a generic approach later.

wmoreiraa · 2023-02-09T12:39:24Z

@merelcht @datajoely, I've finished the test cases on the Generic Approach. Should I open a second PR then?

datajoely · 2023-02-09T12:40:13Z

As merel said let's get this one in and then do a follow-up?

wmoreiraa · 2023-02-09T12:45:51Z

Fine for me, ima just wait and then fork again.

merelcht · 2023-02-09T13:15:38Z

Fine for me, ima just wait and then fork again.

Thank you! I've just resolved some minor merge conflicts and will merge this PR as soon as the builds finish successfully 🙂

Signed-off-by: Merel Theisen <[email protected]>

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Yassine Alouini <[email protected]>

* [kedro-docker] Layers size optimization (#92) * [kedro-docker] Layers size optimization Signed-off-by: Mariusz Strzelecki <[email protected]> * Adjust test requirements Signed-off-by: Mariusz Strzelecki <[email protected]> * Skip coverage check on tests dir (some do not execute on Windows) Signed-off-by: Mariusz Strzelecki <[email protected]> * Update .coveragerc with the setup Signed-off-by: Mariusz Strzelecki <[email protected]> * Fix bandit so it does not scan kedro-datasets Signed-off-by: Mariusz Strzelecki <[email protected]> * Fixed existence test Signed-off-by: Mariusz Strzelecki <[email protected]> * Check why dir is not created Signed-off-by: Mariusz Strzelecki <[email protected]> * Kedro starters are fixed now Signed-off-by: Mariusz Strzelecki <[email protected]> * Increased no-output-timeout for long spark image build Signed-off-by: Mariusz Strzelecki <[email protected]> * Spark image optimized Signed-off-by: Mariusz Strzelecki <[email protected]> * Linting Signed-off-by: Mariusz Strzelecki <[email protected]> * Switch to slim image always Signed-off-by: Mariusz Strzelecki <[email protected]> * Trigger build Signed-off-by: Mariusz Strzelecki <[email protected]> * Use textwrap.dedent for nicer indentation Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Use textwrap.dedent for nicer indentation" This reverts commit 3a1e3f8. Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Revert "Use textwrap.dedent for nicer indentation"" This reverts commit d322d35. Signed-off-by: Mariusz Strzelecki <[email protected]> * Make tests read more lines (to skip all deprecation warnings) Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release Kedro-Docker 0.3.1 (#94) * Add release notes for kedro-docker 0.3.1 Signed-off-by: Jannic Holzer <[email protected]> * Update version in kedro_docker module Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Bump version and update release notes (#96) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Make the SQLQueryDataSet compatible with mssql. Signed-off-by: Yassine Alouini <[email protected]> * Add one test + update RELEASE.md. Signed-off-by: Yassine Alouini <[email protected]> * Add missing pyodbc for tests. Signed-off-by: Yassine Alouini <[email protected]> * Mock connection as well. Signed-off-by: Yassine Alouini <[email protected]> * Add more dates parsing for mssql backend (thanks to [email protected]) Signed-off-by: Yassine Alouini <[email protected]> * Fix an error in docstring of MetricsDataSet (#98) Signed-off-by: Yassine Alouini <[email protected]> * Bump relax pyarrow version to work the same way as Pandas (#100) * Bump relax pyarrow version to work the same way as Pandas We only use PyArrow for `pandas.ParquetDataSet` as such I suggest we keep our versions pinned to the same range as [Pandas does](https://github.com/pandas-dev/pandas/blob/96fc51f5ec678394373e2c779ccff37ddb966e75/pyproject.toml#L100) for the same reason. As such I suggest we remove the upper bound as we have users requesting later versions in [support channels](https://kedro-org.slack.com/archives/C03RKP2LW64/p1674040509133529) * Updated release notes Signed-off-by: Yassine Alouini <[email protected]> * Add missing type in catalog example. Signed-off-by: Yassine Alouini <[email protected]> * Add one more unit tests for adapt_mssql. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Add missing mocker from date test. Signed-off-by: Yassine Alouini <[email protected]> * [TEST] Add a wrong input test. Signed-off-by: Yassine Alouini <[email protected]> * Add pyodbc dependency. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Remove dict() in tests. Signed-off-by: Yassine Alouini <[email protected]> * Change check to check on plugin name (#103) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Set coverage in pyproject.toml (#105) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Move coverage settings to pyproject.toml (#106) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Replace kedro.pipeline with modular_pipeline.pipeline factory (#99) * Add non-spark related test changes Replace kedro.pipeline.Pipeline with kedro.pipeline.modular_pipeline.pipeline factory. This is for symmetry with changes made to the main kedro library. Signed-off-by: Adam Farley <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Fix outdated links in Kedro Datasets (#111) * fix links * fix dill links Signed-off-by: Yassine Alouini <[email protected]> * Fix docs formatting and phrasing for some datasets (#107) * Fix docs formatting and phrasing for some datasets Signed-off-by: Deepyaman Datta <[email protected]> * Manually fix files not resolved with patch command Signed-off-by: Deepyaman Datta <[email protected]> * Apply fix from #98 Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release `kedro-datasets` `version 1.0.2` (#112) * bump version and update release notes * fix pylint errors Signed-off-by: Yassine Alouini <[email protected]> * Bump pytest to 7.2 (#113) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Prefix Docker plugin name with "Kedro-" in usage message (#57) * Prefix Docker plugin name with "Kedro-" in usage message Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` (#56) * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [kedro-datasets ] Add `Polars.CSVDataSet` (#95) Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Remove deprecated `test_requires` from `setup.py` in Kedro-Docker (#54) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Fix ds to data_set. Signed-off-by: Yassine Alouini <[email protected]> --------- Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Co-authored-by: Mariusz Strzelecki <[email protected]> Co-authored-by: Jannic <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: OKA Naoya <[email protected]> Co-authored-by: Joel <[email protected]> Co-authored-by: adamfrly <[email protected]> Co-authored-by: Sajid Alam <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Co-authored-by: Walber Moreira <[email protected]>

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Danny Farah <[email protected]>

* [kedro-docker] Layers size optimization (kedro-org#92) * [kedro-docker] Layers size optimization Signed-off-by: Mariusz Strzelecki <[email protected]> * Adjust test requirements Signed-off-by: Mariusz Strzelecki <[email protected]> * Skip coverage check on tests dir (some do not execute on Windows) Signed-off-by: Mariusz Strzelecki <[email protected]> * Update .coveragerc with the setup Signed-off-by: Mariusz Strzelecki <[email protected]> * Fix bandit so it does not scan kedro-datasets Signed-off-by: Mariusz Strzelecki <[email protected]> * Fixed existence test Signed-off-by: Mariusz Strzelecki <[email protected]> * Check why dir is not created Signed-off-by: Mariusz Strzelecki <[email protected]> * Kedro starters are fixed now Signed-off-by: Mariusz Strzelecki <[email protected]> * Increased no-output-timeout for long spark image build Signed-off-by: Mariusz Strzelecki <[email protected]> * Spark image optimized Signed-off-by: Mariusz Strzelecki <[email protected]> * Linting Signed-off-by: Mariusz Strzelecki <[email protected]> * Switch to slim image always Signed-off-by: Mariusz Strzelecki <[email protected]> * Trigger build Signed-off-by: Mariusz Strzelecki <[email protected]> * Use textwrap.dedent for nicer indentation Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Use textwrap.dedent for nicer indentation" This reverts commit 3a1e3f8. Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Revert "Use textwrap.dedent for nicer indentation"" This reverts commit d322d35. Signed-off-by: Mariusz Strzelecki <[email protected]> * Make tests read more lines (to skip all deprecation warnings) Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release Kedro-Docker 0.3.1 (kedro-org#94) * Add release notes for kedro-docker 0.3.1 Signed-off-by: Jannic Holzer <[email protected]> * Update version in kedro_docker module Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Bump version and update release notes (kedro-org#96) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Make the SQLQueryDataSet compatible with mssql. Signed-off-by: Yassine Alouini <[email protected]> * Add one test + update RELEASE.md. Signed-off-by: Yassine Alouini <[email protected]> * Add missing pyodbc for tests. Signed-off-by: Yassine Alouini <[email protected]> * Mock connection as well. Signed-off-by: Yassine Alouini <[email protected]> * Add more dates parsing for mssql backend (thanks to [email protected]) Signed-off-by: Yassine Alouini <[email protected]> * Fix an error in docstring of MetricsDataSet (kedro-org#98) Signed-off-by: Yassine Alouini <[email protected]> * Bump relax pyarrow version to work the same way as Pandas (kedro-org#100) * Bump relax pyarrow version to work the same way as Pandas We only use PyArrow for `pandas.ParquetDataSet` as such I suggest we keep our versions pinned to the same range as [Pandas does](https://github.com/pandas-dev/pandas/blob/96fc51f5ec678394373e2c779ccff37ddb966e75/pyproject.toml#L100) for the same reason. As such I suggest we remove the upper bound as we have users requesting later versions in [support channels](https://kedro-org.slack.com/archives/C03RKP2LW64/p1674040509133529) * Updated release notes Signed-off-by: Yassine Alouini <[email protected]> * Add missing type in catalog example. Signed-off-by: Yassine Alouini <[email protected]> * Add one more unit tests for adapt_mssql. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Add missing mocker from date test. Signed-off-by: Yassine Alouini <[email protected]> * [TEST] Add a wrong input test. Signed-off-by: Yassine Alouini <[email protected]> * Add pyodbc dependency. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Remove dict() in tests. Signed-off-by: Yassine Alouini <[email protected]> * Change check to check on plugin name (kedro-org#103) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Set coverage in pyproject.toml (kedro-org#105) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Move coverage settings to pyproject.toml (kedro-org#106) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Replace kedro.pipeline with modular_pipeline.pipeline factory (kedro-org#99) * Add non-spark related test changes Replace kedro.pipeline.Pipeline with kedro.pipeline.modular_pipeline.pipeline factory. This is for symmetry with changes made to the main kedro library. Signed-off-by: Adam Farley <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Fix outdated links in Kedro Datasets (kedro-org#111) * fix links * fix dill links Signed-off-by: Yassine Alouini <[email protected]> * Fix docs formatting and phrasing for some datasets (kedro-org#107) * Fix docs formatting and phrasing for some datasets Signed-off-by: Deepyaman Datta <[email protected]> * Manually fix files not resolved with patch command Signed-off-by: Deepyaman Datta <[email protected]> * Apply fix from kedro-org#98 Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release `kedro-datasets` `version 1.0.2` (kedro-org#112) * bump version and update release notes * fix pylint errors Signed-off-by: Yassine Alouini <[email protected]> * Bump pytest to 7.2 (kedro-org#113) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Prefix Docker plugin name with "Kedro-" in usage message (kedro-org#57) * Prefix Docker plugin name with "Kedro-" in usage message Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` (kedro-org#56) * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [kedro-datasets ] Add `Polars.CSVDataSet` (kedro-org#95) Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Remove deprecated `test_requires` from `setup.py` in Kedro-Docker (kedro-org#54) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Fix ds to data_set. Signed-off-by: Yassine Alouini <[email protected]> --------- Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Co-authored-by: Mariusz Strzelecki <[email protected]> Co-authored-by: Jannic <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: OKA Naoya <[email protected]> Co-authored-by: Joel <[email protected]> Co-authored-by: adamfrly <[email protected]> Co-authored-by: Sajid Alam <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Co-authored-by: Walber Moreira <[email protected]> Signed-off-by: Danny Farah <[email protected]>

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Danny Farah <[email protected]>

* [kedro-docker] Layers size optimization (kedro-org#92) * [kedro-docker] Layers size optimization Signed-off-by: Mariusz Strzelecki <[email protected]> * Adjust test requirements Signed-off-by: Mariusz Strzelecki <[email protected]> * Skip coverage check on tests dir (some do not execute on Windows) Signed-off-by: Mariusz Strzelecki <[email protected]> * Update .coveragerc with the setup Signed-off-by: Mariusz Strzelecki <[email protected]> * Fix bandit so it does not scan kedro-datasets Signed-off-by: Mariusz Strzelecki <[email protected]> * Fixed existence test Signed-off-by: Mariusz Strzelecki <[email protected]> * Check why dir is not created Signed-off-by: Mariusz Strzelecki <[email protected]> * Kedro starters are fixed now Signed-off-by: Mariusz Strzelecki <[email protected]> * Increased no-output-timeout for long spark image build Signed-off-by: Mariusz Strzelecki <[email protected]> * Spark image optimized Signed-off-by: Mariusz Strzelecki <[email protected]> * Linting Signed-off-by: Mariusz Strzelecki <[email protected]> * Switch to slim image always Signed-off-by: Mariusz Strzelecki <[email protected]> * Trigger build Signed-off-by: Mariusz Strzelecki <[email protected]> * Use textwrap.dedent for nicer indentation Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Use textwrap.dedent for nicer indentation" This reverts commit 3a1e3f8. Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Revert "Use textwrap.dedent for nicer indentation"" This reverts commit d322d35. Signed-off-by: Mariusz Strzelecki <[email protected]> * Make tests read more lines (to skip all deprecation warnings) Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release Kedro-Docker 0.3.1 (kedro-org#94) * Add release notes for kedro-docker 0.3.1 Signed-off-by: Jannic Holzer <[email protected]> * Update version in kedro_docker module Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Bump version and update release notes (kedro-org#96) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Make the SQLQueryDataSet compatible with mssql. Signed-off-by: Yassine Alouini <[email protected]> * Add one test + update RELEASE.md. Signed-off-by: Yassine Alouini <[email protected]> * Add missing pyodbc for tests. Signed-off-by: Yassine Alouini <[email protected]> * Mock connection as well. Signed-off-by: Yassine Alouini <[email protected]> * Add more dates parsing for mssql backend (thanks to [email protected]) Signed-off-by: Yassine Alouini <[email protected]> * Fix an error in docstring of MetricsDataSet (kedro-org#98) Signed-off-by: Yassine Alouini <[email protected]> * Bump relax pyarrow version to work the same way as Pandas (kedro-org#100) * Bump relax pyarrow version to work the same way as Pandas We only use PyArrow for `pandas.ParquetDataSet` as such I suggest we keep our versions pinned to the same range as [Pandas does](https://github.com/pandas-dev/pandas/blob/96fc51f5ec678394373e2c779ccff37ddb966e75/pyproject.toml#L100) for the same reason. As such I suggest we remove the upper bound as we have users requesting later versions in [support channels](https://kedro-org.slack.com/archives/C03RKP2LW64/p1674040509133529) * Updated release notes Signed-off-by: Yassine Alouini <[email protected]> * Add missing type in catalog example. Signed-off-by: Yassine Alouini <[email protected]> * Add one more unit tests for adapt_mssql. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Add missing mocker from date test. Signed-off-by: Yassine Alouini <[email protected]> * [TEST] Add a wrong input test. Signed-off-by: Yassine Alouini <[email protected]> * Add pyodbc dependency. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Remove dict() in tests. Signed-off-by: Yassine Alouini <[email protected]> * Change check to check on plugin name (kedro-org#103) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Set coverage in pyproject.toml (kedro-org#105) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Move coverage settings to pyproject.toml (kedro-org#106) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Replace kedro.pipeline with modular_pipeline.pipeline factory (kedro-org#99) * Add non-spark related test changes Replace kedro.pipeline.Pipeline with kedro.pipeline.modular_pipeline.pipeline factory. This is for symmetry with changes made to the main kedro library. Signed-off-by: Adam Farley <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Fix outdated links in Kedro Datasets (kedro-org#111) * fix links * fix dill links Signed-off-by: Yassine Alouini <[email protected]> * Fix docs formatting and phrasing for some datasets (kedro-org#107) * Fix docs formatting and phrasing for some datasets Signed-off-by: Deepyaman Datta <[email protected]> * Manually fix files not resolved with patch command Signed-off-by: Deepyaman Datta <[email protected]> * Apply fix from kedro-org#98 Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release `kedro-datasets` `version 1.0.2` (kedro-org#112) * bump version and update release notes * fix pylint errors Signed-off-by: Yassine Alouini <[email protected]> * Bump pytest to 7.2 (kedro-org#113) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Prefix Docker plugin name with "Kedro-" in usage message (kedro-org#57) * Prefix Docker plugin name with "Kedro-" in usage message Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` (kedro-org#56) * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [kedro-datasets ] Add `Polars.CSVDataSet` (kedro-org#95) Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Remove deprecated `test_requires` from `setup.py` in Kedro-Docker (kedro-org#54) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Fix ds to data_set. Signed-off-by: Yassine Alouini <[email protected]> --------- Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Co-authored-by: Mariusz Strzelecki <[email protected]> Co-authored-by: Jannic <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: OKA Naoya <[email protected]> Co-authored-by: Joel <[email protected]> Co-authored-by: adamfrly <[email protected]> Co-authored-by: Sajid Alam <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Co-authored-by: Walber Moreira <[email protected]> Signed-off-by: Danny Farah <[email protected]>

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Danny Farah <[email protected]>

* [kedro-docker] Layers size optimization (kedro-org#92) * [kedro-docker] Layers size optimization Signed-off-by: Mariusz Strzelecki <[email protected]> * Adjust test requirements Signed-off-by: Mariusz Strzelecki <[email protected]> * Skip coverage check on tests dir (some do not execute on Windows) Signed-off-by: Mariusz Strzelecki <[email protected]> * Update .coveragerc with the setup Signed-off-by: Mariusz Strzelecki <[email protected]> * Fix bandit so it does not scan kedro-datasets Signed-off-by: Mariusz Strzelecki <[email protected]> * Fixed existence test Signed-off-by: Mariusz Strzelecki <[email protected]> * Check why dir is not created Signed-off-by: Mariusz Strzelecki <[email protected]> * Kedro starters are fixed now Signed-off-by: Mariusz Strzelecki <[email protected]> * Increased no-output-timeout for long spark image build Signed-off-by: Mariusz Strzelecki <[email protected]> * Spark image optimized Signed-off-by: Mariusz Strzelecki <[email protected]> * Linting Signed-off-by: Mariusz Strzelecki <[email protected]> * Switch to slim image always Signed-off-by: Mariusz Strzelecki <[email protected]> * Trigger build Signed-off-by: Mariusz Strzelecki <[email protected]> * Use textwrap.dedent for nicer indentation Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Use textwrap.dedent for nicer indentation" This reverts commit 3a1e3f8. Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Revert "Use textwrap.dedent for nicer indentation"" This reverts commit d322d35. Signed-off-by: Mariusz Strzelecki <[email protected]> * Make tests read more lines (to skip all deprecation warnings) Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release Kedro-Docker 0.3.1 (kedro-org#94) * Add release notes for kedro-docker 0.3.1 Signed-off-by: Jannic Holzer <[email protected]> * Update version in kedro_docker module Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Bump version and update release notes (kedro-org#96) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Make the SQLQueryDataSet compatible with mssql. Signed-off-by: Yassine Alouini <[email protected]> * Add one test + update RELEASE.md. Signed-off-by: Yassine Alouini <[email protected]> * Add missing pyodbc for tests. Signed-off-by: Yassine Alouini <[email protected]> * Mock connection as well. Signed-off-by: Yassine Alouini <[email protected]> * Add more dates parsing for mssql backend (thanks to [email protected]) Signed-off-by: Yassine Alouini <[email protected]> * Fix an error in docstring of MetricsDataSet (kedro-org#98) Signed-off-by: Yassine Alouini <[email protected]> * Bump relax pyarrow version to work the same way as Pandas (kedro-org#100) * Bump relax pyarrow version to work the same way as Pandas We only use PyArrow for `pandas.ParquetDataSet` as such I suggest we keep our versions pinned to the same range as [Pandas does](https://github.com/pandas-dev/pandas/blob/96fc51f5ec678394373e2c779ccff37ddb966e75/pyproject.toml#L100) for the same reason. As such I suggest we remove the upper bound as we have users requesting later versions in [support channels](https://kedro-org.slack.com/archives/C03RKP2LW64/p1674040509133529) * Updated release notes Signed-off-by: Yassine Alouini <[email protected]> * Add missing type in catalog example. Signed-off-by: Yassine Alouini <[email protected]> * Add one more unit tests for adapt_mssql. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Add missing mocker from date test. Signed-off-by: Yassine Alouini <[email protected]> * [TEST] Add a wrong input test. Signed-off-by: Yassine Alouini <[email protected]> * Add pyodbc dependency. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Remove dict() in tests. Signed-off-by: Yassine Alouini <[email protected]> * Change check to check on plugin name (kedro-org#103) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Set coverage in pyproject.toml (kedro-org#105) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Move coverage settings to pyproject.toml (kedro-org#106) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Replace kedro.pipeline with modular_pipeline.pipeline factory (kedro-org#99) * Add non-spark related test changes Replace kedro.pipeline.Pipeline with kedro.pipeline.modular_pipeline.pipeline factory. This is for symmetry with changes made to the main kedro library. Signed-off-by: Adam Farley <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Fix outdated links in Kedro Datasets (kedro-org#111) * fix links * fix dill links Signed-off-by: Yassine Alouini <[email protected]> * Fix docs formatting and phrasing for some datasets (kedro-org#107) * Fix docs formatting and phrasing for some datasets Signed-off-by: Deepyaman Datta <[email protected]> * Manually fix files not resolved with patch command Signed-off-by: Deepyaman Datta <[email protected]> * Apply fix from kedro-org#98 Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release `kedro-datasets` `version 1.0.2` (kedro-org#112) * bump version and update release notes * fix pylint errors Signed-off-by: Yassine Alouini <[email protected]> * Bump pytest to 7.2 (kedro-org#113) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Prefix Docker plugin name with "Kedro-" in usage message (kedro-org#57) * Prefix Docker plugin name with "Kedro-" in usage message Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` (kedro-org#56) * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [kedro-datasets ] Add `Polars.CSVDataSet` (kedro-org#95) Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Remove deprecated `test_requires` from `setup.py` in Kedro-Docker (kedro-org#54) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Fix ds to data_set. Signed-off-by: Yassine Alouini <[email protected]> --------- Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Co-authored-by: Mariusz Strzelecki <[email protected]> Co-authored-by: Jannic <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: OKA Naoya <[email protected]> Co-authored-by: Joel <[email protected]> Co-authored-by: adamfrly <[email protected]> Co-authored-by: Sajid Alam <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Co-authored-by: Walber Moreira <[email protected]> Signed-off-by: Danny Farah <[email protected]>

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Danny Farah <[email protected]>

* [kedro-docker] Layers size optimization (kedro-org#92) * [kedro-docker] Layers size optimization Signed-off-by: Mariusz Strzelecki <[email protected]> * Adjust test requirements Signed-off-by: Mariusz Strzelecki <[email protected]> * Skip coverage check on tests dir (some do not execute on Windows) Signed-off-by: Mariusz Strzelecki <[email protected]> * Update .coveragerc with the setup Signed-off-by: Mariusz Strzelecki <[email protected]> * Fix bandit so it does not scan kedro-datasets Signed-off-by: Mariusz Strzelecki <[email protected]> * Fixed existence test Signed-off-by: Mariusz Strzelecki <[email protected]> * Check why dir is not created Signed-off-by: Mariusz Strzelecki <[email protected]> * Kedro starters are fixed now Signed-off-by: Mariusz Strzelecki <[email protected]> * Increased no-output-timeout for long spark image build Signed-off-by: Mariusz Strzelecki <[email protected]> * Spark image optimized Signed-off-by: Mariusz Strzelecki <[email protected]> * Linting Signed-off-by: Mariusz Strzelecki <[email protected]> * Switch to slim image always Signed-off-by: Mariusz Strzelecki <[email protected]> * Trigger build Signed-off-by: Mariusz Strzelecki <[email protected]> * Use textwrap.dedent for nicer indentation Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Use textwrap.dedent for nicer indentation" This reverts commit 3a1e3f8. Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Revert "Use textwrap.dedent for nicer indentation"" This reverts commit d322d35. Signed-off-by: Mariusz Strzelecki <[email protected]> * Make tests read more lines (to skip all deprecation warnings) Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release Kedro-Docker 0.3.1 (kedro-org#94) * Add release notes for kedro-docker 0.3.1 Signed-off-by: Jannic Holzer <[email protected]> * Update version in kedro_docker module Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Bump version and update release notes (kedro-org#96) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Make the SQLQueryDataSet compatible with mssql. Signed-off-by: Yassine Alouini <[email protected]> * Add one test + update RELEASE.md. Signed-off-by: Yassine Alouini <[email protected]> * Add missing pyodbc for tests. Signed-off-by: Yassine Alouini <[email protected]> * Mock connection as well. Signed-off-by: Yassine Alouini <[email protected]> * Add more dates parsing for mssql backend (thanks to [email protected]) Signed-off-by: Yassine Alouini <[email protected]> * Fix an error in docstring of MetricsDataSet (kedro-org#98) Signed-off-by: Yassine Alouini <[email protected]> * Bump relax pyarrow version to work the same way as Pandas (kedro-org#100) * Bump relax pyarrow version to work the same way as Pandas We only use PyArrow for `pandas.ParquetDataSet` as such I suggest we keep our versions pinned to the same range as [Pandas does](https://github.com/pandas-dev/pandas/blob/96fc51f5ec678394373e2c779ccff37ddb966e75/pyproject.toml#L100) for the same reason. As such I suggest we remove the upper bound as we have users requesting later versions in [support channels](https://kedro-org.slack.com/archives/C03RKP2LW64/p1674040509133529) * Updated release notes Signed-off-by: Yassine Alouini <[email protected]> * Add missing type in catalog example. Signed-off-by: Yassine Alouini <[email protected]> * Add one more unit tests for adapt_mssql. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Add missing mocker from date test. Signed-off-by: Yassine Alouini <[email protected]> * [TEST] Add a wrong input test. Signed-off-by: Yassine Alouini <[email protected]> * Add pyodbc dependency. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Remove dict() in tests. Signed-off-by: Yassine Alouini <[email protected]> * Change check to check on plugin name (kedro-org#103) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Set coverage in pyproject.toml (kedro-org#105) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Move coverage settings to pyproject.toml (kedro-org#106) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Replace kedro.pipeline with modular_pipeline.pipeline factory (kedro-org#99) * Add non-spark related test changes Replace kedro.pipeline.Pipeline with kedro.pipeline.modular_pipeline.pipeline factory. This is for symmetry with changes made to the main kedro library. Signed-off-by: Adam Farley <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Fix outdated links in Kedro Datasets (kedro-org#111) * fix links * fix dill links Signed-off-by: Yassine Alouini <[email protected]> * Fix docs formatting and phrasing for some datasets (kedro-org#107) * Fix docs formatting and phrasing for some datasets Signed-off-by: Deepyaman Datta <[email protected]> * Manually fix files not resolved with patch command Signed-off-by: Deepyaman Datta <[email protected]> * Apply fix from kedro-org#98 Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release `kedro-datasets` `version 1.0.2` (kedro-org#112) * bump version and update release notes * fix pylint errors Signed-off-by: Yassine Alouini <[email protected]> * Bump pytest to 7.2 (kedro-org#113) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Prefix Docker plugin name with "Kedro-" in usage message (kedro-org#57) * Prefix Docker plugin name with "Kedro-" in usage message Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` (kedro-org#56) * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [kedro-datasets ] Add `Polars.CSVDataSet` (kedro-org#95) Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Remove deprecated `test_requires` from `setup.py` in Kedro-Docker (kedro-org#54) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Fix ds to data_set. Signed-off-by: Yassine Alouini <[email protected]> --------- Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Co-authored-by: Mariusz Strzelecki <[email protected]> Co-authored-by: Jannic <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: OKA Naoya <[email protected]> Co-authored-by: Joel <[email protected]> Co-authored-by: adamfrly <[email protected]> Co-authored-by: Sajid Alam <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Co-authored-by: Walber Moreira <[email protected]> Signed-off-by: Danny Farah <[email protected]>

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Danny Farah <[email protected]>

* [kedro-docker] Layers size optimization (kedro-org#92) * [kedro-docker] Layers size optimization Signed-off-by: Mariusz Strzelecki <[email protected]> * Adjust test requirements Signed-off-by: Mariusz Strzelecki <[email protected]> * Skip coverage check on tests dir (some do not execute on Windows) Signed-off-by: Mariusz Strzelecki <[email protected]> * Update .coveragerc with the setup Signed-off-by: Mariusz Strzelecki <[email protected]> * Fix bandit so it does not scan kedro-datasets Signed-off-by: Mariusz Strzelecki <[email protected]> * Fixed existence test Signed-off-by: Mariusz Strzelecki <[email protected]> * Check why dir is not created Signed-off-by: Mariusz Strzelecki <[email protected]> * Kedro starters are fixed now Signed-off-by: Mariusz Strzelecki <[email protected]> * Increased no-output-timeout for long spark image build Signed-off-by: Mariusz Strzelecki <[email protected]> * Spark image optimized Signed-off-by: Mariusz Strzelecki <[email protected]> * Linting Signed-off-by: Mariusz Strzelecki <[email protected]> * Switch to slim image always Signed-off-by: Mariusz Strzelecki <[email protected]> * Trigger build Signed-off-by: Mariusz Strzelecki <[email protected]> * Use textwrap.dedent for nicer indentation Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Use textwrap.dedent for nicer indentation" This reverts commit 3a1e3f8. Signed-off-by: Mariusz Strzelecki <[email protected]> * Revert "Revert "Use textwrap.dedent for nicer indentation"" This reverts commit d322d35. Signed-off-by: Mariusz Strzelecki <[email protected]> * Make tests read more lines (to skip all deprecation warnings) Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release Kedro-Docker 0.3.1 (kedro-org#94) * Add release notes for kedro-docker 0.3.1 Signed-off-by: Jannic Holzer <[email protected]> * Update version in kedro_docker module Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Bump version and update release notes (kedro-org#96) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Make the SQLQueryDataSet compatible with mssql. Signed-off-by: Yassine Alouini <[email protected]> * Add one test + update RELEASE.md. Signed-off-by: Yassine Alouini <[email protected]> * Add missing pyodbc for tests. Signed-off-by: Yassine Alouini <[email protected]> * Mock connection as well. Signed-off-by: Yassine Alouini <[email protected]> * Add more dates parsing for mssql backend (thanks to [email protected]) Signed-off-by: Yassine Alouini <[email protected]> * Fix an error in docstring of MetricsDataSet (kedro-org#98) Signed-off-by: Yassine Alouini <[email protected]> * Bump relax pyarrow version to work the same way as Pandas (kedro-org#100) * Bump relax pyarrow version to work the same way as Pandas We only use PyArrow for `pandas.ParquetDataSet` as such I suggest we keep our versions pinned to the same range as [Pandas does](https://github.com/pandas-dev/pandas/blob/96fc51f5ec678394373e2c779ccff37ddb966e75/pyproject.toml#L100) for the same reason. As such I suggest we remove the upper bound as we have users requesting later versions in [support channels](https://kedro-org.slack.com/archives/C03RKP2LW64/p1674040509133529) * Updated release notes Signed-off-by: Yassine Alouini <[email protected]> * Add missing type in catalog example. Signed-off-by: Yassine Alouini <[email protected]> * Add one more unit tests for adapt_mssql. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Add missing mocker from date test. Signed-off-by: Yassine Alouini <[email protected]> * [TEST] Add a wrong input test. Signed-off-by: Yassine Alouini <[email protected]> * Add pyodbc dependency. Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Remove dict() in tests. Signed-off-by: Yassine Alouini <[email protected]> * Change check to check on plugin name (kedro-org#103) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Set coverage in pyproject.toml (kedro-org#105) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Move coverage settings to pyproject.toml (kedro-org#106) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Replace kedro.pipeline with modular_pipeline.pipeline factory (kedro-org#99) * Add non-spark related test changes Replace kedro.pipeline.Pipeline with kedro.pipeline.modular_pipeline.pipeline factory. This is for symmetry with changes made to the main kedro library. Signed-off-by: Adam Farley <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Fix outdated links in Kedro Datasets (kedro-org#111) * fix links * fix dill links Signed-off-by: Yassine Alouini <[email protected]> * Fix docs formatting and phrasing for some datasets (kedro-org#107) * Fix docs formatting and phrasing for some datasets Signed-off-by: Deepyaman Datta <[email protected]> * Manually fix files not resolved with patch command Signed-off-by: Deepyaman Datta <[email protected]> * Apply fix from kedro-org#98 Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Release `kedro-datasets` `version 1.0.2` (kedro-org#112) * bump version and update release notes * fix pylint errors Signed-off-by: Yassine Alouini <[email protected]> * Bump pytest to 7.2 (kedro-org#113) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Prefix Docker plugin name with "Kedro-" in usage message (kedro-org#57) * Prefix Docker plugin name with "Kedro-" in usage message Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` (kedro-org#56) * Keep Kedro-Docker plugin docstring from appearing in `kedro -h` Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [kedro-datasets ] Add `Polars.CSVDataSet` (kedro-org#95) Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * Remove deprecated `test_requires` from `setup.py` in Kedro-Docker (kedro-org#54) Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> * [FIX] Fix ds to data_set. Signed-off-by: Yassine Alouini <[email protected]> --------- Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Mariusz Strzelecki <[email protected]> Signed-off-by: Yassine Alouini <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Co-authored-by: Mariusz Strzelecki <[email protected]> Co-authored-by: Jannic <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: OKA Naoya <[email protected]> Co-authored-by: Joel <[email protected]> Co-authored-by: adamfrly <[email protected]> Co-authored-by: Sajid Alam <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Co-authored-by: Walber Moreira <[email protected]> Signed-off-by: Danny Farah <[email protected]>

astrojuanlu · 2023-04-16T16:42:57Z

Most up to date effort: gh-170.

wmoreiraa changed the title ~~polars csvdataset~~ [kedro-datasets ] Polars csvdataset Jan 7, 2023

wmoreiraa marked this pull request as ready for review January 18, 2023 19:17

merelcht requested a review from AhdraMeraliQB January 20, 2023 11:15

merelcht changed the title ~~[kedro-datasets ] Polars csvdataset~~ [kedro-datasets ] Add Polars.CSVDataSet Jan 20, 2023

merelcht reviewed Jan 20, 2023

View reviewed changes

wmoreiraa and others added 8 commits January 23, 2023 07:36

polars csvdataset

37d81eb

Signed-off-by: wmoreiraa <[email protected]>

Polars setup.py

d7ddf5d

Signed-off-by: wmoreiraa <[email protected]>

workaoround since boto3 cant store encoded bytesio or .encode Body

56fa6f5

Signed-off-by: wmoreiraa <[email protected]>

updated docstrings and polars version

c603f22

Signed-off-by: wmoreiraa <[email protected]>

bump package version and release

235122a

Signed-off-by: wmoreiraa <[email protected]>

Bump version and update release notes (#96)

e941b85

Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: wmoreiraa <[email protected]>

Fix an error in docstring of MetricsDataSet (#98)

863c456

Signed-off-by: wmoreiraa <[email protected]>

merelcht reviewed Jan 23, 2023

View reviewed changes

Fixing release notes, package version and docstring

264189d

Signed-off-by: wmoreiraa <[email protected]>

merelcht reviewed Jan 24, 2023

View reviewed changes

kedro-datasets/RELEASE.md Show resolved Hide resolved

merelcht approved these changes Jan 24, 2023

View reviewed changes

Update kedro-datasets/RELEASE.md

6feec66

Add dataset headers Signed-off by: wmoreiraa <[email protected]> Co-authored-by: Merel Theisen <[email protected]>

merelcht requested a review from SajidAlamQB January 30, 2023 12:39

astrojuanlu mentioned this pull request Feb 2, 2023

Adding Polars' dataframe as a dataset #110

Closed

merelcht removed the request for review from AhdraMeraliQB February 6, 2023 15:18

AhdraMeraliQB approved these changes Feb 8, 2023

View reviewed changes

kedro-datasets/kedro_datasets/polars/csv_dataset.py Outdated Show resolved Hide resolved

Fix docs

a501082

Changing equals to frame_equal Co-authored-by: Ahdra Merali <[email protected]>

Merge branch 'main' into main

2698418

Fix linting

cb023e0

Signed-off-by: Merel Theisen <[email protected]>

merelcht merged commit 61e0f08 into kedro-org:main Feb 9, 2023

wmoreiraa mentioned this pull request Feb 13, 2023

polars generic dataset #116

Closed

3 tasks

yassineAlouini pushed a commit to yassineAlouini/kedro-plugins that referenced this pull request Feb 24, 2023

[kedro-datasets ] Add Polars.CSVDataSet (kedro-org#95)

7861bad

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Yassine Alouini <[email protected]>

yassineAlouini pushed a commit to yassineAlouini/kedro-plugins that referenced this pull request Feb 24, 2023

[kedro-datasets ] Add Polars.CSVDataSet (kedro-org#95)

758a82a

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Yassine Alouini <[email protected]>

dannyrfar pushed a commit to dannyrfar/kedro-plugins that referenced this pull request Mar 13, 2023

[kedro-datasets ] Add Polars.CSVDataSet (kedro-org#95)

564f8e5

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Danny Farah <[email protected]>

dannyrfar pushed a commit to dannyrfar/kedro-plugins that referenced this pull request Mar 21, 2023

[kedro-datasets ] Add Polars.CSVDataSet (kedro-org#95)

a3bb3f9

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Danny Farah <[email protected]>

dannyrfar pushed a commit to dannyrfar/kedro-plugins that referenced this pull request Mar 21, 2023

[kedro-datasets ] Add Polars.CSVDataSet (kedro-org#95)

72670d9

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Danny Farah <[email protected]>

dannyrfar pushed a commit to dannyrfar/kedro-plugins that referenced this pull request Mar 21, 2023

[kedro-datasets ] Add Polars.CSVDataSet (kedro-org#95)

4b5da98

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Danny Farah <[email protected]>

dannyrfar pushed a commit to dannyrfar/kedro-plugins that referenced this pull request Mar 21, 2023

[kedro-datasets ] Add Polars.CSVDataSet (kedro-org#95)

f516cc2

Signed-off-by: wmoreiraa <[email protected]> Signed-off-by: Danny Farah <[email protected]>

wmoreiraa mentioned this pull request Apr 1, 2023

Polars Generic Dataset #153

Closed

3 tasks

astrojuanlu mentioned this pull request Apr 16, 2023

feat: Polars generic dataset (take 3) #170

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[kedro-datasets ] Add `Polars.CSVDataSet` #95

[kedro-datasets ] Add `Polars.CSVDataSet` #95

wmoreiraa commented Jan 7, 2023 •

edited

Loading

wmoreiraa commented Jan 9, 2023

merelcht commented Jan 16, 2023

wmoreiraa commented Jan 17, 2023

wmoreiraa commented Jan 18, 2023

merelcht commented Jan 18, 2023

merelcht commented Jan 18, 2023

merelcht left a comment

wmoreiraa commented Jan 23, 2023

merelcht left a comment

merelcht left a comment

wmoreiraa commented Jan 24, 2023

merelcht commented Jan 24, 2023

datajoely commented Feb 2, 2023

AhdraMeraliQB left a comment

datajoely commented Feb 8, 2023

wmoreiraa commented Feb 8, 2023

merelcht commented Feb 9, 2023

wmoreiraa commented Feb 9, 2023

datajoely commented Feb 9, 2023

wmoreiraa commented Feb 9, 2023

merelcht commented Feb 9, 2023

astrojuanlu commented Apr 16, 2023

[kedro-datasets ] Add Polars.CSVDataSet #95

[kedro-datasets ] Add Polars.CSVDataSet #95

Conversation

wmoreiraa commented Jan 7, 2023 • edited Loading

Description

Development notes

TODO on this PR:

Checklist

wmoreiraa commented Jan 9, 2023

merelcht commented Jan 16, 2023

wmoreiraa commented Jan 17, 2023

wmoreiraa commented Jan 18, 2023

merelcht commented Jan 18, 2023

merelcht commented Jan 18, 2023

merelcht left a comment

Choose a reason for hiding this comment

wmoreiraa commented Jan 23, 2023

merelcht left a comment

Choose a reason for hiding this comment

merelcht left a comment

Choose a reason for hiding this comment

wmoreiraa commented Jan 24, 2023

merelcht commented Jan 24, 2023

datajoely commented Feb 2, 2023

AhdraMeraliQB left a comment

Choose a reason for hiding this comment

datajoely commented Feb 8, 2023

wmoreiraa commented Feb 8, 2023

merelcht commented Feb 9, 2023

wmoreiraa commented Feb 9, 2023

datajoely commented Feb 9, 2023

wmoreiraa commented Feb 9, 2023

merelcht commented Feb 9, 2023

astrojuanlu commented Apr 16, 2023

[kedro-datasets ] Add `Polars.CSVDataSet` #95

[kedro-datasets ] Add `Polars.CSVDataSet` #95

wmoreiraa commented Jan 7, 2023 •

edited

Loading