Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FEAT-#5936: support pandas 2.0.2 #5995

Merged
merged 190 commits into from
Jun 7, 2023
Merged

FEAT-#5936: support pandas 2.0.2 #5995

merged 190 commits into from
Jun 7, 2023

Conversation

anmyachev
Copy link
Collaborator

@anmyachev anmyachev commented Apr 12, 2023 •
edited
Loading

What do these changes do?

  • first commit message and PR title follow format outlined here

    NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

  • passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
  • passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
  • signed commit with git commit -s
  • Resolves Support pandas 2.x #5936
  • tests added and passing
  • module layout described at docs/development/architecture.rst is up-to-date

anmyachev added 30 commits April 12, 2023 21:36
@anmyachev
FEAT-modin-project#5936: support pandas 2.0.0
6887fb0 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
remove Int64Index, UInt64Index, Float64Index
c0c9cf2 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
remove pandas.datetime
93b7f66 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
remove convert_to_index_sliceable
eca0422 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
remove 'ensure_clean_dir'
929edcc 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
Merge branch 'master' of https://github.com/modin-project/modin into …
025d14a 
…pandas-2.0.0
@anmyachev
disable 'xarray' until it adds support for pandas 2.0.0
417bf0c 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
remove 'mad' [2]
b9a83b5 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
add 'dtype_backend' parameter for all functions and remove is_monotonic
af9b32e 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
remove 'tshift' and 'iteritems'
5626de0 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
remove 'slice_shift'
9ee440c 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
remove 'append'
3cb0007 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
add new parameter: 'axis' for 'add_suffix', 'add_prefix'
2b0fbfc 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
use copy=None instead of copy=True
510b907 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
remove deprecated parameters in 'between_time'
3cddf29 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'drop_duplicates'
bc5e97f 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'dropna'
0d5672e 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'factorize'
f586ad1 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'groupby'
be041e8 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'kurt'
bdead97 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'mask'
78217ef 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'mean' and 'max'
f0af110 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'min', 'median'
7daad2d 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'resample', 'rank'
1a37565 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'sem', 'skew'
3b40481 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'var', 'sum', 'std'
79c4b63 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'to_json'
24f6f5c 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'all', 'any'
db67661 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'describe', 'clip', 'count'
886a7cf 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update 'resample', 'expanding'
2cc84a8 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
update comments
276b383 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
Copy link
Collaborator Author

@vnlitvinov ready for review

@vnlitvinov
Copy link
Collaborator

@anmyachev there are a few open questions left from previous review, but most of them are minor.
I think the only major blocker for me is my last comment, about updating QC in-place.

@anmyachev
address review comments
8121867 
Signed-off-by: Anatoly Myachev <[email protected]>
dchigarev
dchigarev previously approved these changes Jun 6, 2023
View reviewed changes
Copy link
Collaborator

@dchigarev dchigarev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good from my side

thank you for this massive work!

anmyachev and others added 2 commits June 6, 2023 16:43
@anmyachev @dchigarev
Update modin/pandas/series.py
7b8d207 
Co-authored-by: Dmitry Chigarev <[email protected]>
@anmyachev
fix
c1e7cb8 
Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev
Copy link
Collaborator Author

@dchigarev ready for review

dchigarev
dchigarev previously approved these changes Jun 6, 2023
View reviewed changes
YarShev
YarShev previously approved these changes Jun 6, 2023
View reviewed changes
@YarShev
Copy link
Collaborator

YarShev commented Jun 6, 2023

@vnlitvinov, any comments?

@anmyachev anmyachev changed the title FEAT-#5936: support pandas 2.0.0 FEAT-#5936: support pandas 2.0.2 Jun 6, 2023
@anmyachev anmyachev dismissed stale reviews from YarShev and dchigarev via 3e7e55a June 7, 2023 11:53
@mvashishtha
Copy link
Collaborator

@vnlitvinov, any comments?

@YarShev @dchigarev I see that @vnlitvinov comment about _update_inplace is still open

https://github.com/modin-project/modin/pull/5995/files/7edd9bdd8bcaa16466f666949d5314d460f610ea..6d7e8d260bf08a7bb304c258e902c237736aa043#r1219040039

@anmyachev
Copy link
Collaborator Author

@vnlitvinov, any comments?

@YarShev @dchigarev I see that @vnlitvinov comment about _update_inplace is still open

https://github.com/modin-project/modin/pull/5995/files/7edd9bdd8bcaa16466f666949d5314d460f610ea..6d7e8d260bf08a7bb304c258e902c237736aa043#r1219040039

@mvashishtha I fixed it but forgot to reply to the comment

@mvashishtha mvashishtha merged commit 1d8a179 into modin-project:master Jun 7, 2023
@anmyachev anmyachev deleted the pandas-2.0.0 branch June 7, 2023 19:23
vnlitvinov
vnlitvinov reviewed Jun 8, 2023
View reviewed changes
modin/pandas/test/test_io.py
ReadSqlEngine.put(read_sql_engine)
if ReadSqlEngine.get() == "Connectorx":
modin_df = pd.read_sql(sql=query, con=conn)
else:
modin_df = pd.read_sql(
sql=query, con=ModinDatabaseConnection("sqlalchemy", conn)
)
ReadSqlEngine.put(old_sql_engine)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be done in try..finally to be absolutely sure we return the value back... not critical in tests I suppose, so mostly a note for the future patterns

modin/pandas/test/test_io.py
Comment on lines +2258 to +2260
def comparator(df1, df2):
df_equals(df1, df2)
df_equals(df1.dtypes, df2.dtypes)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feels like it should be extracted to a common place, this comparator is written in a few places already; could go in some tests refactoring PR I suppose

@vnlitvinov
Copy link
Collaborator

@anmyachev thanks for the tremendous amount of effort into making this PR by the way!

@vnlitvinov vnlitvinov mentioned this pull request Jun 13, 2023
Open
@anmyachev anmyachev mentioned this pull request Jul 2, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vnlitvinov vnlitvinov vnlitvinov left review comments

@YarShev YarShev YarShev approved these changes

@dchigarev dchigarev dchigarev approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support pandas 2.x
6 participants
@anmyachev @AndreyPavlenko @vnlitvinov @YarShev @mvashishtha @dchigarev