Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix GCSToGCSOperator behavior difference for moving single object #40162

Merged
merged 2 commits into from
Jun 21, 2024

Conversation

boraberke
Copy link
Contributor

GCSToGCSOperator did not retain the nested folder structure when moving a single object. This PR merges the logic for single and multiple objects and handles folder structure similar to what gsutil does.
closes: #37576


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Jun 10, 2024
@potiuk
Copy link
Member

potiuk commented Jun 12, 2024

I think this one requires at the very least some more explanation added to https://github.com/apache/airflow/blob/main/airflow/providers/google/CHANGELOG.rst

It's on the border of breaking change, but we can also treat it as a bugfix, but we should have a user-targeted explanation on what they should do if they experience the problem. Could you please add such explanation at teh top of the Changelog?

@boraberke boraberke force-pushed the fix/gcs-to-gcs-operator branch from f0202a8 to c4d92e7 Compare June 21, 2024 08:56
@boraberke
Copy link
Contributor Author

I think this one requires at the very least some more explanation added to https://github.com/apache/airflow/blob/main/airflow/providers/google/CHANGELOG.rst

It's on the border of breaking change, but we can also treat it as a bugfix, but we should have a user-targeted explanation on what they should do if they experience the problem. Could you please add such explanation at teh top of the Changelog?

Hi @potiuk, thanks for the feedback! I added a note about the change. It was a bit unclear without an example as it is a very specific case, so I also added a simple example to make it clear.

LMK if I need to change anything.

@potiuk
Copy link
Member

potiuk commented Jun 21, 2024

nice!

@potiuk potiuk merged commit 2f2796f into apache:main Jun 21, 2024
51 checks passed
MaksYermak added a commit to VladaZakharova/airflow that referenced this pull request Jul 3, 2024
potiuk pushed a commit that referenced this pull request Jul 3, 2024
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
…pache#40162)

* Merge different behavior of `GCSToGCSOperator` for single and multiple objects

* Add behavior change note to changelog
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

moving single object with GCSToGCSOperator differs from gsutil mv command
2 participants