Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ordering key in GCP PubSub operator PubSubPublishMessageOperator #39940

Closed
2 tasks done
mehdigati opened this issue May 30, 2024 · 1 comment · Fixed by #39955
Closed
2 tasks done

Add ordering key in GCP PubSub operator PubSubPublishMessageOperator #39940

mehdigati opened this issue May 30, 2024 · 1 comment · Fixed by #39955
Assignees
Labels
area:providers good first issue kind:feature Feature Requests provider:google Google (including GCP) related issues

Comments

@mehdigati
Copy link
Contributor

Description

Currently, the PubSubPublishMessageOperator in the Google Cloud Platform (GCP) integration for Apache Airflow does not allow specifying an ordering key for published messages. The ordering key is a feature provided by Cloud Pub/Sub that allows messages to be delivered in the order they were published within a single ordering key.

Use case/motivation

There are several use cases where maintaining the order of published messages is crucial, such as:

  1. Event processing pipelines: When processing a sequence of events, it's essential to maintain the order in which they occurred to ensure proper state management and avoid data corruption.
  2. Distributed transactions: In scenarios involving distributed transactions across multiple services, preserving the order of messages can be critical for maintaining data consistency.
  3. Deduplication: When implementing deduplication mechanisms, the ordering of messages can be a key factor in determining which messages should be considered duplicates.

By adding support for ordering keys in the PubSubPublishMessageOperator, Airflow users would be able to leverage this Cloud Pub/Sub feature and ensure that their published messages are delivered in the desired order, enabling more robust and reliable data processing pipelines.

Related issues

None that I'm aware of.

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@mehdigati mehdigati added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet labels May 30, 2024
Copy link

boring-cyborg bot commented May 30, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers good first issue kind:feature Feature Requests provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants