Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

openmetadata_managed_apis uses deprecated Airflow class #12190

Closed
arroadie opened this issue Jun 28, 2023 · 2 comments · Fixed by #12398
Closed

openmetadata_managed_apis uses deprecated Airflow class #12190

arroadie opened this issue Jun 28, 2023 · 2 comments · Fixed by #12398
Assignees

Comments

@arroadie
Copy link

Affected module
Ingestion

Describe the bug
WIth the change added here, the SchedulerJob class was renamed to SchedulerJobRunner. That change breaks the interaction between openmetadata and airflow in version 2.6 (haven't pinpointed the exact version)

To Reproduce

Fresh installation of airflow running tag 2.6.2 with the latest plugin version added will result in the following

image

Expected behavior
The plugin should reference the proper class or in it's effect limit the compatibility to the latest version in which the SchedulerJob class is available.

Version:

  • OS: docker on macos (apache/airflow:2.6.2)
  • Python version: Python 3.7.17
  • OpenMetadata version: [e.g. 0.8]
    openmetadata-airflow-managed-apis 0.11.5.3
    openmetadata-managed-apis 1.0.5.0
  • OpenMetadata Ingestion package version: [e.g. openmetadata-ingestion[docker]==XYZ] openmetadata-ingestion 0.11.5.3

Additional context
I'm using my own docker compose setup, so it's using a previously set version of airflow (2.6.2) instead of the baked in setup provided by openmetadata

@pmbrull
Copy link
Collaborator

pmbrull commented Jul 3, 2023

Hi, note that we don't support yet Airflow 2.5 or 2.6. The work to do here is to enable a wider version range.

If you want to use our APIs you should stick to 2.3.3 (the one we use in our image) or 2.4 (I think some community users tested that already)

Thanks

@arroadie
Copy link
Author

arroadie commented Jul 7, 2023

Oh, in this case I believe it would be awesome to add a compatibility matrix somewhere in the documentation since in some places where airflow is mentioned it says it's supported on oficial airflow releases (not necessarily for this library, but for the pipeline connector).
Anyways, awesome work with the project. kudos to the team.
Will try later with a different version since I can't downgrade our current airflow runtime just to test openmetadata and there's no sense in adding a new version just for it as well.
Cheers

@pmbrull pmbrull moved this to 🏗 In progress in Release 1.1.1 Jul 10, 2023
pmbrull added a commit that referenced this issue Jul 13, 2023
… new ingestion base (#12398)

* Bump Airflow version

* Support Airflow 2.6

* Lint

* Bump airflow version

* Fix airflow 2.6 lineage

* Fix airflow 2.6 lineage
@github-project-automation github-project-automation bot moved this from 🏗 In progress to 🔖 Ready in Release 1.1.1 Jul 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Status: 🔖 Ready
2 participants