Improve Dagster integration. Support SDA

[geoHeil]

https://openmetadata.slack.com/archives/C02B6955S4S/p1670503431014219

The dagster integration is only using op and not SDA lineage.

Therefore, DBT and ingestion tools like Airbyte are not represented correctly.

Please can you ensure that proper SDA (software-defined assets) support is present in OM to ingest the lineage in the same way as dagster has:

• for assets inside OM
• for external additional assets i.e. things outside the database but present in. the orchestrator
• reconcile/unify the E2E asset lineage to show the end-to-end flow of data in OM.

[geoHeil]

similar is: #8495 however, dagster does not seem to be implemented by that change

[geoHeil]

The current integration has some problems:

1. The lineage view is too small
2. The assets are not integrated with OM
3. The assets of dagster (outside of i.e. DBT) but also their lineage from dagster is not mapped to the one in OM
4. There is a catch_all __ASSET_JOB_0 pipeline which lumps together (including assets) from several (default UI triggered) runs in dagster

[PascalStehling]

Hi,

are there any Plans to include dagster support for higher Versions, as the currently supported Version is Version 1.0.13, which was released in October 2022 and since then there were alot of changes and improvements.

[geoHeil]

I think it is not breaking at least until 1.4

however the support is still limited i.e. the dagster lineage external to DBT / the database is not yet merged with what is retrieved by other ingestors