Skip currently refreshing assets with Auto-Materialization #19214
-
Hey, we want to use auto-materialization to refresh dbt root assets (most upstream assets) with a cron schedule, but struggle with parallel runs of the same asset (eg. two separate runs triggered by fresh upstream data). Is there any skip rule we can add to prevent launching new runs for an asset if it is still being refreshed? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments
-
In Dagster This will skip sending a run from AMP if there is already a run in progress for the particular asset. |
Beta Was this translation helpful? Give feedback.
-
This is now the out-of-the-box behavior of the new If you are building a custom Automation Condition, then you can always add |
Beta Was this translation helpful? Give feedback.
In Dagster
1.6.12
, we shipped aAutoMaterializeRule.skip_on_run_in_progress()
: #20498.This will skip sending a run from AMP if there is already a run in progress for the particular asset.