Can I programmatically set up concurrency limits for assets?

[dagsir[bot]]

Hello everyone,
I'm currently utilizing a Docker Compose setup with the latest version of Dagster and have a couple of queries I hope you could assist me with:

1. I'm looking to enable auto-materialization by default. Could anyone guide me on the best approach to achieve this within my current setup?
2. Additionally, I'm interested in implementing a concurrency limit for assets. Would someone be able to offer advice on how to set this up programmatically rather than through the UI?
   Thank you for the support, and I must say, Dagster is proving to be an exceptional tool!

---

The question was originally asked in Dagster Slack.

[alangenfeld]

I'm looking to enable auto-materialization by default. Could anyone guide me on the best approach to achieve this within my current setup?

I don't quite follow, are you not using a database backed instance to persist information over time?

Here are the private methods that get invoked to "unpause" the auto materialization daemon https://github.com/dagster-io/dagster/blob/1.5.9/python_modules/dagster/dagster/_daemon/asset_daemon.py#L73

Additionally, I'm interested in implementing a concurrency limit for assets. Would someone be able to offer advice on how to set this up programmatically rather than through the UI?

Theres the cli

$ dagster instance concurrency --help
Usage: dagster instance concurrency [OPTIONS] COMMAND [ARGS]...

  Commands for working with the instance-wide op concurrency (Experimental).

Options:
  -h, --help  Show this message and exit.

Commands:
  get  Get op concurrency limits
  set  Set op concurrency limits

and the private APIs that it calls https://github.com/dagster-io/dagster/blob/1.5.9/python_modules/dagster/dagster/_cli/instance.py#L139 . CLI should be stable, private APIs may change.

[0xAndrii]

I'm looking to enable auto-materialization by default. Could anyone guide me on the best approach to achieve this within my current setup?

I don't quite follow, are you not using a database backed instance to persist information over time?

Here are the private methods that get invoked to "unpause" the auto materialization daemon https://github.com/dagster-io/dagster/blob/1.5.9/python_modules/dagster/dagster/_daemon/asset_daemon.py#L73

Additionally, I'm interested in implementing a concurrency limit for assets. Would someone be able to offer advice on how to set this up programmatically rather than through the UI?

Theres the cli

$ dagster instance concurrency --help
Usage: dagster instance concurrency [OPTIONS] COMMAND [ARGS]...

  Commands for working with the instance-wide op concurrency (Experimental).

Options:
  -h, --help  Show this message and exit.

Commands:
  get  Get op concurrency limits
  set  Set op concurrency limits

and the private APIs that it calls https://github.com/dagster-io/dagster/blob/1.5.9/python_modules/dagster/dagster/_cli/instance.py#L139 . CLI should be stable, private APIs may change.

The link to the private methods for "unpausing" the auto materialization daemon is exactly what I was looking for, especially the ASSET_DAEMON_PAUSED_KEY.

Regarding the concurrency limits, I initially thought I could set it through dagster.yaml, but now I see the CLI option dagster instance concurrency is the way to go. Will do it through cmd docker command in order to automate.

I appreciate your help in navigating these features!