How to access the context variable in a graph_multi_asset

[pbower]

In Dagster, how does one access the 'Context' variable within a graph_multi_asset?

E.g., I have context running through the underlying ops/ assets.
But, then I need to do this:

" base_path = Path(context.partition_key)"

I can't do it though, because dagster errors with: " dagster._core.errors.DagsterInvalidDefinitionError: "context" is not a valid name in Dagster. It conflicts with a Dagster or python reserved keyword."

So How does one access context metadata (per partition) within the graph_multi_asset?

Thanks heaps!!

[garethbrickman]

In Dagster, the context parameter is used to pass execution-time information to assets and ops. However, when defining a graph_multi_asset, you cannot directly use context as a parameter name because it is a reserved keyword in Dagster. This is why you are encountering the DagsterInvalidDefinitionError.

To access context metadata within a graph_multi_asset, you should use the parameters that are passed to the underlying ops or assets. These parameters can include the partition key or other context-related information that you need.

If you need to access the partition key within a graph_multi_asset, you would typically do so within the individual ops that make up the graph, rather than at the graph level itself. Each op can access the partition key through its context parameter, assuming the op is part of a partitioned asset.

Here's an example of how you might access the partition key within an op that is part of a graph_multi_asset:

from dagster import op, graph

@op
def my_op(context):
    base_path = Path(context.partition_key)
    # ... your logic here ...

@graph
def my_graph():
    my_op()

# Then you would define a `graph_multi_asset` that includes `my_graph`

In this example, my_op is an op that accesses the partition key from its context. When you define a graph_multi_asset, you would include my_graph, which in turn includes my_op.

If you need to pass additional context metadata to the ops within a graph_multi_asset, you can do so by defining resources or configuration that the ops can access through their context.

[pbower]

Ok great, thanks for this. So I am then of the understanding that a Graph_multi_asset doesn't (shouldn't) have additional programming statements within its function, other than the sequencing of the ops / graphs?
In other words, put all that stuff in the ops, and keep that as the container?

Anyway, I think it's answered. Thanks for your help.

[garethbrickman]

Yes, it's inadvisable to put code inside a @job- or @graph- decorated function that does anything other than define dependencies between ops. Doing so can lead to unexpected behavior. More information about this in #12554

[cldixon]

This is very helpful @garethbrickman! As a follow up question, what if we use the context object for logging, and want to log within each op function? Is there a recommended pattern for this? Thank you.

[garethbrickman]

@cldixon You can post a log to the event log from in an op like this:

import dagster as dg

@dg.op
def my_op(context: dg.OpExecutionContext):
    context.log.info("This is an info level log message from my_op.")

And from in an asset like this:

import dagster as dg

@dg.asset
def my_asset(context: dg.AssetExecutionContext):
    context.log.info("This is an info level log message from my_asset.")