Duplicate Outputs[Feature Request]

[mheydasch]

Is your feature request related to a problem? Please describe.
Will we ever get the option to reuse Outputs?
I am getting extremely frustrated with having to work around the issue, that every output can only be used once.
Several examples:
I have a filter for data that I want to plot.
I have a text field to add comments to specific plotted tracks.
These comments can be used for further filtering (i.e. don't show tracks with comment 'dead')
I store the adjusted data in a hidden div and retrieve it for my graphs.
Because of the duplicate outputs issue I have to do both filters in a single callback.
Problem:

1. Requires quite some conditioning.
2. I cannot revert the previous filter as this will overwrite all the comments.

Second example:
I want to display images based on hoverdata.
I upload a dictionary with keys to filepaths and upload and display them based on hoverdata.
This of course is not very fast, so I would like to each time an image is uploaded update the dictionary with key to uploaded image instead.
Of course this also does not work for the same reason.

Third example:
I would like to be able to change the brightness/contrast of the displayed images.
So i would have a graph displaying the image, and a slider that retrieves the image and selects the max displayed brightness values, that then sends the altered image back to the graph displaying it.
Also can't do this because of the duplicate Outputs issue.

Yes I can think about work arounds for all of these, but that would mean, that each callback will have to be the possibility to do several very different things, getting multiple inputs from completely different objects etc. which just does not make any sense to me.
Allowing for multiple outputs would make life a lot easier.

[neilpanchal]

I think this is a major pain point of Dash. As the number of components (and their callbacks) get large, it becomes unmanageable. We can use the work around of a single callback and using switch statements but the complexity rises rapidly with n^2 where n is the number of inputs.

Having separate independent callbacks that can update the same data (duplicate outputs) is absolutely fundamental to writing moderately complex apps.

[luiztauffer]

From what I understand, the problem would be something like this (also stated here and here):
callback1(A, B) → C
callback2(A, D) → C

where one has the same input A modifying the output C in two potentially different ways.
Now, could this be solved with something such as a priority index for callbacks? Or this wouldn’t work or create other types of problems?

There is pattern matching, which indeed helps in many cases now, for example when you go creating new components by user interaction, you just need to set specific patterns for the id keys.

[Clonkk]

This feature is a must have for large application. There are several reason for having a single piece of data have different point of origin. Right now the options are :

• Have on big callback with imbricated & complex if / else on the context (which is hard to maintain).
• Duplicate data with the risk of losing synchronisation between data supposed to be identical.

[tr8dr]

@plotly this is a major issue for anything but trivial applications. Is there a plan to address this?

I spent some years using R-shiny and moved to dash. This is not at all an issue in Shiny, so one would hope that dash would meet parity with Shiny on this front (or do better as is often the case in Dash).

[tr8dr]

Instead of dealing with the complexities of:

callback1(A, B) → C
callback2(A, D) → C

how about just solving the following for now (this would solve a lot of problems I encounter):

callback(A, S1, S2, S3) → C
callback(B, S1, S2, S3) → C

In the above case there is no overlap of event triggers (where A & B are inputs, and S1, S2, S3 is state information), just the output. This should be an easier case to solve.

[nicolaskruchten]

Here is a gist https://gist.github.com/nicolaskruchten/8aa6ae9df62d2c45ef87ad28efd06a31 that implements a simple "router" pattern for both cases listed above using dash.callback_context:

callback1(In_A, State) → Out_1
callback2(In_B, State) → Out_1

and

callback3(In_A, In_C, State) -> Out_2
callback4(In_B, In_C, State) -> Out_2

The way I've structured the code, the callbacks that folks here would like to have as separate functions are separate functions (e.g. output_given_a), and I've included as a comment above them what the corresponding desired @app.callback(...) call would be if multiple callbacks could target the same output. The router-callbacks immediately below the pairs of functions actually implements the logic. The bottom router implements the priority of input C.

One downside of this pattern is that the functions don't have their associated callback information immediately above them in the source file, but the corresponding upside is that all the information one needs to understand what functions could be involved in updating a given input is centralized within the actual router callbacks.

[tr8dr]

@nicolaskruchten Thanks for that. Indeed I have had to do something like that, but the code starts becoming unreadable and unwieldy; as the # of inputs increases, trivial routing becomes complex combinatorial dispatching. Worse, some combination of inputs may want to write to some combination of outputs, again creating combinatorial complexity.

Two callback related issues that should be addressed are:

1. this problem
   • perhaps the simpler of the problems (where there are no overlapping input triggers) can be solved without as much effort?
2. allow "null" output designation (for example clicking a button that sends something to a backend, with no effect on UI)
   • I have to create empty divs in the DOM in order to fake this

Is there an issue on the react / client side which would make #1 difficult. or is this a python side issue? If you were to point me in the right direction, would be happy to create a fix in a fork.

[alexcjohnson]

@tr8dr no-output callbacks are a great idea 🏆 . That's worth a separate issue to discuss and track, but in principle I see no problem with it. Care to open that issue??

Re: the cases you've described above, there are still challenges even with the simplest cases A->C and B->C:

• The initial call (on page load or layout-chunk load) - if all the callbacks have prevent_initial_call=False this is no issue, and in principle we could allow exactly one of them to have prevent_initial_call=True but then it would need logic to see if it had a trigger or not (the initial call has no trigger) and go down the initial content path. So there's still some branching to do.
• If both A and B are themselves callback outputs, with a common ancestor or even disjoint ancestors subject to initial calls. Perhaps not very common but it would be extremely complicated for us to discover these cases, and easy for users to start out not in this situation and then get to this point as their app grows. At that point you'd encounter an error that would force you to refactor all of these callbacks back into one big one with an internal router pattern. Same goes for initial calls.

My feeling is we'd be better off thinking carefully about how to take a pattern like in @nicolaskruchten's gist and build it into Dash via helpful utilities, so that we can maintain the unambiguous callback graph and build this logic explicitly into the callbacks themselves, just in a more concise and maintainable way.

This is not at all an issue in Shiny

Can you elaborate, or point us to an example? I had the impression that Shiny was in fact more or less the opposite, ie all the logic that Dash would call callbacks is collected into a single function that it's the developer's job to dispatch from. But I've never made a Shiny app myself, only looked at docs and examples.

[tr8dr]

@tr8dr no-output callbacks are a great idea 🏆 . That's worth a separate issue to discuss and track, but in principle I see no problem with it. Care to open that issue??

Re: the cases you've described above, there are still challenges even with the simplest cases A->C and B->C:

• The initial call (on page load or layout-chunk load) - if all the callbacks have prevent_initial_call=False this is no issue, and in principle we could allow exactly one of them to have prevent_initial_call=True but then it would need logic to see if it had a trigger or not (the initial call has no trigger) and go down the initial content path. So there's still some branching to do.
• If both A and B are themselves callback outputs, with a common ancestor or even disjoint ancestors subject to initial calls. Perhaps not very common but it would be extremely complicated for us to discover these cases, and easy for users to start out not in this situation and then get to this point as their app grows. At that point you'd encounter an error that would force you to refactor all of these callbacks back into one big one with an internal router pattern. Same goes for initial calls.

Is the concern regarding event cycles (leading to infinite looping)? discovering of the reactive DAG? other? I don't know enough about your event mechanism to comment. Is the goal to keep users from shooting themselves in the foot or should power users be allowed some additional flexibilities at their own risk.

My feeling is we'd be better off thinking carefully about how to take a pattern like in @nicolaskruchten's gist and build it into Dash via helpful utilities, so that we can maintain the unambiguous callback graph and build this logic explicitly into the callbacks themselves, just in a more concise and maintainable way.

How about generating the DAG or dispatch code. Perhaps have a different decorator for the more complex behavior. Could have something like:

@app.callback(
    SharedOutput('C', 'children'),
    [Input('A', 'value'), Input('D','value')])
def update_C1(a,d):
    pass

@app.callback(
    SharedOutput('C', 'children'),
    [Input('E, 'value'), Input('F','value')])
def update_C2(e,f):
    pass

Code or a DAG would be generated (equivalent to the proposed set of dispatch functions for the example), except the user (me) does not see the underlying complexities and is only exposed to the simpler application logic functions. This would require a scan of all callbacks and determine which outputs are shared, creating a dispatch and input collection function under the covers.

There generated code would:

1. scan for all @app.callback's
2. find all callbacks / functions with shared targets
3. group by shared target
4. determine the superset of all inputs needed for shared target
5. determine the superset of all states needed for shared target
6. generate logic to dispatch to 1 of k functions sharing a given target

To simplify, perhaps would not allow multiple output targets for a function in the shared output scenario. Allowing multiple would present a problem as function foo() might output to shared(B) and C, but function bar() only output to shared(B). If the dispatch need to output to B and C under the covers, output C would be undefined if the 2nd function is dispatched.

This is not at all an issue in Shiny

Can you elaborate, or point us to an example? I had the impression that Shiny was in fact more or less the opposite, ie all the logic that Dash would call callbacks is collected into a single function that it's the developer's job to dispatch from. But I've never made a Shiny app myself, only looked at docs and examples.

In the simplest case, one does define a "reactive" body of code for a particular output. However it is possible to create alternative reactive functions that output to the same target, however this is not a widely used feature (more for "power" users).

As for callbacks, shiny makes most things more implicit (and natural) with respect to reactive events. Instead of explicitly listing inputs, one interacts with reactive variables in a closure. Rather than requiring a function to be parameterized with all possible inputs towards the output calculation, the code-path and use of particular variables triggers or does not trigger the output. A dynamic DAG is determined at runtime, allowing this behavior.

All that said, I prefer python, and like the ecosystem dash presents. I think callbacks / the reactive side of things is more primitive (or perhaps better stated, much more low level) than that presented in Shiny.

[nicolaskruchten]

Thanks for your comments @tr8dr :)

Is the goal to keep users from shooting themselves in the foot or should power users be allowed some additional flexibilities at their own risk.

I actually think that the current state of affairs does in fact allow power users a lot of flexibility, by allowing them to build the kinds of routers that I showed and you already use. That power does unfortunately come at the expense of verbosity and lack of core-library support, but it certainly is there today, and I think that this maybe isn't as clearly called out in the docs as it should be, hence my gist :)

[MartinPyka]

Hi everybody,

just FYI, a work around for the multiple output limitation is provided by dash-extensions (https://github.com/thedirtyfew/dash-extensions/), which, I guess, is a nice wrapper that should already make the code much more readable. But I still think, it would be nice, to have it supported by Dash in the first place to avoid any work arounds. Therefore, I support this feature request!

Best,
Martin

[alexcjohnson]

Thanks @tr8dr - very helpful.

The concern here is not about cycles but ambiguity. We want it to be clear, both to the dash renderer and more importantly to the app developer, which code is going to set the value of a given output at any given time. In both of the cases I describe above a single event could fire both flavors of the callback, so how do we choose which response to take (or, pushing the problem up a level, which flavor callback to fire)?

We could defer to the order callbacks were registered with the app, firing the first (or perhaps the last? I can imagine arguments either way) callback with a triggering input. Fairly simple, but we don't have anything else today that depends on the code order, and I can see this itself becoming hard to manage as the app grows, especially if you're importing callbacks from multiple files.

@nicolaskruchten had an idea to extend your SharedOutput proposal with an explicit priority field. That way we could dictate the order: the smallest priority value wins, independent of code order (then I guess we fall back on code order if priorities match?).

This would be relatively straightforward to add into the Python back end. If we want it to work with clientside callbacks we'd need to implement it separately there, but it could still be a layer on top of the existing callback dispatching logic. Or if we want it to support serverside and clientside both outputting to the same prop, we would need to build it directly into the callback dispatch logic. But that would have the advantage of not needing to send all the inputs, just the inputs for the specific callback we're going to fire.

One more question occurs to me: what if the callback returns no_update when there's another we could call as well. Do we then fire the other one?

[tr8dr]

@alexcjohnson With regard to the proposed SharedOutput(), It would not be a bother if required a priority field as you suggest. One could potential default the priorities based on lexical order in the code (my initial thinking on how this would work). However a combination of lexical and specific priority could easily collide, so perhaps best to go with one or the other.

Re: adding on the python side; though I don't know your codebase all that well, generating code on the fly in python is straightforward given python's meta programming capabilities. Sounds like a great idea. I would hazard a guess that doing the fully optimal client and server side implementation would be quite a bit more complex.

[emilhe]

I have just updated dash-extensions (version 0.0.47) to include a new MultiplexerTransform. It makes it possible to target an output by multiple callbacks with nearly zero code changes,

import dash_html_components as html
from dash_extensions.enrich import Output, DashProxy, Input, MultiplexerTransform

app = DashProxy(prevent_initial_callbacks=True, transforms=[MultiplexerTransform()])
app.layout = html.Div([html.Button("left", id="left"), html.Button("right", id="right"), html.Div(id="log")])


@app.callback(Output("log", "children"), Input("left", "n_clicks"))
def left(_):
    return "left"


@app.callback(Output("log", "children"), Input("right", "n_clicks"))
def right(_):
    return "right"


if __name__ == '__main__':
    app.run_server()

The logic behind it is in fact really simple. When n > 1 callbacks target the same element as output, n Store elements are created, and the callbacks are redirect to target these intermediate outputs. Finally, a callback is added with the intermediate outputs as inputs and the original output as output. Maybe the multiplexer approach implemented directly in Dash?

[nicolaskruchten]

Some more discussion here in the forum: https://community.plotly.com/t/duplicate-callback-outputs-solution-api-discussion/55909

[emilhe]

I guess a really simple solution could be to port the concept of the MultiplexerTransform directly in the dash callback logic. I imagine it would require two changes,

• Add a new optional priority keyword to the Output object
• Insert intermediate "virtual nodes" in the DAG whenever an output is targeted multiple times

Considering the simple example by @luiztauffer,

(A,B) -> C
(A,D) -> C

this would map into

(A,B) -> C'
(A,D) -> C''
(C', C'') -> D

with some logic ensuring that the of the updates (A,B) -> C' and (A,D) -> C'' is performed in prioritized order. Do you think this would be doable, @nicolaskruchten @alexcjohnson ?

[FatHare]

Hi, i created a package to solve this problem: https://github.com/FatHare/dash_duplicate_output. I hope he helps you.

[alexcjohnson]

Closed by @T4rk1n in #2414 and released in Dash 2.9.0 today. We ended up implementing this with Patch partial prop outputs as a simple allow_duplicate flag in each output because in the context of partial prop updates even priority doesn’t quite make sense. Note also that the feature is not limited to Patch outputs, full outputs can use it too!

I know this has long long been a requested feature, we just wanted to be very careful with it because of the potential for ambiguous results and confusing debugging. Please try it and let us know how it goes!

[nicolaskruchten]

So how is prioritization handled? :)

[CNFeffery]

@alexcjohnson same problem as #2414 (comment), allow_duplicate not work in clientside_callback

[leo-smi]

@alexcjohnson same problem as #2414 (comment), allow_duplicate not work in clientside_callback

I think this will be solved in 2.9.3 see: https://community.plotly.com/t/dash-2-9-2-released-partial-property-updates-with-patch-duplicate-outputs-dcc-geolocation-scatter-group-attributes-and-more/72114/16?u=leosmi