Regressions in visualizations and drop-downs behavior

[sylwiabr]

Tasks

Preview Give feedback

1. Investigate reasons for Request timeouts during AttachVisualizationHandler #7005
   -language-server x-chore +2
   
2. Improve performance of graph changes #6954
   CI: No changelog needed CI: Ready to merge +2
   

Discord username

No response

What type of issue is this?

Permanent – Occurring repeatably

Is this issue blocking you from using Enso?

• Yes, I can't use Enso because of this issue.

Is this a regression?

• Yes, previous version of Enso did not have this issue.

What issue are you facing?

• Visualizations are failing much more often on starting the existing project - they don't show up.
• The drop-downs are missing when starting the existing project - editing the node is bringing them back.
• Changing the drop-down is not changing the value on the visualization - user has to perform another action (i.e. open CB) to get the update on the visualization.

The package from 2.06 is working perfectly fine.
The package from the 6.06 has those issue.
The Package from 09.06 has even bigger problems with showing drop-downs.
I am assuming this was at the beginning caused by 72b202b .
Please confirm the root cause of the issues and revert the problematic commits.

We were testing different versions of the IDE with newest Engine to make sure it is the regression on the IDE side.

Expected behaviour

visualizations should work fluently and update when drop-down is changed. Drop-downs should appear on the project start.

How we can reproduce it?

No response

Screenshots or screencasts

Screencast form the IDE from 02.06 + Engine from 09.06

02.06.mov

Screencast form the IDE from 06.06 + Engine from 09.06

06.06.mov

Screencast form the IDE from 09.06 + Engine from 09.06

09.06_720.mov

Logs

No response

Enso Version

nightly 09.06

Browser or standalone distribution

Standalone distribution (local project)

Browser Version or standalone distribution

standalone

Operating System

Windows

Operating System Version

No response

Hardware you are using

No response

[farmaazon]

Tested newest develop (372bc8f) and we for some reason attach visualization on all nodes during project open. It clutters the engine with "attachVisualization" requests, and some returns timeouts (the engine's timeouts, not IDE's).

So the problem is twofold:

1. The IDE should not attach visualization to all nodes.
2. Even if IDE attach visualization to all nodes, the engine should not time out (after all, we need visualizations for widgets, and user always could enable vis on all nodes by themselves).

@hubertp Ad 2. Do you have any issue for those "attaching visualization" timeouts? They seem to come back like a chorus on various vis problems.

[farmaazon]

Regarding point 1 from above comment, the offending commit seems to be 72b202b @MichaelMauderer

However, given the above note, this bug should be considered only a catalyst, not the root cause. I would not revert it @sylwiabr

[jdunkerley]

I think the attachVisualization to all nodes is possibly due to the widget evaluation rather than the vizzes.

[farmaazon]

I think the attachVisualization to all nodes is possibly due to the widget evaluation rather than the vizzes.

No, we attach them also on nodes. It's easy to spot when you actually click the eye icon while tracking network log: first visualization show does not send attach visualization (because it's already attached).

Also, when browsing all "attach visualization" on project startup you can clearly see the difference between widget visualizations (which call "get_widget_json" as preprocessor) and "normal" visualizations (calling "default_preprocessor"). Below we have 11 of them, before Michael's commit there was only one.

{"jsonrpc":"2.0","id":12,"method":"executionContext/attachVisualisation","params":{"expressionId":"9576c7ed-54e3-4864-9734-d8869bd6d637","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"1d3f10e9-b84d-4c2f-aa1a-812731a2947b"}}	461	
14:40:24.558
{"jsonrpc":"2.0","id":13,"method":"executionContext/attachVisualisation","params":{"expressionId":"c7ef638f-8938-4dd7-9d4d-194a8c92b637","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"37476e0f-7768-4d5b-ae2d-6b720210a640"}}	461	
14:40:24.559
{"jsonrpc":"2.0","id":14,"method":"executionContext/attachVisualisation","params":{"expressionId":"41be788b-bb66-4179-bc9b-59389902431f","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"da9662dc-c7a7-4514-8cb4-917b441aa710"}}	461	
14:40:24.559
{"jsonrpc":"2.0","id":15,"method":"executionContext/attachVisualisation","params":{"expressionId":"dd707700-073a-4d8d-bca6-4a394bfc63bb","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"7c96bbbd-d062-4992-a967-6c918e21a1be"}}	461	
14:40:24.559
{"jsonrpc":"2.0","id":16,"method":"executionContext/attachVisualisation","params":{"expressionId":"cdd2fd37-2f55-4790-8e45-1935064ea3a2","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"1dffca2d-c52c-421d-9556-df0462f450d2"}}	461	
14:40:24.559
{"jsonrpc":"2.0","id":17,"method":"executionContext/attachVisualisation","params":{"expressionId":"ce5f5911-5ba4-46d0-b7a3-7084a2cc25ee","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"10466149-78fa-4be2-b83b-ecc6466d9526"}}	461	
14:40:24.559
{"jsonrpc":"2.0","id":18,"method":"executionContext/attachVisualisation","params":{"expressionId":"61e9fa31-4721-4580-aa53-5463a0d616f6","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"bd8f12cb-1661-474f-bddb-a13926669076"}}	461	
14:40:24.560
{"jsonrpc":"2.0","id":19,"method":"executionContext/attachVisualisation","params":{"expressionId":"97c1884a-36a5-43a8-887e-d1f9dc458391","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"38848864-3a5d-4183-8596-79f41bcb6dbd"}}	461	
14:40:24.560
{"jsonrpc":"2.0","id":20,"method":"executionContext/attachVisualisation","params":{"expressionId":"28187a71-7ee5-41af-b2b2-8e9ccf9babbb","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"6698cea1-44eb-4886-ab78-4ca84e6033fa"}}	461	
14:40:24.560
{"jsonrpc":"2.0","id":21,"method":"executionContext/attachVisualisation","params":{"expressionId":"522cdf6c-a73e-4d75-9326-91d3a67d7bf8","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"aa0775c1-4e23-4d16-9b2c-f5844000cc6b"}}	461	
14:40:24.560
{"jsonrpc":"2.0","id":22,"method":"executionContext/attachVisualisation","params":{"expressionId":"c97d9d26-a36c-4abc-8a15-46aa4c3a1eac","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"default_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"1cac3fd9-d7f7-4f58-97c5-239010e48395"}}	461	
14:40:24.560
{"jsonrpc":"2.0","id":23,"method":"executionContext/attachVisualisation","params":{"expressionId":"ce5f5911-5ba4-46d0-b7a3-7084a2cc25ee","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Preprocessor","module":"Standard.Visualization.Preprocessor","name":"error_preprocessor"},"positionalArgumentsExpressions":[]},"visualisationId":"d10af4cf-98f7-4a00-9e39-ed5ae3fd2fc5"}}	459	
14:40:24.768
{"jsonrpc":"2.0","id":25,"method":"executionContext/attachVisualisation","params":{"expressionId":"63a63ea9-7c48-44a9-82de-89b6ca380875","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'read'","['self', 'format', 'on_problems']"]},"visualisationId":"662e9690-2bd8-4d94-8446-40d0b9d40f0b"}}	490	
14:40:26.056
{"jsonrpc":"2.0","id":26,"method":"executionContext/attachVisualisation","params":{"expressionId":"28c18919-4a37-460a-ba1a-d036e769c580","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'/'","['self', 'subpath']"]},"visualisationId":"dca397c3-d6cc-4004-8a3d-471da06f40bf"}}	473	
14:40:26.056
{"jsonrpc":"2.0","id":27,"method":"executionContext/attachVisualisation","params":{"expressionId":"a9e2cef8-85d3-4b80-9b3b-6448987b0e97","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'/'","['self', 'subpath']"]},"visualisationId":"4524b67e-f3ed-4713-b8b7-3e76216cd230"}}	473	
14:40:26.056
{"jsonrpc":"2.0","id":28,"method":"executionContext/attachVisualisation","params":{"expressionId":"37365f39-87cf-4263-8156-291f414a31bb","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'read'","['self', 'format', 'on_problems']"]},"visualisationId":"7b76f5d1-1bf7-437f-a39a-25fa694d83f4"}}	490	
14:40:26.056
{"jsonrpc":"2.0","id":29,"method":"executionContext/attachVisualisation","params":{"expressionId":"0e0900f6-7de2-4aa9-ae52-2ac7af9ace13","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'parse'","['self', 'text', 'pattern']"]},"visualisationId":"06ff1f4b-b6c8-4faa-a587-364a5adc56f5"}}	485	
14:40:26.057
{"jsonrpc":"2.0","id":30,"method":"executionContext/attachVisualisation","params":{"expressionId":"c90a59d4-85d4-40b9-8a37-9debea55bce1","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'parse'","['self', 'text', 'pattern']"]},"visualisationId":"1ea4da5c-4452-49aa-88a8-1899af6ced7f"}}	485	
14:40:26.057
{"jsonrpc":"2.0","id":31,"method":"executionContext/attachVisualisation","params":{"expressionId":"c231aff1-b6ff-4a34-bd76-1c3321f0845d","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'read'","['self', 'query', 'limit']"]},"visualisationId":"2b537462-85b9-46c8-b0f4-2d0f648602ee"}}	483	
14:40:26.733
{"jsonrpc":"2.0","id":32,"method":"executionContext/attachVisualisation","params":{"expressionId":"9c755f2b-316d-4301-a4c6-f6932d9855cf","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'use_first_row_as_names'","['self', 'on_problems']"]},"visualisationId":"306eb9de-9fe7-48b7-aa53-91504e8f8ad0"}}	498	
14:40:26.733
{"jsonrpc":"2.0","id":33,"method":"executionContext/attachVisualisation","params":{"expressionId":"c1e60426-a3d1-44c9-82f6-e607ab7f3f49","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'split_to_columns'","['self', 'column', 'delimiter', 'column_count', 'on_problems']"]},"visualisationId":"e50c0dac-0adc-4585-9aaf-b2626ca921f2"}}	531	
14:40:26.733
{"jsonrpc":"2.0","id":34,"method":"executionContext/attachVisualisation","params":{"expressionId":"639e6ff6-e8f9-4e8e-9c8a-d8ce71874cd1","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'parse'","['self', 'columns', 'type', 'format', 'error_on_missing_columns', 'on_problems']"]},"visualisationId":"f9f9d603-a512-47b5-8b43-bab43704c03b"}}	538	
14:40:26.734
{"jsonrpc":"2.0","id":35,"method":"executionContext/attachVisualisation","params":{"expressionId":"74d9e004-0203-460e-b80d-58784f152302","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'Date'","[]"]},"visualisationId":"28350ad7-06d8-424b-a445-34110d4cc91a"}}	459	
14:40:26.734
{"jsonrpc":"2.0","id":36,"method":"executionContext/attachVisualisation","params":{"expressionId":"0f27c6dc-6b8b-431b-b4b3-f64d7ab1b977","visualisationConfig":{"executionContextId":"3f5a7430-b839-4206-bdf5-15a4fb4860c4","expression":{"definedOnType":"Standard.Visualization.Widgets","module":"Standard.Visualization.Widgets","name":"get_widget_json"},"positionalArgumentsExpressions":["'get'","['self', 'selector', 'if_missing']"]},"visualisationId":"20d14ea0-9e66-4d03-b83e-7271645118f7"}}	490	
14:40:26.734

[jdunkerley]

As requested we attempted to open all the visualizations on the little workflow and after 9 were open opening any more failed and the viz closed.

After that it became the same as with develop where if we do another action (like changing a locale) the previous change then calculates.

2023-06-09_13-58-23.mp4

[wdanilo]

@farmaazon thanks for the details. I don't understand one comment here: "However, given the above note, this bug should be considered only a catalyst, not the root cause." – so is Michaels commit introducing a bug or not? Earlier you told that before the commit visualizations were not attached to all the nodes. I'm just trying to understand better what happens :)

[wdanilo]

@jdunkerley What you shown on the video is worrying. Why changing one node would affect another to recompute? I'm not sure whether this is the same bug as this one.

[farmaazon]

@farmaazon thanks for the details. I don't understand one comment here: "However, given the above note, this bug should be considered only a catalyst, not the root cause." – so is Michaels commit introducing a bug or not? Earlier you told that before the commit visualizations were not attached to all the nodes. I'm just trying to understand better what happens :)

Look at my comment #7001 (comment) The second point was always a problem, Michael's bug just made it visible on every project.

So Michale's commit introduced bug, but not the bug described in this issue.

[wdanilo]

@farmaazon all clear now! Thank you!

[kazcw]

1. The IDE should not attach visualization to all nodes.

This part is fixed in my upcoming PR #6954, as it was the root of major performance issues.

[jdunkerley]

@jdunkerley What you shown on the video is worrying. Why changing one node would affect another to recompute? I'm not sure whether this is the same bug as this one.

Only occurs once you overload the vizzes.
Agree was present before the other regression but only surfaced when I used new version

[Akirathan]

@hubertp Ad 2. Do you have any issue for those "attaching visualization" timeouts? They seem to come back like a chorus on various vis problems.

#7005 is the task to investigate the timeouts on the LS side. Since it is not yet clear whether it is LS to blame, I am also attaching -language-server label.

[farmaazon]

So I mark those two tasks as tracked by this one (see the top of the description).

[kazcw]

Both the frontend and backend tasks to fix this are closed, so I'll mark this fixed.