Visualizations updates are not stable and leaving visualizations blank

[sylwiabr]

Discord username

No response

What type of issue is this?

Transient – Occurring only once

Is this issue blocking you from using Enso?

• Yes, I can't use Enso because of this issue.

Is this a regression?

• Yes, previous version of Enso did not have this issue.

What issue are you facing?

During building up use-cases and demos we observed the decrease in visualizations stability. That include:

• visualization is blanked after finishing editing a node
• blank visualizations after nodes collapsing,
• blank visualization on an existing node with viz opened while uploading the new file,
• no update for the full screen visualization if it was opened before IDE got the data - it is never updated and needs reopening,
• clicking on the drop-down to select a visualization is vanishing the visualization.

Expected behaviour

Visualizations should be updated with the data as soon as data are available.

How we can reproduce it?

No response

Screenshots or screencasts

No response

Enso Version

nightly 04.05

Browser or standalone distribution

Standalone distribution

Browser Version or standalone distribution

standalone

Operating System

MacOS

Operating System Version

No response

Hardware you are using

No response

[wdanilo]

Question – we had so many issues about it already. Do we have any insights why it is happening? Why can't we just fix the GUI - backend communication once for all and prevent such issues from happening ever again? Basically, my question here is - what is exactly the cause of this / similar to this issues? Is it some kind of race condition / bugs in communication / something else?

CC @4e6 as you'd probably know the answer to these questions :)

[4e6]

The IDE has basically two feedback channels from the engine - expression and visualization updates. And when anything goes wrong, the user can only observe it by seeing the wrong updates. Here are the last issues related to visualizations

• DataflowAnalysis preserves dependencies order #6493
  The issue with the dropdowns (which are utilizing the visualization updates channel) was fixed in the compiler pass
• Visualization preprocessor should not catch errors #6423
  The issue with table visualizations blanking was caused by the library function catching the interruption exception
• Keep scheduled execution job on text edit #6354
  Some visualizations that were not showing up after the project startup was caused by the job engine

All causes were completely different

[4e6]

The issue I can reproduce on the current develop seems like on the IDE side

Peek.2023-05-05.15-37.mp4

[sylwiabr]

@4e6 thank you for checking. Package from 29.04 does not have those issues, package from 02.05 does. This means the regression was caused by #6458 or #6461. As #6461 was just touching debug scenes not application probably the issue was introduced with #6458 - @kazcw can you have a look at this?

[kazcw]

Looking into it.

[4e6]

May be related #6526

[kazcw]

The bug still occurs on develop after reverting #6458 (and dependent PR #6569).

I suspect it's older than 02.05. The bug is sensitive to the exact testing procedure: Immediately after creating a new node the bug is present when editing the node, but if I open a project and edit an already existing node the bug isn't triggered. (This made attempts to narrow down the cause "interesting").

[enso-bot]

Michael Mauderer reports a new STANDUP for today (2023-05-08):

Progress: Started investigating why visualizations are blank by reproducing the issue and adding more logging. It should be finished by 2023-05-10.

Next Day: Next day I will be working on the #6561 task. Continue investigation Address question about recurring issues around visualisations.

[enso-bot]

Michael Mauderer reports a new STANDUP for yesterday (2023-05-09):

Progress: Found that the blank visualization is caused by a spurious FRP update that sets the visualization path and thus re-creates the visualization. But since the node is not re-evaluated as it was already evaluated for preview, the engine sends no data update. It should be finished by 2023-05-10.

Next Day: Next day I will be working on the #6561 task. Check if this solves all described scenarios. Investigate missing colouring that also happens in this scenario.

[enso-bot]

Michael Mauderer reports a new STANDUP for today (2023-05-15):

Progress: Still investigating the missing data update when full-screen visualization is opened before data arrives. So far it seems to not only be related to FRP sheninigans but an inconsistent state in the controller: the visualisaiton seems to dissapear from the execution context just before it is meant to receive an update. But I am not sure why yet. It should be finished by 2023-05-15.

Next Day: Next day I will be working on the #6561 task. Investigate remaining issues.

[enso-bot]

Michael Mauderer reports a new 🔴 DELAY for today (2023-05-16):

Summary: There is 4 days delay in implementation of the Visualizations updates are not stable and leaving visualizations blank (#6561) task.
It will cause 0 days delay for the delivery of this weekly plan.

Delay Cause: Multiple bugs on this issue. The current investigation also seems to be caused by multiple issues: (1) a rare crash of the table visualization (2) a race condition that causes the visualization not to be parented correctly (3) a race condition that sometimes removes the visualization.

Possible solutions: Implement easier bugs.

[enso-bot]

Michael Mauderer reports a new STANDUP for today (2023-05-16):

Progress: Still investigating the missing data update when full-screen visualization is opened before data arrives. So far, I solved an issue that the visualizations sometimes was not parented to the correct layer due to a race condition. Still investigating an issue where the visualization is not correctly restored after resetting it just before going full-screen. It should be finished by 2023-05-19.

Next Day: Next day I will be working on the #6561 task. Investigate remaining issues.

[enso-bot]

Michael Mauderer reports a new STANDUP for yesterday (2023-05-17):

Progress: After fixing some FRP bugs in the visualization pipeline, I found an issue with the controller not emitting the correct events during full-screen visualization. Investigating root cause. It should be finished by 2023-05-19.

Next Day: Next day I will be working on the #6561 task. Investigate remaining issues.

[enso-bot]

Michael Mauderer reports a new STANDUP for the provided date (2023-05-19):

Progress: After some more investigation, I found that the actual root cause was a logic bug with the display/hide button, which w\s not triggered correctly when going full-screen. Starting to implement a + fix + some refactoring that will simplify the FRP logic around showing/hiding visualizations. It should be finished by 2023-05-19.

Next Day: Next day I will be working on the #6561 task. Investigate remaining issues.

[enso-bot]

Michael Mauderer reports a new 🔴 DELAY for today (2023-05-23):

Summary: There is 3 days delay in implementation of the Visualizations updates are not stable and leaving visualizations blank (#6561) task.
It will cause 0 days delay for the delivery of this weekly plan.

Delay Cause: Multiple bugs on this issue caused by excess complexity in the FRP networks around the visualization logic. Logic is distributed around multiple components (action bar / nodes / graph editor) and thus prone to breaking as different usages make different assumptions. Cleaning this up should decrease future maintenance burden and bugs in this code.

Possible solutions: Implement easier bugs.

[enso-bot]

Michael Mauderer reports a new 🔴 DELAY for today (2023-05-23):

Summary: There is 2 days delay in implementation of the Visualizations updates are not stable and leaving visualizations blank (#6561) task.
It will cause 0 days delay for the delivery of this weekly plan.

Delay Cause: Forgot to account for weekend days in the last delay.

[enso-bot]

Michael Mauderer reports a new STANDUP for yesterday (2023-05-22):

Progress: Continued work on fixing FRP issues and simplifying the logic. Found an existing issue with the show/hide buttons not distinguishing between API events and user events, which sometimes lead to multiple show/hide events for visualizations. It should be finished by 2023-05-24.

Next Day: Next day I will be working on the #6561 task. Open PR for this round of fixes.

[enso-bot]

Michael Mauderer reports a new STANDUP for yesterday (2023-05-23):

Progress: Finalised the changes cleaning up code and testing against regressions. It should be finished by 2023-05-24.

Next Day: Next day I will be working on the #6561 task. Starting next task.

[enso-bot]

Michael Mauderer reports a new STANDUP for yesterday (2023-05-24):

Progress: When testing, I noticed another regression that is now fixed. Opened PR. and started checking other open issues that could be resolved by this PR. It should be finished by 2023-05-24.

Next Day: Next day I will be working on the #6561 task. Starting next task.