State and Contexts are not preserved through the Polyglot boundary

[radeusgd]

Whenever I call an Enso callback from within a polyglot call (i.e. I pass an Enso callback to a Java code as Function<A,B> and then execute that function in the Java code), the data about the State and more importantly Contexts is lost.

Repro

Create a new project, with a file polyglot/java/foo/foo/Foo.java:

package foo;

import java.util.function.Function;
import org.graalvm.polyglot.Value;

public class Foo {
    public static Value bar(Function<Integer, Value> f) {
        Value result = f.apply(42);
        return result;
    }
}

then in polyglot/java/foo run javac foo/Foo.java to get polyglot/java/foo/foo/Foo.class.

Now modify src/Main.enso to be:

from Standard.Base import all
import Standard.Base.Runtime.Context
import Standard.Base.Runtime.State

polyglot java import foo.Foo

run_inside_polyglot_callback ~action =
    callback _ = 
        action
    Foo.bar callback

report_states =
    IO.println "Context.Output.is_enabled = "+Context.Output.is_enabled.to_text
    state = Panic.catch Any (State.get Integer).to_text caught_panic->
        caught_panic.payload.to_display_text
    IO.println "State.get Integer = "+state

main =
    State.run Integer 123 <|
        IO.println "Direct:"
        report_states
        IO.println '\n'

        IO.println "Inside polyglot callback:"
        run_inside_polyglot_callback <| report_states
        IO.println '\n'

        IO.println '---------\n'
        IO.println "Now disabling context."
        Context.Output.with_disabled <|
            IO.println "Direct:"
            report_states
            IO.println '\n'

            IO.println "Inside polyglot callback:"
            run_inside_polyglot_callback <| report_states
            IO.println '\n'

and run the project.

Expected results

Direct:
Context.Output.is_enabled = True
State.get Integer = 123


Inside polyglot callback:
Context.Output.is_enabled = True
State.get Integer = 123


---------

Now disabling context.
Direct:
Context.Output.is_enabled = False
State.get Integer = 123


Inside polyglot callback:
Context.Output.is_enabled = False
State.get Integer = 123

Actual results

Direct:
Context.Output.is_enabled = True
State.get Integer = 123


Inside polyglot callback:
Context.Output.is_enabled = True
State.get Integer = State is not initialized for type Integer.


---------

Now disabling context.
Direct:
Context.Output.is_enabled = False
State.get Integer = 123


Inside polyglot callback:
Context.Output.is_enabled = True
State.get Integer = State is not initialized for type Integer.

We can see that inside of the callbacks that were executed by a Java call both the State and output context are lost - the state is treated as uninitialized and the Output context gets 're-enabled'.

I don't think this is correct.

[radeusgd]

I'm implementing some synchronization primitives in #7072 using Java and callbacks and then inside of such synchronized methods the execution context information gets reset, leading to invalid results in my computations.

I will add workarounds for this, but I think the current behaviour is incorrect and not intuitive.

If we have to keep it, we at least need to very carefully document this oddity somewhere.

[JaroslavTulach]

Currently State is passed as an argument throughout the Enso interpret execution. Such argument is obviously lost whenever a callback to and from polyglot call is made. To fix that we would need to keep state as thread local - either as a value in EnsoContext or as real ThreadLocal variable.

We should use ContextThreadLocal to store the State per EnsoContext and per thread.

[radeusgd]

Bumping this issue as I ran into it again when working on #10609 - I was surprised why a State read was not finding the value that I was sure is there, and I found out it was because of this bug.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2025-02-16):

Progress: .

• review storages: Implement Iterators on Storage, arithmetic and comparisons use the StorageIterators #12274 (review)
• comparing EE vs. CE performance: https://docs.google.com/document/d/13hNdFkfpvyk7xSnyFW1RmkyxZzmzRjSegI1uK0rKUyM/edit?usp=sharing
• benchmarking lengthOfStrings
• benchmarking - Reported Further rework State manipulation Nodes #12294
• benchmarking - new Text PR: Removing volatile and synchronization in Text #12295 It should be finished by 2025-02-18.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2025-02-17):

Progress: .

• Merged State: Removing State from method signatures #12233
• Enso binary launcher integrated: Enabling native launchers for the release #12296
• review Vue.js benchmarks app: VueJS SPA engine-benchmark-results#2 (review)
• review: Make test failure source locations lazy #12260 (review)
• demo & other meetings It should be finished by 2025-02-18.