[query] Expose references via ExecuteContext

[ehigham]

This change is split out from a larger refactoring effort on the various Backend
implementations. The goals of this effort are to provide query-level
configuration to the backend that's currently tied to the lifetime of a backend,
reduce code duplication and reduce state duplication.

In this change, I'm restoring references to the execute context [1] and
decoupling them from the backend. In a future change, they'll be lifted out of
the backend implementations altogether. This is to reduce the surface area of
the Backend interface to the details that are actually different.

Both the Local and Spark backend have state that's manipulated from python via
various py methods. These pollute the Backend interface [2] and so have been
extracted into the trait Py4JBackendExtensions. In future changes, this will
become a facade that owns state set in python.

Notes
[1] "Restoring" old behaviour I foolishly removed in fe5ed32
[2] "Pollute" in that they obfuscate what's different about backend query plan
and execution

[ehigham]

• [query] Move is.hail.check to test code #14801 : 2 dependent PRs (#14693 , #14698 )
• [query] Move LoweredTableReaderCoercer into ExecuteContext #14696
• [query] Remove persistedIr from Backend interface #14692
• [query] Remove lookupOrCompileCachedFunction from Backend interface #14691
• [query] Remove BlockMatrix persist from Backend interface #14690
• [query] Expose references via ExecuteContext #14686 👈 (View in Graphite)
• [query] Extract Backend Methods called from Python into Py4JBackendExtensions #14767
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[chrisvittal]

Looks good to me. Won't change my approval, I just have one question.

[patrick-schultz]

Looks good overall, just a few questions

[patrick-schultz]

This is a bit awkward, and I don't love HailFeatureFlags having both a mutable interface and a + operator that returns a new object. How about if ExecuteContext.local took only the bindings to add/update?

[ehigham]

I don't see what's awkward, can you ellaborate?

This creates a local context where the flags are updated.
local itself accepts more than flags and takes care of the whole

val whatWasItBefore = get
set newValue
try {
...
} finally {
set whatItWasBefore
}

noise.
If you wanted to modify the flags for the rest of the execution, you could just do

ctx.flags.set("foo", "bar")
...

This interface gives a simple way of doing both

[patrick-schultz]

Huh, I could have sworn I replied to this, but my comment seems to have disappeared.

Sorry, I should have been more specific. I just meant that it might be simpler for ExecuteContext.local to take a overrideFlags: Seq[(String, String)] = Seq.empty, so you could call ctx.local(overrideFlags = Seq("shuffle_cutoff_to_local_sort" -> "40")). I.e. local overides individual flags in the scope, rather than the entire flags map.

But this is minor, so I'm happy to approve as is if you don't want to change it.

[ehigham]

Thanks, I see what you're getting at. But how would I unset flags? By mapping the value to null? yuck!
I think the current thing is simpler and so I'm inclided to keep it.