distsql: tracking issue for enabling DistSQL by default

[RaduBerinde]

TODO list for getting all our ducks in a row for enabling DistSQL in "auto" mode. Feel free to add as needed. CC @andreimatei @cuongdo

Correctness:

• logic tests on multi-node cluster with fake span resolver distsql: make TestLogicDistSQL run on a 3-node cluster w/fake span resolver #13377
• nightly bigtests on distsql (@andreimatei)
• tests for auto mode determination (probably via a special EXPLAIN mode - distsql: add EXPLAIN(DIST_SQL) #13674)
• versioning distsql: versioning story #13801
• retryable errors should be marshalled correctly and tested distsql: txn retry errors are not caught by distsql #14504 @andreimatei
• DistSQL reads can't be run in txns that performed writes distsql: TxnCoordSender doesn't allow distribution if transaction changed data #13376 @andreimatei

Stability:

• enable on all production clusters in auto mode @asubiotto
  • currently running photos load on navy with DISTSQL=ON
• run continuous TPC-H load against at least one production cluster @cuongdo
• check that index backfill works fine when nodes crash (distsql: avoid dead nodes #13655) @RaduBerinde

Performance:

• update caches for span resolver (@andreimatei)
• sanity check execution plans for various queries (@asubiotto)

Features:

• marshall evalCtx data - particularly timezone info required for date operations distsql: expressions requiring the EvalCtx can't be evaluated #13821

After initial "auto" release:

• marshall ObservedTimestamps distsql: ObservedTimestamps are not marshalled back to the gateway #15013
• memory accounting distsql: memory accounting #15304
• distributed query cancellation distsql: distSQLReceiver should respond to context cancellation #15671
• better test infrastructure distsql: testing infrastructure #13665
• have some basic benchmarks in place for running periodically (@asubiotto)
• gather some distsql on vs off data from the benchmarks @arjunravinarayan is working on.

[jseldess]

Wonder if we should add external docs to this checklist, @RaduBerinde?

[RaduBerinde]

@jseldess Yes, I think we should have some docs mentioning this. I don't know if going in depth about how distsql works is useful, but what we should convey is that we have a new distributed framework for running a subclass of (read-only) queries, and it can be disabled per-session using SET DISTSQL = OFF (this might be useful if a user is having some kind of problem with a query and wants to try it without distsql). The docs can also mention EXPLAIN (DISTSQL) can be used to see if a query will be run via distsql.

[petermattis]

@RaduBerinde Is there anything left to do here? There are still some unfinished items, but we've already enable DistSQL by default.

[rjnn]

The only agenda item (that's not in the "after auto release" section) that has not been closed is "DistSQL reads can't be run in txns that performed writes", which we've chosen not to do for now.

So this issue should be closed, although we should first probably add/document a separate tracking issue for "distributed query cancellation" which does not have a linked issue (the other open agenda item is marshalling ObservedTimestamps which does have a linked issue).

[andreimatei]

Created #15671 for the cancellation.