sql: the Insert planNode runs everything in one humongous batch

[andreimatei]

Currently the tableWriter (and I assume also tableUpdater) does everything in one batch. This can lead to large Raft proposals (and it can run into the kv.raft.command.max_size limit). If the Insert is configured with autocommit, then the batch is first checked against the kv.transaction.max_intents limit.

As a fun fact, it's currently possible to not hit the max_size limit, but subsequently hit the kv.transaction.max_intents limit at commit time. I think this is not ideal since, if all the intents that EndTransaction needs to carry were produced by a single batch, and that batch was applied successfully, the EndTransaction would probably apply successfully too.

I think it'd be a significant improvement if the Insert node would chunk up its writes so that large batches and proposals are avoided.
If we do this, and if we allow parallel execution of batches, there might be sequencing concerns to keep in mind for the UPDATE case - if a single UPDATE stmt updates a row twice, is it specified which one has to win?

Also see #15713

[jordanlewis]

@andreimatei did this get finished recently? I heard a rumor...

[andreimatei]

yes, closed in #23373