tracing: support span linking for spans shared by multiple traces

[erikgrinaker]

During query processing, we frequently wait for asynchronous operations that are shared by multiple callers. These operations are currently absent from traces. For example:

• Raft processing: a single round of Raft can include log entries for many commands coming from many queries and clients.

• Asynchronous transaction writes: most transaction writes are pipelined, i.e. sent off asynchronously.

• Range descriptor lookups: if a descriptor lookup is already in flight, other callers will coalesce their lookups, i.e. wait for the result of the in-flight request.

In these cases (and probably many others), query tracing goes dark once it hits these operations, which is unfortunate -- especially in the case of Raft, where it would be very useful to follow e.g. the replication and fsync latencies of each query.

What we probably need to do here is to start a new span for these operations, and then return that span up to all callers who are waiting for the results of the operations, such that they can link the shared span into their own traces. Note that the callers may well arrive after the operation has started, as in the case of range descriptor lookup coalescing (i.e. a query can find an in-flight lookup that started 100ms ago, and wait for that to complete).

This may also require support for this in the trace data formats. @andreimatei mentioned in #70741 (review) that there is some support for this in OpenTelemetry, but that it's still somewhat in flux.

Jira issue: CRDB-10256

[erikgrinaker]

@ajwerner notes that there is some support for this in Raft already, but it requires an increased log level as it has some overhead:

cockroach/pkg/kv/kvserver/raft.go

Lines 256 to 267 in c2af5b8

	// traceMessageSends records the provided event for all proposals contained in
	// in entries contained in msgs. The vmodule level for raft must be at
	// least 1.
	func (r *Replica) traceMessageSends(msgs []raftpb.Message, event string) {
	if log.V(1) || r.store.TestingKnobs().TraceAllRaftEvents {
	var ids []kvserverbase.CmdIDKey
	for _, m := range msgs {
	ids = extractIDs(ids, m.Entries)
	}
	traceProposals(r, ids, event)
	}
	}

[erikgrinaker]

@andreimatei says regarding the above Raft tracing:

re raft application tracing needing a high-vmodule setting - this might be as easy to fix as a change from if log.V(2) to if log.ExpensiveLogsEnabled(2); perhaps check if that's the case

Tried this, didn't seem to work: no Raft stuff in the session traces.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!