TiDB Server crashing w/ OOM despite the tidb_mem_quota_query limit

[coderplay]

Bug Report

1. Minimal reproduce step (Required)

huge hash join

2. What did you expect to see? (Required)

We are mimicking our production workloads to a dev TiDB cluster. One of the biggest queries is a huge hash join, which can consume ~50GB of memory. We reduce the tidb_mem_quota_query to 1GB to avoid system's oom-killing, but the setting didn't work.

After diving deeper into the hashjoinExecutor implementation, we figure out that's because below memory usages are not counted in the memory tracker:

1. The hash table used for joining: https://github.com/pingcap/tidb/blob/master/executor/hash_table.go#L187
2. The entryStore for above hash table: https://github.com/pingcap/tidb/blob/master/executor/hash_table.go#L324
3. The newly created chunks: https://github.com/pingcap/tidb/blob/master/executor/join.go#L281
   Besides above func (worker *copIteratorWorker) handleCopResponse can take another 2~4 GB in our case.

I am attaching a heap profiling to this ticket, please check it out. We took the profiling when the TiDB server was still alive, taking around 15.8GB heap memory, but eventually, it can consume ~50GB.

3. What did you see instead (Required)

tidb server get killed by system's oom-killer

4. What is your TiDB version? (Required)

5.2.2 on kubernetes

[XuHuaiyu]

Hi, @coderplay
The largest part of the HashJoin memory usage is the underlying data of the hashtable, which has been counted in the memory tracker, and can be spilled to the disk. But as you mentioned in the description, the entryStore for the hash table, which is actually a pointer to the underlying data has not been tracked.
We are still working on improving the coverage of the memory tracker.

[XuHuaiyu]

We may try to set tidb_enable_rate_limit_action to false to see whether the SQL can be cancelled successfully.

[coderplay]

@XuHuaiyu

The largest part of the HashJoin memory usage is the underlying data of the hashtable, which has been counted in the memory tracker,

IIUC, the hashtable isn't accounted, see https://github.com/pingcap/tidb/blob/master/executor/hash_table.go#L187. Could you please let me know which lines of code it does the accounting?

Please be aware that there are 4 big memory consumers in this issue.

[XuHuaiyu]

1. The underlying data of hashtable is accounted in

   tidb/executor/hash_table.go

   Line 166 in 954e1e1

   err := c.rowContainer.Add(chk)

hashtable stores the key and rowPtr, the memory of it is not tracked:

tidb/executor/hash_table.go

Lines 82 to 83 in 954e1e1

	// hashTable stores the map of hashKey and RowPtr
	hashTable baseHashTable

2. The newly created chunk is not accounted
   

   tidb/executor/join.go

   Line 281 in 954e1e1

   chk := chunk.NewChunkWithCapacity(e.buildSideExec.base().retFieldTypes, e.ctx.GetSessionVars().MaxChunkSize)

3. The copResponse is accounted in

   tidb/store/copr/coprocessor.go

   Line 550 in 954e1e1

   worker.memTracker.Consume(consumed)

[coderplay]

@XuHuaiyu yep, we are on the same page now. Looking forward to the fix.

[wshwsh12]

There are three memory untracked issues mentioned in this issue

1. For the memory usage of hash table and entryStore, we have tracked them in the pr executor: add some memory tracker in HashJoin #33918.
2. For the memory usage of NewChunkWithCapacity in func buildHashTableForList, the memory usage will be tracked when trying to add chunk into rowContainer in HashJoinExec.buildHashTable(https://github.com/pingcap/tidb/blob/master/executor/join.go#L783)
3. For the memory usage of handleCopResponse, currently we will control it through tidb_enable_rate_limit_action. But it seem not to work well at the moment, we will refactor the logic in the near future.

[XuHuaiyu]

Close this issue, because we fix this problem in the duplicate issue
#35627