Skip to content

Commit

Permalink
sql: chunk deletions
Browse files Browse the repository at this point in the history
In the absence of a fast path deletion, `DELETE` would generate one
potentially giant batch and OOM the gateway node. This became obvious
quickly via heap profiling.

Added chunking of the deletions to `tableDeleter`. SQL folks may have
stronger opinions on how to achieve this, or a better idea of a
preexisting chunking mechanism that works more reliably. If nothing
else, this change serves as a prototype to fix #17921.

With this change, `roachtest run drop` works (as in, it doesn't
out-of-memory right away; the run takes a long time so I can't yet
confirm that it actually passes).

Release note (sql change): deleting many rows at once now consumes less
memory.
  • Loading branch information
tbg committed Feb 23, 2018
1 parent 43cac57 commit a2da685
Showing 1 changed file with 15 additions and 2 deletions.
17 changes: 15 additions & 2 deletions pkg/sql/tablewriter.go
Original file line number Diff line number Diff line change
Expand Up @@ -715,8 +715,9 @@ func (tu *tableUpserter) close(ctx context.Context) {

// tableDeleter handles writing kvs and forming table rows for deletes.
type tableDeleter struct {
rd sqlbase.RowDeleter
alloc *sqlbase.DatumAlloc
rd sqlbase.RowDeleter
alloc *sqlbase.DatumAlloc
batchSize int

// Set by init.
txn *client.Txn
Expand All @@ -736,6 +737,18 @@ func (td *tableDeleter) init(txn *client.Txn, evalCtx *tree.EvalContext) error {
func (td *tableDeleter) row(
ctx context.Context, values tree.Datums, traceKV bool,
) (tree.Datums, error) {
// Rudimentarily chunk the deletions to avoid memory blowup in queries such
// as `DELETE FROM mytable`.
const maxBatchSize = 10000
if td.batchSize >= maxBatchSize {
if err := td.txn.Run(ctx, td.b); err != nil {
return nil, err
}
td.b = td.txn.NewBatch()
td.batchSize = 0
}
td.batchSize++

return nil, td.rd.DeleteRow(ctx, td.b, values, sqlbase.CheckFKs, traceKV)
}

Expand Down

0 comments on commit a2da685

Please sign in to comment.