[BUG] Possible memory leak when merging DataFrames

[ayushdg]

Describe the bug
There might be a possible memory leak (or gc weirdness) when merging two DataFrames.

Steps/Code to reproduce bug

import cudf
import numpy as np
n_elem = 100_000_000

def func(n_elem):
    a = cudf.DataFrame({"a": np.arange(n_elem), "b": np.random.randint(0, 5, n_elem)})

    b = cudf.DataFrame(
        {
            "a": np.arange(
                n_elem - int(n_elem / 10), stop=2 * n_elem - int(n_elem / 10)
            ),
            "c": np.random.randint(0, 5, n_elem),
        }
    )

    a = a.merge(b, on="a", how="inner")
    return None

x1 = func(n_elem=n_elem)
x2 = func(n_elem=n_elem)

Memory usage after calling func (which returns None) with 0.12: 6.5GB
Memory usage after calling func with 0.11: 340 MB

Expected behavior
Similar memory usage with both versions.

Environment overview (please complete the following information)

• Environment location: Docker
• Method of cuDF install: 0.11 release docker, 0.12 from source at: adabc0e

Additional context
For v0.12, an arbitrary number of operations later the extra memory gets freed. If I explicitly call gc.collect() the memory gets freed.

When running func in a loop (tested on 10 iterations), it occasionally works with memory usage peaking around ~13-14GB and occasionally fails with OOM errors.

With 0.11 GPU mem usage doesn't exceed ~3-4GB during this process.

[shwina]

The two big changes since 0.11 that affect merge are (1) the libcudf++ refactor and (2) implicit typecasting. Possibly a first step would be to compare the behaviour of the following:

1. cudf 0.11
2. cudf 0.11 + libcudf++ refactor
3. cudf 0.11 + libcudf++ refactor + implicit typecasting

cc: @brandon-b-miller @kkraus14

[brandon-b-miller]

Looking into this on my side.

[brandon-b-miller]

I think this is related the the fact that casting_rules within _typecast_before_merge is persisting references to the mergees, working on a PR.