Failed to cudaMalloc 1610612736 bytes on device 0 (error 2 out of memory)

[2081147463]

Summary

def compute_pseudo_labels(features, cluster, k1):
mat_dist = compute_jaccard_distance(features, k1=k1, k2=6)

if (search_option==0):
# GPU + PyTorch CUDA Tensors (1)
res = faiss.StandardGpuResources()
res.setDefaultNullStreamAllDevices()
_, initial_rank = search_raw_array_pytorch(res, target_features, target_features, k1)
initial_rank = initial_rank.cpu().numpy()

faiss.bruteForceKnn(res, metric,
xb_ptr, xb_row_major, nb,
xq_ptr, xq_row_major, nq,
d, k, D_ptr, I_ptr)

def bruteForceKnn(resources, metric, vectors, vectorsRowMajor, numVectors, queries, queriesRowMajor, numQueries, dims, k, outDistances, outIndices):
return _swigfaiss.bruteForceKnn(resources, metric, vectors, vectorsRowMajor, numVectors, queries, queriesRowMajor, numQueries, dims, k, outDistances, outIndices)

Platform

OS: UBantu 18.04

Faiss version: 1.7.2

Installed from: conda

Faiss compilation options:

Running on:

• [] CPU
• [✔] GPU

Interface:

• [] C++
• [✔] Python

Reproduction instructions

I run a ReID code.
nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.54 Driver Version: 510.54 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| 0% 40C P8 9W / 300W | 19MiB / 11264MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1350 G /usr/lib/xorg/Xorg 9MiB |
| 0 N/A N/A 1604 G /usr/bin/gnome-shell 7MiB |
+-----------------------------------------------------------------------------+
I have 11G but out of memory
I don`t understand
please help me

RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /__w/faiss-wheels/faiss-wheels/faiss/faiss/gpu/StandardGpuResources.cpp:415: Error: 'err == cudaSuccess' failed: Failed to cudaMalloc 1610612736 bytes on device 0 (error 2 out of memory
Outstanding allocations: