issues with retrieving large image data from GPU #344
Comments
|
Hi @pr4deepr , I'm good and hope you too! I have a suspicion, but want to check if the AI knows as well ;-) git-bob, please comment on potential reasons for the OpenCL error |
|
This message was generated by git-bob (version: 0.19.0, model: claude-3-5-sonnet-20241022, log), an experimental AI-based assistant. It can make mistakes and has limitations. Check its messages carefully. The A few things that could help debug this:
print(f"GPU memory before: {cle.get_gpu().get_info(cl.device_info.GLOBAL_MEM_SIZE)/(1024**3):.1f} GB")
deskewed = cle.deskew_x(image, ...)
print(f"GPU memory after: {cle.get_gpu().get_info(cl.device_info.GLOBAL_MEM_SIZE)/(1024**3):.1f} GB")
@haesleinhuepf - would you know if there's a way to check host memory availability through PyOpenCL or if there are any existing methods in pyclesperanto for chunked data transfer? |
|
I agree with git-bob (even though the code snippet might not work) |
|
If you're talking about the RAM, the PC has about 700GB RAM, so plenty available. |
|
I'm wondering what other resources could be depleted. git-bob answer with a list |
|
This message was generated by git-bob (version: 0.19.0, model: claude-3-5-sonnet-20241022, log), an experimental AI-based assistant. It can make mistakes and has limitations. Check its messages carefully. Following potential OpenCL resources could be depleted and cause the OUT_OF_RESOURCES error:
To investigate further, I suggest checking if using @haesleinhuepf - would you know if there are OpenCL command queue or pinned memory limitations we should consider here? |
At that point, the deskewing might not have happened yet because OpenCL queues work like lazy dask processing. The image is 13 GB large if it is of type float. @pr4deepr what GPU are you using? |
|
It's a Nvidia Quadro RTX 6000, 24GB of RAM |
|
Is it got to do with fact that image has to be of a certain size to work with GPU? I forgot the exact details. It does work with smaller images, like half size in xy.. I assumed it would throw a memory error when I run 'cle.deskew' command, wasn't used to getting an error when pulling the image back.. |
It starts computing and may return an error, when you ask to do the next operation on the data, e.g. when pulling it. Could you please run this to print out specs of the GPU(s)? I presume it outputs a line starting with Also this might be helpful to see which GPU you are using (just for completeness): |
|
FYI, on a numpy array, you can use |
I am pretty sure that the issue is the GPU RAM on two levels:
If you want to process your image with this GPU you would need to tile it. |
|
cl_info() cl.get_device
yea, entry for graphics card is:
so, 6GB.. If I wanted to handle this error in the code, can I use the condition: |
|
It should be possible to process the output string of the cl_info function using Python and regular expressions to have an if-statement that checks if the number in the line behind git-bob comment how to do this |
|
This message was generated by git-bob (version: 0.19.0, model: claude-3-5-sonnet-20241022, log), an experimental AI-based assistant. It can make mistakes and has limitations. Check its messages carefully. To parse the maximum allocation size from the cl_info output, you could do something like this: import re
def get_max_allocation_size():
info = cle.cl_info()
match = re.search(r'MAX_MEM_ALLOC_SIZE:(\d+)', info)
if match:
return int(match.group(1))
return None
# Example usage:
max_size = get_max_allocation_size()
needed_size = output_shape[0] * output_shape[1] * output_shape[2] * 4 # float32 = 4 bytes
if needed_size > max_size:
print(f"Output image size ({needed_size/1e9:.1f} GB) exceeds maximum GPU allocation size ({max_size/1e9:.1f} GB)")
# Handle tiling hereThis would help catch the allocation limit before attempting the operation. |
|
Thanks a lot for the help!! |
Hi @haesleinhuepf
Hope you have been well.
I'm following up on an issue we had with large image data
BioimageAnalysisCoreWEHI/napari_lattice#105
When I run this code
(294, 2304, 5184)The deskew process works.
When I pull the image from the GPU,
deskewed_np = cle.pull_zyx(deskewed)I get this error
It looks like we are running out of memory, but I assumed as the image is already on GPU, it shouldn't be a problem?
Cheers
Pradeep
The text was updated successfully, but these errors were encountered: