torch.cuda.OutOfMemoryError: Allocation on device

[807502278]

got prompt
!!! Exception during processing !!! Allocation on device
Traceback (most recent call last):
File "/root/autodl-tmp/ComfyUI/execution.py", line 324, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "/root/autodl-tmp/ComfyUI/execution.py", line 199, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "/root/autodl-tmp/ComfyUI/execution.py", line 170, in _map_node_over_list
process_inputs(input_dict, i)
File "/root/autodl-tmp/ComfyUI/execution.py", line 159, in process_inputs
results.append(getattr(obj, func)(**inputs))
File "/root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-HunyuanVideoWrapper/nodes.py", line 817, in encode
latents = vae.encode(image).latent_dist.sample(generator)
File "/root/miniconda3/lib/python3.10/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
return method(self, *args, **kwargs)
File "/root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-HunyuanVideoWrapper/hyvideo/vae/autoencoder_kl_causal_3d.py", line 289, in encode
return self.spatial_tiled_encode(x, return_dict=return_dict)
File "/root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-HunyuanVideoWrapper/hyvideo/vae/autoencoder_kl_causal_3d.py", line 405, in spatial_tiled_encode
tile = self.encoder(tile)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-HunyuanVideoWrapper/hyvideo/vae/vae.py", line 133, in forward
sample = down_block(sample)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-HunyuanVideoWrapper/hyvideo/vae/unet_causal_3d_blocks.py", line 721, in forward
hidden_states = resnet(hidden_states, temb=None, scale=scale)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-HunyuanVideoWrapper/hyvideo/vae/unet_causal_3d_blocks.py", line 408, in forward
hidden_states = self.conv1(hidden_states)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-HunyuanVideoWrapper/hyvideo/vae/unet_causal_3d_blocks.py", line 78, in forward
return self.conv(x)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 610, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 605, in _conv_forward
return F.conv3d(
torch.cuda.OutOfMemoryError: Allocation on device

Got an OOM, unloading all loaded models.
Prompt executed in 2.16 seconds

[807502278]

device: V100-32GB
environment: Linux+Torch 2.3