[hardware][cuda] use device id under CUDA_VISIBLE_DEVICES for get_device_capability

[youkaichao]

fixes #6213

[mgoin]

This looks reasonable, thanks for quick fix!

[comaniac]

Should we replace all usages of torch.cuda.get_device_capability() with this? For example attention selector and ROCm attention now still use the native torch one, which may result in the same issue.

[youkaichao]

Should we replace all usages of torch.cuda.get_device_capability() with this? For example attention selector and ROCm attention now still use the native torch one, which may result in the same issue.

I replaced most of them in #6080 . The rest are mostly used under is_hip, and they still use torch.cuda.get_device_capability() . In rocm, this function does not initialize device.

[mgoin]

@youkaichao I forget where but I ran into CI issues with current_platform.get_device_capability() hitting an exception on CPU tests because current_platform returns as None in that case. Should we make better dummy behavior for CPU/non-supported platforms, such as returning 0?

[youkaichao]

I forget where but I ran into CI issues with current_platform.get_device_capability() hitting an exception on CPU tests because current_platform returns as None in that case.

links would be appreciated. I don't think cpu code should call this function. even before this change, torch.cuda.get_device_capability() would also error out in cpu case.

[comaniac]

How about this one?

https://github.com/vllm-project/vllm/blob/main/vllm/attention/selector.py#L151

[youkaichao]

@comaniac these should be called after process initialize cuda, so they are fine. we can also replace them, though.

[comaniac]

@comaniac these should be called after process initialize cuda, so they are fine. we can also replace them, though.

Yeah it would be safer to replace this one as well.

[T-Atlas]

LGTM

[comaniac]

Merge first. I'll change the use in selector in another PR.