Question about "cuequivariance or cuequivariance_torch is not available. Cuequivariance acceleration will be disabled."

[treasureUu]

May I ask about the Cuda GPU training ?

I know the error code I've listed is a bit long, but I would be very grateful if someone could help identify where there might be a problem.

Last week, I installed MACE on the supercomputing server at my school. However, the GPU training works not well.

I used: pip install cuequivariance_ops_torch_cu11
Because, when it is pip install cuequivariance_ops_torch_cu12

ERROR: Could not find a version that satisfies the requirement cuequivariance_ops_torch_cu12 (from versions: none)
ERROR: No matching distribution found for cuequivariance_ops_torch_cu12

**1. When I set:

device: cuda
enable_cueq: True

I got the error: cuequivariance or cuequivariance_torch is not available. Cuequivariance acceleration will be disabled. The details shows below:**

/scratch/user/djy/mace_env/lib/python3.10/site-packages/e3nn/o3/_wigner.py:10: UserWarning: Environment variable TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD detected, since the`weights_only` argument was not explicitly passed to `torch.load`, forcing weights_only=False.
  _Jd, _W3j_flat, _W3j_indices = torch.load(os.path.join(os.path.dirname(__file__), 'constants.pt'))
**cuequivariance or cuequivariance_torch is not available. Cuequivariance acceleration will be disabled.**
2025-02-16 03:51:34.326 INFO: ===========VERIFYING SETTINGS===========
2025-02-16 03:51:34.327 INFO: MACE version: 0.3.10
2025-02-16 03:51:34.489 INFO: CUDA version: 12.4, CUDA device: 0
/scratch/user/djy/mace_djy/try_crystal_Cr_random01_comp_expa_optim_cuda_E0s_average_force10energy100_weight_fineturn/run_train.py:143: UserWarning: Environment variable TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD detected, since the`weights_only` argument was not explicitly passed to `torch.load`, forcing weights_only=False.
  model_foundation = torch.load(
2025-02-16 03:51:35.137 INFO: Using foundation model /scratch/user/djy/mace_djy/mace-models/2023-12-03-mace-128-L1_epoch-199.model as initial checkpoint.
2025-02-16 03:51:35.138 WARNING: Using multiheads finetuning with a foundation model that is not a Materials Project model, need to provied a path to a pretraining file with --pt_train_file.

2025-02-16 03:51:38.479 INFO: Total number of parameters: 699674
2025-02-16 03:51:38.479 INFO:
2025-02-16 03:51:38.479 INFO: ===========OPTIMIZER INFORMATION===========
2025-02-16 03:51:38.479 INFO: Using ADAM as parameter optimizer
2025-02-16 03:51:38.479 INFO: Batch size: 2
2025-02-16 03:51:38.479 INFO: Number of gradient updates: 2244
2025-02-16 03:51:38.480 INFO: Learning rate: 0.0005, weight decay: 5e-07
2025-02-16 03:51:38.480 INFO: WeightedEnergyForcesLoss(energy_weight=100.000, forces_weight=10.000)
2025-02-16 03:51:38.480 INFO: Converting model to CUEQ for accelerated training
2025-02-16 03:51:38.729 INFO: Creating new model with cuequivariance settings
/scratch/user/djy/mace_env/lib/python3.10/site-packages/mace/modules/models.py:69: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  "atomic_numbers", torch.tensor(atomic_numbers, dtype=torch.int64)
/scratch/user/djy/mace_env/lib/python3.10/site-packages/torch/jit/_check.py:178: UserWarning: The TorchScript type system doesn't support instance-level annotations on empty non-base types in `__init__`. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in `torch.jit.Attribute`.
  warnings.warn(
/scratch/user/djy/mace_env/lib/python3.10/site-packages/torch/jit/_check.py:178: UserWarning: The TorchScript type system doesn't support instance-level annotations on empty non-base types in `__init__`. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in `torch.jit.Attribute`.

  warnings.warn(
Traceback (most recent call last):
  File "/scratch/user/djy/mace_djy/try_crystal_Cr_random01_comp_expa_optim_cuda_E0s_average_force10energy100_weight_fineturn/run_train.py", line 858, in <module>
    main()
  File "/scratch/user/djy/mace_djy/try_crystal_Cr_random01_comp_expa_optim_cuda_E0s_average_force10energy100_weight_fineturn/run_train.py", line 66, in main
    run(args)
  File "/scratch/user/djy/mace_djy/try_crystal_Cr_random01_comp_expa_optim_cuda_E0s_average_force10energy100_weight_fineturn/run_train.py", line 593, in run
    model = run_e3nn_to_cueq(deepcopy(model), device=device)
  File "/scratch/user/djy/mace_env/lib/python3.10/site-packages/mace/cli/convert_e3nn_cueq.py", line 151, in run
    transfer_weights(source_model, target_model, max_L, correlation)
  File "/scratch/user/djy/mace_env/lib/python3.10/site-packages/mace/cli/convert_e3nn_cueq.py", line 111, in transfer_weights
    target_model.load_state_dict(target_dict)
  File "/scratch/user/djy/mace_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2581, in load_state_dict
    raise RuntimeError(
RuntimeError: Error(s) in loading state_dict for ScaleShiftMACE:
        Unexpected key(s) in state_dict: "products.0.symmetric_contractions.weight", "products.1.symmetric_contractions.weight".

2. When I used the CPU, it still reminded me that cuequivariance or cuequivariance_torch is not available. Cuequivariance acceleration will be disabled. But it can run I confused.

 _Jd, _W3j_flat, _W3j_indices = torch.load(os.path.join(os.path.dirname(__file__), 'constants.pt'))
**cuequivariance or cuequivariance_torch is not available. Cuequivariance acceleration will be disabled.**
2025-02-16 03:24:01.924 INFO: ===========VERIFYING SETTINGS===========
2025-02-16 03:24:01.926 INFO: MACE version: 0.3.10
2025-02-16 03:24:01.927 INFO: Using CPU
2025-02-16 03:24:05.588 INFO: Using foundation model mace-mp-0 medium as initial checkpoint.

Could someone help me with this issue? Thank you!

[Swiftily]

Run conda list or pip list and check the cuda version of pytorch, it must match the version of cuequivariance_torch. I mean though you say pip install cuequivariance_ops_torch_cu12 gives an error, it doesn't make sense to install cuequivariance_ops_torch_cu11 if your pytorch utilizes cu12.

[Swiftily]

Ok I've seen it :
2025-02-16 03:51:34.489 INFO: CUDA version: 12.4, CUDA device: 0
So your cuda verison is 12.4 and my suggestion is downloading the .whl file from PyPI and install it with pip.
Try this website https://files.pythonhosted.org/packages/32/e0/fd6b7402d445eaace56a2cb06eacf53f6a8563c64baf79b26c8ed365592d/cuequivariance_ops_torch_cu12-0.2.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Something like this:
pip install xxx.whl
One more thing, run uname -a to verify your HPC's platform is x86_64 architecture cause now cueq acceleration only supports this platform

[treasureUu]

Obviously, your current glibc version is 2.17, but cueq-cu12 requires glibc 2.27 or 2.28. So you have two options:

1.Create a new environment: Install a compatible version of PyTorch with CUDA 11 and use cueq-cu11. 2.Upgrade glibc: It is not recommended. This could lead other programs to crash.

Thank you very much for your guidance; it's really helpful! I plan to follow your first suggestion, but I have a question regarding the use of CUDA. When I run the command nvidia-smi, it shows CUDA Version: 12.2.

On the school's server, I need to load a CUDA module. Should I load CUDA version 12.4.0, or could I also use CUDA 11.8.0? Both options are available to me.
I've tried loading the CUDA version 12.4.0 module and the CUDA 11.8.0 module separately, but neither can run, I got the same error: cuequivariance or cuequivariance_torch is not available. Cuequivariance acceleration will be disabled.

I am unclear about how these versions relate to the CUDA version displayed by the nvidia-smi command. Could you please help me understand this connection?

[Swiftily]

I think you've been misled by some online tutorials. PyTorch doesn't use the system's CUDA module; it only uses the runtime bundled with the installation. Unless you must compile the source code with CUDA for some specific needs, you'll never use CUDA toolkit or module. To create an environment for PyTorch, you can just do these following things:

1. run nvidia-smi. If these is output then we can say your machine has installed the driver of GPU( NOT CUDA);
2. Go to the official website of PyTorch and select a compatible version. This step you only need run something like this
   pip install torch==2.4.1 --index-url https://download.pytorch.org/whl/cu118
   It will install all you need.
   Remember that the output of nvidia-smi, here the "CUDA Version: 12.2", has nothing to do with PyTorch.
   A simple path you can try:
   conda create --name mace python=3.10
conda activate mace
pip install torch==2.4.1 --index-url https://download.pytorch.org/whl/cu118
pip install mace-torch
pip install cuequivariance-torch
pip install cuequivariance_ops_torch_cu11
   You don't need to load CUDA module during this.

[treasureUu]

I think you've been misled by some online tutorials. PyTorch doesn't use the system's CUDA module; it only uses the runtime bundled with the installation. Unless you must compile the source code with CUDA for some specific needs, you'll never use CUDA toolkit or module. To create an environment for PyTorch, you can just do these following things:

1. run nvidia-smi. If these is output then we can say your machine has installed the driver of GPU( NOT CUDA);
2. Go to the official website of PyTorch and select a compatible version. This step you only need run something like this
   pip install torch==2.4.1 --index-url https://download.pytorch.org/whl/cu118
   It will install all you need.
   Remember that the output of nvidia-smi, here the "CUDA Version: 12.2", has nothing to do with PyTorch.
   A simple path you can try:
   conda create --name mace python=3.10
conda activate mace
pip install torch==2.4.1 --index-url https://download.pytorch.org/whl/cu118
pip install mace-torch
pip install cuequivariance-torch
pip install cuequivariance_ops_torch_cu11
   You don't need to load CUDA module during this.

Thank you for the suggestions; they were very helpful!
I carefully followed each step, and the error 'cuequivariance or cuequivariance_torch is not available. Cuequivariance acceleration will be disabled' no longer appears.

However, a new error has arisen: 😔

2025-02-23 23:53:32.780 INFO: Total number of parameters: 699674
2025-02-23 23:53:32.781 INFO:
2025-02-23 23:53:32.781 INFO: ===========OPTIMIZER INFORMATION===========
2025-02-23 23:53:32.781 INFO: Using ADAM as parameter optimizer
2025-02-23 23:53:32.781 INFO: Batch size: 2
2025-02-23 23:53:32.781 INFO: Number of gradient updates: 2244
2025-02-23 23:53:32.782 INFO: Learning rate: 0.0005, weight decay: 5e-07
2025-02-23 23:53:32.782 INFO: WeightedEnergyForcesLoss(energy_weight=100.000, forces_weight=10.000)
2025-02-23 23:53:32.782 INFO: Converting model to CUEQ for accelerated training
2025-02-23 23:53:33.981 INFO: Creating new model with cuequivariance settings
/scratch/user/djy/mace_envdjy/lib/python3.10/site-packages/mace/modules/models.py:69: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  "atomic_numbers", torch.tensor(atomic_numbers, dtype=torch.int64)
Traceback (most recent call last):
  File "/scratch/user/djy/mace_djy/try_crystal_Cr_random01_comp_expa_optim_cuda_E0s_average_force10energy100_weight_fineturn/run_train.py", line 858, in <module>
    main()
  File "/scratch/user/djy/mace_djy/try_crystal_Cr_random01_comp_expa_optim_cuda_E0s_average_force10energy100_weight_fineturn/run_train.py", line 66, in main
    run(args)
  File "/scratch/user/djy/mace_djy/try_crystal_Cr_random01_comp_expa_optim_cuda_E0s_average_force10energy100_weight_fineturn/run_train.py", line 593, in run
    model = run_e3nn_to_cueq(deepcopy(model), device=device)
  File "/scratch/user/djy/mace_envdjy/lib/python3.10/site-packages/mace/cli/convert_e3nn_cueq.py", line 148, in run
    target_model = source_model.__class__(**config).to(device)
  File "/scratch/user/djy/mace_envdjy/lib/python3.10/site-packages/mace/modules/models.py", line 337, in __init__
    super().__init__(**kwargs)
  File "/scratch/user/djy/mace_envdjy/lib/python3.10/site-packages/mace/modules/models.py", line 85, in __init__
    self.node_embedding = LinearNodeEmbeddingBlock(
  File "/scratch/user/djy/mace_envdjy/lib/python3.10/site-packages/mace/modules/blocks.py", line 50, in __init__
    self.linear = Linear(
  File "/scratch/user/djy/mace_envdjy/lib/python3.10/site-packages/mace/modules/wrapper_ops.py", line 102, in __new__
    instance = cuet.Linear(
TypeError: Linear.__init__() got an unexpected keyword argument 'optimize_fallback'

[Swiftily]

This might be caused by a naming error in a MACE attribute. I have already reported the bug to maintainer. To resolve your issue, you can uninstall the current cuequivariance-torch and cuequivariance-ops-torch-cu11 versions, and install version 0.1.0 instead.

[treasureUu]

This might be caused by a naming error in a MACE attribute. I have already reported the bug to maintainer. To resolve your issue, you can uninstall the current cuequivariance-torch and cuequivariance-ops-torch-cu11 versions, and install version 0.1.0 instead.

I'm super excited to say that your method worked perfectly! 🤩

I checked the /mace_env/lib/python3.10/site-packages directory for the versions of cuequivariance I had installed earlier, such as cuequivariance_ops_torch_cu11-0.2.0.dist-info, and they were indeed version 0.2.0.

First, I uninstalled the previous versions:

pip uninstall cuequivariance-torch 
pip uninstall cuequivariance-ops-torch-cu11

Then, following your suggestion, I installed version 0.1.0:

pip install cuequivariance-torch==0.1.0
pip install cuequivariance-ops-torch-cu11==0.1.0

Thank you so much for your help over these past few days; 😄I can now use the GPU for computations normally.