cudaPackages: fix #220357; use -Xfatbin=-compress-all; prune default cudaCapabilities #220402
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ConnorBaker
force-pushed
the
fix/cuda-nvcc-compress-fatbins
branch
3 times, most recently
from
a84ce56 to
c6888a7
Compare
ConnorBaker
changed the title
ConnorBaker
changed the title
Closed
|
cc @NixOS/cuda-maintainers |
This was referenced Mar 10, 2023
samuela
reviewed
Mar 11, 2023
ConnorBaker
force-pushed
the
fix/cuda-nvcc-compress-fatbins
branch
from
c6888a7 to
bdfb61e
Compare
|
Result of 7 packages marked as broken and skipped:
31 packages failed to build:
36 packages built:
|
|
Failures: look reasonable to me. |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/tweag-nix-dev-update-45/26397/1 |
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
-Xfatbin=-compress-alltoNVCC_PREPEND_FLAGSto help ensure the generated libraries aren't so large that they fail to linkdontDefaultAfterattribute togpus.nixgpus.nixby makingmaxCudaVersiona string ornull, indicating there is no maximum supported versionmaxCudaVersionattributes ingpus.nixevery timeWe're experiencing the same problems Apache MXNet did due to the the number of capabilities we build for currently: apache/mxnet#19123. Unfortunately, just using
-Xfatbin=-compress-allisn't enough -- the number of CUDA capabilities we support is still so large that, without some pruning, Magma will fail to build.I think it'd be fair to remove support for Kepler and Maxwell since they're 11 and 9 years old, respectively. Kepler has been deprecated for most of CUDA 11 and is fully removed as of CUDA 12. Maxwell is still supported in CUDA 12, but they are increasingly old and rare.
Before:
After:
An added benefit of this PR: the closure and NAR serializations of CUDA-binaries and libraries are smaller.
Closure size with cudaCapabilities = [ "8.6" ];
From
/nix/store/ib9ckb796i7nqj0fy6iankvhyfc6az70-magma-2.7.1 429.4M 2.6Gto
/nix/store/r9dc4yf6nq17dl8bdav4hy7q0vd80s64-magma-2.7.1 233.6M 2.4GWith the following example (which, admittedly is just targeting a single capability), the size of the Magma NAR is nearly halved, from 429.4M to 233.6M.
With
the before closure:
and afterwards:
Closure size with cudaCapabilities = [ "6.0" "6.1" "6.2" "7.0" "7.2" "7.5" "8.0" "8.6" ];
NOTE: Even with the reduced number of capabilities this PR introduces, without
-Xfatbin=-compress-allMagma still fails to build.Master failed to build so there's only an after for this example:
/nix/store/dhqykhps4khbzr8aks84qlyxnhqxi066-magma-2.7.1 1.7G 3.9GWith a config to mimic the
cudaCapabilitiesthis PR would generatethe build of Magma against master failed with the linking error again! The build against this PR however, succeeded:
Closure size with cudaCapabilities = [ "7.5" "8.0" "8.6" ];
From
/nix/store/8q090bm2r3n4b58ygsbsnlj68rxb92vx-magma-2.7.1 1.2G 3.4Gto
/nix/store/1f78c0ihfrxdfbmc56jp4s9ql4kdd3bw-magma-2.7.1 659.9M 2.8GAnd one last comparison, this time building against only three capabilities so
mastercan succeed as well. With this configuration:Before closure:
After closure:
Things done
sandbox = trueset innix.conf? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)