Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement chunks builtin for makeDerivationParallel #995

Closed
dsalaza4 opened this issue Jan 13, 2023 · 0 comments
Closed

Implement chunks builtin for makeDerivationParallel #995

dsalaza4 opened this issue Jan 13, 2023 · 0 comments
Assignees

Comments

@dsalaza4
Copy link
Contributor

dsalaza4 commented Jan 13, 2023

makeDerivationParallel is great for local parallelization in just one machine.

Many times, the next step for improving performance is distributing workloads into several machines.

A chunks builtin would be great for this:

{
  chunks,
  inputs,
  makeDerivation,
  makeDerivationParallel,
}: let
numbers = [0 1 2 3 4 5 6 7 8 9];
myChunks =  chunks numbers 3; # [[0 1 2 3] [4 5 6] [7 8 9]]

buildNumber = n: makeDerivation {
  name = "build-number-${n}";
  env.envNumber = n;
  builder = ''
    echo "$envNumber"
    touch "$out"
  '';
};
in
  makeDerivationParallel {
    dependencies = builtins.map buildNumber (inputs.nixpkgs.lib.lists.elemAt myChunks 0);
    name = "build-numbers-0";
  }

With this logic one would be able to easily distribute workloads into several CI jobs, achieving distributed parallelization.

@dsalaza4 dsalaza4 self-assigned this Jan 13, 2023
dsalaza4 added a commit that referenced this issue Jan 13, 2023
github-actions bot added a commit that referenced this issue Jan 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant