Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stdenv.mkDerivation: patches with renames fail to apply #370363

Closed
phanirithvij opened this issue Jan 2, 2025 · 8 comments
Closed

stdenv.mkDerivation: patches with renames fail to apply #370363

phanirithvij opened this issue Jan 2, 2025 · 8 comments
Labels
0.kind: bug Something is broken 6.topic: stdenv Standard environment

Comments

@phanirithvij
Copy link
Member

phanirithvij commented Jan 2, 2025 •
edited
Loading

Describe the bug

stdenv.mkDerivation patches fail to apply with the following case

Steps To Reproduce

Steps to reproduce the behavior:

{
  nixpkgs ? fetchTarball "https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz",
  pkgs ? import nixpkgs { },
}:
pkgs.stdenv.mkDerivation {
  preferLocalBuild = true;
  allowSubstitutes = false;
  phases = "unpackPhase patchPhase installPhase";
  installPhase = "cp -R ./ $out";
  src = nixpkgs;
  name = "nixpkgs-patched";
  patches =
    builtins.map pkgs.fetchpatch2 [
      {
        url = "https://github.com/NixOS/nixpkgs/pull/369415.diff?full_index=1";
        hash = "sha256-KN1megdQEm1R1L+0GfZF1hj+8UMm0gsnEtplDGIMvHY=";
      }
      {
        url = "https://github.com/NixOS/nixpkgs/pull/370188.diff?full_index=1";
        hash = "sha256-++5G/uuadsdTelaB/D5oVmN4J4dr/7CVOG6YF0L3Wac=";
      }
    ]
    # can't use fetchpatch2 with patches which have file renames, see nixpkgs#32084 nixpkgs#266556
    ++ builtins.map pkgs.fetchurl [
      {
        url = "https://github.com/kachick/nixpkgs/pull/3.diff?full_index=1";
        hash = "sha256-CGvpk5ZEI3FVe/uH+0I5PkNTg97eqxDp9Y9HuCE7MjE=";
      }
    ];
}
See error: 
$ ls $(nix-build applypatches.nix --no-out-link)/pkgs/by-name/dp/dprint/plugins/
this derivation will be built:
  /nix/store/i220wjlyf3pqg9cb9frmv0vxxw1y9smk-nixpkgs-patched.drv
building '/nix/store/i220wjlyf3pqg9cb9frmv0vxxw1y9smk-nixpkgs-patched.drv'...
Running phase: unpackPhase
unpacking source archive /nix/store/il9cqlynm0bh10i1hg1hcl0xz48v0lkf-8fwsiv0hd7nw1brkvka0jf1frk3m7qkr-source
source root is 8fwsiv0hd7nw1brkvka0jf1frk3m7qkr-source
Running phase: patchPhase
applying patch /nix/store/yqmrjc13zf449xv31c6b9ar0z9vwdm1w-369415.diff?full_index=1
patching file pkgs/top-level/all-packages.nix
Hunk #1 succeeded at 5963 (offset 44 lines).
patching file pkgs/by-name/dp/dprint/plugins/default.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-biome.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-dockerfile.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-json.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-jupyter.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-markdown.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-ruff.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-toml.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-typescript.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-malva.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-markup_fmt.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-pretty_graphql.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-pretty_yaml.nix
patching file pkgs/by-name/dp/dprint/plugins/update-plugins.py
applying patch /nix/store/a8nfhzrqs3ly4zjqrf9ibp2g4fsi03x6-370188.diff?full_index=1
patching file pkgs/by-name/dp/dprint/plugins/default.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-biome.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-dockerfile.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-json.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-jupyter.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-markdown.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-ruff.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-toml.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-typescript.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-malva.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-markup_fmt.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-pretty_graphql.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-pretty_yaml.nix
patching file pkgs/by-name/dp/dprint/plugins/example.css
patching file pkgs/by-name/dp/dprint/plugins/example.graphql
patching file pkgs/by-name/dp/dprint/plugins/example.html
patching file pkgs/by-name/dp/dprint/plugins/example.ipynb
patching file pkgs/by-name/dp/dprint/plugins/example.yaml
applying patch /nix/store/0b8j4p002ixixhs9rlvq2sd4y3yrygv9-3.diff?full_index=1
patching file pkgs/by-name/dp/dprint/plugins/default.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-biome.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-dockerfile.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-json.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-jupyter.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-markdown.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-ruff.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-toml.nix
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-typescript.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-malva.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-markup_fmt.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-pretty_graphql.nix
patching file pkgs/by-name/dp/dprint/plugins/g-plane-pretty_yaml.nix
patching file pkgs/by-name/dp/dprint/plugins/testdata/example.css (renamed from pkgs/by-name/dp/dprint/plugins/example.css)
Cannot rename file without two valid file names
patching file pkgs/by-name/dp/dprint/plugins/testdata/example.graphql (renamed from pkgs/by-name/dp/dprint/plugins/example.graphql)
patching file pkgs/by-name/dp/dprint/plugins/testdata/example.html (renamed from pkgs/by-name/dp/dprint/plugins/example.html)
patching file pkgs/by-name/dp/dprint/plugins/testdata/example.ipynb (renamed from pkgs/by-name/dp/dprint/plugins/example.ipynb)
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
patching file pkgs/by-name/dp/dprint/plugins/testdata/example.yaml (renamed from pkgs/by-name/dp/dprint/plugins/example.yaml)
patching file pkgs/by-name/dp/dprint/plugins/update-plugins.py
error: builder for '/nix/store/i220wjlyf3pqg9cb9frmv0vxxw1y9smk-nixpkgs-patched.drv' failed with exit code 1;

I am trying fetchurl above because fetchpatch2 doesn't work for renames, it is a known issue. #32084

Additional context

the below does run successfully

# nixpkgs local checkout, git fetch once to get latest master
git checkout aa1627035626b9a47989b83c70d13859ea2d88da
curl -L https://github.com/NixOS/nixpkgs/pull/369415.diff | git apply -
curl -L https://github.com/NixOS/nixpkgs/pull/370188.diff | git apply -
curl -L https://github.com/kachick/nixpkgs/pull/3.diff | git apply -
nix build -f default.nix dprint-plugins --print-out-paths --no-link

Initially I was using applyPatches, which is just a mkDerivation call.

Expected behavior

Expect patches with renames to be handled properly in stdenv.mkDerivation.

Metadata

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.66, NixOS, 25.05 (Warbler), 25.05pre-git`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.11`
 - nixpkgs: `/nix/store/8fwsiv0hd7nw1brkvka0jf1frk3m7qkr-source`

Notify maintainers

@NixOS/stdenv

Apologies in advance if it is a known issue, In my search I couldn't find one
@phanirithvij phanirithvij added 0.kind: bug Something is broken 6.topic: stdenv Standard environment labels Jan 2, 2025
@phanirithvij phanirithvij changed the title patches with renames fail to apply stdenv.mkDerivation: patches with renames fail to apply Jan 2, 2025
@emilazy
Copy link
Member

emilazy commented Jan 2, 2025

I am trying fetchurl above because fetchpatch2 doesn't work for renames, it is a known issue. #32084

That’s fetchpatch. fetchpatch2 can do renames.

@phanirithvij
Copy link
Member Author

phanirithvij commented Jan 2, 2025 •
edited
Loading

I've tried with fetchpatch2 first but got wrong output

ls $(nix-build applypatches.nix --no-out-link)/pkgs/by-name/dp/dprint/plugins

# wrong output compared to the git apply one above

{
  nixpkgs ? fetchTarball "https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz",
  pkgs ? import nixpkgs { },
}:
pkgs.applyPatches {
  src = nixpkgs;
  name = "nixpkgs-patched";
  patches =
    builtins.map pkgs.fetchpatch2 [
      {
        url = "https://github.com/NixOS/nixpkgs/pull/369415.diff?full_index=1";
        hash = "sha256-KN1megdQEm1R1L+0GfZF1hj+8UMm0gsnEtplDGIMvHY=";
      }
      {
        url = "https://github.com/NixOS/nixpkgs/pull/370188.diff?full_index=1";
        hash = "sha256-++5G/uuadsdTelaB/D5oVmN4J4dr/7CVOG6YF0L3Wac=";
      }
      {
        url = "https://github.com/kachick/nixpkgs/pull/3.diff?full_index=1";
        hash = "sha256-QIpzjnsr8d+cXwjdXgehxg2FFai3MTt55yA/ZUMGSfQ=";
      }
    ];
}

@emilazy
Copy link
Member

emilazy commented Jan 2, 2025 •
edited
Loading

You need to update the hashes when you change fetcher.

(I’m not 100% sure that patch(1) can handle renames, though, but I think it can?)

@phanirithvij
Copy link
Member Author

phanirithvij commented Jan 2, 2025 •
edited
Loading

You need to update the hashes when you change fetcher.

I have.

unrelated but I also edited the original post to add installPhase, etc. it was previously a broken derivation.

(I’m not 100% sure that patch(1) can handle renames, though, but I think it can?)

I haven't tried it with patch command, I was thinking git apply uses it under the hood. idk

@r-vdp
Copy link
Contributor

r-vdp commented Jan 7, 2025 •
edited
Loading

If you manually run the commands that fetchpatch2 runs (lsdiff and filterdiff), you can see that indeed pure renames without content changes are dropped by both commands. This is an upstream issue though. Solving this in nixpkgs would probably be hacky.

This is also mentioned in #266556.

@phanirithvij
Copy link
Member Author

phanirithvij commented Jan 8, 2025 •
edited
Loading

I have used fetchurl and not fetchpatch2 for the last patch. (only the last patch above has renames)

so it should not use lsdiff or filterdiff. (stdenv.mkDerivation uses the patch -p1 command. pkgs/stdenv/generic/setup.sh)

And I checked with patch -p1 and it is failing. So git apply is the only correct way to use it.

from the above pr you linked

fixes found in the wild:

anduril/jetpack-nixos@8bf417e

that seems like a solution

# in current master first pr/patch is already merged
$ curl -L https://github.com/NixOS/nixpkgs/pull/370188.diff | patch -p1
...
$ curl -L https://github.com/kachick/nixpkgs/pull/3.diff | patch -p1
patching file pkgs/by-name/dp/dprint/plugins/default.nix
Hunk #1 FAILED at 15.
Hunk #2 FAILED at 32.
2 out of 2 hunks FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/default.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-biome.nix
Hunk #1 FAILED at 15.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/dprint-plugin-biome.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-dockerfile.nix
Hunk #1 FAILED at 8.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/dprint-plugin-dockerfile.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-json.nix
Hunk #1 FAILED at 8.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/dprint-plugin-json.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-jupyter.nix
Hunk #1 FAILED at 8.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/dprint-plugin-jupyter.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-markdown.nix
Hunk #1 FAILED at 8.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/dprint-plugin-markdown.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-ruff.nix
Hunk #1 FAILED at 11.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/dprint-plugin-ruff.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-toml.nix
Hunk #1 FAILED at 8.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/dprint-plugin-toml.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/dprint-plugin-typescript.nix
Hunk #1 FAILED at 15.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/dprint-plugin-typescript.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/g-plane-malva.nix
Hunk #1 FAILED at 13.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/g-plane-malva.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/g-plane-markup_fmt.nix
Hunk #1 FAILED at 18.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/g-plane-markup_fmt.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/g-plane-pretty_graphql.nix
Hunk #1 FAILED at 11.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/g-plane-pretty_graphql.nix.rej
patching file pkgs/by-name/dp/dprint/plugins/g-plane-pretty_yaml.nix
Hunk #1 FAILED at 11.
1 out of 1 hunk FAILED -- saving rejects to file pkgs/by-name/dp/dprint/plugins/g-plane-pretty_yaml.nix.rej
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
Cannot rename file without two valid file names
patching file pkgs/by-name/dp/dprint/plugins/update-plugins.py

@r-vdp
Copy link
Contributor

r-vdp commented Jan 8, 2025

Ah right, I misunderstood the issue.

Weird though, because I apply many patches to nixpkgs in the same way and I'm not running into the same issue.

@phanirithvij
Copy link
Member Author

phanirithvij commented Jan 8, 2025 •
edited
Loading

I don't have the time to find the commits which produced this issue. (the pr got merged so the hashes changed)

hope this is a one off and already covered in the known issues about fetchpatch, fetchpatch2 linked above.

@phanirithvij phanirithvij closed this as not planned Won't fix, can't repro, duplicate, stale Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
0.kind: bug Something is broken 6.topic: stdenv Standard environment
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@r-vdp @emilazy @phanirithvij