numpy 2 migrator fails during CHECKING RECIPE SOLVABLE

[h-vetinari]

I had been wondering why no PRs for the numpy 2 migration had been opened yet.

Looking at the most recent bot run, I see that the rerender works fine, but then it fails during CHECKING RECIPE SOLVABLE:

Running migrations for MigrationYaml-numpy2: 280
  
  2024-05-04 18:52:17,961 INFO     conda_forge_tick.auto_tick || MIGRATIONYAML-numpy2 IS MIGRATING pandas:main
  
  error: pathspec 'rebuild-numpy2-0-1_h4f1b37' did not match any file(s) known to git
  2024-05-04 18:52:20,057 INFO     conda_forge_tick.migrators.cross_compile || new conda-forge.yml for pandas:={'azure': {'store_build_artifacts': True}, 'build_platform': {'linux_ppc64le': 'linux_64', 'osx_arm64': 'osx_64', 'linux_aarch64': 'linux_64'}, 'conda_build': {'pkg_format': '2'}, 'conda_forge_output_validation': True, 'github': {'branch_name': 'main', 'tooling_branch_name': 'main'}, 'provider': {'linux_aarch64': 'default', 'linux_ppc64le': 'azure', 'win': 'azure'}, 'test': 'native_and_emulated'}
  2024-05-04 18:52:20,075 INFO     conda_forge_tick.auto_tick || Rerendering the feedstock
  INFO:conda_smithy.configure_feedstock:Downloading conda-forge-pinning-2024.05.04.18.06.14
  INFO:conda_smithy.configure_feedstock:Extracting conda-forge-pinning to /tmp/tmp_arr1gyv/conda-smithy
  INFO:conda_smithy.configure_feedstock:__pycache__ rendering is skipped
  INFO:conda_smithy.configure_feedstock:README rendering is skipped
  INFO:conda_smithy.configure_feedstock:numpy2.yaml from feedstock is ignored and upstream version is used
  INFO:conda_smithy.configure_feedstock:python312.yaml from feedstock is ignored and upstream version is used
  INFO:conda_smithy.configure_feedstock:pypy38.yaml from feedstock is ignored and upstream version is used
  WARNING: Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
  WARNING: Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
  WARNING: No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.23
  Adding in variants from internal_defaults
  Adding in variants from /tmp/tmp_arr1gyv/conda-smithy/conda_build_config.yaml
  INFO:conda_smithy.configure_feedstock:Applying migrations: /tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/pypy38.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/python312.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/numpy2.yaml
  Adding in variants from argument_variants
  INFO:conda_smithy.configure_feedstock:Applying migrations: /tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/pypy38.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/python312.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/numpy2.yaml
  INFO:conda_smithy.configure_feedstock:Applying migrations: /tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/pypy38.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/python312.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/numpy2.yaml
  INFO:conda_smithy.configure_feedstock:Applying migrations: /tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/pypy38.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/python312.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/numpy2.yaml
  INFO:conda_smithy.configure_feedstock:Applying migrations: /tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/pypy38.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/python312.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/numpy2.yaml
  INFO:conda_smithy.configure_feedstock:Applying migrations: /tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/pypy38.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/python312.yaml,/tmp/tmp_arr1gyv/conda-smithy/share/conda-forge/migrations/numpy2.yaml
  INFO:conda_smithy.configure_feedstock:Re-rendered with conda-build 24.3.0, conda-smithy 3.35.0, and conda-forge-pinning 2024.05.04.18.06.14
  INFO:conda_smithy.configure_feedstock:You can commit the changes with:
  
      git commit -m "MNT: Re-rendered with conda-build 24.3.0, conda-smithy 3.35.0, and conda-forge-pinning 2024.05.04.18.06.14"
  
  INFO:conda_smithy.configure_feedstock:These changes need to be pushed to github!
  
  INFO:conda_forge_feedstock_check_solvable.mamba_solver:94:CHECKING FEEDSTOCK: pandas-feedstock
  INFO:conda_forge_feedstock_check_solvable.mamba_solver:94:CHECKING RECIPE SOLVABLE: linux_64_numpy1.22python3.9.____73_pypypython_implpypy.yaml
  2024-05-04 18:54:33,880 WARNING  conda_oci_mirror.logger || ORAS_USER or ORAS_PASS is missing, push may have issues.
  /home/runner/micromamba/envs/cf-scripts/lib/python3.11/site-packages/conda_oci_mirror/cache/sysroot_linux-64-2.12-he073ed8_17/info.tar.gz already exists with expected hash, not re-downloading.
  INFO:conda_forge_feedstock_check_solvable.mamba_solver:94:RUN EXPORTS: downloading package https://conda-static.anaconda.org/conda-forge/noarch/sysroot_linux-64-2.12-he073ed8_17.conda
  INFO:conda_forge_feedstock_check_solvable.mamba_solver:94:RUN EXPORT CACHE STATUS: CacheInfo(hits=0, misses=14, maxsize=10240, currsize=14)
  INFO:conda_forge_feedstock_check_solvable.mamba_solver:94:MAMBA SOLVER MEM USAGE: 3436 MB
  INFO:conda_forge_feedstock_check_solvable.mamba_solver:94:CHECKING RECIPE SOLVABLE: linux_64_numpy2.0python3.10.____cpythonpython_implcpython.yaml
  WARNING:conda_forge_feedstock_check_solvable.mamba_solver:94:MAMBA failed to solve specs 
  
  ['libgcc-ng >=12',
   'pip',
   'tomli',
   'numpy 2.0.*',
   'meson-python 0.13.1.*',
   'libstdcxx-ng >=12',
   'ninja',
   'versioneer',
   'meson 1.2.1.*',
   'python 3.10.* *_cpython',
   'cython >=3.0.5']
  
  for channels 
  
  ['file:///home/runner/work/_temp/tmp5xedr441', 'conda-forge', 'msys2']
  
  The reported errors are:
  
  Could not solve for environment specs
  The following package could not be installed
  └─ numpy 2.0.*  is not installable because it requires
     └─ _numpy_rc, which does not exist (perhaps a missing channel).

I'm not sure why/how this isn't picking up the channel_sources? Could this be related to conda-forge/conda-smithy#1911?

CC @beckermr

[beckermr]

This issue needs to be opened on conda-forge-check-solvable. You can see from the test you pasted that the numpy rc channel is not being used.

[beckermr]

We need to try that migration on the feedstock locally to see what ends up in the ci support files.

[h-vetinari]

You can check on the scipy or pythran feedstocks, those have already been rerendered (with use_local: true). Obviously the channel_sources are there, otherwise the ci would never have passed

[beckermr]

Ah right. Thanks for the reminder. Someone will have to debug this by hand on a failing case then.

[beckermr]

Yep. I just rerendered locally and the channel sources are wrong. It may indeed be the issue in smithy we had worked on.

[h-vetinari]

Should we pause the migration in the meantime?

[jakirkham]

If it's not producing any migration PRs, it is at some level already paused. It's just taking time on the bot that presumably could be allocated to something else. So yeah let's pause it

[beckermr]

See this PR: conda-forge/conda-forge-pinning-feedstock#5842

there is a bug in the migrator itself.

[h-vetinari]

there is a bug in the migrator itself.

Thank you very much, and sorry for the hassle!

(Did I ever mention my severe dislike for yaml as a format 🤔😅)

[beckermr]

@h-vetinari it appears to put in a manual exclude ~3 days ago for pandas and numpy2. Did the change in the migrator above not fix things?

[beckermr]

xref: conda-forge/conda-forge-pinning-feedstock@b2bf0f3

[beckermr]

In general, it'd be better to solve this issue if it is ongoing instead of special-casing around it. IDK for this case what is happening though.

[h-vetinari]

I added pandas because it was one of the biggest blockers, and after it got put into the "not resolvable" bucket, it would have taken a long time for it to be retried (AFAIU, all "awaiting PR" feedstocks get prioritized before retrying failed ones). We can remove it again, but that was in the interest of unblocking one of the biggest bottlenecks in the migration, rather than waiting for potentially weeks (we haven't manage to exhause the "awaiting PR" queue yet; still at ~200).

[beckermr]

Sure that makes sense, but just to confirm, the fix to the numpy2 migrator above didn't fix the solver error?

[h-vetinari]

just to confirm, the fix to the numpy2 migrator above didn't fix the solver error?

If you're referring to conda-forge/conda-forge-pinning-feedstock@b2bf0f3, then that kind of sidestepped the solver error, because now the bot will remove pandas from the list of "feedstocks I need to migrate", but will not consider dependent feedstocks blocked.

So it did achieve its intended effect (same thing for scipy).

[beckermr]

Sorry no I meant the fix I put in for the ordering but in the yaml before that.

[h-vetinari]

I still don't fully understand, but I think all that came afterwards. What happened is that after we restarted the migration with conda-forge/conda-forge-pinning-feedstock#5851 (5 days ago), pandas showed up as "not resolvable". That made no sense to me but I didn't investigate further, I just opened conda-forge/pandas-feedstock#207 (merged 4 days ago), and after I realised the bot will not pick up that change by itself without retrying, I put in conda-forge/conda-forge-pinning-feedstock@b2bf0f3 a couple hours later to unblock.

[h-vetinari]

after I realised the bot will not pick up that change by itself without retrying

This could be a good inspiration for retrying a feedstock faster, especially if it has a lot of children: if the main branch of that feedstock has received new commits since the last resolver/bot error, retry again immediately, without waiting for the rest of the queue (because often, it'll have been a manual fix for the migrator issue)

[beckermr]

Ok this is enough info for me to track this down. Thanks.

[h-vetinari]

I don't know how the bot works on pause/unpause, but it's possible that the "not resolvable" status was stale from when we briefly started the migrator ~2 weeks ago, before we paused it again due to the py312 issue