Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAINT: Use mamba in CIs #11471

Merged
merged 11 commits into from
Feb 21, 2023
Merged

MAINT: Use mamba in CIs #11471

merged 11 commits into from
Feb 21, 2023

Conversation

larsoner
Copy link
Member

@larsoner larsoner commented Feb 13, 2023

One way to move toward being able to recommend mamba is by including it in environment.yml, so I added it here. I made our solvers use mamba by including it in the setup-miniconda action according to the instructions here. Note that they even say it's experimental, and I'm assuming that's because mamba is moreso than the action itself:

Example 6: Mamba
Experimental! Use mamba to handle conda installs in a faster way. mamba-version accepts a version string x.y (including "*"). It requires you specify conda-forge as part of the channels, ideally with the highest priority.

FWIW @drammock Azure was already using mamba env update on Windows, IIRC to speed up the solve time.

@larsoner larsoner changed the title MAINT: Include mamba in env MAINT: Include mamba in CIs and environment.yml Feb 13, 2023
@larsoner larsoner changed the title MAINT: Include mamba in CIs and environment.yml MAINT: Use mamba in CIs Feb 13, 2023
@larsoner
Copy link
Member Author

Okay, just asking it to install mamba as the solver broke things / created conflicts. This does not bode well for people trying conda install -c conda-forge mamba in their base environments :(

I'm going to try it now with better proper channel priority (as suggested in the setup-miniconda docs) and hope that works. I guess that's more in line with our instructions anyway, which suggest --strict-channel-priority at least some of the time.

@larsoner
Copy link
Member Author

... apparently this works on macOS but not Linux. I'm tempted to close for now and revisit in a few months hoping some conda, mamba, and/or setup-miniconda fixes land in the meantime

@drammock
Copy link
Member

drammock commented Feb 16, 2023

using mambaforge fixed compat-old and compat-minimal. Only notebook is failing now, with:

 Warning: 
  UnsatisfiableError: The following specifications were found
  to be incompatible with the existing python installation in your environment:
  
  Specifications:
  
    - defaults/linux-64::brotlipy==0.7.0=py310h7f8727e_1002[md5=a8dfa5aefcc49d5e78000b16d2b7714e] -> python[version='2.7.*|3.5.*|3.6.*|3.6.12|3.6.12|3.7.10|3.7.10|3.7.12|3.8.12|3.8.16|3.9.16|>=3.11,<3.12.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0|3.9.10|>=3.6,<3.7.0a0|3.7.9|3.6.9|3.6.9|3.6.9|>=2.7,<2.8.0a0|3.6.9|>=3.5,<3.6.0a0|3.4.*',build='0_73_pypy|4_73_pypy|0_73_pypy|1_73_pypy|0_73_pypy|5_73_pypy|5_73_pypy|3_73_pypy|2_73_pypy|1_73_pypy']
    - defaults/linux-64::conda-content-trust==0.1.3=py310h06a4308_0[md5=69f1c06e7c9ce52d4a9b5787c74cac20] -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.11,<3.12.0a0|>=3.8,<3.9.0a0|>=3.9,<3.10.0a0|>=3.7,<3.8.0a0|>=3.6,<3.7.0a0|>=3.5,<3.6.0a0|3.4.*']
    - defaults/linux-64::cryptography==38.0.1=py310h9ce1e76_0[md5=f9aba54a3e93124c31846c700f502f9d] -> python[version='3.6.12|3.6.12|3.7.10|3.7.10|3.7.12|3.8.12|3.8.16|3.9.16|>=3.11,<3.12.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0|3.9.10|>=3.6,<3.7.0a0|3.7.9|3.6.9|3.6.9|3.6.9|>=2.7,<2.8.0a0|3.6.9',build='0_73_pypy|4_73_pypy|0_73_pypy|1_73_pypy|0_73_pypy|5_73_pypy|5_73_pypy|3_73_pypy|2_73_pypy|1_73_pypy']
    - defaults/linux-64::ruamel.yaml==0.17.21=py310h5eee18b_0[md5=0a54e321078e043b72b6f68d10472b62] -> python[version='>=3.11,<3.12.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0']
  
  Your python: defaults/linux-64::python==3.10.8=h7a1cb2a_1[md5=c50aebbb632aea22b13bd654dfe80a5b]

[... a lot more omitted]

If I'm reading it correctly, those versions of those packages simply don't allow python 3.10 (though they do allow 3.11?) Should I try using 3.11 in the notebook image, or is that known to fail for other reasons?

@drammock
Copy link
Member

All green! mambaforge FTW. @larsoner merge if happy.

@drammock
Copy link
Member

actually, we should check the job timings. maybe I'm reading things wrong but it looks like the usage minutes went up for the last run of CIs on this PR, when compared to recent runs on #11473 (a randomly chosen PR for comparison). IDK how to query GH Actions timings efficiently though, maybe you do?

@larsoner
Copy link
Member Author

Okay I restarted linux_conda, compat_minimal, and compat_old and will check against runs from #11479 which are happening concurrently

@larsoner
Copy link
Member Author

Timings are 6min / 2min / 2min, respectively, on this PR, and 8min / 1min / 1min on #11479 (roughly). But macOS improves from 11 to 7 min on this PR, too, which is nice. Since it's at least as fast overall and tests mamba let's merge and continue!

@larsoner larsoner merged commit c93c848 into mne-tools:main Feb 21, 2023
@larsoner larsoner deleted the mamba branch February 21, 2023 16:35
@agramfort
Copy link
Member

🙏 @larsoner

larsoner added a commit to drammock/mne-python that referenced this pull request Feb 22, 2023
* upstream/main: (46 commits)
  Fix docstrings by replacing str with path-like and fix double backticks for formatting (mne-tools#11499)
  Use pathlib.Path instead of os.path to handle files and folders [circle deploy] (mne-tools#11473)
  MAINT: Fix Circle [circle deploy] (mne-tools#11497)
  MAINT: Use mamba in CIs (mne-tools#11471)
  Updating documentation to clarify full vs half-bandwidth and defaults in time_frequency.multitaper.py (mne-tools#11479)
  Fix typo in tutorial (mne-tools#11498)
  Typo fix and added colons before code (mne-tools#11496)
  [MRG] ENH/DOC: demo custom spectrum creation (mne-tools#11493)
  Accept only left-clicks for adding annotations (mne-tools#11491)
  [BUG, MRG] Fix pial surface loading, logging in locate_ieeg (mne-tools#11489)
  [ENH] Added unit_role to add non-breaking space between magnitude  and units (mne-tools#11469)
  MAINT: Fix CircleCI build (mne-tools#11488)
  [DOC] Updated decoding.SSD documentation and internal variable naming (mne-tools#11475)
  Typo fix (mne-tools#11485)
  [MRG] Forward argument axes from plot_sensors to DigMontage.plot (mne-tools#11470)
  [MRG] Improve error message raised on channels missing from DigMontage (mne-tools#11472)
  MAINT: Deal with pkg_resources usage bugs (mne-tools#11478)
  Add object array support and docstring (mne-tools#11465)
  [ENH] Adjusted SSD algorithm to support non-full rank data (mne-tools#11458)
  [BUG] fix nibabel reference (mne-tools#11467)
  ...
larsoner added a commit to larsoner/mne-python that referenced this pull request Mar 1, 2023
* upstream/main: (264 commits)
  BUG: Fix deprecated API usage in example (mne-tools#11512)
  Deprecate 'kind' and 'path' in favor of 'fname' in the layout reader (mne-tools#11500)
  EGI/MFF events outside EEG recording should not break the code (mne-tools#11505)
  fixed annotations error on export (mne-tools#11435)
  DOC: Update installer links [skip azp] [skip actions] [skip cirrus] (mne-tools#11506)
  BUG: updates for MPL 3.7 compatibility (mne-tools#11409)
  Fix docstrings by replacing str with path-like and fix double backticks for formatting (mne-tools#11499)
  Use pathlib.Path instead of os.path to handle files and folders [circle deploy] (mne-tools#11473)
  MAINT: Fix Circle [circle deploy] (mne-tools#11497)
  MAINT: Use mamba in CIs (mne-tools#11471)
  Updating documentation to clarify full vs half-bandwidth and defaults in time_frequency.multitaper.py (mne-tools#11479)
  Fix typo in tutorial (mne-tools#11498)
  Typo fix and added colons before code (mne-tools#11496)
  [MRG] ENH/DOC: demo custom spectrum creation (mne-tools#11493)
  Accept only left-clicks for adding annotations (mne-tools#11491)
  [BUG, MRG] Fix pial surface loading, logging in locate_ieeg (mne-tools#11489)
  [ENH] Added unit_role to add non-breaking space between magnitude  and units (mne-tools#11469)
  MAINT: Fix CircleCI build (mne-tools#11488)
  [DOC] Updated decoding.SSD documentation and internal variable naming (mne-tools#11475)
  Typo fix (mne-tools#11485)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants