Unneeded feedstock #2

FarhanTejani · 2018-07-19T02:00:19Z

I am arguing for the deletion of this feedstock for the following reasons:

it is the only PyTorch related feedstock in conda-forge (what use is torchvision without pytorch?)
it is rarely used with < 1.5k downloads for the latest version
it only contains the Linux build, no Mac or Windows
the PyTorch org channel already maintains all PyTorch related conda packages including this one
the PyTorch website and torchvision repo both direct users to install torchvision from their own channel so this feedstock will not be getting any exposure

nehaljwani · 2018-07-19T04:07:30Z

cc @jakirkham

I am okay with the proposal.

hmaarrfk · 2019-01-31T01:05:47Z

We got pytorch-cpu compiling on conda-forge

https://github.com/conda-forge/pytorch-cpu-feedstock

I still can't get windows to compile, but if somebody would like to help on that front it would be appreciated.

If you would make this recipe depend on pytorch-cpu then it would work out for conda-forge, at least on linux.

nlgranger · 2019-06-14T15:53:28Z

Presently, this package seems to depend on 'pytorch' whereas conda-forge only provides 'pytorch-cpu', this is confusing for users like me.

hmaarrfk · 2019-06-14T21:00:41Z

The reason I packaged it that way is that there is a huge difference in performance between the two. I figured at least this way, if people depend on pytorch, they will know that only pytorch-cpu exists on conda-forge. We can make a meta package pytorch that will dependent on either pytorch-cpu or pytorch-gpu,

I might be tempted to work on this soon.

jph00 · 2019-09-29T12:41:37Z

Having torchvision in conda-forge is big problem. It's not compatible with the official pytorch conda packages, is very out of date, and results in conflicts when attempting to install software that depends on torchvision, if conda-forge is configured with a high channel priority (as in the recommended install steps on the conda-forge site). In addition, most PyTorch users use GPU, which isn't supported in the conda-forge package.

What's the process for getting packages removed from conda-forge that cause problems? This issue was raised over a year ago, was ok'd by @nehaljwani, and then nothing happened. There's zero upside to having this on conda-forge, and many downsides.

hmaarrfk · 2019-09-29T14:23:23Z

I think the package could be updated to a more modern version.

It seems that the setup process now requires pytorch.

Conda forge is community lead and complicated software like pytorch is simply challenging to package in an ecosystem.

People expecting GPU performance will not find a package called pytorch from conda forge

jph00 · 2019-09-29T16:40:50Z

You're breaking a lot of downstream software by leaving this here, meaning that anyone wanting to use pytorch in its normal configuration can't use conda-forge. I strongly hope you reconsider, for the health of the conda-forge community and python more generally.

If you don't, I very much hope that the project leadership will consider removing it, or at least publish clear guidelines explaining how ecosystem-breaking packages like this can be appealed and on what basis such appeals will be considered.

hmaarrfk · 2019-09-29T16:46:03Z

I'm just having trouble understanding the use case.

Are you using

conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

.
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/mark/miniconda3/envs/pytorch

  added / updated specs:
    - cudatoolkit=10.0
    - pytorch
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cudatoolkit-10.0.130       |                0       261.2 MB
    intel-openmp-2019.4        |              243         729 KB
    libblas-3.8.0              |      12_openblas          10 KB  conda-forge
    libcblas-3.8.0             |      12_openblas          10 KB  conda-forge
    liblapack-3.8.0            |      12_openblas          10 KB  conda-forge
    mkl-2019.4                 |              243       131.2 MB
    ninja-1.9.0                |       h6bb024c_0         1.6 MB  conda-forge
    pytorch-1.2.0              |py3.7_cuda10.0.130_cudnn7.6.2_0       428.0 MB  pytorch
    torchvision-0.4.0          |       py37_cu100         9.8 MB  pytorch
    ------------------------------------------------------------
                                           Total:       832.5 MB

Then maybe

conda update --all

and that is breaking your system?

Why not add the pytorch channel with higher priority than the conda-forge channel? That seems like it should solve the issues.

FWIW: I've also started to build pytorch 1.2 CPU, conda-forge/pytorch-cpu-feedstock#8 Their tar balls didn't include the versioned dependencies, making it much harder to auto update

hmaarrfk · 2019-09-29T16:51:44Z

it seems that pytorch-cpu is a build time requirement for torchvision(-cpu????) they moved to tracking features, enabling a more streamlined install.

I agree that torchvision probably shouldn't have been added in before pytorch(-cpu) was built.

That said, I think that

conda config --add channels conda-forge
conda config --add channels pytorch

should enable pytorch to have highest priority, making the pytorch install good, while using conda-forge packages

The pytorch channel seems quite limited to software directly related to pytorch, and doesn't seem like it will add many conflicts
https://anaconda.org/pytorch/repo

jph00 · 2019-09-29T17:08:21Z

Yes it's possible to add an extra install step, which would be required by every single dependency on pytorch, to work around this breakage.

But better would be to remove this package so that everyone else doesn't have to work around the problem. It adds nothing, almost no-one uses it, and causes a lot of pain. I spent the last 24 hours trying to work around it in a way that doesn't require me to tell every single user to change their conda setup.

hmaarrfk · 2019-09-29T17:29:03Z

I really do understand your frustration. I'm sorry it has caused you so many problems.

I'm just trying to better understand your install steps.

Following the instructions from the pytorch website, I get the following:

(pytorch) mark@mark-owl:~$ conda install pytorch torchvision cpuonly -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/mark/miniconda3/envs/pytorch

  added / updated specs:
    - cpuonly
    - pytorch
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cpuonly-1.0                |                0           2 KB  pytorch
    intel-openmp-2019.4        |              243         729 KB
    mkl-2019.4                 |              243       131.2 MB
    ninja-1.9.0                |       h6bb024c_0         1.6 MB  conda-forge
    pytorch-1.2.0              |      py3.7_cpu_0        31.3 MB  pytorch
    torchvision-0.4.0          |         py37_cpu        13.2 MB  pytorch
    ------------------------------------------------------------
                                           Total:       178.1 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h516909a_1
  ca-certificates    conda-forge/linux-64::ca-certificates-2019.9.11-hecc5488_0
  certifi            conda-forge/linux-64::certifi-2019.9.11-py37_0
  cffi               conda-forge/linux-64::cffi-1.12.3-py37h8022711_0
  cpuonly            pytorch/noarch::cpuonly-1.0-0
  freetype           conda-forge/linux-64::freetype-2.10.0-he983fc9_1
  intel-openmp       pkgs/main/linux-64::intel-openmp-2019.4-243
  jpeg               conda-forge/linux-64::jpeg-9c-h14c3975_1001
  libblas            conda-forge/linux-64::libblas-3.8.0-12_openblas
  libcblas           conda-forge/linux-64::libcblas-3.8.0-12_openblas
  libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1006
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libgfortran-ng     pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0
  liblapack          conda-forge/linux-64::liblapack-3.8.0-12_openblas
  libopenblas        conda-forge/linux-64::libopenblas-0.3.7-h6e990d7_1
  libpng             conda-forge/linux-64::libpng-1.6.37-hed695b0_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  libtiff            conda-forge/linux-64::libtiff-4.0.10-h57b8799_1003
  lz4-c              conda-forge/linux-64::lz4-c-1.8.3-he1b5a44_1001
  mkl                pkgs/main/linux-64::mkl-2019.4-243
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  ninja              conda-forge/linux-64::ninja-1.9.0-h6bb024c_0
  numpy              conda-forge/linux-64::numpy-1.17.2-py37h95a1406_0
  olefile            conda-forge/noarch::olefile-0.46-py_0
  openssl            conda-forge/linux-64::openssl-1.1.1c-h516909a_0
  pillow             conda-forge/linux-64::pillow-6.1.0-py37h6b7be26_1
  pip                conda-forge/linux-64::pip-19.2.3-py37_0
  pycparser          conda-forge/linux-64::pycparser-2.19-py37_1
  python             conda-forge/linux-64::python-3.7.3-h33d41f4_1
  pytorch            pytorch/linux-64::pytorch-1.2.0-py3.7_cpu_0
  readline           conda-forge/linux-64::readline-8.0-hf8c457e_0
  setuptools         conda-forge/linux-64::setuptools-41.2.0-py37_0
  six                conda-forge/linux-64::six-1.12.0-py37_1000
  sqlite             conda-forge/linux-64::sqlite-3.29.0-hcee41ef_1
  tk                 conda-forge/linux-64::tk-8.6.9-hed695b0_1003
  torchvision        pytorch/linux-64::torchvision-0.4.0-py37_cpu
  wheel              conda-forge/linux-64::wheel-0.33.6-py37_0
  xz                 conda-forge/linux-64::xz-5.2.4-h14c3975_1001
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1006
  zstd               conda-forge/linux-64::zstd-1.4.0-h3b9ef0a_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
ninja-1.9.0          | 1.6 MB    | ############################################################### | 100%
mkl-2019.4           | 131.2 MB  | ############################################################### | 100%
cpuonly-1.0          | 2 KB      | ############################################################### | 100%
intel-openmp-2019.4  | 729 KB    | ############################################################### | 100%
torchvision-0.4.0    | 13.2 MB   | ############################################################### | 100%
pytorch-1.2.0        | 31.3 MB   | ############################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(pytorch) mark@mark-owl:~$ conda update --all
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

(pytorch) mark@mark-owl:~$ conda install torchvision
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

conda update --all leaves the torchvision package alone
conda install torchvision leaves the torchvision package alone

If your users have an old version of the package, then maybe provide the command:

conda install "torchvision>=0.4" "pytorch>=1.2" -c pytorch

or similar.

Can you provide me with a list install steps that are able to recreate the issue for you?

I don't have the powers to delete this package, (and I will likely reupload the 0.4 package when pytorch 1.2 gets built), so I am very interested in learning more about your usecase

Feel free to ping the core team to delete the package, but they are swamped
https://conda-forge.org/docs/orga/guidelines.html#fixing-broken-packages
and if they miss your ping, then it can take even more time.

hmaarrfk · 2019-09-29T17:32:54Z

And this is my config

(pytorch) mark@mark-owl:~$ conda config --show
add_anaconda_token: True
add_pip_as_python_dependency: True
aggressive_update_packages:
  - ca-certificates
  - certifi
  - openssl
allow_conda_downgrades: False
allow_cycles: True
allow_non_channel_urls: False
allow_softlinks: False
always_copy: False
always_softlink: False
always_yes: None
anaconda_upload: None
auto_activate_base: True
auto_update_conda: True
bld_path:
changeps1: True
channel_alias: https://conda.anaconda.org
channel_priority: flexible
channels:
  - conda-forge
  - defaults
client_ssl_cert: None
client_ssl_cert_key: None
clobber: False
conda_build: {}
create_default_packages: []
croot: /home/mark/miniconda3/conda-bld
custom_channels:
  pkgs/main: https://repo.anaconda.com
  pkgs/r: https://repo.anaconda.com
  home/mark/miniconda3/conda-bld: file://
  pkgs/pro: https://repo.anaconda.com
custom_multichannels:
  defaults:
    - https://repo.anaconda.com/pkgs/main
    - https://repo.anaconda.com/pkgs/r
  local:
    - file:///home/mark/miniconda3/conda-bld
debug: False
default_channels:
  - https://repo.anaconda.com/pkgs/main
  - https://repo.anaconda.com/pkgs/r
default_python: 3.7
default_threads: None
deps_modifier: not_set
dev: False
disallowed_packages: []
download_only: False
dry_run: False
enable_private_envs: False
env_prompt: ({default_env})
envs_dirs:
  - /home/mark/miniconda3/envs
  - /home/mark/.conda/envs
error_upload_url: https://conda.io/conda-post/unexpected-error
execute_threads: 1
extra_safety_checks: False
force: False
force_32bit: False
force_reinstall: False
force_remove: False
ignore_pinned: False
json: False
local_repodata_ttl: 1
migrated_channel_aliases: []
migrated_custom_channels: {}
non_admin_enabled: True
notify_outdated_conda: True
offline: False
override_channels_enabled: True
path_conflict: clobber
pinned_packages: []
pip_interop_enabled: False
pkgs_dirs:
  - /home/mark/miniconda3/pkgs
  - /home/mark/.conda/pkgs
proxy_servers: {}
quiet: False
remote_connect_timeout_secs: 9.15
remote_max_retries: 3
remote_read_timeout_secs: 60.0
repodata_fns:
  - current_repodata.json
  - repodata.json
repodata_threads: None
report_errors: None
restore_free_channel: False
rollback_enabled: True
root_prefix: /home/mark/miniconda3
safety_checks: warn
sat_solver: pycosat
separate_format_cache: False
shortcuts: True
show_channel_urls: None
solver_ignore_timestamps: False
ssl_verify: True
subdir: linux-64
subdirs:
  - linux-64
  - noarch
target_prefix_override:
track_features: []
unsatisfiable_hints: True
update_modifier: update_specs
use_index_cache: False
use_local: False
use_only_tar_bz2: False
verbosity: 0
verify_threads: 1
whitelist_channels: []

hmaarrfk · 2019-09-29T18:41:55Z

Just a note: It seems that as of 0.3, torchvision builds depend on cpu/gpu configurations.

jph00 · 2019-09-29T19:45:14Z

Thank you - I will follow those directions to request removal. Other than that, I don't have more time to spend on this unfortunately, since I've decided to continue to avoid conda-forge for now. I've tested it every 6 months since it was first released, to see if it gets to a point that it works without pain for users. Note that the issue is for maintaining and updating packages that *depend* on pytorch - not pytorch itself.

hmaarrfk · 2019-09-29T20:53:28Z

Depending on pytorch should be fine no?

I named it pytorch-cpu to explicitly avoid people meaning the GPU version and getting something they don't want.

Finally, you can specify the version of the package you need. In that case, if you specify - torchvision >=0.4 in your meta.yaml, then your package will not be able to install with the version from conda forge.

Users will have to have the pytorch channel enabled, but I guess that can be done with a -c flag.

hmaarrfk · 2019-09-29T21:05:30Z

I named it pytorch-cpu to explicitly avoid people meaning the GPU version and getting something they don't want.

At the time, this followed pytorch's convention, with the exception of the macos build.

jph00 · 2019-09-29T22:24:00Z

I don't have the powers to delete this package, (and I will likely reupload the 0.4 package when pytorch 1.2 gets built), so I am very interested in learning more about your usecase Feel free to ping the core team to delete the package, but they are swamped https://conda-forge.org/docs/orga/guidelines.html#fixing-broken-packages and if they miss your ping, then it can take even more time.

According to those instructions the request needs to come from you.

hmaarrfk · 2019-09-29T23:02:07Z

I know, even though I'm not a maintainer here, I guess I could ping. But honestly, I'm not convinced it is the best way to help you.

@jph00 I still think the correct way for you to resolve your package correctly is to restrict your required version of torchvision in the meta.yaml file.

Hi @conda-forge/core,

Please move this package from main to broken on the conda-forge channel on anaconda.org
because:

It was built with an external dependency.

jph00 · 2019-09-30T02:55:04Z

Many thanks. I don't actually need any help because we've solved the problem by avoiding conda-forge. However I am interested in seeing conda-forge play more nicely with the pytorch ecosystem, and have better policies and processes with interfacing with organizations that want to support the conda community more generally - especially since things like RapidsAI are now using conda-forge deps. If every package that uses pytorch pins the torchvision version then that would be a workaround for this particular problem. But it's not a great solution to the more general policies and processes issue which led us here.

hmaarrfk · 2019-09-30T03:27:10Z

does your software really support that many versions of pytorch/torchvision? With such a fast moving software, ~~I is really suprising~~ I find that really amazing!

rgommers · 2019-12-03T06:56:31Z

Hi @conda-forge/core,

Please move this package from main to broken on the conda-forge channel on anaconda.org
because:

Note: this got no response, this package is still installable. Ping @jakirkham and @jjhelmus, adding to the discussion for tomorrow. Deletion of this feedstock seems preferable to adding a broken label, but either of those would be helpful.

Maltimore · 2020-03-09T09:25:30Z

I believe the package is now finally broken, since it cannot import pillow 7.0.0, but pillow doesn't seem to be pinned to a lower version in this repo.

python-pillow/Pillow#4130

Maybe now would be a good time to reconsider deleting this package.

nehaljwani · 2020-03-09T09:39:33Z

@ocefpaf What's your opinion on this? I am OK with moving all builds to archive and marking this feedstock as deprecated (but don't have the powers to do so 😉)

ocefpaf · 2020-03-09T13:10:12Z

@ocefpaf What's your opinion on this? I am OK with moving all builds to archive and marking this feedstock as deprecated (but don't have the powers to do so )

All the packages here are in the epoch cf201901 label. Only people who know what to do can install that. @Maltimore if you are installing it from that label you want to reproduce an old env and you are expected to know that you may get broken env. There is no need to remove it from there and no users that relies on the main channel will get it.

Regarding archiving this feedstock, that is up to you @nehaljwani, as the sole maintainer here. If you want to do so I can archive it for you.

nehaljwani · 2020-03-09T13:21:48Z

Thanks for the quick response, @ocefpaf ! Please archive this feedstock.

isuruf · 2021-01-17T19:06:06Z

Unarchiving the feedstock as we have pytorch packages available in conda-forge and it's okay to have torchvision packages in conda-forge.

nehaljwani closed this as completed Jul 19, 2018

nehaljwani reopened this Jul 19, 2018

hmaarrfk mentioned this issue Sep 30, 2019

GPU Support conda-forge/pytorch-cpu-feedstock#7

Closed

isuruf closed this as completed Jan 17, 2021

conda-forge locked and limited conversation to collaborators Jan 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unneeded feedstock #2

Unneeded feedstock #2

FarhanTejani commented Jul 19, 2018 •

edited

Loading

nehaljwani commented Jul 19, 2018

hmaarrfk commented Jan 31, 2019 •

edited

Loading

nlgranger commented Jun 14, 2019

hmaarrfk commented Jun 14, 2019

jph00 commented Sep 29, 2019

hmaarrfk commented Sep 29, 2019

jph00 commented Sep 29, 2019

hmaarrfk commented Sep 29, 2019 •

edited

Loading

hmaarrfk commented Sep 29, 2019

jph00 commented Sep 29, 2019

hmaarrfk commented Sep 29, 2019

hmaarrfk commented Sep 29, 2019

hmaarrfk commented Sep 29, 2019

jph00 commented Sep 29, 2019 via email

hmaarrfk commented Sep 29, 2019 •

edited

Loading

hmaarrfk commented Sep 29, 2019

jph00 commented Sep 29, 2019 via email

hmaarrfk commented Sep 29, 2019

jph00 commented Sep 30, 2019 via email

hmaarrfk commented Sep 30, 2019 •

edited

Loading

rgommers commented Dec 3, 2019

Maltimore commented Mar 9, 2020

nehaljwani commented Mar 9, 2020

ocefpaf commented Mar 9, 2020

nehaljwani commented Mar 9, 2020

isuruf commented Jan 17, 2021

Unneeded feedstock #2

Unneeded feedstock #2

Comments

FarhanTejani commented Jul 19, 2018 • edited Loading

nehaljwani commented Jul 19, 2018

hmaarrfk commented Jan 31, 2019 • edited Loading

nlgranger commented Jun 14, 2019

hmaarrfk commented Jun 14, 2019

jph00 commented Sep 29, 2019

hmaarrfk commented Sep 29, 2019

jph00 commented Sep 29, 2019

hmaarrfk commented Sep 29, 2019 • edited Loading

hmaarrfk commented Sep 29, 2019

jph00 commented Sep 29, 2019

hmaarrfk commented Sep 29, 2019

hmaarrfk commented Sep 29, 2019

hmaarrfk commented Sep 29, 2019

jph00 commented Sep 29, 2019 via email

hmaarrfk commented Sep 29, 2019 • edited Loading

hmaarrfk commented Sep 29, 2019

jph00 commented Sep 29, 2019 via email

hmaarrfk commented Sep 29, 2019

jph00 commented Sep 30, 2019 via email

hmaarrfk commented Sep 30, 2019 • edited Loading

rgommers commented Dec 3, 2019

Maltimore commented Mar 9, 2020

nehaljwani commented Mar 9, 2020

ocefpaf commented Mar 9, 2020

nehaljwani commented Mar 9, 2020

isuruf commented Jan 17, 2021

FarhanTejani commented Jul 19, 2018 •

edited

Loading

hmaarrfk commented Jan 31, 2019 •

edited

Loading

hmaarrfk commented Sep 29, 2019 •

edited

Loading

hmaarrfk commented Sep 29, 2019 •

edited

Loading

hmaarrfk commented Sep 30, 2019 •

edited

Loading