Add lightning #22249

weiji14 · 2023-03-08T20:15:05Z

Use Lightning Apps to build everything from production-ready, multi-cloud ML systems to simple research demos! Repo at https://github.com/Lightning-AI/lightning

Xref previous attempts at #19716 and #19715. Requires #22247.

Checklist

Use Lightning Apps to build everything from production-ready, multi-cloud ML systems to simple research demos! Repo at https://github.com/Lightning-AI/lightning

conda-forge-webservices · 2023-03-08T20:15:22Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/lightning) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipes/lightning:

Recipe with the same name exists in bioconda: please discuss with @conda-forge/bioconda-recipes.

recipes/lightning/meta.yaml

weiji14 · 2023-03-08T20:25:47Z

For recipes/lightning:

* Recipe with the same name exists in bioconda: please discuss with @conda-forge/bioconda-recipes.

Discussion at https://github.com/orgs/conda-forge/teams/bioconda-recipes/discussions/15

weiji14 · 2023-03-12T20:49:39Z

Build failing at https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=671385&view=logs&j=6f142865-96c3-535c-b7ea-873d86b887bd&t=22b0682d-ab9e-55d7-9c79-49f3c3ba4823&l=3700 as follows:

export PREFIX=/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh
export SRC_DIR=/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/test_tmp
import: 'lightning'
Traceback (most recent call last):
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/lightning/pytorch/strategies/bagua.py", line 29, in <module>
    from lightning.pytorch.plugins.precision import PrecisionPlugin
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/lightning/pytorch/plugins/__init__.py", line 7, in <module>
    from lightning.pytorch.plugins.precision.apex_amp import ApexMixedPrecisionPlugin
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/lightning/pytorch/plugins/precision/__init__.py", line 18, in <module>
    from lightning.pytorch.plugins.precision.fsdp_native_native_amp import FullyShardedNativeNativeMixedPrecisionPlugin
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/lightning/pytorch/plugins/precision/fsdp_native_native_amp.py", line 24, in <module>
    from torch.distributed.fsdp.fully_sharded_data_parallel import MixedPrecision
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/torch/distributed/fsdp/__init__.py", line 1, in <module>
    from .flat_param import FlatParameter
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/torch/distributed/fsdp/flat_param.py", line 26, in <module>
    from ._fsdp_extensions import _ext_post_unflatten_transform, _ext_pre_flatten_transform
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/torch/distributed/fsdp/_fsdp_extensions.py", line 7, in <module>
    from torch.distributed.fsdp._shard_utils import _create_chunk_sharded_tensor
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/torch/distributed/fsdp/_shard_utils.py", line 10, in <module>
    from torch.distributed._shard.sharded_tensor import (
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/torch/distributed/_shard/__init__.py", line 1, in <module>
    from .api import (
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/torch/distributed/_shard/api.py", line 6, in <module>
    from torch.distributed._shard.sharded_tensor import (
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/torch/distributed/_shard/sharded_tensor/__init__.py", line 8, in <module>
    import torch.distributed._shard.sharding_spec as shard_spec
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/torch/distributed/_shard/sharding_spec/__init__.py", line 1, in <module>
    from .api import (
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/torch/distributed/_shard/sharding_spec/api.py", line 16, in <module>
    import torch.distributed._shard.sharded_tensor.metadata as sharded_tensor_meta
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/site-packages/torch/distributed/_shard/sharded_tensor/metadata.py", line 70, in <module>
    @dataclass
     ^^^^^^^^^
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/dataclasses.py", line 1221, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/dataclasses.py", line 1211, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/dataclasses.py", line 959, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/conda/staged-recipes/build_artifacts/lightning_1678550325764/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/python3.11/dataclasses.py", line 816, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'torch.distributed._shard.sharded_tensor.metadata.TensorProperties'> for field tensor_properties is not allowed: use default_factory
Tests failed for lightning-1.9.4-pyhd8ed1ab_0.conda - moving package to /home/conda/staged-recipes/build_artifacts/broken
WARNING:conda_build.build:Tests failed for lightning-1.9.4-pyhd8ed1ab_0.conda - moving package to /home/conda/staged-recipes/build_artifacts/broken
TESTS FAILED: lightning-1.9.4-pyhd8ed1ab_0.conda
##[error]Bash exited with code '1'.

The error looks like Lightning-AI/pytorch-lightning#15614 and is because of building on Python 3.11 (which is not officially supported by Pytorch < 2.0 yet). Can try the suggestion at conda-forge/pytorch-lightning-feedstock#107 (comment) to put an upper bound on python of <3.11 for host, ~~run~~ and test/requires (note though, that things should still work on Python 3.11 (?) but will need a patched version of Pytorch that fixes the dataclass issue).

Due to Pytorch<2.0 not supporting dataclass properly yet. Xref conda-forge/pytorch-lightning-feedstock#107 (comment).

Needed to fix `RuntimeError: Form data requires "python-multipart" to be installed` when running `lightning --help`.

weiji14 · 2023-03-12T21:48:19Z

Windows build failing due to pytorch missing in conda-forge channels (see conda-forge/pytorch-cpu-feedstock#32). Should be ok though since this is a noarch build, similar situation to #15917 (comment).

recipes/lightning/meta.yaml

weiji14 · 2023-03-14T21:33:33Z

@conda-forge/help-python, ready for review!

synapticarbors · 2023-03-14T21:37:51Z

Is this the successor to https://github.com/conda-forge/pytorch-lightning-feedstock? If so, should there be some coordination with the maintainers of that feedstock?

weiji14 · 2023-03-14T21:44:22Z

Is this the successor to https://github.com/conda-forge/pytorch-lightning-feedstock? If so, should there be some coordination with the maintainers of that feedstock?

No, lightning is supposed to be pytorch-lightning + other things. It's confusing because the organization got renamed from Pytorch Lightning to Lightning AI last year, and they changed the repo to https://github.com/Lightning-AI/lightning which is now a monorepo with pytorch-lightning, lightning and other stuff.

synapticarbors · 2023-03-14T21:55:33Z

When I look at https://github.com/Lightning-AI/lightning, it says: "NEWS: PyTorch Lightning has been renamed Lightning!" and there are commits like https://github.com/Lightning-AI/lightning/pull/17074/files, which seem to indicate that pytorch lightning is just being renamed. I just want to suggest coordinating with the other feedstock maintainers, especially since one of them is the creator of pytorch lightning.

weiji14 · 2023-03-14T22:08:49Z

When I look at https://github.com/Lightning-AI/lightning, it says: "NEWS: PyTorch Lightning has been renamed Lightning!" and there are commits like https://github.com/Lightning-AI/lightning/pull/17074/files, which seem to indicate that pytorch lightning is just being renamed. I just want to suggest coordinating with the other feedstock maintainers, especially since one of them is the creator of pytorch lightning.

Oo ok, that's news to me too. So apparently they might be deprecating pytorch-lightning for lightning in June 2023? Maybe it is worth checking... Cc @Borda?

Edit: Saw the news article at https://lightning.ai/pages/community/lightning-releases/pytorch-lightning-2-0. Changing this to draft for now.

Borda · 2023-03-16T07:28:28Z

Oo ok, that's news to me too. So apparently they might be deprecating pytorch-lightning for lightning in June 2023? Maybe it is worth checking... Cc @Borda?

this was updated and we do not have any plan for depreciation PL at this moment, so both will live :)

Borda

let's start with 2.0

recipes/lightning/meta.yaml

Co-authored-by: Jirka Borovec <[email protected]>

Set minimum python version to 3.8 and so on.

Similar to 019c0ee, but just setting python<3.11 on test/requires.

Needed to fix `RuntimeError: Form data requires "python-multipart" to be installed` when running `lightning --help`.

weiji14 · 2023-03-16T08:51:49Z

Oo ok, that's news to me too. So apparently they might be deprecating pytorch-lightning for lightning in June 2023? Maybe it is worth checking... Cc @Borda?

this was updated and we do not have any plan for depreciation PL at this moment, so both will live :)

Thanks for the clarification. In that case, this should be ready for review again @synapticarbors, cc @conda-forge/help-python. I've updated the version from 1.9.4 to 2.0.0 now.

weiji14 · 2023-03-23T20:36:06Z

Gentle ping again @conda-forge/help-python. CI test failure on Windows is due to missing pytorch build as mentioned above at #22249 (comment).

dopplershift

Recipe looks good. Just need confirmation from @lantiga that they are willing to be listed as a maintainer for this recipe.

dopplershift · 2023-04-05T17:41:10Z

This is timing out on the build, was this a problem previously?

weiji14 · 2023-04-05T20:06:24Z

This is timing out on the build, was this a problem previously?

Hmm, the build itself seems to be fine, but it is timing out on the lightning --help test step:

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
export PREFIX=/home/conda/staged-recipes/build_artifacts/lightning_1680635102681/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh
export SRC_DIR=/home/conda/staged-recipes/build_artifacts/lightning_1680635102681/test_tmp
import: 'lightning'
import: 'lightning'
+ pip check
No broken requirements found.
+ lightning --help
##[error]The operation was canceled.
Finishing: Run docker build

I've re-triggered the CI build, let's see if it was just randomly stuck. Also removed @lantiga for now since he hasn't responded, we can add him back later.

weiji14 · 2023-04-06T02:50:39Z

recipes/lightning/meta.yaml

+    - lightning
+  commands:
+    - pip check
+    - lightning --help


This is timing out on the build, was this a problem previously?

Ok, after building and installing lightning==2.0.0 locally and running lightning --help, there is a prompt that requires user input:

A newer version of lightning is available (2.0.1). Would you like to upgrade? [Y/n]:

This is caused by https://github.com/Lightning-AI/lightning/blob/2.0.0/src/lightning/app/cli/lightning_cli.py#L84. Need to find a way to disable this prompt on the CLI, otherwise it'll block the build from completing.

Using LIGHTING_TESTING=1 lightning --help in 4661300 worked. @dopplershift, should be ok to merge now!

Silences the `A newer version of lightning is available` user prompt that stops the build from completing on the command-line. Xref https://github.com/Lightning-AI/lightning/blob/2.0.0/src/lightning/app/cli/lightning_cli.py#L76-L84

Add lightning

641289e

Use Lightning Apps to build everything from production-ready, multi-cloud ML systems to simple research demos! Repo at https://github.com/Lightning-AI/lightning

weiji14 commented Mar 8, 2023

View reviewed changes

recipes/lightning/meta.yaml Outdated Show resolved Hide resolved

Merge branch 'main' into lightning

583b218

weiji14 added 2 commits March 13, 2023 10:01

Pin to python<3.11 in host and test/requires dependency list

019c0ee

Due to Pytorch<2.0 not supporting dataclass properly yet. Xref conda-forge/pytorch-lightning-feedstock#107 (comment).

Add python-multipart to runtime requirements

22aa4ee

Needed to fix `RuntimeError: Form data requires "python-multipart" to be installed` when running `lightning --help`.

weiji14 marked this pull request as ready for review March 12, 2023 21:31

weiji14 commented Mar 12, 2023

View reviewed changes

recipes/lightning/meta.yaml Outdated Show resolved Hide resolved

github-actions bot added python review-requested labels Mar 14, 2023

weiji14 marked this pull request as draft March 15, 2023 00:23

Add @Borda and @lantiga to recipe maintainer list

cfb723b

Borda mentioned this pull request Mar 16, 2023

conda install lightning -c conda-forge ? Lightning-AI/pytorch-lightning#17101

Closed

Borda approved these changes Mar 16, 2023

View reviewed changes

Borda reviewed Mar 16, 2023

View reviewed changes

recipes/lightning/meta.yaml Outdated Show resolved Hide resolved

recipes/lightning/meta.yaml Outdated Show resolved Hide resolved

weiji14 and others added 2 commits March 16, 2023 21:10

Bump from lightning 1.9.4 to 2.0.0

f44dc44

Co-authored-by: Jirka Borovec <[email protected]>

Update version pins for lightning 2.0

9f50c0f

Set minimum python version to 3.8 and so on.

weiji14 marked this pull request as ready for review March 16, 2023 08:18

weiji14 added 2 commits March 16, 2023 21:42

Pin to python<3.11 in test/requires dependency list

be4940f

Similar to 019c0ee, but just setting python<3.11 on test/requires.

Add python-multipart to runtime requirements

f69cb25

Needed to fix `RuntimeError: Form data requires "python-multipart" to be installed` when running `lightning --help`.

Merge branch 'main' into lightning

f833921

dopplershift approved these changes Apr 4, 2023

View reviewed changes

dopplershift closed this Apr 4, 2023

dopplershift reopened this Apr 4, 2023

Remove lantiga

ac5b591

weiji14 commented Apr 6, 2023

View reviewed changes

awaelchli mentioned this pull request Apr 6, 2023

DOCS: Fix conda install info Lightning-AI/pytorch-lightning#17289

Closed

dopplershift approved these changes Apr 6, 2023

View reviewed changes

dopplershift merged commit b6bde15 into conda-forge:main Apr 6, 2023

weiji14 deleted the lightning branch April 6, 2023 19:30

weiji14 mentioned this pull request Jun 26, 2023

Does this package need to exist, with pytorch-lightning package already in conda-forge? conda-forge/lightning-feedstock#5

Closed

1 task

gonzalocasas mentioned this pull request Feb 18, 2024

Add AIXD toolkit #25442

Merged

10 tasks

gonzalocasas mentioned this pull request May 7, 2024

Add aixd_ara: grasshopper plugin for the aixd toolkit #26277

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lightning #22249

Add lightning #22249

weiji14 commented Mar 8, 2023 •

edited

Loading

conda-forge-webservices bot commented Mar 8, 2023

weiji14 commented Mar 8, 2023

weiji14 commented Mar 12, 2023 •

edited

Loading

weiji14 commented Mar 12, 2023

weiji14 commented Mar 14, 2023

synapticarbors commented Mar 14, 2023

weiji14 commented Mar 14, 2023

synapticarbors commented Mar 14, 2023

weiji14 commented Mar 14, 2023 •

edited

Loading

Borda commented Mar 16, 2023

Borda left a comment

weiji14 commented Mar 16, 2023

weiji14 commented Mar 23, 2023

dopplershift left a comment

dopplershift commented Apr 5, 2023

weiji14 commented Apr 5, 2023

weiji14 Apr 6, 2023 •

edited

Loading

weiji14 Apr 6, 2023

Add lightning #22249

Add lightning #22249

Conversation

weiji14 commented Mar 8, 2023 • edited Loading

conda-forge-webservices bot commented Mar 8, 2023

weiji14 commented Mar 8, 2023

weiji14 commented Mar 12, 2023 • edited Loading

weiji14 commented Mar 12, 2023

weiji14 commented Mar 14, 2023

synapticarbors commented Mar 14, 2023

weiji14 commented Mar 14, 2023

synapticarbors commented Mar 14, 2023

weiji14 commented Mar 14, 2023 • edited Loading

Borda commented Mar 16, 2023

Borda left a comment

Choose a reason for hiding this comment

weiji14 commented Mar 16, 2023

weiji14 commented Mar 23, 2023

dopplershift left a comment

Choose a reason for hiding this comment

dopplershift commented Apr 5, 2023

weiji14 commented Apr 5, 2023

weiji14 Apr 6, 2023 • edited Loading

Choose a reason for hiding this comment

weiji14 Apr 6, 2023

Choose a reason for hiding this comment

weiji14 commented Mar 8, 2023 •

edited

Loading

weiji14 commented Mar 12, 2023 •

edited

Loading

weiji14 commented Mar 14, 2023 •

edited

Loading

weiji14 Apr 6, 2023 •

edited

Loading