Code

Feat: Add torch elastic task type#1583

Code

Closed

fg91 wants to merge 40 commits intomasterflyteorg/flytekit:masterfrom fabio/feat/torch-elastic-pluginflyteorg/flytekit:fabio/feat/torch-elastic-pluginCopy head branch name to clipboard

+1,972-326

Commits

Commits on Apr 23, 2023

• Add torch elastic task type

  Show description for c3de752
  

  Fabio Grätz

  authored and

  fg91

  committed
  c3de752

  Copy full SHA for c3de752

• updated

  Show description for fd0ae59
  

  kumare3

  authored and

  fg91

  committed
  fd0ae59

  Copy full SHA for fd0ae59

• updated

  Show description for dd2a334
  

  kumare3

  authored and

  fg91

  committed
  dd2a334

  Copy full SHA for dd2a334

• Don't pass min_nodes, max_nodes but nnodes only

  Show description for 855bf84
  

  Fabio Grätz

  authored and

  fg91

  committed
  855bf84

  Copy full SHA for 855bf84

• Add docstrings

  Show description for 4245bf6
  

  Fabio Grätz

  authored and

  fg91

  committed
  4245bf6

  Copy full SHA for 4245bf6

• Cleanup test

  Show description for a6ac14d
  

  Fabio Grätz

  authored and

  fg91

  committed
  a6ac14d

  Copy full SHA for a6ac14d

• Replace wrong occurences of mpi in torch-elastic plugin

  Show description for 35930b9
  

  Fabio Grätz

  authored and

  fg91

  committed
  35930b9

  Copy full SHA for 35930b9

• Extend kfpytorch plugin README

  Show description for 4245d39
  

  Fabio Grätz

  authored and

  fg91

  committed
  4245d39

  Copy full SHA for 4245d39

• Move elastic task into existing kf-pytorch plugin

  Show description for 40c83dd
  

  Fabio Grätz

  authored and

  fg91

  committed
  40c83dd

  Copy full SHA for 40c83dd

• Add Elastic config to plugin's __init__

  Show description for 440ea47
  

  Fabio Grätz

  authored and

  fg91

  committed
  440ea47

  Copy full SHA for 440ea47

• Set elastic config in pytorchjob proto

  Show description for 2d916af
  

  Fabio Grätz

  authored and

  fg91

  committed
  2d916af

  Copy full SHA for 2d916af

• Unify sqlalchemy Dockerfiles (#1585)

  Show description for 1c32952
  

  eapolinario

  authored and

  fg91

  committed
  1c32952

  Copy full SHA for 1c32952

• removed unnecessary model files and simplified codebase

  Show description for cc960b6
  

  kumare3

  authored and

  fg91

  committed
  cc960b6

  Copy full SHA for cc960b6

• Configure rdzv endpoint

  Show description for 7f31a5f
  

  Fabio Grätz

  authored and

  fg91

  committed
  7f31a5f

  Copy full SHA for 7f31a5f

• Configure worker count also for elastic training

  Show description for bf97ece
  

  Fabio Grätz

  authored and

  fg91

  committed
  bf97ece

  Copy full SHA for bf97ece

• Fix exception scope and handle non-rank-0 outputs

  Show description for fcc54f4
  fg91

  committed
  fcc54f4

  Copy full SHA for fcc54f4

• Remove todo item about handling exception scope, now done

  Show description for e5f7c58
  fg91

  committed
  e5f7c58

  Copy full SHA for e5f7c58

• Add note about c10d backend

  Show description for 59ba4a5
  fg91

  committed
  59ba4a5

  Copy full SHA for 59ba4a5

• Add classifier for Python 3.11 (#1589)

  Show description for 7367757
  

  honnix

  authored and

  fg91

  committed
  7367757

  Copy full SHA for 7367757

• Pyflyte build imageSpec (#1555)

  Show description for 041d188
  

  pingsutw

  authored and

  fg91

  committed
  041d188

  Copy full SHA for 041d188

• Feature/3506: Improve Type Conversion errors, use rich to prettify error messages (#1582)

  Show description for 9c39e36
  

  cosmicBboy

  authored and

  fg91

  committed
  9c39e36

  Copy full SHA for 9c39e36

• Add support nested FlyteFile in pyflyte run (#1587)

  Show description for 538a7f3
  

  pingsutw

  authored and

  fg91

  committed
  538a7f3

  Copy full SHA for 538a7f3

• Set cache_regions=True in the case of s3fs (#1592)

  Show description for 50c2e38
  

  eapolinario

  authored and

  fg91

  committed
  50c2e38

  Copy full SHA for 50c2e38

• Set default ComSpec when running on Windows (#1595)

  Show description for 12660ad
  

  pingsutw

  authored and

  fg91

  committed
  12660ad

  Copy full SHA for 12660ad

• Feature: add activate-launchplan command to pyflyte (#1588)

  Show description for 77513f0
  

  wckdman

  authored and

  fg91

  committed
  77513f0

  Copy full SHA for 77513f0

• nit (#1596)

  Show description for 25b00b1
  

  wild-endeavor

  authored and

  fg91

  committed
  25b00b1

  Copy full SHA for 25b00b1

• Add log streaming to papermill plugin (#1129)

  Show description for 57853a7
  

  3 peopleauthored and

  fg91

  committed
  57853a7

  Copy full SHA for 57853a7

• Set lower bound for s3fs (#1598)

  Show description for e701f96
  

  pingsutw

  authored and

  fg91

  committed
  e701f96

  Copy full SHA for e701f96

• Implement horovod task in mpi plugin (#1575)

  Show description for 9c9fb5d
  

  2 peopleauthored and

  fg91

  committed
  9c9fb5d

  Copy full SHA for 9c9fb5d

• add ImageRender class for Flyte Decks (#1599)

  Show description for c7d69fa
  

  peridotml

  authored and

  fg91

  committed
  c7d69fa

  Copy full SHA for c7d69fa

• Bump tensorflow from 2.10.0 to 2.11.1 in /plugins/flytekit-mlflow (#1586)

  Show description for 0db0009
  

  dependabot[bot]

  authored and

  fg91

  committed
  0db0009

  Copy full SHA for 0db0009

• Handle batchable lists of length 0 (#1600)

  Show description for 6a85430
  

  eapolinario

  authored and

  fg91

  committed
  6a85430

  Copy full SHA for 6a85430

• pyflyte run imperative workflow (#1597)

  Show description for 5b5525e
  

  pingsutw

  authored and

  fg91

  committed
  5b5525e

  Copy full SHA for 5b5525e

• Backfill should fill in the right input vars (#1593)

  Show description for b43d044
  

  kumare3

  authored and

  fg91

  committed
  b43d044

  Copy full SHA for b43d044

• Pyflyte prettified (#1602)

  Show description for c46e2e6
  

  kumare3

  authored and

  fg91

  committed
  c46e2e6

  Copy full SHA for c46e2e6

• Let user set min and max replicas explicitly and not via nnodes

  Show description for 63db8b4
  fg91

  committed
  63db8b4

  Copy full SHA for 63db8b4

• Catch torch import error and configure for flytekitplugins-kfpytorch['elastic']

  Show description for 5d24b29
  fg91

  committed
  5d24b29

  Copy full SHA for 5d24b29

• Add more tests

  Show description for c58eaa2
  fg91

  committed
  c58eaa2

  Copy full SHA for c58eaa2

• Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py

  Show description for 379a68b
  fg91

  committed
  379a68b

  Copy full SHA for 379a68b

• Lint

  Show description for d5ab6ac
  fg91

  committed
  d5ab6ac

  Copy full SHA for d5ab6ac