Code

Improve All-Reduce fault-tolerance#423

Code

Merged

justheuristic merged 61 commits intomasterlearning-at-home/hivemind:masterfrom fault_tolerant_allreducelearning-at-home/hivemind:fault_tolerant_allreduceCopy head branch name to clipboardDec 14, 2021

+471-138

Commits

Commits on Dec 11, 2021

• proper scheduling for gradient averaging

  justheuristic

  committed
  eced497

  Copy full SHA for eced497

• more agressive pre-scheduling

  justheuristic

  committed
  9e5561d

  Copy full SHA for 9e5561d

Commits on Dec 12, 2021

• fault-tolerant all-reduce with next chunk timeout

  justheuristic

  committed
  82c9473

  Copy full SHA for 82c9473

• fault-tolerant all-reduce with next chunk timeout

  justheuristic

  committed
  b1b3336

  Copy full SHA for b1b3336

• DRY

  justheuristic

  committed
  4991e71

  Copy full SHA for 4991e71

• fault-tolerant allreduce: global test 1

  justheuristic

  committed
  965afa8

  Copy full SHA for 965afa8

• revert peer ids

  justheuristic

  committed
  874338e

  Copy full SHA for 874338e

• blackisort

  justheuristic

  committed
  6838fe1

  Copy full SHA for 6838fe1

• cancel

  justheuristic

  committed
  1b0e1bd

  Copy full SHA for 1b0e1bd

Commits on Dec 13, 2021

• next_chunk_timeout in optimizer

  justheuristic

  committed
  ceb2a0a

  Copy full SHA for ceb2a0a

• black

  justheuristic

  committed
  94972a7

  Copy full SHA for 94972a7

• Update hivemind/averaging/allreduce.py

  Show description for eaa1603
  

  justheuristic

  and

  borzunov

  authored
  eaa1603

  Copy full SHA for eaa1603

• Update hivemind/averaging/partition.py

  Show description for 6f1e22a
  

  justheuristic

  and

  borzunov

  authored
  6f1e22a

  Copy full SHA for 6f1e22a

• Update hivemind/averaging/averager.py

  Show description for 35d8306
  

  justheuristic

  and

  borzunov

  authored
  35d8306

  Copy full SHA for 35d8306

• track pending tasks

  justheuristic

  committed
  298f437

  Copy full SHA for 298f437

• review

  

  borzunov

  authored and

  justheuristic

  committed
  df67842

  Copy full SHA for df67842

• review

  

  borzunov

  authored and

  justheuristic

  committed
  a3897a8

  Copy full SHA for a3897a8

• review

  

  borzunov

  authored and

  justheuristic

  committed
  e51ee75

  Copy full SHA for e51ee75

• review

  

  borzunov

  authored and

  justheuristic

  committed
  2386611

  Copy full SHA for 2386611

• review

  

  borzunov

  authored and

  justheuristic

  committed
  8eac4a0

  Copy full SHA for 8eac4a0

• review

  

  borzunov

  authored and

  justheuristic

  committed
  cc348a8

  Copy full SHA for cc348a8

• black it

  justheuristic

  committed
  ac2c004

  Copy full SHA for ac2c004

• Update hivemind/averaging/allreduce.py

  Show description for abc67e4
  

  justheuristic

  and

  borzunov

  authored
  abc67e4

  Copy full SHA for abc67e4

• review

  

  borzunov

  authored and

  justheuristic

  committed
  92ab91e

  Copy full SHA for 92ab91e

• Merge remote-tracking branch 'origin/fault_tolerant_allreduce' into fault_tolerant_allreduce

  justheuristic

  committed
  be13c1f

  Copy full SHA for be13c1f

• review

  

  borzunov

  authored and

  justheuristic

  committed
  aa0f2b1

  Copy full SHA for aa0f2b1

• Update hivemind/averaging/partition.py

  Show description for 539a573
  

  justheuristic

  and

  borzunov

  authored
  539a573

  Copy full SHA for 539a573

• reblack

  justheuristic

  committed
  dfa0d1a

  Copy full SHA for dfa0d1a

• Merge remote-tracking branch 'origin/fault_tolerant_allreduce' into fault_tolerant_allreduce

  justheuristic

  committed
  c4f4c1e

  Copy full SHA for c4f4c1e

• review

  

  borzunov

  authored and

  justheuristic

  committed
  ddcb240

  Copy full SHA for ddcb240

• remove if

  justheuristic

  committed
  4df6ea7

  Copy full SHA for 4df6ea7

• review

  

  borzunov

  authored and

  justheuristic

  committed
  4ce62f6

  Copy full SHA for 4ce62f6

• review

  justheuristic

  committed
  a42d20c

  Copy full SHA for a42d20c

• review

  

  borzunov

  authored and

  justheuristic

  committed
  53071e9

  Copy full SHA for 53071e9

• Update hivemind/averaging/allreduce.py

  Show description for c5be676
  

  justheuristic

  and

  borzunov

  authored
  c5be676

  Copy full SHA for c5be676

• review

  

  borzunov

  authored and

  justheuristic

  committed
  e3506d8

  Copy full SHA for e3506d8

• review

  

  borzunov

  authored and

  justheuristic

  committed
  8d2427e

  Copy full SHA for 8d2427e

• review

  justheuristic

  committed
  5adfec7

  Copy full SHA for 5adfec7

• review

  

  borzunov

  authored and

  justheuristic

  committed
  3bb5e03

  Copy full SHA for 3bb5e03

• add test case

  justheuristic

  committed
  e50e459

  Copy full SHA for e50e459

• isort

  justheuristic

  committed
  f564176

  Copy full SHA for f564176

Commits on Dec 14, 2021

• test for slowness

  

  borzunov

  authored and

  justheuristic

  committed
  6e86743

  Copy full SHA for 6e86743

• report integrity

  justheuristic

  committed
  12b708c

  Copy full SHA for 12b708c

• report integrity

  

  borzunov

  authored and

  justheuristic

  committed
  cb71c7f

  Copy full SHA for cb71c7f

• nirvana tests

  justheuristic

  committed
  1d56876

  Copy full SHA for 1d56876

• nirvana tests

  justheuristic

  committed
  c09ed95

  Copy full SHA for c09ed95

• nirvana tests

  justheuristic

  committed
  aadf8c3

  Copy full SHA for aadf8c3

• nirvana tests

  justheuristic

  committed
  ddeb508

  Copy full SHA for ddeb508

• nirvana tests

  justheuristic

  committed
  2b0d850

  Copy full SHA for 2b0d850

• nirvana tests

  justheuristic

  committed
  671ca20

  Copy full SHA for 671ca20

• nirvana tests

  justheuristic

  committed
  8862a85

  Copy full SHA for 8862a85

• nirvana tests

  justheuristic

  committed
  f61ba96

  Copy full SHA for f61ba96

• nirvana test - wait for client-mode peers to begin sending data (within timeout)

  justheuristic

  committed
  5e2f33f

  Copy full SHA for 5e2f33f

• nirvana test - wait for client-mode peers to begin sending data (within timeout)

  justheuristic

  committed
  53ffcca

  Copy full SHA for 53ffcca

• nirvana test - wait for client-mode peers to begin sending data (within timeout)

  justheuristic

  committed
  65ae51e

  Copy full SHA for 65ae51e

• nirvana test - wait for client-mode peers to begin sending data (within timeout)

  justheuristic

  committed
  91e8c3c

  Copy full SHA for 91e8c3c

• nirvana test - wait for client-mode peers to begin sending data (within timeout)

  justheuristic

  committed
  10121f8

  Copy full SHA for 10121f8

• fix edge case where all peers are auxiliaries

  justheuristic

  committed
  3d8fb18

  Copy full SHA for 3d8fb18

• blacken

  justheuristic

  committed
  4b675aa

  Copy full SHA for 4b675aa

• better loggign

  justheuristic

  committed
  b03ccd9

  Copy full SHA for b03ccd9

• learned from 160 peers

  justheuristic

  committed
  a803aa8

  Copy full SHA for a803aa8