Implement core functionality of hivemind.Optimizer

[justheuristic]

This PR will implement a subset of #400 via hivemind.optim.experimental.Optimizer

This is a draft PR that implements hivemind optimizer core components. It will be merged piece-wise in #404 #407 and 2 more PRs.

This PR implements the main class of hivemind.Optimizer

• implement GradientAverager with pre-scheduling
• test that pre-scheduling works as intended
• implement ProgressTracker
• test that ProgressTracker correctly accumulates progress from several peers
• implement HivemindOptimzier with synchronous or DPU steps
• check that the new optimizer trains albert-based model similarly to CollaborativeOptimizer

[codecov]

Codecov Report

Merging #403 (f0160ce) into master (00538db) will decrease coverage by 1.15%.
The diff coverage is 64.56%.

@@            Coverage Diff             @@
##           master     #403      +/-   ##
==========================================
- Coverage   84.87%   83.72%   -1.16%     
==========================================
  Files          76       77       +1     
  Lines        7327     7783     +456     
==========================================
+ Hits         6219     6516     +297     
- Misses       1108     1267     +159     

Impacted Files	Coverage Δ
hivemind/__init__.py	100.00% <ø> (ø)
hivemind/averaging/partition.py	98.01% <ø> (ø)
hivemind/optim/collaborative.py	23.64% <0.00%> (ø)
hivemind/optim/grad_scaler.py	30.13% <10.86%> (-5.58%)	⬇️
hivemind/averaging/averager.py	85.50% <38.88%> (-1.75%)	⬇️
hivemind/optim/experimental/optimizer.py	64.17% <64.17%> (ø)
hivemind/optim/experimental/grad_averager.py	92.78% <75.00%> (+0.07%)	⬆️
hivemind/optim/experimental/state_averager.py	86.76% <78.12%> (-2.74%)	⬇️
hivemind/averaging/control.py	94.17% <80.00%> (-0.73%)	⬇️
hivemind/optim/experimental/progress_tracker.py	98.28% <91.30%> (+0.80%)	⬆️
... and 7 more

[justheuristic]

Verified stability as of commit 8840aaa

Using @SeanNaren 's hivemind-lightning

Since that commit, all changes were cosmetic (e.g. docs, renaming internal functions)