Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Technical debt: RemoteMixtureOfExperts (v0.8) #64

Closed
5 of 7 tasks
justheuristic opened this issue Jul 3, 2020 · 1 comment
Closed
5 of 7 tasks

Technical debt: RemoteMixtureOfExperts (v0.8) #64

justheuristic opened this issue Jul 3, 2020 · 1 comment
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@justheuristic
Copy link
Member

justheuristic commented Jul 3, 2020

  • beam search uses tuple endpoints (i.e. address, port), while dht switched to string endpoints
  • beam search needs one extra step in beam search because prefix.123.321 != expert.123.321
  • we may no longer need parallel autograd if it is implemented in pytorch (not the case)
    • remove hivemind.utils.autograd in favor of _RemoteExpertCallMany
  • add a more feature-rich test for moe.py (with several DHT nodes and experts)
  • cancel unused queries in first_k_active?
  • when declaring experts, introduce some kind of "grace period" - only "declare" prefixes that have not been updated for that period. (rationale: first prefixes are likely to be already updated by other peers)
@justheuristic justheuristic added enhancement New feature or request help wanted Extra attention is needed labels Jul 3, 2020
@justheuristic justheuristic linked a pull request Jul 5, 2020 that will close this issue
8 tasks
@justheuristic
Copy link
Member Author

merged as #80

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant