Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compiling libp2p daemon on setup #153

Merged
merged 2 commits into from
Mar 1, 2021
Merged

Compiling libp2p daemon on setup #153

merged 2 commits into from
Mar 1, 2021

Conversation

dvmazur
Copy link
Collaborator

@dvmazur dvmazur commented Feb 26, 2021

No description provided.

@justheuristic
Copy link
Member

image
Initial code review:

  • Please implement safe go version comparison via packaging.version
  • If possible, replace os.system calls to curl/tar with urllib and tarfile python modules
  • os.system('go install ...
    • If this command fails, make sure to interrupt installation
    • It would be great to capture logs from this call and print them to main stdout/err

@mryab mryab requested review from mryab and justheuristic February 28, 2021 16:51
@dvmazur dvmazur changed the base branch from master to libp2p February 28, 2021 18:48
@dvmazur dvmazur marked this pull request as ready for review February 28, 2021 19:41
@justheuristic
Copy link
Member

[Merging into libp2p branch]

@justheuristic justheuristic merged commit c24936b into libp2p Mar 1, 2021
@justheuristic justheuristic deleted the libp2p-daemon-build branch March 1, 2021 20:17
@justheuristic justheuristic mentioned this pull request Mar 2, 2021
Merged
dvmazur added a commit that referenced this pull request Mar 10, 2021
@dvmazur
Compiling libp2p daemon on setup (#153)
ee3fb4c 
* add setup.py prototype

* refactor
justheuristic added a commit that referenced this pull request Mar 10, 2021
@dvmazur @leshanbog
@justheuristic @borzunov @mryab @yhn112 @romakail @skobellev
Rebase master onto libp2p (#179)
35938ec 
* copytree implementation for py37 compatibility (#162)

* copytree implementation for py37 compatibility

* Running tests for python3.7

* Increment version

* Python3.7 notions

* Remove pickle.loads in averager (#160)

* Security update: remove pickle.loads in averager
* add py37 to circleci config

Co-authored-by: Alexander Borzunov <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>

* Support edge cases for DHT key/subkey/value, add tests, update .gitignore for pb2 (#167)

* fix bug with subkey equals zero
* add autogenerated protobuf files to .gitignore
* test store and get "tricky" values in dht

* Fix the remaining tests for py37 (#166)

* DecentralizedAverager is now compatible with python37's acyncio exception
    * the problem was: grpc.aio with python37 raised concurrent.futures.CancelledError in some cases;
    * we relied on isinstance(asyncio.CancelledError, Exception) == False
    * but isinstance(concurrent.futures.CancelledError, Exception) == True
*  DecentralizedAverager now shuts down if dereferenced in the main process
    * though it won't shutdown if dereferenced in forks for obvious reasons
* HIVEMIND_THREADS now actually works
* test_averaging now shuts down dht and averager instances to avoid leaking processes

Co-authored-by: Max Ryabinin <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>

* Move Averager metadata serialization out of user scope (#168)

* move metadata serialization outside user scope
* test_overcrowded: reduce the default number of peers

* Handle edge cases in DecentralizedAverager (#171)

* move metadata serialization outside user scope
* retry averager.step on network errors
* raise AllreduceException on partial tensor
* test split/combine tensors, combine corrupted stream

Co-authored-by: Max Ryabinin <[email protected]>

* Fix a typo in quickstart.md (#174)

* Serialize DHTID source with msgpack (#172)

* Change DHTID serializer

* Remove unused serializers

* Add msgpack tuple serialization

* Move CLI server launch script to hivemind/hivemind_cli (#173)

* Cast environment variables to correct types

* Compiling libp2p daemon on setup (#153)

* add setup.py prototype

* refactor

* feat: add p2p daemon (#164)

* Add p2p daemon

* Test p2p daemon exits correctly

* Impose restriction on elapsed time

Co-authored-by: Ilya Kobelev <[email protected]>

* compare golang versions using packaging.version

* fix typo

Co-authored-by: justheuristic <[email protected]>

* move p2pd executable to hivemind/hivemind_cli

Co-authored-by: Alexey Bukhtiyarov <[email protected]>
Co-authored-by: justheuristic <[email protected]>
Co-authored-by: Alexander Borzunov <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>
Co-authored-by: Michael Diskin <[email protected]>
Co-authored-by: romakail <[email protected]>
Co-authored-by: Ilya <[email protected]>
Co-authored-by: Ilya Kobelev <[email protected]>
justheuristic pushed a commit that referenced this pull request Apr 13, 2021
@dvmazur @justheuristic
Compiling libp2p daemon on setup (#153)
62279c3 
* add setup.py prototype

* refactor
justheuristic added a commit that referenced this pull request Apr 13, 2021
@dvmazur @leshanbog
@justheuristic @borzunov @mryab @yhn112 @romakail @skobellev
Rebase master onto libp2p (#179)
4409cd6 
* copytree implementation for py37 compatibility (#162)

* copytree implementation for py37 compatibility

* Running tests for python3.7

* Increment version

* Python3.7 notions

* Remove pickle.loads in averager (#160)

* Security update: remove pickle.loads in averager
* add py37 to circleci config

Co-authored-by: Alexander Borzunov <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>

* Support edge cases for DHT key/subkey/value, add tests, update .gitignore for pb2 (#167)

* fix bug with subkey equals zero
* add autogenerated protobuf files to .gitignore
* test store and get "tricky" values in dht

* Fix the remaining tests for py37 (#166)

* DecentralizedAverager is now compatible with python37's acyncio exception
    * the problem was: grpc.aio with python37 raised concurrent.futures.CancelledError in some cases;
    * we relied on isinstance(asyncio.CancelledError, Exception) == False
    * but isinstance(concurrent.futures.CancelledError, Exception) == True
*  DecentralizedAverager now shuts down if dereferenced in the main process
    * though it won't shutdown if dereferenced in forks for obvious reasons
* HIVEMIND_THREADS now actually works
* test_averaging now shuts down dht and averager instances to avoid leaking processes

Co-authored-by: Max Ryabinin <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>

* Move Averager metadata serialization out of user scope (#168)

* move metadata serialization outside user scope
* test_overcrowded: reduce the default number of peers

* Handle edge cases in DecentralizedAverager (#171)

* move metadata serialization outside user scope
* retry averager.step on network errors
* raise AllreduceException on partial tensor
* test split/combine tensors, combine corrupted stream

Co-authored-by: Max Ryabinin <[email protected]>

* Fix a typo in quickstart.md (#174)

* Serialize DHTID source with msgpack (#172)

* Change DHTID serializer

* Remove unused serializers

* Add msgpack tuple serialization

* Move CLI server launch script to hivemind/hivemind_cli (#173)

* Cast environment variables to correct types

* Compiling libp2p daemon on setup (#153)

* add setup.py prototype

* refactor

* feat: add p2p daemon (#164)

* Add p2p daemon

* Test p2p daemon exits correctly

* Impose restriction on elapsed time

Co-authored-by: Ilya Kobelev <[email protected]>

* compare golang versions using packaging.version

* fix typo

Co-authored-by: justheuristic <[email protected]>

* move p2pd executable to hivemind/hivemind_cli

Co-authored-by: Alexey Bukhtiyarov <[email protected]>
Co-authored-by: justheuristic <[email protected]>
Co-authored-by: Alexander Borzunov <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>
Co-authored-by: Michael Diskin <[email protected]>
Co-authored-by: romakail <[email protected]>
Co-authored-by: Ilya <[email protected]>
Co-authored-by: Ilya Kobelev <[email protected]>
MaximKsh pushed a commit that referenced this pull request Apr 20, 2021
@dvmazur
Compiling libp2p daemon on setup (#153)
133a6fc 
* add setup.py prototype

* refactor
MaximKsh pushed a commit that referenced this pull request Apr 20, 2021
@dvmazur @leshanbog
@justheuristic @borzunov @mryab @yhn112 @romakail @skobellev
Rebase master onto libp2p (#179)
c0c16eb 
* copytree implementation for py37 compatibility (#162)

* copytree implementation for py37 compatibility

* Running tests for python3.7

* Increment version

* Python3.7 notions

* Remove pickle.loads in averager (#160)

* Security update: remove pickle.loads in averager
* add py37 to circleci config

Co-authored-by: Alexander Borzunov <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>

* Support edge cases for DHT key/subkey/value, add tests, update .gitignore for pb2 (#167)

* fix bug with subkey equals zero
* add autogenerated protobuf files to .gitignore
* test store and get "tricky" values in dht

* Fix the remaining tests for py37 (#166)

* DecentralizedAverager is now compatible with python37's acyncio exception
    * the problem was: grpc.aio with python37 raised concurrent.futures.CancelledError in some cases;
    * we relied on isinstance(asyncio.CancelledError, Exception) == False
    * but isinstance(concurrent.futures.CancelledError, Exception) == True
*  DecentralizedAverager now shuts down if dereferenced in the main process
    * though it won't shutdown if dereferenced in forks for obvious reasons
* HIVEMIND_THREADS now actually works
* test_averaging now shuts down dht and averager instances to avoid leaking processes

Co-authored-by: Max Ryabinin <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>

* Move Averager metadata serialization out of user scope (#168)

* move metadata serialization outside user scope
* test_overcrowded: reduce the default number of peers

* Handle edge cases in DecentralizedAverager (#171)

* move metadata serialization outside user scope
* retry averager.step on network errors
* raise AllreduceException on partial tensor
* test split/combine tensors, combine corrupted stream

Co-authored-by: Max Ryabinin <[email protected]>

* Fix a typo in quickstart.md (#174)

* Serialize DHTID source with msgpack (#172)

* Change DHTID serializer

* Remove unused serializers

* Add msgpack tuple serialization

* Move CLI server launch script to hivemind/hivemind_cli (#173)

* Cast environment variables to correct types

* Compiling libp2p daemon on setup (#153)

* add setup.py prototype

* refactor

* feat: add p2p daemon (#164)

* Add p2p daemon

* Test p2p daemon exits correctly

* Impose restriction on elapsed time

Co-authored-by: Ilya Kobelev <[email protected]>

* compare golang versions using packaging.version

* fix typo

Co-authored-by: justheuristic <[email protected]>

* move p2pd executable to hivemind/hivemind_cli

Co-authored-by: Alexey Bukhtiyarov <[email protected]>
Co-authored-by: justheuristic <[email protected]>
Co-authored-by: Alexander Borzunov <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>
Co-authored-by: Michael Diskin <[email protected]>
Co-authored-by: romakail <[email protected]>
Co-authored-by: Ilya <[email protected]>
Co-authored-by: Ilya Kobelev <[email protected]>
MaximKsh pushed a commit that referenced this pull request May 14, 2021
@dvmazur
Compiling libp2p daemon on setup (#153)
295a20f 
* add setup.py prototype

* refactor
MaximKsh pushed a commit that referenced this pull request May 14, 2021
@dvmazur @leshanbog
@justheuristic @borzunov @mryab @yhn112 @romakail @skobellev
Rebase master onto libp2p (#179)
decd6d7 
* copytree implementation for py37 compatibility (#162)

* copytree implementation for py37 compatibility

* Running tests for python3.7

* Increment version

* Python3.7 notions

* Remove pickle.loads in averager (#160)

* Security update: remove pickle.loads in averager
* add py37 to circleci config

Co-authored-by: Alexander Borzunov <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>

* Support edge cases for DHT key/subkey/value, add tests, update .gitignore for pb2 (#167)

* fix bug with subkey equals zero
* add autogenerated protobuf files to .gitignore
* test store and get "tricky" values in dht

* Fix the remaining tests for py37 (#166)

* DecentralizedAverager is now compatible with python37's acyncio exception
    * the problem was: grpc.aio with python37 raised concurrent.futures.CancelledError in some cases;
    * we relied on isinstance(asyncio.CancelledError, Exception) == False
    * but isinstance(concurrent.futures.CancelledError, Exception) == True
*  DecentralizedAverager now shuts down if dereferenced in the main process
    * though it won't shutdown if dereferenced in forks for obvious reasons
* HIVEMIND_THREADS now actually works
* test_averaging now shuts down dht and averager instances to avoid leaking processes

Co-authored-by: Max Ryabinin <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>

* Move Averager metadata serialization out of user scope (#168)

* move metadata serialization outside user scope
* test_overcrowded: reduce the default number of peers

* Handle edge cases in DecentralizedAverager (#171)

* move metadata serialization outside user scope
* retry averager.step on network errors
* raise AllreduceException on partial tensor
* test split/combine tensors, combine corrupted stream

Co-authored-by: Max Ryabinin <[email protected]>

* Fix a typo in quickstart.md (#174)

* Serialize DHTID source with msgpack (#172)

* Change DHTID serializer

* Remove unused serializers

* Add msgpack tuple serialization

* Move CLI server launch script to hivemind/hivemind_cli (#173)

* Cast environment variables to correct types

* Compiling libp2p daemon on setup (#153)

* add setup.py prototype

* refactor

* feat: add p2p daemon (#164)

* Add p2p daemon

* Test p2p daemon exits correctly

* Impose restriction on elapsed time

Co-authored-by: Ilya Kobelev <[email protected]>

* compare golang versions using packaging.version

* fix typo

Co-authored-by: justheuristic <[email protected]>

* move p2pd executable to hivemind/hivemind_cli

Co-authored-by: Alexey Bukhtiyarov <[email protected]>
Co-authored-by: justheuristic <[email protected]>
Co-authored-by: Alexander Borzunov <[email protected]>
Co-authored-by: Max Ryabinin <[email protected]>
Co-authored-by: Michael Diskin <[email protected]>
Co-authored-by: romakail <[email protected]>
Co-authored-by: Ilya <[email protected]>
Co-authored-by: Ilya Kobelev <[email protected]>
MaximKsh pushed a commit that referenced this pull request May 14, 2021
@dvmazur
Compiling libp2p daemon on setup (#153)
cef18da 
* add setup.py prototype

* refactor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mryab mryab Awaiting requested review from mryab

@justheuristic justheuristic Awaiting requested review from justheuristic

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dvmazur @justheuristic