Ietf (dashpay#119)

* wip schemes
* wip: all schemes tests pass
* del relic in prep for submodule add
* add relic submodule, major wip commit with basic functionality
* manual merge hdkey, better logic for g1/g2 leading bits
* fix compresspoint g2
* g1/g2 serialization correct
* Get IETF branch building under MacOS and Linux (dashpay#120)
* remove travis
* bring the ci forward 2 months
* bring setup.py forward
* add flake8 ini
* add mypi.ini
* first attempt at FetchContent chia-relic-src
* Final cmake fixes
* Add lgtm.yml
* explicit order of ops
* clean up code for mypy/flake8 - black
* enough to start looking at plotter
* lgtm assignment operator
* add 0x80 compression  bit for 0 and 48 byte
* reverted bit changes and commented out G2 tests
* switch repos for relic
* Remove Unknown distribution option: 'build_requires' warning
* msvc fixes
* Add libsodium to all platform wheels. README nit
* Fix ARM, use nuget on Windows
* Try libsodium-devel, bump verbosity to confirm sodium
* ARM install epel first - Fix find sodium macos
* no need for floor
* add fingerprint, derive_child
* fix __bytes__
* add n of n sigs in AugScheme, use MPL suffix
* standardizing around __bytes__
* libsodium on linux (x86_64 & ARM)
* allow agg sig of 0 msgs
* alias g1/g2 element
* Try find gmp from chiavdf
* Build gmp and sodium from scratch
* Fix ARM sodium and gmp - start debugging macos
* delete dist dir on ARM shr's
* Move all but Windows to build verbosity 0
* Changes required for chia-blockchain (deepcopy)
* ADD hd keys tests
* added gmp include folder to python-bindings cmake
* pyecc tests
* Move ci's to pytest
* Add test vectors ietf
* g2 elements start with 0b000 now
* Use FromSeed
* Revert bad test vectors

Co-authored-by: Alex Wice <[email protected]>
Co-authored-by: Gene Hoffman <[email protected]>
Co-authored-by: Bill Blanke <[email protected]>
Co-authored-by: Gene Hoffman <[email protected]>

Author: 4 people

.clang-format

@@ -0,0 +1,17 @@
+BasedOnStyle: Google
+UseTab: Never
+ColumnLimit: 80
+IndentWidth: 4
+TabWidth: 4
+AllowShortIfStatementsOnASingleLine: false
+IndentCaseLabels: false
+AccessModifierOffset: -4
+BinPackArguments: false
+BinPackParameters: false
+AlignAfterOpenBracket: AlwaysBreak
+IndentCaseLabels: true
+AllowAllParametersOfDeclarationOnNextLine: false
+BreakBeforeBraces: Custom
+BraceWrapping:
+    AfterFunction: true
+PenaltyReturnTypeOnItsOwnLine: 1000

.flake8

@@ -1,5 +1,4 @@
 [flake8]
 max-line-length = 120
-exclude = ./typings/**/*
+exclude = ./typings/**/* python-impl/fields.py
 ignore = E203,W503
-

.github/workflows/build-aarch64.yml

@@ -18,7 +18,7 @@ jobs:
         access_token: ${{ github.token }}
 
     - name: Checkout code
-      uses: actions/checkout@v1
+      uses: actions/checkout@v2
       with:
         fetch-depth: 0
         # we need fetch-depth 0 so setuptools_scm can resolve tags
@@ -47,13 +47,13 @@ jobs:
       run: |
         source venv/bin/activate
         pip install flake8
-        flake8 src setup.py
+        flake8 src setup.py python-bindings python-impl
 
     - name: Lint source with mypy
       run: |
         source venv/bin/activate
         pip install mypy
-        mypy --config-file mypi.ini python-impl python-bindings
+        mypy --config-file mypi.ini python-bindings python-impl
 
     - name: Build wheel and test
       run: |
@@ -64,16 +64,22 @@ jobs:
         CIBW_BUILD: cp38-*
         CIBW_MANYLINUX_AARCH64_IMAGE: manylinux2014
         # we need boost
+        CIBW_BUILD_VERBOSITY_LINUX: 0
         CIBW_BEFORE_BUILD_LINUX: >
           yum -y install epel-release
-          && yum -y install cmake3
+          && yum -y install cmake3 lzip
           && ln -s /usr/bin/cmake3 /usr/local/bin/cmake
           && python -m pip install --upgrade pip
           && cmake --version
           && uname -a
-        CIBW_BUILD_VERBOSITY_LINUX: 0
+          && curl -L https://gmplib.org/download/gmp/gmp-6.2.0.tar.lz | lzip -dc | tar x
+          && cd gmp-6.2.0 && ./configure --enable-fat
+          && make && make install && cd .. && rm -rf gmp-6.2.0
+          && curl -L https://download.libsodium.org/libsodium/releases/libsodium-1.0.18-stable.tar.gz | tar xz
+          && cd libsodium-stable && ./configure
+          && make && make install && cd .. && rm -rf libsodium-stable
         CIBW_TEST_REQUIRES: pytest
-        CIBW_TEST_COMMAND: python {project}/python-bindings/test.py
+        CIBW_TEST_COMMAND: pytest -v {project}/python-bindings/test.py
 
     - name: Upload artifacts
       uses: actions/upload-artifact@v1
@@ -107,6 +113,7 @@ jobs:
         source venv/bin/activate
         twine upload --non-interactive --skip-existing --verbose 'dist/*'
 
-    - name: Clean up venv
+    - name: Clean up
       run: |
         rm -rf venv
+        rm -rf dist

.github/workflows/build.yml

@@ -17,7 +17,8 @@ jobs:
       with:
         access_token: ${{ github.token }}
 
-    - uses: actions/checkout@v1
+    - name: Checkout code
+      uses: actions/checkout@v2
       with:
         fetch-depth: 0
         # we need fetch-depth 0 so setuptools_scm can resolve tags
@@ -45,12 +46,12 @@ jobs:
     - name: Lint source with flake8
       run: |
         pip install flake8
-        flake8 src setup.py
+        flake8 src setup.py python-bindings python-impl
 
     - name: Lint source with mypy
       run: |
         pip install mypy
-        mypy --config-file mypi.ini python-impl python-bindings
+        mypy --config-file mypi.ini python-bindings python-impl
 
     - name: Build source distribution with MacOS
       if: startsWith(matrix.os, 'mac')
@@ -62,27 +63,42 @@ jobs:
       run: |
         python -m cibuildwheel --output-dir dist
       env:
+        CIBW_BUILD_VERBOSITY_MACOS: 0
+        CIBW_BUILD_VERBOSITY_LINUX: 0
+        CIBW_BUILD_VERBOSITY_WINDOWS: 1
         # Python 3.7 and 3.8
         CIBW_BUILD: cp37-* cp38-*
         # don't build i686 targets, can't seem to find cmake for these
         CIBW_SKIP: '*-manylinux_i686 *-win32'
         # we need boost
         CIBW_TEST_REQUIRES: pytest
         CIBW_MANYLINUX_X86_64_IMAGE: manylinux2010
+        CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.3-Linux-`uname -m`/bin:$PATH"
         CIBW_BEFORE_BUILD_LINUX: >
-          curl -L https://github.com/Kitware/CMake/releases/download/v3.17.0/cmake-3.17.0-Linux-`uname -m`.sh > cmake.sh
+          yum -y install lzip
+          && curl -L https://github.com/Kitware/CMake/releases/download/v3.17.3/cmake-3.17.3-Linux-`uname -m`.sh > cmake.sh
           && yes | sh cmake.sh | cat
-          && rm -f /usr/bin/cmake
+          && rm -f /usr/bin/cmake && hash -r
           && python -m pip install --upgrade pip
           && which cmake
           && cmake --version
-        CIBW_BEFORE_BUILD_MACOS: python -m pip install --upgrade pip
+          && curl -L https://gmplib.org/download/gmp/gmp-6.2.0.tar.lz | lzip -dc | tar x
+          && cd gmp-6.2.0 && ./configure --enable-fat
+          && make && make install && cd .. && rm -rf gmp-6.2.0
+          && curl -L https://download.libsodium.org/libsodium/releases/libsodium-1.0.18-stable.tar.gz | tar xz
+          && cd libsodium-stable && ./configure
+          && make && make install && cd .. && rm -rf libsodium-stable
+        # GH hosted runner has gmp and libsodium natively
+        CIBW_BEFORE_BUILD_MACOS: >
+          python -m pip install --upgrade pip
         CIBW_ENVIRONMENT_MACOS: MACOSX_DEPLOYMENT_TARGET=10.14
-        CIBW_BUILD_VERBOSITY_MACOS: 0
         CIBW_BEFORE_BUILD_WINDOWS: >
-          python -m pip install --upgrade pip && pwd
-          && git clone https://github.com/Chia-Network/relic_gmp_64.git
-          && ls -l relic_gmp_64
+          python -m pip install --upgrade pip
+          && pwd
+          && nuget install libsodium -Version 1.0.18
+          && ls -l libsodium.1.0.18/runtimes/win-x64/native/
+          && git clone https://github.com/Chia-Network/relic_ietf_64.git
+          && ls -l relic_ietf_64
           && git clone https://github.com/Chia-Network/mpir_gc_x64.git
           && ls -l mpir_gc_x64
         CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: >
@@ -98,12 +114,11 @@ jobs:
           && delocate-wheel -v -i mpir_gc_x64/mpir_skylake.dll {wheel}
           && delocate-wheel -v -i mpir_gc_x64/mpir_skylake_avx.dll {wheel}
           && cp {wheel} {dest_dir} && ls -l {dest_dir}
-          && rm -fr relic_gmp_64
+          && rm -fr relic_ietf_64
           && rm -fr mpir_gc_x64
         # The pure python implementation, is not used anywhere and will be replaced
         # pytest {project}/python-impl/tests.py
-        CIBW_TEST_COMMAND: python {project}/python-bindings/test.py
-        CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.0-Linux-`uname -m`/bin:$PATH"
+        CIBW_TEST_COMMAND: pytest -v {project}/python-bindings/test.py
         CIBW_REPAIR_WHEEL_COMMAND_MACOS: >
           pip uninstall -y delocate
           && pip install git+https://github.com/Chia-Network/delocate.git
@@ -114,9 +129,10 @@ jobs:
     - name: Mac OS build C++ and test
       if: startsWith(matrix.os, 'macos')
       run: |
-        brew install pkg-config
-
+        ls -l
+        export MACOSX_DEPLOYMENT_TARGET=10.14
         mkdir -p build
+        ls -l build
         cd build
         cmake ../
         cmake --build . -- -j 6
@@ -128,6 +144,11 @@ jobs:
       run: |
         sudo apt-get update
         sudo apt-get install valgrind libsodium-dev -y
+        sudo apt-get install snap -y
+        sudo apt-get remove --purge cmake -y
+        sudo snap install cmake --classic
+        hash -r
+        cmake --version
         mkdir -p build
         cd build
         cmake ../

.gitignore

@@ -53,3 +53,6 @@ blsbench.*
 *.pb.h
 
 **/.DS_Store
+
+*.whl
+venv

.gitmodules

@@ -1,3 +1,6 @@
 [submodule "contrib/pybind11"]
 	path = contrib/pybind11
 	url = https://github.com/pybind/pybind11.git
+[submodule "contrib/relic"]
+	path = contrib/relic
+	url = https://github.com/Chia-Network/relic

CMakeLists.txt

@@ -1,5 +1,4 @@
-# add_compile_definitions() was introduced in cmake 3.12 but this compiles on 3.10
-CMAKE_MINIMUM_REQUIRED(VERSION 3.10.0 FATAL_ERROR)
+CMAKE_MINIMUM_REQUIRED(VERSION 3.11.0 FATAL_ERROR)
 set(CMAKE_CXX_STANDARD 11)
 set(CMAKE_POSITION_INDEPENDENT_CODE ON)
 
@@ -49,6 +48,7 @@ set(VERBS "off" CACHE STRING "")
 set(ALLOC "AUTO" CACHE STRING "")
 set(SHLIB "OFF" CACHE STRING "")
 set(MULTI "PTHREAD" CACHE STRING "")
+set(DOCUM "off" CACHE STRING "")
 
 set(FP_PRIME 381 CACHE INTEGER "")
 
@@ -78,12 +78,13 @@ set(QUIET 1 CACHE INTEGER "")
 set(PP_EXT "LAZYR" CACHE STRING "")
 set(PP_METHD "LAZYR;OATEP" CACHE STRING "")
 
-add_subdirectory(contrib/relic)
+#add_subdirectory(contrib/relic)
 add_subdirectory(src)
 
 if (EMSCRIPTEN)
   add_subdirectory(js-bindings)
 else()
   # emscripten can't build python bindings, it produces only javascript
+  # add_subdirectory(contrib/pybind11)
   add_subdirectory(python-bindings)
 endif()

README.md

@@ -17,11 +17,10 @@ Implements BLS signatures with aggregation as in
 , using [relic toolkit](https://github.com/relic-toolkit/relic)
 for cryptographic primitives (pairings, EC, hashing).
 The [BLS12-381](https://github.com/zkcrypto/pairing/tree/master/src/bls12_381)
-curve is used. The spec is
+curve is used. The original spec is
 [here](https://github.com/Chia-Network/bls-signatures/tree/master/SPEC.md).
-This library will be migrating to the
-[IETF BLS RFC](https://datatracker.ietf.org/doc/draft-irtf-cfrg-bls-signature/)
-shortly.
+This library now implements
+[IETF BLS RFC](https://datatracker.ietf.org/doc/draft-irtf-cfrg-bls-signature/).
 
 Features:
 * Non-interactive signature aggregation on identical or distinct messages
@@ -266,15 +265,12 @@ function. Note: relic is used with the Apache 2.0 license.
 Libsodium and GMP are optional dependencies: libsodium gives secure memory
 allocation, and GMP speeds up the library by ~ 3x. To install them, either
 download them from github and follow the instructions for each repo, or use
-a package manager like APT or brew.
+a package manager like APT or brew. You can follow the recipe used to build
+python wheels for multiple platforms in `.github/workflows/`
 
 ### Discussion
-Discussion about this library and other Chia related development is on Keybase.
-Install Keybase, and run the following to join the Chia public channels:
-
-```bash
-keybase team request-access chia_network.public
-```
+Discussion about this library and other Chia related development is in Chia's
+[public Keybase channels](https://keybase.io/team/chia_network.public).
 
 ### Code style
 * Always use uint8_t for bytes
@@ -296,10 +292,10 @@ build binary wheels for MacOS, Linux (x64 and aarch64), and Windows and publish
 them with a source wheel on PyPi. See `.github/workflows/build.yml`. CMake uses
 [FetchContent](https://cmake.org/cmake/help/latest/module/FetchContent.html)
 to download [pybind11](https://github.com/pybind/pybind11) for the Python
-bindings. Building is then managed by
-[cibuildwheel](https://github.com/joerick/cibuildwheel). Further installation
-is then available via `pip install blspy` e.g. The ci builds include GMP and
-libsoduium.
+bindings and relic from a chia relic forked reporitory. Building is then
+managed by [cibuildwheel](https://github.com/joerick/cibuildwheel).
+Further installation is then available via `pip install blspy` e.g. The ci
+builds include GMP and soduium.
 
 ### Contributing and workflow
 Contributions are welcome and more details are available in chia-blockchain's