Fixing CI (#363)

* Touch

* Touch back

* Updated GitHub CI

- Ran black black-nb mypy
- Removed python 3.5
- Fixed issue with MacOS Build using --features=-supports_dynamic_linker
- Updated Google DP to the latest commit
- Renamed Bazel target from bindings_test to pydp
- Updated pyproject.toml
- Fixed com_google_protobuf issue in WORKSPACE
- Disabled docker build on every commit / push

* Added pre-commit yaml file

- Ran pre-commit to fix lots of code issues
- Removed unused # type: ignore
- Updated setup.cfg with isort, mypy and flake8 rules
- Ran isort
- Disabled notebook tests
- Fixed matrix.os template variable in tests.yml

* Update docker to use python 3.9

- Fix mypy issues with missing typeshed
- Fixing tests.yml

* Fixing Dockerfile path to bazelisk

- Fixing issue with converting matrix.os to bazel config name

* Add ubuntu to .bazelrc file

* Fixing matrix.os conversion to bazel config

* Fixing multi-os issues in tests.yml

* More multi-os fixes

* Fixing windows bazel

* Fix windows

* Adding bazel caching for linux

- Fixing install wheel on windows
- Fixing print version

* More CI fixes

* tunshell

* Fixing macos wheel

* remove tunshell

* Quickfix serialized test data

* Added bazel-cache

- Combined publish actions into a single github action
- Added nightly tests for older versions of python
- Updated some package meta-data
- Bumped version to 1.0.3
- Added poetry.lock file to speed up CI

* Removed unused exclusion rule from pre-commit

* Adding other OSes to the Bazel Cache step

* Update test to include information about cached result from
tests/algorithms/test_bounded_mean/test_bounded_mean_int64_data.proto

* Refactored Docker and Docs actions

- Updated poetry lock file

* Renaming a few action names

- Testing versions cron

* tunshell into windows machine

* tunshell into mac

* Revert cron job for versions

- Remove tunshell for mac

* Trying specific version of cache plugin for windows tar cache error

* Added proper bazel cache directories for each OS plus Windows fix

* Trying different tar.exe fix for Windows

* Forgot to remove --output_base ./bazel-cache

* Trying windows tar fix

- Busting cache on linux and mac

* Trying older bazel cache strategy

* Debugging Windows Bazel Cache

* Trying actions/[email protected]

* Trying a different windows bazel cache approach

* Fix test yaml

- Trying Windows Bash tunshell

* woops checkout not cache

* Wrong runner

* Windows good times

* Windows cache might be working

* Try without symlinks?

* Putting cache code in all workflows

- Time to test Windows cache retrieval

* Added linting requirement back to main tests

* Fixed PR change requests

* Replace bazel with bazelisk

- Use dp submodule SHA

* debugging versions check

* Adding python binary and site-packages for different versions on unix

* Test all versions on mac and linux

* Adding Windows support for python path detection

* Reversed quotes and removed \ line endings

* Changing the var name on windows

* Using old syntax for the PYTHONPATH on windows

* Re-enabled windows on versions

* Replacing incorrect Capital R in replace

Co-authored-by: Lukasz Kozuchowski <[email protected]>

.bazelrc

@@ -1,3 +1,3 @@
-build:Linux --cxxopt='-std=c++17'
-build:macOS --cxxopt='-std=c++17'
-build:Windows --cxxopt='/std:c++17'
+build:linux --cxxopt='-std=c++17'
+build:macos --cxxopt='-std=c++17' --features=-supports_dynamic_linker
+build:windows --cxxopt='/std:c++17'

.dockerignore

@@ -1,2 +1,4 @@
 .git
 /third_party
+/build
+/bazel-*

.github/workflows/build-docs.yml

@@ -0,0 +1,112 @@
+name: Build Dev Docs
+
+on:
+  push:
+    branches:
+      - dev
+
+jobs:
+  build-docs:
+    strategy:
+      fail-fast: false
+      max-parallel: 1
+      matrix:
+        os: [ubuntu-latest]
+        python-version: [3.9]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Cache Bazel Unix
+        uses: actions/[email protected]
+        with:
+          path: |
+            ./bazel-PyDP
+            ./bazel-bin
+            ./bazel-cache
+            ./bazel-out
+          key: ${{ runner.os }}-bazel-${{ hashFiles('WORKSPACE') }}
+
+      - name: Build Google DP Unix
+        if: runner.os != 'Windows'
+        timeout-minutes: 20
+        run: |
+          PYTHONHOME=$(which python)
+          PYTHONPATH=$(python -c "import sys; print([x for x in sys.path if 'site-packages' in x][0]);")
+          BAZEL_CONFIG_OS=$(python -c "print('${{ matrix.os }}'.split('-')[0].lower().replace('ubuntu', 'linux'))")
+          echo "Running: ${{ matrix.os }}"
+          echo "Using BAZEL_CONFIG_OS: $BAZEL_CONFIG_OS"
+          bazel --output_base ./bazel-cache build src/python:pydp \
+                --config $BAZEL_CONFIG_OS \
+                --verbose_failures --action_env=PYTHON_BIN_PATH=$PYTHONHOME \
+                --action_env=PYTHON_LIB_PATH=$PYTHONPATH
+          cp -f ./bazel-bin/src/bindings/_pydp.so ./src/pydp
+
+      - name: Upgrade pip
+        run: |
+          pip install --upgrade --user pip
+
+      - name: Install Poetry
+        run: |
+          pip install poetry
+
+      - name: Get poetry cache dir
+        id: poetry-cache
+        run: |
+          echo "::set-output name=dir::$(poetry config cache-dir)"
+
+      - name: poetry cache
+        uses: actions/cache@v2
+        with:
+          path: ${{ steps.poetry-cache.outputs.dir }}
+          key: ${{ runner.os }}-pip-py${{ matrix.python-version }}-${{ hashFiles('**/pyproject.toml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-py${{ matrix.python-version }}-
+
+      - name: Install dependencies
+        run: |
+          poetry install
+
+      - name: Build PyDP Linux / Windows
+        if: runner.os != 'macOS'
+        run: |
+          poetry run python setup.py build bdist_wheel
+
+      - name: Install Wheel Unix
+        if: runner.os != 'Windows'
+        run: |
+          pip install `find -L ./ -name "*.whl"`
+
+      - name: Import Package
+        run: |
+          python -c "import pydp; print(pydp.__version__)"
+
+      - uses: ammaraskar/sphinx-action@master
+        with:
+          docs-folder: "docs/"
+
+      - name: Commit documentation changes
+        run: |
+          git clone https://github.com/ammaraskar/sphinx-action-test.git --branch gh-pages --single-branch gh-pages
+          cp -r docs/_build/html/* gh-pages/
+          cd gh-pages
+          git config --local user.email "[email protected]"
+          git config --local user.name "GitHub Action"
+          git add .
+          git commit -m "Update documentation" -a || true
+          # The above command will fail if no changes were present, so we ignore that
+
+      - name: Push changes
+        uses: ad-m/github-push-action@master
+        with:
+          branch: gh-pages
+          directory: gh-pages
+          force: true
+          github_token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/publish-docker.yml

@@ -0,0 +1,48 @@
+name: Publish Docker Image
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    name: Build Docker Container
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8, 3.9]
+    env:
+      IMAGE: openmined/pydp
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Show python version and build image name
+        id: build_name
+        run: |
+          python -c "import sys; print(sys.version)"
+          echo "::set-output name=image_name::${IMAGE}-${GITHUB_REF##*/}-${{matrix.python-version}}"
+
+      - name: Build images
+        run: docker build -t ${{steps.build_name.outputs.image_name}} --build-arg PYTHON_VERSION=${{matrix.python-version}} -f Dockerfile .
+
+      - name: Log into Docker
+        run: echo "${{ secrets.DOCKER_PASSWORD }}" | docker login -u ${{ secrets.DOCKER_LOGIN }} --password-stdin
+
+      - name: Tag and Push Image
+        run: |
+          docker tag ${{steps.build_name.outputs.image_name}} $IMAGE:${{ github.event.release.tag_name }}-py${{ matrix.python-version }}
+          docker push $IMAGE:${{ github.event.release.tag_name }}-py${{ matrix.python-version }}
+
+      - name: Tag Python 3.9 as latest
+        if: matrix.python-version == '3.9'
+        run: |
+          docker tag ${{steps.build_name.outputs.image_name}} $IMAGE:${{ github.event.release.tag_name }}
+          docker push $IMAGE:${{ github.event.release.tag_name }}
+          docker tag ${{steps.build_name.outputs.image_name}} $IMAGE:latest
+          docker push $IMAGE:latest

.github/workflows/publish.yml

@@ -0,0 +1,165 @@
+name: Publish PyDP
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  deploy:
+    strategy:
+      fail-fast: false
+      max-parallel: 12
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python-version: [3.6, 3.7, 3.8, 3.9]
+    runs-on: ${{ matrix.os }}
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Setup msbuild Windows
+        if: runner.os == 'Windows'
+        uses: microsoft/[email protected]
+
+      - name: Fix Paths Windows
+        # Make sure that tar.exe from Git is used not from Windows
+        if: runner.os == 'Windows'
+        run: |
+          @("C:\Program Files\Git\usr\bin") + (Get-Content $env:GITHUB_PATH) | Set-Content $env:GITHUB_PATH -Encoding utf8
+
+      - name: Cache Bazel Unix
+        # Not working on Windows: https://github.com/actions/cache/issues/576
+        if: runner.os != 'Windows'
+        uses: actions/[email protected]
+        with:
+          path: |
+            ./bazel-PyDP
+            ./bazel-bin
+            ./bazel-cache
+            ./bazel-out
+          key: ${{ runner.os }}-bazel-${{ hashFiles('WORKSPACE') }}
+
+      - name: Update environment variables Windows
+        if: runner.os == 'Windows'
+        # See:
+        #   - On Bazel cache: https://docs.bazel.build/versions/master/output_directories.html
+        run: |
+          echo "BAZEL_CACHE_DIR=$env:USERPROFILE\_bazel_$env:USERNAME" >> $env:GITHUB_ENV
+
+      - name: Cache Bazel Windows
+        if: runner.os == 'Windows'
+        # https://stackoverflow.com/questions/66870002/github-actions-cache-maven-m2-repository-on-windows-environment-c-users-run
+        uses: actions/[email protected]
+        with:
+          path: |
+            ./bazel-cache
+          key: ${{ runner.os }}-bazel-${{ hashFiles('WORKSPACE') }}
+
+      - name: Build Google DP Unix
+        if: runner.os != 'Windows'
+        timeout-minutes: 20
+        run: |
+          PYTHONHOME=$(which python)
+          PYTHONPATH=$(python -c "import sys; print([x for x in sys.path if 'site-packages' in x][0]);")
+          BAZEL_CONFIG_OS=$(python -c "print('${{ matrix.os }}'.split('-')[0].lower().replace('ubuntu', 'linux'))")
+          echo "Running: ${{ matrix.os }}"
+          echo "Using BAZEL_CONFIG_OS: $BAZEL_CONFIG_OS"
+          bazel --output_base ./bazel-cache build src/python:pydp \
+                --config $BAZEL_CONFIG_OS \
+                --verbose_failures --action_env=PYTHON_BIN_PATH=$PYTHONHOME \
+                --action_env=PYTHON_LIB_PATH=$PYTHONPATH
+          cp -f ./bazel-bin/src/bindings/_pydp.so ./src/pydp
+
+      - name: Build Google DP Windows
+        if: runner.os == 'Windows'
+        timeout-minutes: 20
+        run: |
+          $PYTHONHOME=$(python -c 'import sys; print(sys.executable);').replace('\', '/')
+          $PYTHONPATH=$(python -c "import sys; print([x for x in sys.path if 'site-packages' in x][0]);").replace('\', '/')
+          echo "PYTHONHOME=$PYTHONHOME"
+          echo "PYTHONPATH=$PYTHONPATH"
+          echo "Running: ${{ matrix.os }}"
+          bazel.exe --output_base ./bazel-cache build src/python:pydp --config windows --verbose_failures --action_env=PYTHON_BIN_PATH=$PYTHONHOME --action_env=PYTHON_LIB_PATH=$PYTHONPATH
+          copy ./bazel-bin/src/bindings/_pydp.so ./src/pydp/_pydp.pyd
+
+      - name: Upgrade pip
+        run: |
+          pip install --upgrade --user pip
+
+      - name: Install Poetry
+        run: |
+          pip install poetry
+
+      - name: Get poetry cache dir
+        id: poetry-cache
+        run: |
+          echo "::set-output name=dir::$(poetry config cache-dir)"
+
+      - name: poetry cache
+        uses: actions/cache@v2
+        with:
+          path: ${{ steps.poetry-cache.outputs.dir }}
+          key: ${{ runner.os }}-pip-py${{ matrix.python-version }}-${{ hashFiles('**/pyproject.toml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-py${{ matrix.python-version }}-
+
+      - name: Install dependencies
+        run: |
+          poetry install
+
+      - name: Build PyDP macOS
+        if: runner.os == 'macOS'
+        run: |
+          poetry run python setup.py build bdist_wheel --plat-name macosx_10_14_x86_64
+
+      - name: Build PyDP Linux / Windows
+        if: runner.os != 'macOS'
+        run: |
+          poetry run python setup.py build bdist_wheel
+
+      - name: Install Wheel Unix
+        if: runner.os != 'Windows'
+        run: |
+          pip install `find -L ./ -name "*.whl"`
+
+      - name: Install Wheel Windows
+        if: runner.os == 'Windows'
+        run: |
+          Get-ChildItem -Path ./ -Filter "*.whl" -Recurse -File | foreach {pip install $_.FullName}
+
+      - name: Import Package
+        run: |
+          python -c "import pydp; print(pydp.__version__)"
+
+      - name: Run Pytest
+        run: |
+          poetry run pytest tests -n auto
+
+      - name: Check Wheel Unix
+        if: runner.os != 'Windows'
+        run: |
+          twine check `find -L ./ -name "*.whl"`
+
+      - name: Check Wheel Windows
+        if: runner.os == 'Windows'
+        run: |
+          Get-ChildItem -Path ./ -Filter "*.whl" -Recurse -File | foreach {twine check $_.FullName}
+
+      - name: Renaming wheel
+        if: runner.os == 'Linux'
+        run: |
+          find . -name '*linux*.whl' -type f -exec bash -c 'mv "$1" "${1/linux/manylinux1}"' -- {} \;
+
+      # - name: Publishing the wheel
+      #   env:
+      #     TWINE_USERNAME: __token__
+      #     TWINE_PASSWORD: ${{ secrets.TOKEN }}
+      #   run: |
+      #     twine upload --skip-existing dist/*.whl