Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Cleaned pyroengine and refactored project #101

Merged
merged 41 commits into from
Aug 5, 2022
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
c5f675b
refactor: Refactored pyroengine package
frgfm Aug 3, 2022
7642103
test: Updated unittests
frgfm Aug 3, 2022
9e57cd3
chore: Updated setup
frgfm Aug 3, 2022
2b08470
refactor: Moved engine wrapper to src/
frgfm Aug 3, 2022
f19e663
docs: Updated documentation theme
frgfm Aug 3, 2022
01301c1
docs: Added makefile
frgfm Aug 3, 2022
c4764fa
ci: Updated CI jobs
frgfm Aug 3, 2022
6d3b19f
refactor: Updated ping script
frgfm Aug 3, 2022
7795727
refactor: Updated Dockerfile
frgfm Aug 3, 2022
3fb6bc2
ci: Updated collect_env
frgfm Aug 3, 2022
077364c
feat: Added possibility to load local files
frgfm Aug 3, 2022
26eec86
docs: Added docker build to makefile
frgfm Aug 3, 2022
59e2e11
docs: Updated documentation
frgfm Aug 3, 2022
c831ea4
refactor: Updated run.py
frgfm Aug 3, 2022
d10daa6
style: Fixed style
frgfm Aug 3, 2022
be02754
test: Added unittest for local file loading
frgfm Aug 3, 2022
7514324
test: Fixed unittests
frgfm Aug 3, 2022
96feab3
ci: Removed CI job
frgfm Aug 3, 2022
fb4d95b
docs: Fixed headers
frgfm Aug 3, 2022
33cc1aa
ci: Fixed RPI CI
frgfm Aug 3, 2022
2e52848
chore: Updated version specifier of pyroclient
frgfm Aug 3, 2022
34b3799
docs: Updated CONTRIBUTING
frgfm Aug 3, 2022
17005c1
docs: Updated README
frgfm Aug 3, 2022
d707a5c
ci: Fixed typos
frgfm Aug 4, 2022
e5dce64
chore: Updated gitignore
frgfm Aug 4, 2022
86287b9
chore: Updated build commands
frgfm Aug 4, 2022
ea017e1
chore: Updated pyproject
frgfm Aug 4, 2022
6f96f0a
style: Fixed style
frgfm Aug 4, 2022
42b586a
ci: Extended header check
frgfm Aug 4, 2022
ea99bfc
refactor: Refactored docker orchestration
frgfm Aug 4, 2022
5f215bb
docs: Updated makefile
frgfm Aug 4, 2022
a73cdc9
docs: Updated README
frgfm Aug 4, 2022
97c8d58
refactor: Refactored states management
frgfm Aug 4, 2022
2111196
test: Updated unittests
frgfm Aug 4, 2022
3c61776
docs: Updated header
frgfm Aug 4, 2022
576e331
feat: Cleaned run script
frgfm Aug 4, 2022
17dcd13
ci: Updated CI
frgfm Aug 4, 2022
57ff22a
refactor: Removed CMD from docker
frgfm Aug 4, 2022
ae5e91a
docs: Updated makefile
frgfm Aug 4, 2022
bd82ce7
style: Fixed imports
frgfm Aug 4, 2022
b66b75a
chore: Updated docker orchestration
frgfm Aug 4, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .coveragerc

This file was deleted.

5 changes: 3 additions & 2 deletions .flake8
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
[flake8]
max-line-length = 120
ignore = F401, E402, E265, F403, W503, W504, F821
exclude = venv*, .circleci, .git, docs
ignore = E203, E402, E265, F403, W503, W504, E731
exclude = .github, .git, venv*, docs, build
per-file-ignores = **/__init__.py:F401
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,12 @@ Steps to reproduce the behavior:
## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pyronear/pyro-engine/master/scripts/collect_env.py)
[environment collection script](https://raw.githubusercontent.com/pyronear/pyro-engine/main/.github/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pyronear/pyro-engine/master/scripts/collect_env.py
wget https://raw.githubusercontent.com/pyronear/pyro-engine/main/.github/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```
Expand Down
219 changes: 219 additions & 0 deletions .github/collect_env.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,219 @@
# Copyright (C) 2020-2022, Pyronear.

# This program is licensed under the Apache License 2.0.
# See LICENSE or go to <https://www.apache.org/licenses/LICENSE-2.0> for full license details.

"""
Based on https://github.com/pytorch/pytorch/blob/master/torch/utils/collect_env.py
This script outputs relevant system environment info
Run it with `python collect_env.py`.
"""

from __future__ import absolute_import, division, print_function, unicode_literals

import locale
import os
import re
import subprocess
import sys
from collections import namedtuple

try:
import pyroengine

ENGINE_AVAILABLE = True
except (ImportError, NameError, AttributeError):
ENGINE_AVAILABLE = False

try:
import onnxruntime

ONNX_AVAILABLE = True
except (ImportError, NameError, AttributeError):
ONNX_AVAILABLE = False

PY3 = sys.version_info >= (3, 0)


# System Environment Information
SystemEnv = namedtuple(
"SystemEnv",
[
"pyroengine_version",
"onnxruntime_version",
"os",
"python_version",
],
)


def run(command):
"""Returns (return-code, stdout, stderr)"""
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
output, err = p.communicate()
rc = p.returncode
if PY3:
enc = locale.getpreferredencoding()
output = output.decode(enc)
err = err.decode(enc)
return rc, output.strip(), err.strip()


def run_and_read_all(run_lambda, command):
"""Runs command using run_lambda; reads and returns entire output if rc is 0"""
rc, out, _ = run_lambda(command)
if rc != 0:
return None
return out


def run_and_parse_first_match(run_lambda, command, regex):
"""Runs command using run_lambda, returns the first regex match if it exists"""
rc, out, _ = run_lambda(command)
if rc != 0:
return None
match = re.search(regex, out)
if match is None:
return None
return match.group(1)


def get_platform():
if sys.platform.startswith("linux"):
return "linux"
elif sys.platform.startswith("win32"):
return "win32"
elif sys.platform.startswith("cygwin"):
return "cygwin"
elif sys.platform.startswith("darwin"):
return "darwin"
else:
return sys.platform


def get_mac_version(run_lambda):
return run_and_parse_first_match(run_lambda, "sw_vers -productVersion", r"(.*)")


def get_windows_version(run_lambda):
return run_and_read_all(run_lambda, "wmic os get Caption | findstr /v Caption")


def get_lsb_version(run_lambda):
return run_and_parse_first_match(run_lambda, "lsb_release -a", r"Description:\t(.*)")


def check_release_file(run_lambda):
return run_and_parse_first_match(run_lambda, "cat /etc/*-release", r'PRETTY_NAME="(.*)"')


def get_os(run_lambda):
platform = get_platform()

if platform == "win32" or platform == "cygwin":
return get_windows_version(run_lambda)

if platform == "darwin":
version = get_mac_version(run_lambda)
if version is None:
return None
return "Mac OSX {}".format(version)

if platform == "linux":
# Ubuntu/Debian based
desc = get_lsb_version(run_lambda)
if desc is not None:
return desc

# Try reading /etc/*-release
desc = check_release_file(run_lambda)
if desc is not None:
return desc

return platform

# Unknown platform
return platform


def get_env_info():
run_lambda = run

if ENGINE_AVAILABLE:
pyroengine_str = pyroengine.__version__
else:
pyroengine_str = "N/A"

if ONNX_AVAILABLE:
onnxruntime_str = onnxruntime.__version__
else:
onnxruntime_str = "N/A"

return SystemEnv(
pyroengine_version=pyroengine_str,
onnxruntime_version=onnxruntime_str,
python_version=".".join(map(str, sys.version_info[:3])),
os=get_os(run_lambda),
)


env_info_fmt = """
PyroEngine version: {pyroengine_version}
ONNX runtime version: {onnxruntime_version}

OS: {os}

Python version: {python_version}
""".strip()


def pretty_str(envinfo):
def replace_nones(dct, replacement="Could not collect"):
for key in dct.keys():
if dct[key] is not None:
continue
dct[key] = replacement
return dct

def replace_bools(dct, true="Yes", false="No"):
for key in dct.keys():
if dct[key] is True:
dct[key] = true
elif dct[key] is False:
dct[key] = false
return dct

def maybe_start_on_next_line(string):
# If `string` is multiline, prepend a \n to it.
if string is not None and len(string.split("\n")) > 1:
return "\n{}\n".format(string)
return string

mutable_dict = envinfo._asdict()

# Replace True with Yes, False with No
mutable_dict = replace_bools(mutable_dict)

# Replace all None objects with 'Could not collect'
mutable_dict = replace_nones(mutable_dict)

return env_info_fmt.format(**mutable_dict)


def get_pretty_env_info():
"""Collects environment information for debugging purposes

Returns:
str: environment information
"""
return pretty_str(get_env_info())


def main():
print("Collecting environment information...")
output = get_pretty_env_info()
print(output)


if __name__ == "__main__":
main()
67 changes: 0 additions & 67 deletions .github/validate_headers.py

This file was deleted.

34 changes: 34 additions & 0 deletions .github/workflows/builds.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: builds

on:
push:
branches: main
pull_request:
branches: main

jobs:
build:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python: [3.8]
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python }}
architecture: x64
- name: Cache python modules
uses: actions/cache@v2
with:
path: ~/.cache/pip
key: ${{ runner.os }}-python-${{ matrix.python }}-${{ hashFiles('pyproject.toml') }}-build
- name: Install package
run: |
python -m pip install --upgrade pip
pip install -e . --upgrade
- name: Import package
run: python -c "import pyroengine; print(pyroengine.__version__)"
17 changes: 17 additions & 0 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: docker

on:
push:
branches: main
pull_request:
branches: main

jobs:
package:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Build docker image
run: docker build . -t pyroengine:python3.8.1-slim
- name: Run docker container
run: docker run pyroengine:python3.8.1-slim python -c 'import pyroengine'
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: doc-deploy
on:
push:
branches: master
branches: main

jobs:
docs-publish:
Expand Down
Loading