Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependency Updates #256

Merged
merged 22 commits into from
Mar 22, 2023
Merged

Dependency Updates #256

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
27021c6
update docs with py>=3.8 and rai>=1.2.5
MattToast Feb 10, 2023
f1280a9
fail smart build with onnx on py310, float protobuf
MattToast Feb 10, 2023
7677e31
float ray
MattToast Feb 10, 2023
efb1f82
remove impossible deps for py310
MattToast Feb 10, 2023
05a5ea5
Make onnx version nullable, add nullish checks where needed
MattToast Feb 10, 2023
a68f74a
move rai checks to RedisAIVersion
MattToast Feb 10, 2023
dc8cbf5
replace undefined symbol
MattToast Feb 10, 2023
0850315
use bound methods
MattToast Feb 10, 2023
896a6e0
very gross fix for osx, onnx, rai1.2.7
MattToast Feb 10, 2023
4800ae0
update ci
MattToast Feb 10, 2023
7ced3bd
center badges
MattToast Feb 10, 2023
fb72bd9
ci stringify 3.10
MattToast Feb 10, 2023
cfa6a19
update compatible compilers
MattToast Feb 13, 2023
2bd77b7
Update changelog
MattToast Feb 13, 2023
b116493
Bring up to date, fix chanelog order
MattToast Feb 14, 2023
30cc1f7
fix comments
MattToast Feb 14, 2023
0eb4e15
style
MattToast Feb 15, 2023
f719bed
Merge branch 'develop' into update-deps
MattToast Mar 6, 2023
519814d
remove duplicated line in changelog
MattToast Mar 6, 2023
0263ccf
better err msgs
MattToast Mar 14, 2023
03ded5b
Bring up to date with upstream
MattToast Mar 17, 2023
16b95a2
Bring upto date with upstream x2
MattToast Mar 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 12 additions & 13 deletions .github/workflows/run_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,16 @@ jobs:
matrix:
os: [macos-10.15, ubuntu-20.04] # Operating systems
compiler: [8] # GNU compiler version
rai: [1.2.3, 1.2.5, 1.2.7] # Redis AI versions
py_v: [3.7, 3.8, 3.9] # Python versions
rai: [1.2.5, 1.2.7] # Redis AI versions
py_v: [3.8, 3.9, '3.10'] # Python versions
exclude:
- os: macos-10.15 # Do not build with Redis AI 1.2.5 on MacOS
# Do not build with Redis AI 1.2.5 on MacOS
- os: macos-10.15
rai: 1.2.5
# Do not build Redis AI 1.2.5 with py3.10
# as wheels for dependecies are not availble
- py_v: '3.10'
rai: 1.2.5
- py_v: 3.7 # ONNX requires python >= 3.8
rai: 1.2.7

env:
SMARTSIM_REDISAI: ${{ matrix.rai }}
Expand Down Expand Up @@ -79,17 +82,13 @@ jobs:
python -m pip install git+https://github.com/CrayLabs/SmartRedis.git@develop#egg=smartredis
python -m pip install .[dev,ml,ray]

- name: Install ML Runtimes with Smart
if: contains( matrix.os, 'macos' )
run: smart build --device cpu -v

- name: Install ML Runtimes with Smart (with pt, tf, and onnx support)
if: contains( matrix.os, 'ubuntu' ) && (matrix.py_v != 3.9 || matrix.rai != '1.2.3')
if: (matrix.py_v != '3.10')
run: smart build --device cpu --onnx -v

- name: Install ML Runtimes with Smart excluding PyTorch for Ubuntu/Python3.9/RAI1.2.3 combo
if: contains( matrix.os, 'ubuntu' ) && matrix.py_v == 3.9 && matrix.rai == '1.2.3'
run: smart build --device cpu --no_pt --onnx -v
- name: Install ML Runtimes with Smart (with pt and tf support)
if: (matrix.py_v == '3.10')
run: smart build --device cpu -v

- name: Run Pytest
run: |
Expand Down
12 changes: 8 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
</div>


<div align="center">

[![License](https://img.shields.io/github/license/CrayLabs/SmartSim)](https://github.com/CrayLabs/SmartSim/blob/master/LICENSE.md)
![GitHub last commit](https://img.shields.io/github/last-commit/CrayLabs/SmartSim)
![GitHub deployments](https://img.shields.io/github/deployments/CrayLabs/SmartSim/github-pages?label=doc%20build)
Expand All @@ -27,6 +29,8 @@
[![codecov](https://codecov.io/gh/CrayLabs/SmartSim/branch/develop/graph/badge.svg?token=96HFI2F45E)](https://codecov.io/gh/CrayLabs/SmartSim)
[![Downloads](https://static.pepy.tech/personalized-badge/smartsim?period=total&units=international_system&left_color=grey&right_color=orange&left_text=Downloads)](https://pepy.tech/project/smartsim)

</div>

------------

# SmartSim
Expand Down Expand Up @@ -683,17 +687,17 @@ from C, C++, Fortran and Python with the SmartRedis Clients:
</thead>
<tbody style="text-align:center">
<tr>
<td rowspan="3">1.2.3-1.2.4</td>
<td rowspan="3">1.2.7</td>
<td>PyTorch</td>
<td>1.7.x</td>
<td>1.11.x</td>
</tr>
<tr>
<td>TensorFlow\Keras</td>
<td>2.4.x-2.5.x</td>
<td>2.8.x</td>
</tr>
<tr>
<td>ONNX</td>
<td>1.9.x</td>
<td>1.11.x</td>
</tr>
<td rowspan="3">1.2.5</td>
<td>PyTorch</td>
Expand Down
17 changes: 12 additions & 5 deletions doc/changelog.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,23 +23,30 @@ This section details changes made in the development branch that have not yet be
Description

- Update to current version of Redis
- Add support for Python 3.10, deprecate support for Python 3.7 and RedisAI 1.2.3
- Allow for models to be launched independently as batch jobs
- Fix bug in colocated database entrypoint when loading PyTorch models
- Add support for RedisAI 1.2.7, pyTorch 1.11.0, Tensorflow 2.8.0, ONNXRuntime 1.11.1
- Allow for models to be launched independently as batch jobs

Detailed Notes

- Update from Redis version 6.0.8 to 7.0.5. (PR258_)
- Fix bug in colocated database entrypoint stemming from uninitialized variables. This bug affects PyTorch models being loaded into the database. (PR237_)
- The release of RedisAI 1.2.7 allows us to update support for recent versions of pyTorch, Tensorflow, and ONNX (PR234_)
- Make installation of correct Torch backend more reliable according to instruction from pyTorch
- Adds support for Python 3.10 without the ONNX machine learning backend. Deprecates support for
Python 3.7 as it will stop receiving security updates. Deprecates support for RedisAI 1.2.3.
Update the build process to be able to correctly fetch supported dependencies. If a user
attempts to build an unsupported dependency, an error message is shown highlighting the
discrepancy. (PR256_)
- Models were given a `batch_settings` attribute. When launching a model through `Experiment.start`
the `Experiment` will first check for a non-nullish value at that attribute. If the check is
satisfied, the `Experiment` will attempt to wrap the underlying run command in a batch job using
the object referenced at `Model.batch_settings` as the batch settings for the job. If the check
is not satisfied, the `Model` is launched in the traditional manner as a job step. (PR245_)
- Fix bug in colocated database entrypoint stemming from uninitialized variables. This bug affects PyTorch models being loaded into the database. (PR237_)
- The release of RedisAI 1.2.7 allows us to update support for recent versions of PyTorch, Tensorflow, and ONNX (PR234_)
- Make installation of correct Torch backend more reliable according to instruction from PyTorch

.. _PR255: https://github.com/CrayLabs/SmartSim/pull/258
.. _PR258: https://github.com/CrayLabs/SmartSim/pull/258
.. _PR256: https://github.com/CrayLabs/SmartSim/pull/256
.. _PR245: https://github.com/CrayLabs/SmartSim/pull/245
.. _PR237: https://github.com/CrayLabs/SmartSim/pull/237
.. _PR234: https://github.com/CrayLabs/SmartSim/pull/234
Expand Down
13 changes: 6 additions & 7 deletions doc/installation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Prerequisites

The base prerequisites to install SmartSim and SmartRedis are:

- Python 3.7-3.9
- Python 3.8-3.10
- Pip
- Cmake 3.13.x (or later)
- C compiler
Expand All @@ -22,7 +22,7 @@ The base prerequisites to install SmartSim and SmartRedis are:
For most developer systems, many of these packages will already
be installed.

GCC 5-9 is recommended. There are known bugs with GCC >= 10.
GCC 5-9 or GCC>=11 is recommended. There are known bugs with GCC 10.

Git LFS can be installed through ``conda install git-lfs``

Expand All @@ -46,11 +46,11 @@ Supported Versions
* - MacOS
- x86_64
- Not supported
- 3.7 - 3.9
- 3.8 - 3.10
* - Linux
- x86_64
- Nvidia
- 3.7 - 3.9
- 3.8 - 3.10


.. note::
Expand All @@ -69,7 +69,6 @@ the version of the ML libraries).
+==================+==========+=============+===============+
| 1.2.7 (default) | 1.11.0 | 2.8.0 | 1.11.1 |
| 1.2.5 | 1.9.0 | 2.6.0 | 1.9.0 |
| 1.2.3 | 1.7.0 | 2.5.2 | 1.9.0 |
+------------------+----------+-------------+---------------+

TensorFlow_ 2.0 and Keras_ are supported through graph freezing_.
Expand Down Expand Up @@ -243,9 +242,9 @@ pre-built wheels that SmartSim does.
* - Platform
- Python Versions
* - MacOS
- 3.7 - 3.9
- 3.8 - 3.10
* - Linux
- 3.7 - 3.9
- 3.8 - 3.10


The Python client for SmartRedis is installed through
Expand Down
4 changes: 2 additions & 2 deletions setup.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ contact_email = [email protected]
license = BSD 2-Clause License
keywords = scientific, ai, workflow, hpc, analysis
classifiers =
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
Programming Language :: Python :: 3.10
License :: OSI Approved :: BSD License
Intended Audience :: Science/Research
Topic :: Scientific/Engineering
Expand All @@ -29,7 +29,7 @@ setup_requires =
setuptools>=39.2
cmake>=3.13
include_package_data = True
python_requires = >=3.7
python_requires = >=3.8,<3.11



Expand Down
8 changes: 4 additions & 4 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ class BuildError(Exception):
# see https://github.com/google/or-tools/issues/616
class InstallPlatlib(install):
def finalize_options(self):
install.finalize_options(self)
super().finalize_options()
if self.distribution.has_ext_modules():
self.install_lib = self.install_platlib

Expand All @@ -118,7 +118,7 @@ def run(self):
database_builder.cleanup()

# run original build_py command
build_py.run(self)
super().run()


# Tested with wheel v0.29.0
Expand All @@ -141,7 +141,7 @@ def has_ext_modules(_placeholder):
"redis==3.5.3",
"tqdm>=4.50.2",
"filelock>=3.4.2",
"protobuf==3.20",
"protobuf~=3.20",
]

# Add SmartRedis at specific version
Expand All @@ -159,7 +159,7 @@ def has_ext_modules(_placeholder):
# see smartsim/_core/_install/buildenv.py for more details
"ml": versions.ml_extras_required(),
"ray": [
"ray==1.6",
"ray~=1.6",
],
}

Expand Down
4 changes: 2 additions & 2 deletions smartsim/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@
# -*- coding: utf-8 -*-
from .version import __version__ as __version__

if sys.version_info < (3, 7): # pragma: no cover
sys.exit("Python 3.7 or greater must be used with SmartSim.")
if sys.version_info < (3, 8): # pragma: no cover
sys.exit("Python 3.8 or greater must be used with SmartSim.")

# Main API module
from .experiment import Experiment
33 changes: 13 additions & 20 deletions smartsim/_core/_cli/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,11 +152,6 @@ def __init__(self):
# REDIS/KeyDB
self.build_database()

if self.verbose:
logger.info("Version Information:")
vers = self.versions.as_dict()
print(tabulate(vers, headers=vers.keys(), tablefmt="github"), "\n")

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like useful debug info -- why did we take it out?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question!! There was another block roughly 10 lines earlier (~line 146) which was making roughly same print out. I figured two copies was redundant and removed the extra.

I also meant to call that out in the PR description because I figured that would be non-obvious from the diff, but didn't because I forgot and am bad at my job, lol.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there was a before/after display around selecting files for the database Redis/KeyDB. Maybe that was still useful, but we're probably beyond that point now

# REDISAI
self.build_redis_ai(
str(args.device),
Expand Down Expand Up @@ -201,23 +196,17 @@ def build_redis_ai(
):

# make sure user isn't trying to do something silly on MacOS
if self.build_env.PLATFORM == "darwin":
if device == "gpu":
logger.error("SmartSim does not support GPU on MacOS")
sys.exit(1)
if onnx and self.versions.REDISAI < "1.2.6":
logger.error("RedisAI < 1.2.6 does not support ONNX on MacOS")
sys.exit(1)
if self.versions.REDISAI == "1.2.4" or self.versions.REDISAI == "1.2.5":
logger.error("RedisAI support for MacOS is broken in 1.2.4 and 1.2.5")
sys.exit(1)
if self.build_env.PLATFORM == "darwin" and device == "gpu":
raise BuildError("SmartSim does not support GPU on MacOS")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still support RAI 1.2.5, so some of these checks are still needed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also something I should have called out in the PR description!!

We were making two very similar sets of checks in two different location: both here and in in Versioner/RedisAIVersion. I decided to try and move as many of the RAI compatibility checks into RedisAIVersioner to try and eliminate the duplication.

Now when RedisAIVersion is instantiated, it will use its knowledge of the platform it is on and the version of python it is running with to try and determine if the RedisAI/ML backends can be installed. If for whatever reason a compatibility inconstancy is found it will raise a SetupError immediately. This means that downstream we should be able to use the instanced RedisAIVersion without the need to make additional compatibility checks.

For instance this will now error out on OSX or with Python==3.10:

$ export SMARTSIM_REDISAI=1.2.5
$ smart build  # same erorr raised at ``pip install`` time as well
# Shortened for brevity
smartsim._core._install.buildenv.SetupError: RedisAI version must be greater than or equal to 1.2.7

But the same scenario will work on linux with Python<=3.9

$ export SMARTSIM_REDISAI=1.2.5
$ smart build
# Shortened for brevity
$ echo $?
0

Do let me know if you think this is acceptable/sufficient change! I could definitely be talked into potentially rolling this back if the changes are deemed "too severe"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds reasonable to me, but let's bring it up in standup tomorrow and make sure the whole team's onboard


# decide which runtimes to build
print("\nML Backends Requested")
print("-----------------------")
print(f" PyTorch {self.versions.TORCH}: {color_bool(torch)}")
print(f" TensorFlow {self.versions.TENSORFLOW}: {color_bool(tf)}")
print(f" ONNX {self.versions.ONNX}: {color_bool(onnx)}\n")
backends_table = [
["PyTorch", self.versions.TORCH, color_bool(torch)],
["TensorFlow", self.versions.TENSORFLOW, color_bool(tf)],
["ONNX", self.versions.ONNX or "Unavailable", color_bool(onnx)],
]
print(tabulate(backends_table, tablefmt="fancy_outline"), end="\n\n")
print(f"Building for GPU support: {color_bool(device == 'gpu')}\n")

self.check_backends_install()
Expand All @@ -231,7 +220,6 @@ def build_redis_ai(
if tf:
self.check_tf_install()

cmd = []
# TORCH
if torch:
if torch_dir:
Expand Down Expand Up @@ -341,6 +329,11 @@ def install_torch(self, device="cpu"):

def check_onnx_install(self):
"""Check Python environment for ONNX installation"""
if not self.versions.ONNX:
raise SetupError(
"An onnx wheel is not available for this version of python for the "
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be friendlier to print the requested Python version here. It would be friendlier yet to state which versions of Python we can support ONNX with.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call! I changed the wording to include the current version of python they are using and a brief summary of what combinations of SS and RAI are supported based on their current platform.

"requested onnx runtime"
)
try:
if not self.build_env.check_installed("onnx", self.versions.ONNX):
msg = (
Expand Down
1 change: 0 additions & 1 deletion smartsim/_core/_cli/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
from smartsim._core._cli.build import Build
from smartsim._core._cli.clean import Clean
from smartsim._core._cli.utils import get_install_path
from smartsim._core._install.buildenv import Versioner


def _usage():
Expand Down
Loading