Fixing CI

[madhavajay]

Description

Fixing CI.

Affected Dependencies

None

How has this been tested?

Locally and with CI.

Checklist

• I have followed the Contribution Guidelines and Code of Conduct
• I have commented my code following the OpenMined Styleguide
• I have labeled this PR with the relevant Type labels
• My changes are covered by tests

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[chinmayshah99]

Does this also solve #358 ?

[madhavajay]

Does this also solve #358 ?

Kind of. So it will build and install in CI, but really we should make sure that the wheel it creates can also install on any version of MacOS. There might be some extra tricks to making that happen, and we will likely need to add a second entry to the matrix to be sure:

macOS Big Sur 11.0 | macos-11.0
macOS Catalina 10.15 | macos-latest or macos-10.15

[madhavajay]

I also think this should be done a different way:
https://github.com/OpenMined/PyDP/blob/dev/tests/notebooks/test_all_notebooks.py

But before refactoring it would be good to get a clear understanding of the goal? I would highly suggest running linting and checks before committing in pre-commit and then again in the linting step of CI, but currently the pre-commit checks aren't all passing because there are a lot of flake8 issues.

Try this:

$ pre-commit run --all-files
Found 216 errors in 29 files (checked 42 source files)

However, I think they should be handled as a seperate set of tasks so for now the key is just to get the tests passing and the build working so that we can do a Python 3.9 release.

Once these are passing for 3.9 I can add a nightly build to run the older versions like 3.6, 3.7 and 3.8 every night, and a simple file which can be used to trigger them manually by doing something like this:

$ date > tests/trigger/versions

[replomancer]

I only investigated test_bounded_mean.py There may be some issues with other tests (they fail for me). Here's what I found so far.

This test:

PyDP/tests/algorithms/test_bounded_mean.py

Lines 69 to 73 in c387752

	def test_bounded_mean_int64(make_loaded_object):
	x = make_loaded_object(5, 100000000, 5)
	res = x.result()
	print(x, type(x), res, type(res))
	assert expect_near(5.0, res, 0.1)

means if you compute a dp mean of many many many fives, the result should be close to five.

Apparently for performance reasons, instead of building big lists of fives, an attempt is made here to load data:

PyDP/tests/algorithms/test_bounded_mean.py

Lines 49 to 62 in c387752

	if os.path.exists(dump_filepath):
	# Search for data dump to import
	data = Summary()
	data.load(dump_filepath)
	x.merge(data)
	else:
	# No data dump found, we have to init the alforithm from scratch
	# Add entries into algorithm
	example_list = [value] * size
	for _ in range(iter):
	x.add_entries(example_list)
	# Dump the initialized algorithm data to retrieve in future tests
	x.serialize().save(dump_filepath)

from here:

PyDP/tests/algorithms/test_bounded_mean/test_bounded_mean_int64_data.proto

Lines 1 to 2 in c387752

	;type.googleapis.com/differential_privacy.BoundedMeanSummary

This file looks a little different when something useful is serialized here. Currently instead of having many many fives, BoundedMean probably gets nothing and produces random results between lower and upper bounds. This test passes after data file is deleted but it takes almost 2 minutes on my machine.

Speaking of lower and upper bounds. Maybe something builds differently for me or somehow I downloaded another C++ lib version but for me this is also broken: (this was a result of some dev changes I was missing)

PyDP/tests/algorithms/test_bounded_mean.py

Line 47 in c387752

x = BoundedMean(1.0, 0, 0, 10, dtype="int64")

i.e. I get

E       RuntimeError: LInf sensitivity has to be positive but is 0

because

        x = BoundedMean(1.0, 0, 0, 10, dtype="int64")

means

        # lower_bound same as upper_bound:
        x = BoundedMean(epsilon=1.0, lower_bound=0, upper_bound=0, l0_sensitivity=10, dtype="int64")

I also get the same sensitivity error from here:

PyDP/tests/algorithms/test_bounded_mean.py

Lines 77 to 78 in c387752

	bm1 = BoundedMean(1, 0, 1, 10)
	bm2 = BoundedMean(1, 0, 1, 10)

I need to increase upper_bound for it to stop crashing.

I may have ideas/comments tomorrow.

[replomancer]

(edit: this was from missing some dev changes)
Other fails I see also result from sensitivity set to zero (LInf sensitivity is computed from other parameters) e.g.

PyDP/tests/algorithms/test_bounded_variance.py

Lines 6 to 11 in c387752

	def test_basic(self):
	example_data = [1, 5, 7, 9, 13]
	epsilon = 1.0
	delta = 0
	lower_bound, upper_bound = 0, 16
	bv = BoundedVariance(epsilon, delta, lower_bound, upper_bound, dtype="float")

Here

BoundedVariance(epsilon, delta, lower_bound, upper_bound, dtype="float") 

actually means

BoundedVariance(epsilon=epsilon, lower_bound=delta, upper_bound=lower_bound, l0_sensitivity=upper_bound, dtype="float") 

so both lower_bound and upper_bound are set to 0.

Similar story in TestBoundedStandardDeviation and TestBoundedSum.

[chinmayshah99]

LGTM! can you remove the tunnel ssh?

[madhavajay]

Versions less than 3.9 are now building correctly and all tests are passing: https://github.com/OpenMined/PyDP/actions/runs/875497191