Speed up categorical regressor with numba #3353
Conversation
Codecov ReportAttention: Patch coverage is
✅ All tests successful. No failed tests found.
Additional details and impacted files@@ Coverage Diff @@
## main #3353 +/- ##
==========================================
+ Coverage 73.09% 75.46% +2.36%
==========================================
Files 113 113
Lines 13139 13148 +9
==========================================
+ Hits 9604 9922 +318
+ Misses 3535 3226 -309
|
tests/test_preprocessing.py
Outdated
| np.testing.assert_array_almost_equal(adata.X, tester) | ||
|
|
||
|
|
||
| def test_regressor_categorical(): |
There was a problem hiding this comment.
I would
- explain why this test exists (to test against a previous implementation? I am impartial whether it's necessary TBH since we are already testing for reproducibility, could see getting rid of this)
- refactor the "Create org regressors" into a helper function like
create_original
There was a problem hiding this comment.
I can see your point here
There was a problem hiding this comment.
Do you have an an opinion on the first point? Is this test necessary? If so, perhaps a comment then?
There was a problem hiding this comment.
I think this is missing: #3353 (comment) and the first part of https://github.com/scverse/scanpy/pull/3353/files#r1836830351
src/scanpy/preprocessing/_simple.py
Outdated
| @@ -722,13 +737,13 @@ def regress_out( | |||
| "we regress on the mean for each category." | |||
| ) | |||
| logg.debug("... regressing on per-gene means within categories") | |||
| regressors = np.zeros(X.shape, dtype="float32") | |||
| # Create numpy array's from categorical variable | |||
| cats = np.int64(len(adata.obs[keys[0]].cat.categories)) | |||
There was a problem hiding this comment.
Also comment why np.int64
There was a problem hiding this comment.
because it has be done because of weird typing from pandas. So this ensures that it works within the kernel
There was a problem hiding this comment.
so len doesn’t return a Python int? That’s a pandas bug.
Co-authored-by: Ilan Gold <[email protected]>
Co-authored-by: Ilan Gold <[email protected]>
Co-authored-by: Ilan Gold <[email protected]>
tests/test_preprocessing.py
Outdated
| np.testing.assert_array_almost_equal(adata.X, tester) | ||
|
|
||
|
|
||
| def test_regressor_categorical(): |
There was a problem hiding this comment.
Do you have an an opinion on the first point? Is this test necessary? If so, perhaps a comment then?
|
I renamed one variable to make is clearer what it is. Added some comments that should add more context what the code is doing. |
Co-authored-by: Ilan Gold <[email protected]>
| X: np.ndarray, number_categories: int, cat_array: np.ndarray | ||
| ) -> np.ndarray: | ||
| # create regressor matrix for categorical variables | ||
| regressors = np.zeros(X.shape, dtype=X.dtype) |
There was a problem hiding this comment.
check dtype for behavior with integer dtype i.e., need to ensure this is a floating point matrix
| (["bulk_labels"], "regress_test_small_cat.npy", 1e-6), | ||
| ], | ||
| ) | ||
| def test_regress_out_reproducible(keys, test_file, atol): |
There was a problem hiding this comment.
Shouldn't we add a test for integer + float as a param to this test?
flying-sheep
changed the title
Co-authored-by: Philipp A. <[email protected]>
Co-authored-by: Ilan Gold <[email protected]>
Use numba to create the regressor for categorical regression