[BUG] Scipy 1.11.0 and 1.11.1 are breaking all our tests

[miguelgfierro]

Description

All recent PRs plus all recent nightly builds are broken.

• Rerun and clean notebooks with Python 3.9 #1947
• Review CPU notebooks and run with Python 3.9 #1950
• New URL of Glove from Huggingface #1949
• Nightly CPU https://github.com/microsoft/recommenders/actions/runs/5372464738
• Nightly GPU https://github.com/microsoft/recommenders/actions/runs/5372457721

In which platform does it happen?

How do we replicate the issue?

Expected behavior (i.e. solution)

Other Comments

[miguelgfierro]

The error we are getting is:

=================================== FAILURES ===================================
___________________________ test_sar_deep_dive_runs ____________________________

notebooks = ***'als_deep_dive': '/mnt/azureml/cr/j/4e944a170be74c0e9727bb7a9e80efcd/exe/wd/examples/02_model_collaborative_filtering...rk_movielens': '/mnt/azureml/cr/j/4e944a170be74c0e9727bb7a9e80efcd/exe/wd/examples/06_benchmarks/movielens.ipynb', ...***
output_notebook = 'output.ipynb', kernel_name = 'python3'

    @pytest.mark.notebooks
    def test_sar_deep_dive_runs(notebooks, output_notebook, kernel_name):
        notebook_path = notebooks["sar_deep_dive"]
>       pm.execute_notebook(notebook_path, output_notebook, kernel_name=kernel_name)

tests/unit/examples/test_notebooks_python.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/azureml-envs/azureml_2248098658e75fe22b1e778dcf414d40/lib/python3.9/site-packages/papermill/execute.py:[128](https://github.com/microsoft/recommenders/actions/runs/5388016531/jobs/9780549827#step:3:135): in execute_notebook
    raise_for_execution_errors(nb, output_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

nb = ***'cells': [***'id': 'ed4e3ed1', 'cell_type': 'markdown', 'source': '<span style="color:red; font-family:Helvetica Neue, ...end_time': '2023-06-27T09:42:50.951864', 'duration': 12.94534, 'exception': True***, 'nbformat': 4, 'nbformat_minor': 5***
output_path = 'output.ipynb'

    def raise_for_execution_errors(nb, output_path):
        """Assigned parameters into the appropriate place in the input notebook
    
        Parameters
        ----------
        nb : NotebookNode
           Executable notebook object
        output_path : str
           Path to write executed notebook
        """
        error = None
        for index, cell in enumerate(nb.cells):
            if cell.get("outputs") is None:
                continue
    
            for output in cell.outputs:
                if output.output_type == "error":
                    if output.ename == "SystemExit" and (output.evalue == "" or output.evalue == "0"):
                        continue
                    error = PapermillExecutionError(
                        cell_index=index,
                        exec_count=cell.execution_count,
                        source=cell.source,
                        ename=output.ename,
                        evalue=output.evalue,
                        traceback=output.traceback,
                    )
                    break
    
        if error:
            # Write notebook back out with the Error Message at the top of the Notebook, and a link to
            # the relevant cell (by adding a note just before the failure with an HTML anchor)
            error_msg = ERROR_MESSAGE_TEMPLATE % str(error.exec_count)
            error_msg_cell = nbformat.v4.new_markdown_cell(error_msg)
            error_msg_cell.metadata['tags'] = [ERROR_MARKER_TAG]
            error_anchor_cell = nbformat.v4.new_markdown_cell(ERROR_ANCHOR_MSG)
            error_anchor_cell.metadata['tags'] = [ERROR_MARKER_TAG]
    
            # put the anchor before the cell with the error, before all the indices change due to the
            # heading-prepending
            nb.cells.insert(error.cell_index, error_anchor_cell)
            nb.cells.insert(0, error_msg_cell)
    
            write_ipynb(nb, output_path)
>           raise error
E           papermill.exceptions.PapermillExecutionError: 
E           ---------------------------------------------------------------------------
E           Exception encountered at "In [9]":
E           ---------------------------------------------------------------------------
E           ValueError                                Traceback (most recent call last)
E           Cell In[9], line 1
E           ----> 1 top_k = model.recommend_k_items(test, top_k=TOP_K, remove_seen=True)
E           
E           File /mnt/azureml/cr/j/4e944a[170](https://github.com/microsoft/recommenders/actions/runs/5388016531/jobs/9780549827#step:3:177)be74c0e9727bb7a9e80efcd/exe/wd/recommenders/models/sar/sar_singlenode.py:533, in SARSingleNode.recommend_k_items(self, test, top_k, sort_top_k, remove_seen)
E               520 def recommend_k_items(self, test, top_k=10, sort_top_k=True, remove_seen=False):
E               521     """Recommend top K items for all users which are in the test set
E               522 
E               523     Args:
E              (...)
E               530         pandas.DataFrame: top k recommendation items for each user
E               531     """
E           --> 533     test_scores = self.score(test, remove_seen=remove_seen)
E               535     top_items, top_scores = get_top_k_scored_items(
E               536         scores=test_scores, top_k=top_k, sort_top_k=sort_top_k
E               537     )
E               539     df = pd.DataFrame(
E               540         ***
E               541             self.col_user: np.repeat(
E              (...)
E               546         ***
E               547     )
E           
E           File /mnt/azureml/cr/j/4e944a170be74c0e9727bb7a9e80efcd/exe/wd/recommenders/models/sar/sar_singlenode.py:346, in SARSingleNode.score(self, test, remove_seen)
E               344 # calculate raw scores with a matrix multiplication
E               345 logger.info("Calculating recommendation scores")
E           --> 346 test_scores = self.user_affinity[user_ids, :].dot(self.item_similarity)
E               348 # ensure we're working with a dense ndarray
E               349 if isinstance(test_scores, sparse.spmatrix):
E           
E           File /azureml-envs/azureml_[224](https://github.com/microsoft/recommenders/actions/runs/5388016531/jobs/9780549827#step:3:231)8098658e75fe22b1e778dcf414d40/lib/python3.9/site-packages/scipy/sparse/_base.py:411, in _spbase.dot(self, other)
E               409     return self * other
E               410 else:
E           --> 411     return self @ other
E           
E           File /azureml-envs/azureml_2248098658e75fe22b1e778dcf414d40/lib/python3.9/site-packages/scipy/sparse/_base.py:622, in _spbase.__matmul__(self, other)
E               620 def __matmul__(self, other):
E               621     if isscalarlike(other):
E           --> 622         raise ValueError("Scalar operands are not allowed, "
E               623                          "use '*' instead")
E               624     return self._mul_dispatch(other)
E           
E           ValueError: Scalar operands are not allowed, use '*' instead

See run https://github.com/microsoft/recommenders/actions/runs/5388016531/jobs/9780549827

[miguelgfierro]

They yanked the wheel and generated a new one yesterday: 1.11.1
This is what I get if I install 1.11.0:

$ pip install "scipy==1.11.0"
Collecting scipy==1.11.0
  Downloading scipy-1.11.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 36.5/36.5 MB 13.7 MB/s eta 0:00:00
Requirement already satisfied: numpy<1.28.0,>=1.21.6 in /home/u/anaconda/envs/recommenders/lib/python3.9/site-packages (from scipy==1.11.0) (1.24.3)
WARNING: The candidate selected for download or install is a yanked version: 'scipy' candidate (version 1.11.0 at https://files.pythonhosted.org/packages/77/d5/c17e175e85f49c94b241a2c2f5e0cf209f47eb85dd24ff0ccca0cb97f9f2/scipy-1.11.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from https://pypi.org/simple/scipy/) (requires-python:<3.13,>=3.9))
Reason for being yanked: License Violation
Installing collected packages: scipy
  Attempting uninstall: scipy
    Found existing installation: scipy 1.10.1
    Uninstalling scipy-1.10.1:
      Successfully uninstalled scipy-1.10.1
Successfully installed scipy-1.11.0

it doesn't seem the problem was a license violation btw :-)

[anargyri]

With conda upgrade it may work, usually conda is more robust than pip.

[miguelgfierro]

Scipy 1.11.1 doesn't work either, I get the same error.