CellProfiler plugins installation instructions - A draft

[callum-jpg]

I've pushed a simple setup.py file to the master branch of the plugins repo. The goal of this setup.py file is to define subsets of dependencies for particular plugins. For example, cellpose and stardist are distinct due to troublesome and potentially conflicting dependencies.

You can also see in the setup.py file the line install_requires = install_deps, where install_deps just refers to cellprofiler and cellprofiler_core, thus ensuring a user has these installed, whether it be from PyPI (4.2.5) or compiled from github (5.0.0). Though, perhaps a minimum version should be set here (ie. >= 4.2.1)

So, here would be the steps for installing the plugins:

1. Install CellProfiler from source (See instructions for: Windows; Mac Intel; Mac Apple Silicon; Linux)
2. In your terminal, run the command git clone https://github.com/CellProfiler/CellProfiler-plugins.git
3. Open CellProfiler and go to Preferences and set the CellProfiler plugins directory to the GitHub repository that you just cloned
4. You are now ready to use the basic CellProfiler plugins (DistanceTransform, PixelShuffle etc.)
5. If you would like to install the dependencies for the RunCellPose plugin, change the directory of your terminal to be inside the cellprofiler plugins repository you downloaded previously with cd CellProfiler-plugins/
6. Install the CellPose specific dependencies with pip install -e .[cellpose]
7. For StarDist, run pip install -e .[stardist] (please note: while having CellPose and StarDist in the same python environment is possible, it has been reported to be troublesome to install, so we recommend to chose either CellPose or StarDist)

@ErinWeisbart @fefossa @bethac07

[ErinWeisbart]

I think we want the repo README to point to our docs for install instructions.
I'm copying the text that was there below so we have it here next to what Callum has written in case it helps us.

Beginner-level instructions

Beginner guidance for using plugins in CellProfiler

Beginner instructions for installing runCellpose, runStarDist, and all other plugins with CellProfiler from source:

• Windows
• Mac_Intel_version

Use

1. Clone this repository into a folder (PLUGIN_DIRECTORY) on your local computer:

   cd PLUGIN_DIRECTORY
   git clone https://github.com/CellProfiler/CellProfiler-plugins.git
   

   Alternatively download zip and manually extract to PLUGIN_DIRECTORY.

2. Install required dependencies:

   cd CellProfiler-plugins
   pip install -r requirements.txt
   

   To install CellProfiler-plugins on a windows machine with support for the deep learning module ClassifyPixels-UNet make sure you have Visual Studio 2017 installed then use

   cd CellProfiler-plugins
   pip install -r requirements-windows.txt
   

3. Configure CellProfiler plugins directory in the GUI via Preferences > CellProfiler plugins directory (you will need to restart CellProfiler for the change to take effect). When running CellProfiler via the command line, use the --plugins-directory flag to specify the plugins directory, for example:

   cellprofiler --run --run-headless --project PROJECT_FILE --plugins-directory PLUGIN_DIRECTORY/CellProfiler-plugins
   

ImageJ requirements

If using the RunImageJScript module, please note:

• You will also need to install Maven
• CellProfiler will need to be built from source due to the requirement of additional libraries

[ErinWeisbart]

@callum-jpg

where install_deps just refers to cellprofiler and cellprofiler_core, thus ensuring a user has these installed, whether it be from PyPI (4.2.5)

Naive questions:

• Does this setup mean that you cannot use plugins with built versions of CP? Because some (most?) of the plugins work just fine with built and it would be nice not to lose that function.
• Plugins with external dependencies CAN work with binary CP distributions #166 How does this issue fit in?

[callum-jpg]

You will still be able to use the "basic" plugins with the binary distribution of CellProfiler. Where I had written about the install_deps mostly refers to if a user is planning on installing dependencies for a plugin like cellpose which currently doesn't work with the binary. In this case, the user would run pip install -e .[cellpose], which would check if the PyPI or github source version of cellprofiler was installed.

Interestingly, now that we know we can have a CellProfiler plugin make calls to docker, we could feasibly have all of the plugins (including cellpose, stardist etc) work with the built version of cellprofiler. In this scenario, it would require docker to be installed and for each non-basic plugin to have an associated docker. This would be achieved by having the cellprofiler plugin make subprocess calls. Later releases of cellprofiler could add docker as a dependency so we can use the docker api in the plugin module. I think installing docker is much, much easier than installing conda+pytorch/tensorflow (though this option would still be available if a user wants).

I think the docker solution is markedly easier and more reproducible than the issue linked above. This way we can confirm on our side that a docker is built correctly and works without running into user-specific dependency installation issues.

Outstanding questions/thoughts with getting docker to work with cellprofiler after chatting with Nodar:

• How is multi-threadhing handled? My thought for this was to assign any image passed to the docker image a UUID so that multiple images can be passed simultaneously. This needs to be tested.
• How does a docker handle receiving multiple subprocess calls?
• We could have a docker image for each plugin that required it. The version of that docker image would be linked to the version of the plugin module being used (ie. what is defined by variable_revision_number)
• We could also build docker images using alpine slim as the base (it's only 5MB) so the burden on an end users system is minimal

[fefossa]

Windows machine:

1. I've created an env conda create --name cp_test_cellpose python=3.8;
2. Cloned CellProfiler
3. Then:

pip install numpy
pip install -e .

4. Got the error:

Obtaining file:///C:/Users/fossa/Documents/github/cp_plugins_test/CellProfiler
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting boto3>=1.12.28 (from CellProfiler==5.0.0b1)
  Downloading boto3-1.26.140-py3-none-any.whl (135 kB)
     ---------------------------------------- 135.6/135.6 kB 1.1 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of cellprofiler to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement cellprofiler-core==5.0.0 (from cellprofiler) (from versions: 4.0.0, 4.0.2, 4.0.3, 4.0.4, 4.0.5, 4.0.6, 4.0.7, 4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.2.0, 4.2.1, 4.2.2, 4.2.3, 4.2.4rc1, 4.2.4, 4.2.5)
ERROR: No matching distribution found for cellprofiler-core==5.0.0

5. So tried installing core:

$ git clone https://github.com/CellProfiler/core.git
$ cd core 
$ pip3 install -e . 

6. Install cellprofiler again:
   cd CellProfiler
   pip install -e .

now if I try cellprofiler to open I get

[CP - WARNING] cellprofiler_core.utilities.core.plugins::load_plugin: Could not load runcellpose
Traceback (most recent call last):
  File "C:\Users\fossa\Documents\github\cp_plugins_test\core\cellprofiler_core\utilities\core\plugins.py", line 43, in load_plugin
    m = importlib.import_module(source)
  File "C:\Users\fossa\Anaconda3\envs\cp_test_cellpose\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\fossa\Documents\github\cp_plugins_test\CellProfiler-plugins\runcellpose.py", line 3, in <module>
    from cellpose import models, io, core, utils
ModuleNotFoundError: No module named 'cellpose'
[CP - WARNING] cellprofiler_core.utilities.core.plugins::load_plugin: Could not load runimagejscript
Traceback (most recent call last):
  File "C:\Users\fossa\Documents\github\cp_plugins_test\core\cellprofiler_core\utilities\core\plugins.py", line 43, in load_plugin
    m = importlib.import_module(source)
  File "C:\Users\fossa\Anaconda3\envs\cp_test_cellpose\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\fossa\Documents\github\cp_plugins_test\CellProfiler-plugins\runimagejscript.py", line 25, in <module>
    import cpij.bridge as ijbridge, cpij.server as ijserver
  File "C:\Users\fossa\Documents\github\cp_plugins_test\CellProfiler-plugins\cpij\bridge.py", line 3, in <module>
    import atexit, cpij.server as ijserver
  File "C:\Users\fossa\Documents\github\cp_plugins_test\CellProfiler-plugins\cpij\server.py", line 9, in <module>
    import jpype, imagej, multiprocessing, socket, threading, time
ModuleNotFoundError: No module named 'imagej'
[CP - WARNING] cellprofiler_core.utilities.core.plugins::load_plugin: Could not load runomnipose
Traceback (most recent call last):
  File "C:\Users\fossa\Documents\github\cp_plugins_test\core\cellprofiler_core\utilities\core\plugins.py", line 43, in load_plugin
    m = importlib.import_module(source)
  File "C:\Users\fossa\Anaconda3\envs\cp_test_cellpose\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\fossa\Documents\github\cp_plugins_test\CellProfiler-plugins\runomnipose.py", line 3, in <module>
    from cellpose_omni import models, io, core, plot
ModuleNotFoundError: No module named 'cellpose_omni'
[CP - WARNING] cellprofiler_core.utilities.core.plugins::load_plugin: Could not load runstardist
Traceback (most recent call last):
  File "C:\Users\fossa\Documents\github\cp_plugins_test\core\cellprofiler_core\utilities\core\plugins.py", line 43, in load_plugin
    m = importlib.import_module(source)
  File "C:\Users\fossa\Anaconda3\envs\cp_test_cellpose\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\fossa\Documents\github\cp_plugins_test\CellProfiler-plugins\runstardist.py", line 5, in <module>
    from stardist.models import StarDist2D, StarDist3D
ModuleNotFoundError: No module named 'stardist'
usage: cellprofiler [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: cellprofiler --help [cmd1 cmd2 ...]
   or: cellprofiler --help-commands
   or: cellprofiler cmd --help

error: no commands supplied

7. So I tried installing the cellpose dependencies:
   pip install -e .[cellpose]

It still gives me the same error: no commands supplied when I type cellprofiler

[callum-jpg]

If you were to run conda install python.app and then pythonw -m cellprofiler, all from within your cp_test_cellpose conda environment, does it work?

[bethac07]

Also, I think the idea was to have the first step pip install -e in the plugins repo, not the CellProfiler repo, right? And then the plugins repo will grab CellProfiler for you from pip?

[fefossa]

If you were to run conda install python.app and then pythonw -m cellprofiler, all from within your cp_test_cellpose conda environment, does it work?

That doesn't work :(

[fefossa]

Since my laptop is having issues, I'm trying the instructions in the assaydev VM:

1. set up like Rebecca's instructions;

2. conda create and activate env with python=3.8

3. pip install numpy

4. git clone cp-plugins repo

5. cd cellprofiler-plugins and pip install -e .

6. calling cellprofiler gives the following:

(cp_plugins) C:\Users\Administrator\CellProfiler-plugins>cellprofiler
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda2\envs\cp_plugins\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\ProgramData\Anaconda2\envs\cp_plugins\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda2\envs\cp_plugins\Scripts\cellprofiler.exe\__main__.py", line 4, in <module>
  File "C:\ProgramData\Anaconda2\envs\cp_plugins\lib\site-packages\cellprofiler\__main__.py", line 13, in <module>
    import bioformats.formatreader
  File "C:\ProgramData\Anaconda2\envs\cp_plugins\lib\site-packages\bioformats\__init__.py", line 21, in <module>
    import javabridge
  File "C:\ProgramData\Anaconda2\envs\cp_plugins\lib\site-packages\javabridge\__init__.py", line 38, in <module>
    from .jutil import start_vm, kill_vm, vm, activate_awt, deactivate_awt
  File "C:\ProgramData\Anaconda2\envs\cp_plugins\lib\site-packages\javabridge\jutil.py", line 150, in <module>
    os.environ["PATH"] = os.environ["PATH"] + os.pathsep + _find_jvm() + \
  File "C:\ProgramData\Anaconda2\envs\cp_plugins\lib\site-packages\javabridge\jutil.py", line 139, in _find_jvm
    raise JVMNotFoundError()
javabridge.jutil.JVMNotFoundError: Can't find the Java Virtual Machine

• I checked where java and it says

C:\Program Files\Eclipse Adoptium\jdk-11.0.19.7-hotspot\bin\java.exe
C:\Program Files (x86)\Common Files\Oracle\Java\javapath\java.exe

• My environment variables. I tried with and without the \bin\ folder and it still doesn't find it
  

[fefossa]

Figure it out!! Deleting setup.py from the CellProfiler-plugins folder solves it.

[callum-jpg]

Interesting. Perhaps we need to restructure the repo like so:

CellProfiler-plugins/
├── setup.py
├── CellProfiler-plugins/
│   ├── plugin1.py
│   ├── plugin2.py
│   ├── plugin3.py
│   └── ...
└── README.md

Though, I suppose you could argue that this becomes ambiguous when telling someone to set the CellProfiler preferences plugin path to "CellProfiler-plugins"

[fefossa]

6. Install the CellPose specific dependencies with pip install -e .[cellpose]

After removing the setup.py, doing 6. was pretty straightforward! =)

[bethac07]

Though, I suppose you could argue that this becomes ambiguous when telling someone to set the CellProfiler preferences plugin path to "CellProfiler-plugins"

Yes, this would be my concern. Having a different setup file (pyproject.toml?) might be better, unless/until we fix the execution issue.

[callum-jpg]

So I gave the pyproject.toml solution a quick go and it doesn't quite work. In the setup.py solution above, we are in essence installing an empty package; pip install -e . is only used to install dependencies. The equivalent pyproject.toml solution doesn't seem to work for me. If you do not define py-modules = [], there's an auto discovery error. If you do define py-modules = [], you get a raise ModuleNotFoundError(module_name) ModuleNotFoundError: 0 error.

I think restructuring the repo to contain the plugin.py files into a distinct directory (and thus separate from setup.py) is the way to go. I'd like to think it'd be intuitive to select the directory that contains the plugin.py files from cellprofiler>preferences, but we can make this extra clear in the documentation.