Multigene Displays - "items in new_categories are not the same as in old categories"

[adkinsrs]

Dataset - P25-P27, mouse, scRNA-seq, cochlear sensory neurons (Goodrich) - in Neuron collection
Genes - itpr1 ryr3 ryr2

Payload (display configuration)

{
    "plot_type": "heatmap",
    "obs_filters": {
        "condition": [
            "Apex",
            "Base",
            "Middle"
        ],
        "cell_type": [
            "1_A",
            "1_B",
            "1_C",
            "T2"
        ]
    },
    "gene_symbols": [
        "Itpr1",
        "Ryr3",
        "Ryr2"
    ],
    "sort_order": {
        "cell_type": [
            "1_A",
            "1_B",
            "1_C",
            "T2"
        ],
        "condition": [
            "Base",
            "Middle",
            "Apex"
        ]
    },
    "primary_col": "cell_type",
    "secondary_col": "condition",
    "clusterbar_fields": [],
    "matrixplot": true,
    "cluster_obs": false,
    "cluster_genes": true,
    "flip_axes": false,
    "distance_metric": "euclidean",
    "analysis": null,
    "colorblind_mode": false
}

Stack Trace

File "/home/jorvis/git/gEAR-IGS/www/api/resources/multigene_dash_data.py", line 332, in post
reordered_col = col.cat.reorder_categories(

File "/opt/Python-3.10.4/lib/python3.10/site-packages/pandas/core/accessor.py", line 112, in f
return self._delegate_method(name, *args, **kwargs)

File "/opt/Python-3.10.4/lib/python3.10/site-packages/pandas/core/arrays/categorical.py", line 2939, in _delegate_method
res = method(*args, **kwargs)

File "/opt/Python-3.10.4/lib/python3.10/site-packages/pandas/core/arrays/categorical.py", line 1278, in reorder_categories
raise ValueError(
ValueError: items in new_categories are not the same as in old categories

[adkinsrs]

This was auto-closed prematurely, as I fixed some other issues but not this issue when doing my pull request

Issues fixed

• Use "shadows" to speed up some of the API calls #899 - Make the check more generic to int-types instead of a specific one
• Ran into an issue (due to my dataset being out-of-date) where the curation had columns that were not in my dataset. Threw errors with a hint to adjust curation
• Better heatmap error with 2 or less genes found in the dataset.

[adkinsrs]

So something I noticed about this dataset.

The cell types in adata.obs are ['A_T1', 'B_T1', 'C_T1', 'T2'] but the specified filters and sort order keys use ["1_A", "1_B", "1_C", "T2"]. So only T2 ends up remaining. There is a step where we reorder the filter keys & sort the dataframe to match the provided sort order, but if every "filter" value is not in the dataframe then Pandas throws an error when trying to reorder. I suspect the curation was made and the dataset was edited or reuploaded at one point

So two possible routes to resolve this

1. Adjust filter and sort keys based on what is remaining in the dataframe. It will prevent the error but may generate an undesired plot
2. Throw an error and request that the curation be adjusted. I'm leaning towards this option.