MMM Evaluation, Diagnostics and MLFlow Registry

[louismagowan]

Description

Third time's the charm...
This PR is a recreation of this one which I was told to close and restart due to git issues on the pymc-marketing repo and then this one after, which was also closed due to some Git issues.

It could be nice to have some standardised model evaluation and diagnostic functions added to pymc-marketing. Ideally they'd be formulated in a way that makes them easy to log in MLFLow later on.
It would also be cool to build on top of the MLFlow module, to create a custom mlflow.pyfunc.PythonModel class to allow users to be able to register their models in the MLFlow registry. This would allow people to serve and maintain their MMMs more easily, and could help with MMM refreshes too.

Standard model metrics could include:

• Bayesian R2
• MAPE, RMSE, MAE
• Normalised RMSE and MAE (to allow comparisons across datasets and methodologies - in particular with Robyn models, who use NRMSE as one of their 2 key metrics)
  etc.
• Wrapper functions to calculate those across the entire distributions of posteriors, not just against the means (so we can have HDI lower, upper for each metric too)

Diagnostic metrics could include:

• Step size, divergences
• LOOCV metrics e.g.

Some additional plots (also useful for diagnosing models):

• Plot prior vs posterior distributions, see how data is shifting things
• Plot HDI forests for a given var, along with r-hat value

Model Registry / Additional Logging Code:

A wrapper for an MMM model to make it conform to the MLFlow api, enabling registering and easier deployment
Also an option to load models from the registry as well / or download the idata from MLflow too.
I'll open this as a Draft for now - since I'll need advice on how where best to put this code, as well as overall design etc.

Related Issue

• Closes Evaluation and Model Diagnostic Functions #911
• Closes Add Support for Model Registering to MLFlow Module #973
• Related to Expose serialized format #891 MLflow deployment example #901

Checklist

• Checked that the pre-commit linting/style checks pass
• Included tests that prove the fix is effective or that the new feature works
• Added necessary documentation (docstrings and/or example notebooks)
• If you are a pro: each commit corresponds to a relevant logical change

Modules affected

• MMM
• CLV

Type of change

• New feature / enhancement
• Bug fix
• Documentation
• Maintenance
• Other (please specify):

---

📚 Documentation preview 📚: https://pymc-marketing--1368.org.readthedocs.build/en/1368/

[ColtAllen]

WAIC is another good metric to include if the use case involves out-of-sample predictions:

python code

Wikipedia

[wd60622]

WAIC is another good metric to include if the use case involves out-of-sample predictions:

python code

Wikipedia

Can we address this in another pr. This is already months deep

[wd60622]

I'd rather this take model_uri which is more similar to the sklearn mlflow function: https://mlflow.org/docs/latest/python_api/mlflow.sklearn.html#mlflow.sklearn.load_model

This would allow loading from run or from registered model.

If we'd like, we can add helpers to construct the run URI and the registered model URI

[juanitorduz]

Some minor comments :) I will let @wd60622 review the mlflow components in detail as I have not been using it lately :)

[juanitorduz]

maybe we could use https://python.arviz.org/en/stable/api/generated/arviz.plot_dist_comparison.html as we do in https://www.pymc-marketing.io/en/latest/notebooks/general/prior_predictive.html ?

[louismagowan]

Happy to take it out, but there are some things I prefer in plot_prior_vs_posterior:

• Option to sort by difference / alphabetically (I find to be useful when diagnosing models with many channels)
• Legend giving you differences in means rather than having to calculate them & add them to the fig

You think it's better to remove though? ☺️

[juanitorduz]

It seems there is one error with a test

____________________ ERROR collecting tests/test_mlflow.py _____________________
ImportError while importing test module '/home/runner/work/pymc-marketing/pymc-marketing/tests/test_mlflow.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_mlflow.py:29: in <module>
    from pymc_marketing.mlflow import (
E   ImportError: cannot import name 'log_summary_metrics' from 'pymc_marketing.mlflow' (/home/runner/work/pymc-marketing/pymc-marketing/pymc_marketing/mlflow.py)

[codecov]

Codecov Report

Attention: Patch coverage is 45.03311% with 83 lines in your changes missing coverage. Please review.

Project coverage is 93.86%. Comparing base (8d94482) to head (407521c).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
pymc_marketing/mlflow.py	43.37%	47 Missing ⚠️
pymc_marketing/mmm/mmm.py	2.70%	36 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1368      +/-   ##
==========================================
- Coverage   95.35%   93.86%   -1.49%     
==========================================
  Files          47       48       +1     
  Lines        4995     5135     +140     
==========================================
+ Hits         4763     4820      +57     
- Misses        232      315      +83     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wd60622]

Thanks for the work on this @louismagowan
I will merge and we will iterate on some of the latest suggestion