Fix bug in the "Causal Ladder" docs page

[drbenvincent]

Corrects an error in the "MMMs and Pearl’s ladder of causal inference" docs page.

The MMM was being fitted to df_counterfactual where it should have been df_actual.

This fix only result in minor changes to the outputs and does not change the results meaningfully. I believe this is because the experiment duration is only a small proportion of the total number of observations.

Tagging @cluhmann who passed this bug report to me.

---

📚 Documentation preview 📚: https://pymc-marketing--1175.org.readthedocs.build/en/1175/

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[drbenvincent]

Just to expand and provide some more explanation on this to get more confidence that the current docs page is correct...

If we provided different data to the model to fit on, then we would expect the causal impact estimates to be different. What happened was that the counterfactual estimates did change when fitting on df_actual, however the causal impact estimates seemingly didn't change at all. This is a bit strange.

So I explored a scenario where I made the marketing intervention really big, a 5x increase in media spend.

With bug - model trained on df_counterfactual:

Without bug - model trained on df_actual:

This is a mixture of good and confusing. What we see is that in both cases the MMM can do a good job of fitting the data it is provided with. However, the causal impact estimates are very similar in both cases. What is going on there?

The fixed version seems correct. y is higher in the expt interval and the model fits that. Correctly predicts lower y in the counterfactual scenario. Correctly says that there's a big positive causal impact.

In the original version with the bug, the model does a good job of estimating the counterfactual y (should be the actual y). Counterfactual prediction is identical to the "actual" prediction because the counterfactual scenario being explored is the same as the one it was trained on. But when we calculate the causal impact, we compare the counterfactual prediction with the actual actual data. In both cases the difference between these are the same (because the model does a good job of fitting what it thinks the actual data is).

The original causal impact was accidentally correct because it correctly fitted what it thought was the actual data (it was actually the counterfactual), but then when we come to calculate the causal impact we compare the (wrong) counterfactual and the true actual data.

In short:

• The original docs showed incorrect counterfactual predictions, but accidentally correct (or nearly correct) causal impact simply because the model did a good job of fitting the counterfactual (which was then compared to the actual to get the causal impact estimate)
• We have high confidence that the docs as they are now are correct. The MMM is trained on the actual data, the counterfactual estimates are now correct (predicting lower y values). And the estimate of causal impact is also correct, being the difference between the now correct counterfactual predictions and the actual data.