support sum(w) < 0 in binned fits (#1022)

This patch makes it possible to fit histograms with negative bin entries
with `ExtendedBinnedNLL` and `BinnedNLL`. These occur, for example, when
sweighted samples are histogrammed. This patch implements an extension
of the Bohm-Zech approach for this case. The ideas behind this extension
are compiled in a write-up that can be found in the documentation under
"Studies: Fitting weighted histograms".

The PR also fixes a mistake in `BinnedNLL`, which used the wrong cost
function for weighted histograms, for ordinary histograms there is no
change. This is a side result from thinking deeply about fitting
weighted histograms. The old cost function lead to biased estimates,
while the new one is now almost unbiased and performs even better than
`ExtendedBinnedNLL` in application to weighted histograms.

Other changes:
- Automatic notebook stripping is implemented via pre-commit; so
notebooks that run for a long time are not stripped
- Switch to SVG plots in notebooks

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

.pre-commit-config.yaml

@@ -45,6 +45,7 @@ repos:
   rev: v18.1.8
   hooks:
   - id: clang-format
+    files: "src"
 
 # CMake formatting
 - repo: https://github.com/cheshirekow/cmake-format-precommit
@@ -63,6 +64,14 @@ repos:
     additional_dependencies: [numpy]
     files: "src"
 
+# Clear Jupyter notebook output and remove empty cells
+# Override this by adding "keep_output": true to "metadata" block
+- repo: https://github.com/kynan/nbstripout
+  rev: 0.7.1
+  hooks:
+  - id: nbstripout
+    args: [--drop-empty-cells]
+
 - repo: https://github.com/python-jsonschema/check-jsonschema
   rev: 0.29.1
   hooks:

doc/_static/interactive_demo.ipynb

@@ -1,5 +1,5 @@
 {
-  "cells": [
+ "cells": [
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -9,24 +9,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "9e09f787c7d94549a40efbc670e9eec7",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "HBox(children=(Output(), VBox(children=(HBox(children=(Button(description='Fit', style=ButtonStyle()), ToggleB…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
+   "outputs": [],
    "source": [
     "import numpy as np\n",
     "from scipy.stats import norm\n",
@@ -48,13 +33,6 @@
     "\n",
     "m.interactive()"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
@@ -75,7 +53,6 @@
    "pygments_lexer": "python3",
    "version": "3.9.13"
   },
-  "orig_nbformat": 4,
   "vscode": {
    "interpreter": {
     "hash": "bdbf20ff2e92a3ae3002db8b02bd1dd1b287e934c884beb29a73dced9dbd0fa3"

doc/notebooks/basic.ipynb

@@ -1,5 +1,5 @@
 {
-  "cells": [
+ "cells": [
   {
    "attachments": {},
    "cell_type": "markdown",
@@ -23,6 +23,7 @@
    "outputs": [],
    "source": [
     "# basic setup of the notebook\n",
+    "%config InlineBackend.figure_formats = ['svg']\n",
     "from matplotlib import pyplot as plt\n",
     "import numpy as np\n",
     "\n",
@@ -164,7 +165,7 @@
    "source": [
     "try:\n",
     "    Minuit(least_squares)\n",
-    "except:\n",
+    "except RuntimeError:\n",
     "    import traceback\n",
     "\n",
     "    traceback.print_exc()"
@@ -178,7 +179,7 @@
    "source": [
     "try:\n",
     "    Minuit(least_squares, a=0, b=0)\n",
-    "except:\n",
+    "except RuntimeError:\n",
     "    import traceback\n",
     "\n",
     "    traceback.print_exc()"
@@ -1636,7 +1637,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.16"
+   "version": "3.12.4"
   },
   "nteract": {
    "version": "0.12.3"