diff --git a/examples/Intro_Amortized_Posterior_Estimation.ipynb b/examples/Intro_Amortized_Posterior_Estimation.ipynb index f9a47848c..04121a471 100644 --- a/examples/Intro_Amortized_Posterior_Estimation.ipynb +++ b/examples/Intro_Amortized_Posterior_Estimation.ipynb @@ -8,7 +8,44 @@ }, "source": [ "

Table of Contents

\n", - "
" + "
\n", + " \n", + "
\n" ] }, { @@ -516,6 +553,8 @@ "id": "biblical-tongue", "metadata": {}, "source": [ + "Note about `z`: The inference network takes summary vectors as input and outputs latent vectors (`z`). The latent space is a lower-dimensional space that is assumed to capture the essential information about the parameters. It is essentially transforming the parameter space to the latent space (Gaussian in this case). \n", + "\n", "We can inspect the shapes of the outputs as well:" ] }, @@ -832,7 +871,7 @@ "id": "departmental-preservation", "metadata": {}, "source": [ - "We can inspect the evolution of the loss via a utility function ``plot_losses``, for which we have imported the ``diagnostics`` module from ``BayesFlow``." + "Bayesian models can be complex and computationally intensive, and metrics like training and validation loss can provide critical insights into the model's performance and stability. A decreasing loss over time indicates that the model is learning effectively, while fluctuations or increases in loss might suggest issues in the training process, such as overfitting, underfitting, or inappropriate learning rate settings. We can inspect the evolution of the loss via a utility function ``plot_losses``, for which we have imported the ``diagnostics`` module from ``BayesFlow``. " ] }, { @@ -866,7 +905,7 @@ "### Validating Consistency\n", "Validating the consistency of our model-amortizer coupling is an important step which should be performed before any real data are presented to the networks. In other words, the model should work in the ''small world'', before going out in the world of real data. This involves testing the model under known conditions and ensuring that it behaves logically and accurately. It is a critical step to avoid surprises when the model is later exposed to real and more complex data. In addition to a smooth loss reduction curve, we can use at least four handy diagnostic utilities. \n", "\n", - "For a better illustration, we will start by generating some test simulations (not seen during training). Note, that we also use the default configurator to prepare these test simulations for interacting with the networks." + "For a better illustration, we will start by generating some test simulations (not seen during training) using the simulator `model`. Note, that we also use the default configurator to prepare these test simulations for interacting with the networks." ] }, { @@ -885,7 +924,8 @@ "metadata": {}, "source": [ "#### Latent space inspection\n", - "Since our training objective prescribes a unit Gaussian to the latent variable $\\boldsymbol{z}$ (see: https://arxiv.org/abs/2003.06281), we expect that, upon good convergence, the latent space will exhibit the prescribed probabilistic structure. Good convergence means that the model has learned an appropriate representation of the data in its latent space. We can quickly inspect this structure by calling the ``plot_latent_space_2d`` function from the `diagnostics` module." + "\n", + "Since our training objective prescribes a unit Gaussian to the latent variable $\\boldsymbol{z}$ (see: https://arxiv.org/abs/2003.06281), we expect that, upon good convergence, the latent space will exhibit the prescribed probabilistic structure. Good convergence means that the model has learned an appropriate representation of the data in its latent space. We can quickly inspect this structure by calling the ``plot_latent_space_2d`` function from the `diagnostics` module. " ] }, { @@ -931,7 +971,7 @@ "\n", "SBC is a technique used to assess whether a probabilistic model correctly infers parameters from data. The basic idea is to simulate a large number of datasets from the model's prior distribution and then perform posterior inference on these simulated datasets. The goal is to check if the inferred posterior distributions are consistent with the priors. Essentially, SBC tests if the model can accurately recover the parameters it used to generate the data in the first place. This process helps identify any systematic biases or inaccuracies in the model's inference process.\n", "\n", - "To perform SBC, we first need to obtain a number of `L` posterior draws from `M` simulated data sets. While the procedure is typically intractable, amortized inference allows us to perform SBC instantly." + "To perform SBC, we first need to obtain `L` number of posterior draws from `M` simulated data sets. While the procedure is typically intractable, amortized inference allows us to perform SBC instantly." ] }, { @@ -1012,12 +1052,18 @@ "metadata": {}, "source": [ "#### Posterior z-score and contraction\n", + "\n", + "* Posterior z-score: It measures how many standard deviations away the mean of the posterior distribution is from the true value of the parameter. A z-score of 0 indicates the mean perfectly aligns with the true value (no bias) while positive/negative z-scores indicate positive/negative bias, respectively.\n", + "* Posterior contraction: It measures how much the posterior distribution contracts around the true value of the parameter as more data is observed. A higher contraction indicates that the data provides strong evidence, narrowing the uncertainty range.\n", + "\n", + "Ideally, we should obtain high contraction and a z-score near 0. This means the model accurately captures the true value with little bias and high confidence.\n", + "\n", "A quick and dirty way to gain an understanding of how good point estimates and uncertainty estimates capture the \"true\" parameters, assuming the generative model is well-specified. For this, we will draw more samples from the posteriors in order to get smaller Monte Carlo error." ] }, { "cell_type": "code", - "execution_count": 32, + "execution_count": null, "id": "descending-election", "metadata": {}, "outputs": [], @@ -1053,7 +1099,7 @@ }, { "cell_type": "code", - "execution_count": 34, + "execution_count": null, "id": "virtual-incidence", "metadata": {}, "outputs": [