-
-
Notifications
You must be signed in to change notification settings - Fork 115
/
Copy pathstansummary.qmd
190 lines (155 loc) · 9.43 KB
/
stansummary.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
---
pagetitle: Summarizing MCMC Output
---
# `stansummary`: MCMC Output Analysis {#stansummary}
The CmdStan `stansummary` program reports statistics for one or more sampler chains
over all sampler and model parameters and quantities of interest.
The statistics reported include both summary statistics of the estimates and
diagnostic statistics on the sampler chains, reported in the following order:
- Mean - sample mean
- MCSE - Monte Carlo Standard Error, a measure of the amount of noise in the sample
- StdDev - sample standard deviation - the standard deviation around the sample mean.
- MAD - Median Absolute Deviation - the median absolute deviation around the sample median.
- Quantiles - default 5%, 50%, 95%
- ESS_bulk
- ESS_tail
- R_hat - $\hat{R}$ statistic, a MCMC convergence diagnostic
When reviewing the `stansummary` output, it is important to check the final three
output columns first - these are the diagnostic statistics on MCMC convergence and
effective sample size.
A $\hat{R}$ statistic of greater than $1$ indicates potential convergence problems and that the sample is not presentative of the target posterior, thus the estimates of the mean and all other summary statistics are likely to be invalid. A value $1.01$ can be used as generic threshold to decide whether more iterations or further convergence analysis is needed, but other thresholds can be used depending on the specific use case.
Estimation by sampling produces an approximate value for the model parameters;
the MCSE statistic indicates the amount of uncertainty in the estimate.
Therefore MCSE column is placed next to the sample mean column,
in order to make it easy to compare this sample with others.
For more information, see the
[Posterior Analysis](https://mc-stan.org/docs/reference-manual/analysis.html)
chapter of the Stan Reference Manual which describes both the theory and practice of MCMC
estimation techniques.
The statistics - Mean, StdDev, MAD, and Quantiles - are computed directly from all draws across all chains.
The diagnostic statistics - ESS_bulk, ESS_tail, and R_hat are computed from the rank-normalized,
folded, and splitted chains according to the definitions by @Vehtari+etal:2021:Rhat.
the MCSE statistic is computed using split chain R_hat and autocorrelations.
The summary statistics and the algorithms used to compute them are described in sections
[Notation for samples](https://mc-stan.org/docs/reference-manual/analysis.html#notation-for-samples-chains-and-draws)
and
[Effective Sample Size](https://mc-stan.org/docs/reference-manual/analysis.html#effective-sample-size.section).
## Building the `stansummary` command
The CmdStan makefile task `build` compiles the `stansummary` utility
into the `bin` directory.
It can be compiled directly using the makefile as follows:
```
> cd <cmdstan-home>
> make bin/stansummary
```
## Running the `stansummary` program
The `stansummary` utility processes one or more output files from a set of
chains from one run of the HMC sampler.
To run `stansummary` on the output file or files generated by a run of
the sampler, on Mac or Linux:
```
<cmdstan-home>/bin/stansummary <file_1.csv> ... <file_N.csv>
```
On Windows, use backslashes to call the `stansummary.exe`.
```
<cmdstan-home>\bin\stansummary.exe <file_1.csv> ... <file_N.csv>
```
For example, after running 4 chains to fit the example model `eight_schools.stan`
to the supplied example data file, we run `stansummary` on the resulting Stan CSV
output files to get the following report:
```
> bin/stansummary eight_*.csv
Inference for Stan model: eight_schools_model
4 chains: each with iter=1000; warmup=1000; thin=1; 1000 iterations saved.
Warmup took (0.031, 0.031, 0.038, 0.031) seconds, 0.13 seconds total
Sampling took (0.032, 0.039, 0.026, 0.029) seconds, 0.13 seconds total
Mean MCSE StdDev MAD 5% 50% 95% ESS_bulk ESS_tail R_hat
lp__ -18 0.36 5.3 5.5 -26 -18 -8.7 214 217 1.0
accept_stat__ 0.80 0.021 0.29 0.068 0.033 0.95 1.00 268 314 1.0
stepsize__ 0.20 nan 0.040 0.029 0.14 0.21 0.25 nan nan nan
treedepth__ 3.7 0.22 0.88 1.5 2.0 4.0 5.0 26 18 1.1
n_leapfrog__ 17 3.1 10 12 3.0 15 31 30 21 1.1
divergent__ 0.022 nan 0.15 0.00 0.00 0.00 0.00 nan nan nan
energy__ 23 0.38 5.7 5.8 13 23 32 223 286 1.0
mu 7.9 0.16 5.3 4.9 -0.35 7.9 16 977 472 1.0
theta[1] 12 0.32 8.8 7.4 -0.19 11 28 727 525 1.0
theta[2] 7.8 0.17 6.4 6.0 -2.8 7.8 18 1298 2137 1.0
theta[3] 5.9 0.20 8.1 6.8 -8.0 6.5 18 1493 1720 1.0
theta[4] 7.5 0.17 6.9 6.3 -3.4 7.6 18 1517 1935 1.0
theta[5] 4.9 0.20 6.3 6.1 -6.1 5.1 15 998 1357 1.0
theta[6] 5.8 0.18 6.8 6.1 -6.3 6.2 16 1434 1431 1.0
theta[7] 11 0.25 7.0 6.4 0.42 10 23 741 562 1.0
theta[8] 8.6 0.21 8.1 7.2 -3.6 8.3 22 1444 1850 1.0
tau 7.3 0.31 5.8 4.4 1.5 5.7 18 226 1230 1.0
Samples were drawn using hmc with nuts.
For each parameter, ESS_bulk and ESS_tail measure the effective sample size for the entire sample
(bulk) and for the .05 and .95 tails (tail),
and R_hat measures the potential scale reduction on split chains. At convergence R_hat will be
very close to 1.00.
```
The console output information consists of
- Model, chains, and timing summaries
- Sampler parameter statistics
- Model parameter statistics
- Sampling algorithm - either nuts (shown here) or static HMC.
There is one row per parameter and the row order in the summary report corresponds
to the column order in the Stan CSV output file.
### Sampler parameters
The initial Stan CSV columns provide information on the sampler state for each draw:
- `lp__` - the total log probability density (up to an additive constant) at each sample
- `accept_stat__ ` - the average Metropolis acceptance probability over each simulated Hamiltonian trajectory
- `stepsize__ ` - integrator step size
- `treedepth__ ` - depth of tree used by NUTS (NUTS sampler)
- `n_leapfrog__ ` - number of leapfrog calculations (NUTS sampler)
- `divergent__ ` - has value `1` if trajectory diverged, otherwise `0`. (NUTS sampler)
- `energy__ ` - value of the Hamiltonian
- `int_time__ ` - total integration time (static HMC sampler)
Because we ran the NUTS sampler, the above summary reports sampler parameters
`treedepth__`, `n_leapfrog__`, and `divergent__`;
the static HMC sampler would report `int_time__` instead.
### Model parameters and quantities of interest
The remaining Stan CSV columns report the values of all parameters, transformed parameters,
and generated quantities in the order in which these variables are declared in the Stan program.
For container variables, i.e., vector, row_vector, matrix, and array variables,
the statistics for each element are reported separately, in row-major order.
The `eight_schools.stan` program parameters block contains the following parameter variable
declarations:
```
real mu;
array[J] real theta;
real<lower=0> tau;
```
In the example data, `J` is $8$; therefore the stansummary listing
reports on `theta[1]` through `theta[8]`.
## Command-line options
The `stansummary` command syntax provides a set of flags to customize the output
which must precede the list of filenames.
When invoked with no arguments or with the `-h` or `--help` option,
the program prints the usage message to the console and exits.
```
Report statistics for one or more Stan CSV files from a HMC sampler run.
Example: stansummary model_chain_1.csv model_chain_2.csv
Options:
-a, --autocorr [n] Display the chain autocorrelation for the n-th
input file, in addition to statistics.
-c, --csv_filename [file] Write statistics to a CSV file.
-h, --help Produce help message, then exit.
-p, --percentiles [values] Percentiles to report as ordered set of
comma-separated numbers from (0.1,99.9), inclusive.
Default is 5,50,95.
-s, --sig_figs [n] Significant figures reported. Default is 2.
Must be an integer from (1, 18), inclusive.
-i, --include_param [name] Include the named parameter in the summary output.
By default, all parameters in the file are summarized,
passing this argument one or more times will filter
the output down to just the requested arguments.
```
Both short an long option names are allowed. Short names are specified as `-<o> <value>`;
long option names can be specified either as `--<option>=<value>` or `--<option> <value>`.
The `--percentiles` argument can also be passed an empty string `""`, which
results in no percentiles being displayed in the output of the command.
The amount of precision
in the sampler output limits the amount of real precision in the summary report.
CmdStan's command line interface also has output argument `sig_figs`.
The default sampler output precision is 6.
The `--sig_figs` argument to the stansummary program should not exceed the `sig_figs` argument to the sampler.