Recompiling stan code fails after changes have been made but then works after restarting R session

[martintburgess]

Summary:

I have a small stan script that compiles and runs fine. I then make some small changes (e.g. adding some additional data parameters). I rerun stan(...) and get the following output:

Error in stanc(file = file, model_code = model_code, model_name = model_name,  : 
  0

Syntax error in 'string', line 23, column 21 to column 22, parsing error:

Expected "generated quantities {" or end of file after end of model block.

If I then, restart my R session and run stan(...) compilation happens without error and sampling runs fine.

Reproducible Steps:

I have the following code in a .stan file:

// The input data is a vector 'y' of length 'N'.
data {
  int<lower=0> N;
  vector[N] y;
}

// The parameters accepted by the model. Our model
// accepts two parameters 'mu' and 'sigma'.
parameters {
  real mu;
  real<lower=0> sigma;
}

// The model to be estimated. We model the output
// 'y' to be normally distributed with mean 'mu'
// and standard deviation 'sigma'.
model {
  // mu ~ normal(15, 5);
  y ~ normal(mu, sigma);
}

I run this using (works fine):

library(rstan)

model_data <- list(y = rnorm(n = 1000, mean = 20, sd = 1),
                   N = 1000)
test <- stan(file = "explore/test.stan", 
             data = model_data,
             chains = 8, 
             control = list(max_treedepth = 15),
             iter = 2000)

I then change my .stan file to:

// The input data is a vector 'y' of length 'N'.
data {
  int<lower=0> N;
  vector[N] y;
  real prior_mu;
  real<lower=0> prior_sigma;
}

// The parameters accepted by the model. Our model
// accepts two parameters 'mu' and 'sigma'.
parameters {
  real mu;
  real<lower=0> sigma;
}

// The model to be estimated. We model the output
// 'y' to be normally distributed with mean 'mu'
// and standard deviation 'sigma'.
model {
  mu ~ normal(prior_mu, prior_sigma);
  y ~ normal(mu, sigma);
}

And try and rerun using (not restarting R):

model_data <- list(y = rnorm(n = 1000, mean = 20, sd = 1),
                   N = 1000,
                   prior_mu = 15,
                   prior_sigma = 5)

test <- stan(file = "explore/test.stan", 
             data = model_data,
             chains = 8, 
             control = list(max_treedepth = 15),
             iter = 2000)

Sometimes (though not all the time) this will produce an output like:

Error in stanc(file = file, model_code = model_code, model_name = model_name,  : 
  0

Syntax error in 'string', line 23, column 21 to column 22, parsing error:

Expected "generated quantities {" or end of file after end of model block.

Though the exact details of line 23, column 21 to column 22, change every time stan is run (without changing the code).

If I restart my R session and run the following code it will compile without error:

library(rstan)
model_data <- list(y = rnorm(n = 1000, mean = 20, sd = 1),
                   N = 1000,
                   prior_mu = 15,
                   prior_sigma = 5)

test <- stan(file = "explore/test.stan", 
             data = model_data,
             chains = 8, 
             control = list(max_treedepth = 15),
             iter = 2000)

Many thanks for your help :)

RStan Version:

packageVersion("rstan")
[1] ‘2.26.6’

StanHeaders_2.26.6

R Version:

R.version.string
[1] "R version 4.1.2 (2021-11-01)"

Operating System:

R version 4.1.2 (2021-11-01)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.1

[WardBrian]

This was the result of a bug in stanc which was fixed here.

I have backported the change (it's only one line) to 2.26 here: https://github.com/stan-dev/stanc3/tree/lexer-patch-v2.26.1

I can provide a built stancjs if it helps @hsbadr

[martintburgess]

Awesome, thank you @WardBrian - this may be a silly question but do you know how I can install the version of stanc which has the fix? Typically I install stan through the R packages RStan and StanHeaders

[martintburgess]

Reinstalling the packages seems to work. I will close this issue.

[WardBrian]

I’m actually going to reopen this, because I don’t believe the issue has been resolved on the RStan end

[helske]

I have the same issue with the rstan & StanHeaders versions installed from https://mc-stan.org/r-packages/. The CRAN versions work fine. For me, if I compile the model with stan_model, on subsequent saves rstan:::rstudio_stanc throws (although not always) the same error as Martin wrote above.

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19042)

Matrix products: default

locale:
[1] LC_COLLATE=Finnish_Finland.1252  LC_CTYPE=Finnish_Finland.1252    LC_MONETARY=Finnish_Finland.1252
[4] LC_NUMERIC=C                     LC_TIME=Finnish_Finland.1252    
system code page: 65001

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rstan_2.26.6       StanHeaders_2.26.6

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.7         compiler_4.0.2     pillar_1.6.4       prettyunits_1.1.1  tools_4.0.2        pkgbuild_1.3.1    
 [7] jsonlite_1.7.2     lifecycle_1.0.1    tibble_3.1.6       gtable_0.3.0       pkgconfig_2.0.3    rlang_0.4.12      
[13] cli_3.1.0          parallel_4.0.2     curl_4.3.2         loo_2.4.1          gridExtra_2.3      dplyr_1.0.7       
[19] generics_0.1.1     vctrs_0.3.8        tidyselect_1.1.1   stats4_4.0.2       grid_4.0.2         glue_1.6.0        
[25] inline_0.3.19      R6_2.5.1           processx_3.5.2     fansi_1.0.0        purrr_0.3.4        ggplot2_3.3.5     
[31] callr_3.7.0        magrittr_2.0.1     codetools_0.2-16   matrixStats_0.61.0 scales_1.1.1       ps_1.6.0          
[37] ellipsis_0.3.2     colorspace_2.0-2   V8_4.0.0           utf8_1.2.2         RcppParallel_5.1.5 munsell_0.5.0     
[43] crayon_1.4.2   

[hsbadr]

@WardBrian Sorry! I should be able to look into this when I return to office early in March, Right now, it'd be helpful if we could test the experimental branch, which is inline with Stan/Math development:

remove.packages(c("rstan", "StanHeaders"))
remotes::install_git("https://github.com/stan-dev/rstan", subdir = "StanHeaders", ref = "experimental") 
remotes::install_git("https://github.com/stan-dev/rstan", subdir = "rstan/rstan", ref = "experimental")

v2.26 is a temporary release to address backward compatibility issues on CRAN. So, it has many patches that cause unexpected issues and may not be worth fixing for the transition.

[andrjohns]

I can confirm that it errors under 2.26.4 but not the experimental branch, using the following code to test:

bad_mod <- "parameters { real no_semicolon } model {}"
good_mod <- "parameters { real semicolon; } model {}"

rstan::stanc(model_code = bad_mod)
rstan::stanc(model_code = good_mod)

Is it possible to backport the relevant changes/patches to 2.26? This error has come up a few times on the forum already, and will probably come up more once it's out on CRAN

[WardBrian]

I would expect it to be resolved on experimental if it’s using a recent stancjs

Yes, I backported it on this branch:
https://github.com/stan-dev/stanc3/tree/lexer-patch-v2.26.1

I can provide a built stancjs from this branch no problem, let me know how would be best to get it to you

[hsbadr]

The problem could in the unnecessary (Stan/Math) patches we made for CRAN (check StanHeaders_2.26 branches). If that build passes CRAN checks for StanHeaders, we can immediately fix those issues by releasing a newer version of RStan and then (unpatched) StanHeaders. There's a few other alternatives that could publish the development version on CRAN, but they've been actively discussed in the RStan meetings, due to imperfect choices (e.g., licensing issues or a new RStan3 package).

@WardBrian If it's a stancjs issue, please create a stanc3 v2.26.2 build and update it in the StanHeaders_2.26 branch of RStan (just replace rstan/rstan/inst/stanc.js).

[hsbadr]

rstan/StanHeaders v2.26.7 (stan-dev/r-packages@f752a4f) is ready for testing:

remove.packages(c("rstan", "StanHeaders"))
install.packages("StanHeaders", repos = c("https://mc-stan.org/r-packages/", getOption("repos")))
install.packages("rstan", repos = c("https://mc-stan.org/r-packages/", getOption("repos")))

[andrjohns]

I can confirm that the error is resolved under the new packages thanks @WardBrian and @hsbadr! Closing this issue.