Merge pull request #1977 from orichters/develop

add readcoupledgdx.R script to show scalars for all coupled runs, add it to tutorial

CHANGELOG.md

@@ -9,6 +9,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ### added
 - Possibility of using updated sources for baseline non-CO2 emissions calculation, controlled by `cm_emifacs_baseyear`
+- Add `readcoupledgdx` script that allows to print scalars from gdx files for all coupled runs
+    [[#1977](https://github.com/remindmodel/remind/pull/1977)]
 
 ### input data/calibration
 - new input data rev7.21 including new MAgPIE data [[#1956](https://github.com/remindmodel/remind/pull/1956)]
@@ -21,7 +23,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 -
 
 ### removed
-- **37_industry** removed superseded realization fixed_shares [[#1941]](https://github.com/remindmodel/remind/pull/1941)
+- **37_industry** removed superseded realization fixed_shares [[#1941](https://github.com/remindmodel/remind/pull/1941)]
 
 ### fixed
 - **37_industry** fix and restructure chemical feedstock balancing to account for all negative emissions from stored non-fossil carbon [[#1829](https://github.com/remindmodel/remind/pull/1829)]

scripts/output/single/checkProjectSummations.R

@@ -81,6 +81,6 @@ for (i in seq_along(checkMappings)) {
 }
 
 if (length(stopmessage) > 0 || length(missingVariables) > 0) {
-  stop("Project-related issues found checks for ", paste(stopmessage, collapse = ", "), " and ",
-       length(missingVariables), " missing variables found, see above.")
+  warning("Project-related issues found checks for ", paste(stopmessage, collapse = ", "), " and ",
+          length(missingVariables), " missing variables found, see above.")
 }

scripts/utils/readcoupledgdx.R

@@ -0,0 +1,44 @@
+suppressMessages(library(tidyverse))
+suppressMessages(library(gdx))
+suppressMessages(library(modelstats))
+options(width = 160)
+vars <- c("pm_gmt_conv", "pm_sccConvergenceMaxDeviation")
+argv <- commandArgs(trailingOnly = TRUE)
+if (length(argv) > 0 && ! isTRUE(argv == "")) vars <- strsplit(argv, ",")[[1]]
+folder <- if (sum(file.exists(c("output", "output.R", "start.R", "main.gms"))) == 4) "output" else "."
+dirs <- grep("^C_.*-rem-[0-9]+$", dir(folder), value = TRUE)
+if (length(dirs) == 0) {
+  message("No run found in ", normalizePath(folder))
+  q()
+}
+maxrem <- max(as.numeric(gsub("^C_.*-rem-", "", dirs)))
+runs <- unique(gsub("-rem-[0-9]+$", "", dirs))
+message("\nNumbers in parentheses indicate runs currently in slurm.")
+message("A minus sign indicates that run does not exist.")
+for (v in vars) {
+  message("\n### ", v)
+  results <- matrix(nrow = length(runs), ncol = maxrem + 1)
+  colnames(results) <- c("run", paste0("rem", seq(maxrem)))
+  results <- as_tibble(results)
+  for (r in seq_along(runs)) {
+    results[[r, "run"]] <- runs[[r]]
+    for (m in seq(maxrem)) {
+      rfolder <- file.path(folder, paste0(runs[r], "-rem-", m))
+      gdx <- file.path(rfolder, "fulldata.gdx")
+      if (file.exists(gdx)) {
+        data <- NULL
+        data <- try(gdx::readGDX(gdx, v, react = "silent"), silent = TRUE)
+        if (inherits(data, "try-error")) data <- "-"
+        if (is.null(data)) data <- "null"
+        if (is.numeric(data)) data <- if (data < 10) signif(data, 2) else round(data, 2)
+        data <- paste(data, collapse = ",")
+        if (! modelstats::foundInSlurm(rfolder) == "no" && data != "null") {
+          data <- paste0("(", data, ")")
+        }
+        results[[r, paste0("rem", m)]] <- data
+      }
+    }
+  }
+  results[is.na(results)] <- "-"
+  print(results, width = 160)
+}

tutorials/04_RunningREMINDandMAgPIE.md

@@ -178,6 +178,15 @@ Rscript scripts/output/comparison/plot_compare_iterations.R folder=another-outpu
 To compare the REMIND runs, a `compareScenario2` for each scenario is produced automatically in the main folder (look for `compScen_rem-1-5_Base.pdf` or similar).
 You can switch that off by setting `run_compareScenario` to `FALSE` in `start_bundle_coupled.R`.
 
+# Compare scalars from all coupled runs
+
+By running something like
+```bash
+Rscript --vanilla scripts/utils/readcoupledgdx.R cm_startyear,o_iterationNumber
+```
+you can get a compact overview of scalars (such as `cm_startyear`, `o_iterationNumber`, `s45_actualbudgetco2` or `cm_peakBudgYr`) for all coupled runs in your folder.
+PIK cluster users can use `readcoupledgdx yourvariable` as a shortcut.
+
 # Technical concept
 
 There are two components of the REMIND-MAgPIE coupling: the prominent dynamic part (models solve iteratively and exchange data via coupling script), the more hidden static part (exogenous assumptions derived from the other model, updated manually from time to time via [`mrcommons`](https://github.com/pik-piam/mrcommons/blob/master/R/readMAgPIE.R)).