Validate relation with EU DCC Schema

CHANGELOG.md

@@ -18,6 +18,8 @@
     2. The derivation script for this value set has been updated to check for deprecated ACT/SCT codes.
        (It also reports the correct row numbers now.)
 
+  The [guidelines document v1.14 has been adopted and published](https://ec.europa.eu/health/sites/default/files/ehealth/docs/digital-green-value-sets_en.pdf) through the [eHealth network page on the EU DCC](https://ec.europa.eu/health/ehealth/covid-19_en).
+
 
 ## Release 2.10.0
 
@@ -145,5 +147,5 @@ The [guidelines document v1.5 has been adopted and published](https://ec.europa.
 
 ## Prior releases
 
-Change log for releases before 2.3.0 are available through the [releases page](https://github.com/ehn-dcc-development/ehn-dcc-valuesets/releases).
+Change log for releases before 2.3.0 are available through the [releases page](https://github.com/ehn-dcc-development/eu-dcc-valuesets/releases).
 

README.md

@@ -1,6 +1,6 @@
 # EU eHealthNetwork Digital COVID Certificate valuesets
 
-Welcome to the EU-DCC valuesets repository. This repository contains a snapshot of the valuesets referenced by the EU Digital COVID Certificate (DCC) [JSON Schema](https://github.com/ehn-dcc-development/ehn-dcc-schema). The most up-to-date version of the valuesets are distributed on the [EU DGC Gateway](https://github.com/eu-digital-green-certificates/dgc-gateway).
+Welcome to the EU-DCC valuesets repository. This repository contains a snapshot of the valuesets referenced by the EU Digital COVID Certificate (DCC) [JSON Schema](https://github.com/ehn-dcc-development/eu-dcc-schema). The most up-to-date version of the valuesets are distributed on the [EU DGC Gateway](https://github.com/eu-digital-green-certificates/dgc-gateway).
 
 All of the file found in this repository should be considered for reference usage only. The current official release is published on the EU DGC Gateway. As access to this gateway is only available to Member States please refer to your own Member State to for access to those files.
 
@@ -26,7 +26,7 @@ vaccine-prophylaxis.json			 | List of the vaccine/prophylaxis types							 | Git
 
 The JSON Schema definition for the structure used for all of the above valuesets can be found in [DCC.ValueSets.schema.json](DCC.ValueSets.schema.json)
 
-Finally the we have [changelog](CHANGELOG.md), [license](LICENSE.md) and [versioning](VERSIONING.md) files and, of course, this [readme](README.md).
+Finally, we have [changelog](CHANGELOG.md), [license](LICENSE.md) and [versioning](VERSIONING.md) files and, of course, this [readme](README.md).
 
 
 ## Working with the valuesets
@@ -68,6 +68,10 @@ We use [JSON Schema 2020-12](https://json-schema.org/specification.html) to defi
 
 For files which have a different schema - such as `vaccine-encoding-instructions` - the validator supports the definition of a custom schema. By convention these schema files are named `<valueset-file-name>.schema.json` and are stored in `./schema`. So for `vaccine-encoding-instructions.json` the schema `vaccine-encoding-instructions.schema.json` is used.
 
+The validator also checks whether the set of actual valuesets coincides with the set expected by the [EU DCC Schema standard](https://github.com/ehn-dcc-development/eu-dcc-schema/blob/release/1.3.2/DCC.ValueSets.schema.json).
+The exact GitHub reference - being a (release) tag, or branch - is configured through the `EU_DCC_SCHEMA_REF` constant defined in [this config file](./schemas/config.js).
+This constant needs to be updated when a new release of the EU DCC Schema is released, or while being worked on.
+
 The validator code has been written in an imperative style for ease of understanding and [can be found in the schema directory](schemas/validate-valuesets.js).
 
 

VERSIONING.md

@@ -13,7 +13,7 @@ Each release is identified through a [semver version number](https://semver.org/
 
 Releases are prepared in branches named `release/<release version>`.
 After preparation has finished, a _Pull Request_ (PR) from the release branch into `main` is created on GitHub.
-That can e.g. be done by clicking the “New Pull Request” button on the [branches page](https://github.com/ehn-dcc-development/ehn-dcc-valuesets/branches).
+That can e.g. be done by clicking the “New Pull Request” button on the [branches page](https://github.com/ehn-dcc-development/eu-dcc-valuesets/branches).
 
 Such a PR serves as the trigger a code review for that release.
 That possibly leads to discussion conducted inside the PR page, and further work being committed to that release branch.
@@ -34,7 +34,7 @@ That can be done on the commandline as follows (assuming the `main` branch is cu
     $ git tag <release version>
     $ git push --tags
 
-After this, a release page can be made for that tag, using the [GitHub releases overview](https://github.com/ehn-dcc-development/ehn-dcc-valuesets/releases).
+After this, a release page can be made for that tag, using the [GitHub releases overview](https://github.com/ehn-dcc-development/eu-dcc-valuesets/releases).
 
 After releasing, the European Commission's (EC) representative in the eHN should be notified of the release, so they can notify TSi to update the EU DCC Gateway.
 

schemas/config.js

@@ -0,0 +1,6 @@
+// module.exports.EU_DCC_SCHEMA_REF = "release/1.3.2";
+
+// to verify that the 1.3.3 version of the EU DCC Schema will work:
+module.exports.EU_DCC_SCHEMA_REF = "feature/vaccine-encoding-instructions-valueset";
+// FIXME  remove and uncomment 1st line before merging
+

schemas/validate-valuesets.js

@@ -1,19 +1,18 @@
-const fs = require('fs');
-const exec = require('child_process').exec;
+const fs = require("fs");
 const Ajv2020 = require("ajv/dist/2020"); // see: https://ajv.js.org/json-schema.html#draft-2020-12
 const addFormats = require("ajv-formats");
 
 // Default schema to be used if no file-specific one is defined
-const DEFAULT_SCHEMA = 'DCC.ValueSets.schema.json';
+const DEFAULT_SCHEMA = "DCC.ValueSets.schema.json";
 
 // Directories containing schemas/valuesets
 const SCHEMA_DIR = "./schemas/";
 const VALUESET_DIR = "./";
 
 // These files will be excluded from the schema check
 const FILES_TO_EXCLUDE = [
-    'package.json',
-    'package-lock.json'
+    "package.json",
+    "package-lock.json"
 ];
 
 // Init ajv
@@ -44,8 +43,7 @@ const schemas = fs
     .map(item => item.name);
 
 // Validate all of the valuesets against their schema
-for(const item of valuesets)
-{
+for (const item of valuesets) {
     // Choose + load schema; if a schema exists with named <value-set-file-name>.schema.json then that will be used,
     // this allows us to override the default schema when needed
     //
@@ -62,19 +60,82 @@ for(const item of valuesets)
     const valid = ajv.validate(schema, valueset);
 
     // Handle validation errors
-    if(valid) {
+    if (valid) {
         console.log(`Validating ${item} (with schemas/${schemaFileName}) -> OK`);
     } else {
+        exitWithError = true;
         console.log(`Validating ${item} -> FAILED`)
         console.log(ajv.errors);
+    }
+}
+
+
+const assertSameSet = (actuals, expecteds, what) => {
+    if (
+            actuals.length !== expecteds.length
+        || !actuals.every((l) => expecteds.indexOf(l) > -1)
+        || !expecteds.every((l) => actuals.indexOf(l) > -1)
+    ) {
+        exitWithError = true;
+        const extraActuals = actuals.filter((actual) => expecteds.indexOf(actual) === -1);
+        const extraExpecteds = expecteds.filter((expected) => actuals.indexOf(expected) === -1);
+        console.log(`Mismatch between actual and expected ${what}:`);
+        if (extraActuals.length > 0) {
+            console.log(`\tAmong actual, but not among expected: ${extraActuals.join(", ")}`);
+        }
+        if (extraExpecteds.length > 0) {
+            console.log(`\tAmong expected, but not among actual: ${extraExpecteds.join(", ")}`);
+        }
+    }
+}
+
+
+const SCHEMA_WITH_EXPECTED_VALUESETS = "DCC.ValueSets.schema.json";
+
+const EXPECTED_EXTRA = [
+    "test-manf-example.json"
+];
+const EXPECTED_MISSING = [
+    "test-manf.json"
+];
+
+const removedValuesetsPrefix = (path) => {
+    const prefix = "valuesets/";
+    if (!path.indexOf(prefix) === 0) {
         exitWithError = true;
+        console.log(`Valueset URI doesn't start with expected prefix "${prefix}" --> check ${SCHEMA_WITH_EXPECTED_VALUESETS} in schema repository`);
+        return path;
     }
+    return path.substring(prefix.length);
 }
 
-// Handle exit
-if(exitWithError){
-    console.log("Validation failed!");
-    process.exit(1);
-} else {
-    console.log("Validation succeeded!");
-}
+const {EU_DCC_SCHEMA_REF} = require("./config");
+
+fetch(new URL(`https://raw.githubusercontent.com/ehn-dcc-development/eu-dcc-schema/${EU_DCC_SCHEMA_REF}/${SCHEMA_WITH_EXPECTED_VALUESETS}`))
+    .then((response) => response.json())
+    .then((valuesetsRefSchema) => {
+        const expectedValuesets = Object.values(valuesetsRefSchema["$defs"])
+            .map((valuesetDef) => valuesetDef["valueset-uri"])
+            .map(removedValuesetsPrefix);
+
+        assertSameSet(
+            valuesets.filter((valueset) => expectedValuesets.indexOf(valueset) === -1),
+            EXPECTED_EXTRA,
+            "extra valuesets"
+        );
+        assertSameSet(
+            expectedValuesets.filter((expected) => valuesets.indexOf(expected) === -1),
+            EXPECTED_MISSING,
+            "missing valuesets"
+        );
+
+        // handle exit:
+        if (exitWithError){
+            console.log("Validation failed!");
+            process.exit(1);
+        } else {
+            console.log(`Validating entire set against ${SCHEMA_WITH_EXPECTED_VALUESETS} in schema repository --> OK`);
+            console.log("Validation succeeded!");
+        }
+    });
+

source-update/README.md

@@ -8,7 +8,7 @@
 > This script is now deprecated and will not be kept up-to-date. Currently, due to changes in the source JSON format the script is not functional. The generation of the rapid antigen tests (RATs) value set is managed by the DCC Gateway,
 > see •	https://github.com/eu-digital-green-certificates/dgc-gateway/blob/main/src/main/java/eu/europa/ec/dgc/gateway/service/RatValuesetUpdateService.java
 
-This repository contains scripts that are used to update the value set for the common list of rapid antigen tests (RATs) referenced by the EU Digital COVID Certificate (DCC) [JSON Schema](https://github.com/ehn-dcc-development/ehn-dcc-schema).
+This repository contains scripts that are used to update the value set for the common list of rapid antigen tests (RATs) referenced by the EU Digital COVID Certificate (DCC) [JSON Schema](https://github.com/ehn-dcc-development/eu-dcc-schema).
 
 Release: 2.0.0