handle baseline

.github/PULL_REQUEST_TEMPLATE.md

@@ -3,9 +3,7 @@ Thank you for your Pull Request. Please provide a description and review
 the requirements below.
 
 Contributors guide: https://github.com/theia-ide/theia/blob/master/CONTRIBUTING.md
--->
 
-<!--
 Note: Security vulnerabilities should not be disclosed on GitHub, through a PR or any
 other means. See SECURITY.md at the root of this repository, to learn how to report
 vulnerabilities.
@@ -19,9 +17,8 @@ vulnerabilities.
 
 #### Review checklist
 
-- [ ] as an author, I have thoroughly tested my changes and carefully followed [the review guidelines](https://github.com/theia-ide/theia/blob/master/doc/pull-requests.md#requesting-a-review)
+- [ ] As an author, I have thoroughly tested my changes and carefully followed [the review guidelines](https://github.com/theia-ide/theia/blob/master/doc/pull-requests.md#requesting-a-review)
 
 #### Reminder for reviewers
 
-- as a reviewer, I agree to behave in accordance with [the review guidelines](https://github.com/theia-ide/theia/blob/master/doc/pull-requests.md#reviewing)
-
+- As a reviewer, I agree to behave in accordance with [the review guidelines](https://github.com/theia-ide/theia/blob/master/doc/pull-requests.md#reviewing)

.gitignore

@@ -28,5 +28,5 @@ gh-pages
 dev-packages/electron/compile_commands.json
 *.tsbuildinfo
 .eslintcache
-license-check-summary.txt
 scripts/download
+license-check-summary.txt*

doc/pull-requests.md

@@ -16,37 +16,37 @@ If a rule causes distress during discussions itself, it has to be reviewed on [t
 
 <a name="pr-template"></a>
 - [1.](#pr-template) Each PR description has to follow the following template:
-```
-<!--
-Thank you for your Pull Request. Please provide a description and review
-the requirements below.
 
-Contributors guide: https://github.com/eclipse-theia/theia/blob/master/CONTRIBUTING.md
--->
+  ```md
+  <!--
+  Thank you for your Pull Request. Please provide a description and review
+  the requirements below.
 
-#### What it does
-<!-- Include relevant issues and describe how they are addressed. -->
+  Contributors guide: https://github.com/eclipse-theia/theia/blob/master/CONTRIBUTING.md
+  -->
 
-#### How to test
-<!-- Explain how a reviewer can reproduce a bug, test new functionality or verify performance improvements. -->
+  #### What it does
+  <!-- Include relevant issues and describe how they are addressed. -->
 
-#### Review checklist
+  #### How to test
+  <!-- Explain how a reviewer can reproduce a bug, test new functionality or verify performance improvements. -->
 
-- [ ] as an author, I have thoroughly tested my changes and carefully followed [the review guidelines](https://github.com/eclipse-theia/theia/blob/master/doc/pull-requests.md#requesting-a-review)
+  #### Review checklist
 
-#### Reminder for reviewers
+  - [ ] As an author, I have thoroughly tested my changes and carefully followed [the review guidelines](https://github.com/eclipse-theia/theia/blob/master/doc/pull-requests.md#requesting-a-review)
 
-- as a reviewer, I agree to review in accordance with [the review guidelines](https://github.com/eclipse-theia/theia/blob/master/doc/pull-requests.md#reviewing)
+  #### Reminder for reviewers
 
-```
+  - As a reviewer, I agree to review in accordance with [the review guidelines](https://github.com/eclipse-theia/theia/blob/master/doc/pull-requests.md#reviewing)
+  ```
 
 <a name="design-review"></a>
 - [2.](#design-review) A PR can be opened early for the design review before going into the detailed implementation.
   - A request on the design review should be an explicit comment.
   - Such PR should be marked as a draft or with the WIP prefix.
 
 <a name="fixups"></a>
-- [3.](#fixups) Changes done _after_ the PR has been opened should be kept in separate commits while the review process is not finished. This allows reviewers to re-review only the updated parts of the PR and to determine what needs to be tested again. The "fixup" commits must be squashed before merging in order to keep a clean history. 
+- [3.](#fixups) Changes done _after_ the PR has been opened should be kept in separate commits while the review process is not finished. This allows reviewers to re-review only the updated parts of the PR and to determine what needs to be tested again. The "fixup" commits must be squashed before merging in order to keep a clean history.
 
 ## Requesting a Review
 
@@ -75,6 +75,8 @@ Contributors guide: https://github.com/eclipse-theia/theia/blob/master/CONTRIBUT
 - [5.](#checklist-dependencies) New dependencies are justified and [verified](https://github.com/eclipse-theia/theia/wiki/Registering-CQs#wip---new-ecd-theia-intellectual-property-clearance-approach-experimental).
 <a name="checklist-copied-code"></a>
 - [6.](#checklist-copied-code) Copied code is justified and [approved via a CQ](https://github.com/eclipse-theia/theia/wiki/Registering-CQs#case-3rd-party-project-code-copiedforked-from-another-project-into-eclipse-theia-maintained-by-us).
+  - Look closely at the GitHub actions running for your PR: the 3pp/dash license check should be green.
+  - If red: it most likely mean you need to create a CQ.
 <a name="checklist-copyright"></a>
 - [7.](#checklist-copyright) Each new file has proper copyright with the current year and the name of contributing entity (individual or company).
 <a name="checklist-sign-off"></a>

license-check-baseline.json

@@ -0,0 +1,40 @@
+{
+  "npm/npmjs/-/ajv/6.12.6": null,
+  "npm/npmjs/-/autoprefixer/6.7.7": null,
+  "npm/npmjs/-/big.js/3.2.0": null,
+  "npm/npmjs/-/coa/1.0.4": null,
+  "npm/npmjs/-/eslint-plugin-deprecation/1.2.1": null,
+  "npm/npmjs/-/esprima/4.0.1": null,
+  "npm/npmjs/-/esquery/1.4.0": null,
+  "npm/npmjs/-/extsprintf/1.4.0": null,
+  "npm/npmjs/-/from/0.1.7": null,
+  "npm/npmjs/-/fs-extra/4.0.3": null,
+  "npm/npmjs/-/gauge/2.7.4": null,
+  "npm/npmjs/-/gitconfiglocal/1.0.0": null,
+  "npm/npmjs/-/glob/7.1.3": null,
+  "npm/npmjs/-/js-yaml/3.7.0": null,
+  "npm/npmjs/-/jschardet/2.3.0": null,
+  "npm/npmjs/-/jsdom/11.12.0": null,
+  "npm/npmjs/-/jsmin/1.0.1": null,
+  "npm/npmjs/-/json-schema/0.2.3": null,
+  "npm/npmjs/-/json5/0.5.1": null,
+  "npm/npmjs/-/less-loader/2.2.3": null,
+  "npm/npmjs/-/npmlog/4.1.2": null,
+  "npm/npmjs/-/oniguruma/7.2.3": null,
+  "npm/npmjs/-/parse-json/2.2.0": null,
+  "npm/npmjs/-/postcss-reduce-initial/1.0.1": null,
+  "npm/npmjs/-/q/1.5.1": null,
+  "npm/npmjs/-/rc/1.2.8": null,
+  "npm/npmjs/-/rechoir/0.6.2": null,
+  "npm/npmjs/-/shelljs/0.8.4": null,
+  "npm/npmjs/-/source-list-map/2.0.1": null,
+  "npm/npmjs/-/spdx-correct/3.1.1": null,
+  "npm/npmjs/-/spdx-license-ids/3.0.9": null,
+  "npm/npmjs/-/style-loader/0.13.2": null,
+  "npm/npmjs/-/through/2.3.8": null,
+  "npm/npmjs/-/ts-md5/1.2.9": null,
+  "npm/npmjs/-/tweetnacl/0.14.5": null,
+  "npm/npmjs/-/typescript/3.9.10": null,
+  "npm/npmjs/-/uc.micro/1.0.6": null,
+  "npm/npmjs/-/uri-js/4.4.1": null
+}

scripts/check_3pp_licenses.js

@@ -18,29 +18,179 @@
 const cp = require('child_process');
 const fs = require('fs');
 const path = require('path');
+const readline = require('readline');
 
 const licenseToolJar = path.resolve(__dirname, 'download/license.jar');
 const licenseToolSummary = path.resolve(__dirname, '../license-check-summary.txt');
+const licenseToolBaseline = path.resolve(__dirname, '../license-check-baseline.json');
 const licenseToolUrl = 'https://repo.eclipse.org/service/local/artifact/maven/redirect?r=dash-licenses&g=org.eclipse.dash&a=org.eclipse.dash.licenses&v=LATEST';
 
-console.log('Fetching dash-licenses...');
-if (!fs.existsSync(licenseToolJar)) {
-    fs.mkdirSync(path.dirname(licenseToolJar), { recursive: true });
-    spawn('curl', ['-L', licenseToolUrl, '-o', licenseToolJar]);
+main().catch(error => {
+    console.error(error);
+    process.exit(1);
+});
+
+async function main() {
+    if (!fs.existsSync(licenseToolJar)) {
+        console.warn('Fetching dash-licenses...');
+        fs.mkdirSync(path.dirname(licenseToolJar), { recursive: true });
+        exitOnChildError(spawn(
+            'curl', ['-L', licenseToolUrl, '-o', licenseToolJar],
+        ));
+    }
+    if (fs.existsSync(licenseToolSummary)) {
+        console.warn('Backing up previous summary...')
+        fs.renameSync(licenseToolSummary, `${licenseToolSummary}.old`);
+    }
+    console.warn('Running dash-licenses...');
+    const dashStatus = spawn(
+        'java', ['-jar', licenseToolJar, 'yarn.lock', '-batch', '50', '-timeout', '240', '-summary', licenseToolSummary],
+        { stdio: ['ignore', 'ignore', 'inherit'] },
+    );
+    const error = getChildError(dashStatus);
+    if (error) {
+        console.error(error);
+    }
+    const restricted = await readSummaryRestricted(licenseToolSummary);
+    if (restricted.length > 0) {
+        if (fs.existsSync(licenseToolBaseline)) {
+            console.warn('Checking results against the baseline...');
+            const baseline = readBaseline(licenseToolBaseline);
+            const unhandled = restricted.filter(entry => !baseline.has(entry.entry));
+            if (unhandled.length > 0) {
+                console.error(`ERROR: Found results that aren't part of the baseline!\n`);
+                logRestrictedDependencies(unhandled);
+                process.exit(1);
+            }
+        } else {
+            console.error(`ERROR: Found unhandled restricted dependencies!\n`);
+            logRestrictedDependencies(restricted);
+            process.exit(1);
+        }
+    }
+    console.warn('Done.');
+    process.exit(0);
+}
+
+/**
+ * @param {DashSummaryEntry[]} restricted list of restricted entries to log.
+ * @return {void}
+ */
+function logRestrictedDependencies(restricted) {
+    for (const { entry, license } of restricted) {
+        console.log(`${entry}, ${license}`);
+    }
+}
+
+/**
+ * @param {string} summary path to the summary file.
+ * @returns {Promise<DashSummaryEntry[]>} list of restriced dependencies.
+ */
+async function readSummaryRestricted(summary) {
+    const restricted = [];
+    await readSummary(summary, entry => {
+        if (entry.status.toLocaleLowerCase() === 'restricted') {
+            restricted.push(entry);
+        }
+    });
+    return restricted.sort();
+}
+
+/**
+ * Read each entry from dash's summary file and collect non-ignored restricted entries.
+ * This is essentially a cheap CSV parser.
+ * @param {string} summary path to the summary file.
+ * @param {(line: DashSummaryEntry) => void} callback
+ * @returns {Promise<void>} reading completed.
+ */
+async function readSummary(summary, callback) {
+    return new Promise((resolve, reject) => {
+        // Read each entry from dash's summary file and collect non-ignored restricted entries.
+        // This is essentially a cheap CSV parser.
+        readline.createInterface(fs.createReadStream(summary).on('error', reject))
+            .on('line', line => {
+                const [entry, license, status, source] = line.split(', ');
+                callback({ entry, license, status, source });
+            })
+            .on('close', resolve);
+    });
 }
-console.log('Running dash-licenses...');
-spawn('java', ['-jar', licenseToolJar, 'yarn.lock', '-batch', '50', '-timeout', '240', '-summary', licenseToolSummary]);
 
-function spawn(bin, args, opt) {
-    const exit = cp.spawnSync(bin, args, { stdio: 'inherit', ...opt });
-    if (exit.error) {
-        console.error(exit.error);
+/**
+ * Handle both list and object format for the baseline json file.
+ * @param {string} baseline path to the baseline json file.
+ * @returns {Set<string>} set of ignored restricted dependencies.
+ */
+function readBaseline(baseline) {
+    const json = JSON.parse(fs.readFileSync(baseline, 'utf8'));
+    if (Array.isArray(json)) {
+        return new Set(json);
+    } else if (typeof json === 'object' && json !== null) {
+        return new Set(Object.keys(json));
+    }
+    console.error(`ERROR: Invalid format for "${baseline}"`);
+    process.exit(1);
+}
+
+/**
+ * Spawn a process. Exits with code 1 on spawn error (e.g. file not found).
+ * @param {string} bin
+ * @param {string[]} args
+ * @param {import('child_process').SpawnSyncOptions} [opts]
+ * @returns {import('child_process').SpawnSyncReturns}
+ */
+function spawn(bin, args, opts = {}) {
+    opts = { stdio: 'inherit', ...opts };
+    /** @type {any} */
+    const status = cp.spawnSync(bin, args, opts);
+    // Add useful fields to the returned status object:
+    status.bin = bin;
+    status.args = args;
+    status.opts = opts;
+    // Abort on spawn error:
+    if (status.error) {
+        console.error(status.error);
         process.exit(1);
     }
-    if (typeof exit.signal === 'string') {
-        console.error(`${bin} exited with signal: ${exit.signal}`);
+    return status;
+}
+
+/**
+ * @returns {string | undefined} Error message if the process errored, `undefined` otherwise.
+ */
+function getChildError(status) {
+    if (typeof status.signal === 'string') {
+        return `Command ${prettyCommand(status)} exited with signal: ${status.signal}`;
+    } else if (status.status !== 0) {
+        return `Command ${prettyCommand(status)} exited with code: ${status.status}`;
+    }
+}
+
+/**
+ * @param {any} status
+ * @returns {string} Pretty command with both bin and args as stringified JSON.
+ */
+function prettyCommand(status, indent = 2) {
+    return JSON.stringify([status.bin, ...status.args], undefined, indent);
+}
+
+/**
+ * Exits with code 1 if `status` errored.
+ * @returns {import('child_process').SpawnSyncReturns}
+ */
+function exitOnChildError(status) {
+    const error = getChildError(status);
+    if (error) {
+        console.error(error);
         process.exit(1);
-    } else if (exit.status !== 0) {
-        process.exit(exit.status);
     }
+    return status;
 }
+
+/**
+ * @typedef {object} DashSummaryEntry
+ * @property {string} entry
+ * @property {string} license
+ * @property {string} status
+ * @property {string} source
+ */