[ci] run dash-licenses as part of GH workflow

.github/PULL_REQUEST_TEMPLATE.md

@@ -3,9 +3,7 @@ Thank you for your Pull Request. Please provide a description and review
 the requirements below.
 
 Contributors guide: https://github.com/theia-ide/theia/blob/master/CONTRIBUTING.md
--->
 
-<!--
 Note: Security vulnerabilities should not be disclosed on GitHub, through a PR or any
 other means. See SECURITY.md at the root of this repository, to learn how to report
 vulnerabilities.
@@ -19,9 +17,8 @@ vulnerabilities.
 
 #### Review checklist
 
-- [ ] as an author, I have thoroughly tested my changes and carefully followed [the review guidelines](https://github.com/theia-ide/theia/blob/master/doc/pull-requests.md#requesting-a-review)
+- [ ] As an author, I have thoroughly tested my changes and carefully followed [the review guidelines](https://github.com/theia-ide/theia/blob/master/doc/pull-requests.md#requesting-a-review)
 
 #### Reminder for reviewers
 
-- as a reviewer, I agree to behave in accordance with [the review guidelines](https://github.com/theia-ide/theia/blob/master/doc/pull-requests.md#reviewing)
-
+- As a reviewer, I agree to behave in accordance with [the review guidelines](https://github.com/theia-ide/theia/blob/master/doc/pull-requests.md#reviewing)

.github/workflows/license-check.yml

@@ -0,0 +1,51 @@
+name: 3PP License Check
+
+on:
+  push:
+    branches:
+      - master
+  workflow_dispatch:
+  pull_request:
+    branches:
+      - master
+  schedule:
+    - cron: '0 4 * * *' # Runs every day at 4am: https://docs.github.com/en/actions/reference/events-that-trigger-workflows#scheduled-events-schedule
+
+jobs:
+
+  License-check:
+    name: ${{ matrix.os }}, Node.js v${{ matrix.node }}
+
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-18.04]
+        node: ['12.x']
+        java: ['11']
+
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 60
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 2
+
+      - name: Use Node.js ${{ matrix.node }}
+        uses: actions/setup-node@v1
+        with:
+          node-version: ${{ matrix.node }}
+          registry-url: 'https://registry.npmjs.org'
+
+      - name: Use Java ${{ matrix.java }}
+        uses: actions/setup-java@v2
+        with:
+          distribution: 'adopt'
+          java-version: ${{ matrix.java }}
+
+      - name: Run dash-licenses
+        if: matrix.tests != 'skip'
+        shell: bash
+        run: |
+          yarn license:check

.gitignore

@@ -28,3 +28,5 @@ gh-pages
 dev-packages/electron/compile_commands.json
 *.tsbuildinfo
 .eslintcache
+scripts/download
+license-check-summary.txt*

doc/pull-requests.md

@@ -16,37 +16,37 @@ If a rule causes distress during discussions itself, it has to be reviewed on [t
 
 <a name="pr-template"></a>
 - [1.](#pr-template) Each PR description has to follow the following template:
-```
-<!--
-Thank you for your Pull Request. Please provide a description and review
-the requirements below.
 
-Contributors guide: https://github.com/eclipse-theia/theia/blob/master/CONTRIBUTING.md
--->
+  ```md
+  <!--
+  Thank you for your Pull Request. Please provide a description and review
+  the requirements below.
 
-#### What it does
-<!-- Include relevant issues and describe how they are addressed. -->
+  Contributors guide: https://github.com/eclipse-theia/theia/blob/master/CONTRIBUTING.md
+  -->
 
-#### How to test
-<!-- Explain how a reviewer can reproduce a bug, test new functionality or verify performance improvements. -->
+  #### What it does
+  <!-- Include relevant issues and describe how they are addressed. -->
 
-#### Review checklist
+  #### How to test
+  <!-- Explain how a reviewer can reproduce a bug, test new functionality or verify performance improvements. -->
 
-- [ ] as an author, I have thoroughly tested my changes and carefully followed [the review guidelines](https://github.com/eclipse-theia/theia/blob/master/doc/pull-requests.md#requesting-a-review)
+  #### Review checklist
 
-#### Reminder for reviewers
+  - [ ] As an author, I have thoroughly tested my changes and carefully followed [the review guidelines](https://github.com/eclipse-theia/theia/blob/master/doc/pull-requests.md#requesting-a-review)
 
-- as a reviewer, I agree to review in accordance with [the review guidelines](https://github.com/eclipse-theia/theia/blob/master/doc/pull-requests.md#reviewing)
+  #### Reminder for reviewers
 
-```
+  - As a reviewer, I agree to review in accordance with [the review guidelines](https://github.com/eclipse-theia/theia/blob/master/doc/pull-requests.md#reviewing)
+  ```
 
 <a name="design-review"></a>
 - [2.](#design-review) A PR can be opened early for the design review before going into the detailed implementation.
   - A request on the design review should be an explicit comment.
   - Such PR should be marked as a draft or with the WIP prefix.
 
 <a name="fixups"></a>
-- [3.](#fixups) Changes done _after_ the PR has been opened should be kept in separate commits while the review process is not finished. This allows reviewers to re-review only the updated parts of the PR and to determine what needs to be tested again. The "fixup" commits must be squashed before merging in order to keep a clean history. 
+- [3.](#fixups) Changes done _after_ the PR has been opened should be kept in separate commits while the review process is not finished. This allows reviewers to re-review only the updated parts of the PR and to determine what needs to be tested again. The "fixup" commits must be squashed before merging in order to keep a clean history.
 
 ## Requesting a Review
 
@@ -75,6 +75,8 @@ Contributors guide: https://github.com/eclipse-theia/theia/blob/master/CONTRIBUT
 - [5.](#checklist-dependencies) New dependencies are justified and [verified](https://github.com/eclipse-theia/theia/wiki/Registering-CQs#wip---new-ecd-theia-intellectual-property-clearance-approach-experimental).
 <a name="checklist-copied-code"></a>
 - [6.](#checklist-copied-code) Copied code is justified and [approved via a CQ](https://github.com/eclipse-theia/theia/wiki/Registering-CQs#case-3rd-party-project-code-copiedforked-from-another-project-into-eclipse-theia-maintained-by-us).
+  - Look closely at the GitHub actions running for your PR: the 3pp/dash license check should be green.
+  - If red: it most likely mean you need to create a CQ.
 <a name="checklist-copyright"></a>
 - [7.](#checklist-copyright) Each new file has proper copyright with the current year and the name of contributing entity (individual or company).
 <a name="checklist-sign-off"></a>

license-check-baseline.json

@@ -0,0 +1,40 @@
+{
+  "npm/npmjs/-/ajv/6.12.6": "Approved in CQ: https://dev.eclipse.org/ipzilla/show_bug.cgi?id=22482",
+  "npm/npmjs/-/autoprefixer/6.7.7": null,
+  "npm/npmjs/-/big.js/3.2.0": null,
+  "npm/npmjs/-/coa/1.0.4": null,
+  "npm/npmjs/-/eslint-plugin-deprecation/1.2.1": "Approved as 'works-with': https://dev.eclipse.org/ipzilla/show_bug.cgi?id=22573",
+  "npm/npmjs/-/esprima/4.0.1": null,
+  "npm/npmjs/-/esquery/1.4.0": null,
+  "npm/npmjs/-/extsprintf/1.4.0": null,
+  "npm/npmjs/-/from/0.1.7": null,
+  "npm/npmjs/-/fs-extra/4.0.3": null,
+  "npm/npmjs/-/gauge/2.7.4": null,
+  "npm/npmjs/-/gitconfiglocal/1.0.0": null,
+  "npm/npmjs/-/glob/7.1.3": null,
+  "npm/npmjs/-/js-yaml/3.7.0": "Approved in CQ: https://dev.eclipse.org/ipzilla/show_bug.cgi?id=22482",
+  "npm/npmjs/-/jschardet/2.3.0": "Approved for Eclipse Theia: https://dev.eclipse.org/ipzilla/show_bug.cgi?id=22481",
+  "npm/npmjs/-/jsdom/11.12.0": "CQ under review: https://dev.eclipse.org/ipzilla/show_bug.cgi?id=23640https://dev.eclipse.org/ipzilla/show_bug.cgi?id=23640",
+  "npm/npmjs/-/jsmin/1.0.1": "CQ under review: https://dev.eclipse.org/ipzilla/show_bug.cgi?id=23640https://dev.eclipse.org/ipzilla/show_bug.cgi?id=23640",
+  "npm/npmjs/-/json-schema/0.2.3": null,
+  "npm/npmjs/-/json5/0.5.1": null,
+  "npm/npmjs/-/less-loader/2.2.3": null,
+  "npm/npmjs/-/npmlog/4.1.2": null,
+  "npm/npmjs/-/oniguruma/7.2.3": null,
+  "npm/npmjs/-/parse-json/2.2.0": null,
+  "npm/npmjs/-/postcss-reduce-initial/1.0.1": null,
+  "npm/npmjs/-/q/1.5.1": null,
+  "npm/npmjs/-/rc/1.2.8": null,
+  "npm/npmjs/-/rechoir/0.6.2": null,
+  "npm/npmjs/-/shelljs/0.8.4": null,
+  "npm/npmjs/-/source-list-map/2.0.1": null,
+  "npm/npmjs/-/spdx-correct/3.1.1": null,
+  "npm/npmjs/-/spdx-license-ids/3.0.9": null,
+  "npm/npmjs/-/style-loader/0.13.2": null,
+  "npm/npmjs/-/through/2.3.8": null,
+  "npm/npmjs/-/ts-md5/1.2.9": null,
+  "npm/npmjs/-/tweetnacl/0.14.5": null,
+  "npm/npmjs/-/typescript/3.9.10": null,
+  "npm/npmjs/-/uc.micro/1.0.6": null,
+  "npm/npmjs/-/uri-js/4.4.1": null
+}

package.json

@@ -76,7 +76,8 @@
     "start:electron": "yarn rebuild:electron && yarn --cwd examples/electron start",
     "debug:browser": "yarn rebuild:browser && yarn --cwd examples/browser start:debug",
     "debug:electron": "yarn rebuild:electron && yarn --cwd examples/electron start:debug",
-    "download:plugins": "theia download:plugins"
+    "download:plugins": "theia download:plugins",
+    "license:check": "node scripts/check_3pp_licenses.js"
   },
   "workspaces": [
     "dev-packages/*",

scripts/check_3pp_licenses.js

@@ -0,0 +1,186 @@
+/********************************************************************************
+ * Copyright (c) 2021 Ericsson and others
+ *
+ * This program and the accompanying materials are made available under the
+ * terms of the Eclipse Public License v. 2.0 which is available at
+ * http://www.eclipse.org/legal/epl-2.0.
+ *
+ * This Source Code may also be made available under the following Secondary
+ * Licenses when the conditions for such availability set forth in the Eclipse
+ * Public License v. 2.0 are satisfied: GNU General Public License, version 2
+ * with the GNU Classpath Exception which is available at
+ * https://www.gnu.org/software/classpath/license.html.
+ *
+ * SPDX-License-Identifier: EPL-2.0 OR GPL-2.0 WITH Classpath-exception-2.0
+ ********************************************************************************/
+// @ts-check
+
+const cp = require('child_process');
+const fs = require('fs');
+const path = require('path');
+const readline = require('readline');
+
+const dashLicensesJar = path.resolve(__dirname, 'download/dash-licenses.jar');
+const dashLicensesSummary = path.resolve(__dirname, '../license-check-summary.txt');
+const dashLicensesBaseline = path.resolve(__dirname, '../license-check-baseline.json');
+const dashLicensesUrl = 'https://repo.eclipse.org/service/local/artifact/maven/redirect?r=dash-licenses&g=org.eclipse.dash&a=org.eclipse.dash.licenses&v=LATEST';
+
+main().catch(error => {
+    console.error(error);
+    process.exit(1);
+});
+
+async function main() {
+    if (!fs.existsSync(dashLicensesJar)) {
+        console.warn('Fetching dash-licenses...');
+        fs.mkdirSync(path.dirname(dashLicensesJar), { recursive: true });
+        const curlError = getErrorFromStatus(spawn(
+            'curl', ['-L', dashLicensesUrl, '-o', dashLicensesJar],
+        ));
+        if (curlError) {
+            console.error(curlError);
+            process.exit(1);
+        }
+    }
+    if (fs.existsSync(dashLicensesSummary)) {
+        console.warn('Backing up previous summary...')
+        fs.renameSync(dashLicensesSummary, `${dashLicensesSummary}.old`);
+    }
+    console.warn('Running dash-licenses...');
+    const dashError = getErrorFromStatus(spawn(
+        'java', ['-jar', dashLicensesJar, 'yarn.lock', '-batch', '50', '-timeout', '240', '-summary', dashLicensesSummary],
+        { stdio: ['ignore', 'ignore', 'inherit'] },
+    ));
+    if (dashError) {
+        console.error(dashError);
+    }
+    const restricted = await getRestrictedDependenciesFromSummary(dashLicensesSummary);
+    if (restricted.length > 0) {
+        if (fs.existsSync(dashLicensesBaseline)) {
+            console.warn('Checking results against the baseline...');
+            const baseline = readBaseline(dashLicensesBaseline);
+            const unhandled = restricted.filter(entry => !baseline.has(entry.entry));
+            if (unhandled.length > 0) {
+                console.error(`ERROR: Found results that aren't part of the baseline!\n`);
+                logRestrictedDependencies(unhandled);
+                process.exit(1);
+            }
+        } else {
+            console.error(`ERROR: Found unhandled restricted dependencies!\n`);
+            logRestrictedDependencies(restricted);
+            process.exit(1);
+        }
+    }
+    console.warn('Done.');
+    process.exit(0);
+}
+
+/**
+ * @param {DashSummaryEntry[]} restricted list of restricted entries to log.
+ * @return {void}
+ */
+function logRestrictedDependencies(restricted) {
+    for (const { entry, license } of restricted) {
+        console.log(`${entry}, ${license}`);
+    }
+}
+
+/**
+ * @param {string} summary path to the summary file.
+ * @returns {Promise<DashSummaryEntry[]>} list of restriced dependencies.
+ */
+async function getRestrictedDependenciesFromSummary(summary) {
+    const restricted = [];
+    await readSummary(summary, entry => {
+        if (entry.status.toLocaleLowerCase() === 'restricted') {
+            restricted.push(entry);
+        }
+    });
+    return restricted.sort();
+}
+
+/**
+ * Read each entry from dash's summary file and collect each entry.
+ * This is essentially a cheap CSV parser.
+ * @param {string} summary path to the summary file.
+ * @param {(line: DashSummaryEntry) => void} callback
+ * @returns {Promise<void>} reading completed.
+ */
+async function readSummary(summary, callback) {
+    return new Promise((resolve, reject) => {
+        readline.createInterface(fs.createReadStream(summary).on('error', reject))
+            .on('line', line => {
+                const [entry, license, status, source] = line.split(', ');
+                callback({ entry, license, status, source });
+            })
+            .on('close', resolve);
+    });
+}
+
+/**
+ * Handle both list and object format for the baseline json file.
+ * @param {string} baseline path to the baseline json file.
+ * @returns {Set<string>} set of ignored restricted dependencies.
+ */
+function readBaseline(baseline) {
+    const json = JSON.parse(fs.readFileSync(baseline, 'utf8'));
+    if (Array.isArray(json)) {
+        return new Set(json);
+    } else if (typeof json === 'object' && json !== null) {
+        return new Set(Object.keys(json));
+    }
+    console.error(`ERROR: Invalid format for "${baseline}"`);
+    process.exit(1);
+}
+
+/**
+ * Spawn a process. Exits with code 1 on spawn error (e.g. file not found).
+ * @param {string} bin
+ * @param {string[]} args
+ * @param {import('child_process').SpawnSyncOptions} [opts]
+ * @returns {import('child_process').SpawnSyncReturns}
+ */
+function spawn(bin, args, opts = {}) {
+    opts = { stdio: 'inherit', ...opts };
+    /** @type {any} */
+    const status = cp.spawnSync(bin, args, opts);
+    // Add useful fields to the returned status object:
+    status.bin = bin;
+    status.args = args;
+    status.opts = opts;
+    // Abort on spawn error:
+    if (status.error) {
+        console.error(status.error);
+        process.exit(1);
+    }
+    return status;
+}
+
+/**
+ * @param {import('child_process').SpawnSyncReturns} status
+ * @returns {string | undefined} Error message if the process errored, `undefined` otherwise.
+ */
+function getErrorFromStatus(status) {
+    if (typeof status.signal === 'string') {
+        return `Command ${prettyCommand(status)} exited with signal: ${status.signal}`;
+    } else if (status.status !== 0) {
+        return `Command ${prettyCommand(status)} exited with code: ${status.status}`;
+    }
+}
+
+/**
+ * @param {any} status
+ * @param {number} [indent]
+ * @returns {string} Pretty command with both bin and args as stringified JSON.
+ */
+function prettyCommand(status, indent = 2) {
+    return JSON.stringify([status.bin, ...status.args], undefined, indent);
+}
+
+/**
+ * @typedef {object} DashSummaryEntry
+ * @property {string} entry
+ * @property {string} license
+ * @property {string} status
+ * @property {string} source
+ */