Skip to content

Commit

Permalink
feat: support evaluating spdx license expressions (#1329)
Browse files Browse the repository at this point in the history 
This implements a parser for SPDX license expressions in accordance with
[annex D of the v2
spec](https://spdx.github.io/spdx-spec/v2-draft/SPDX-license-expressions/),
to allow the scanner to properly determine if packages with such
expressive licenses are permitted based on the licenses allowed by the
user.

To do this I've implemented a two-phase parser which starts by
tokenizing the license string and then turns it into an AST of nodes
that can be walked to determine if the full expression is satisfied; for
no particularly good reason I have used a string for the base token type
rather than a `struct`, meaning the tokens value is also it's type - the
tradeoff with this is while it means we don't have to do as much
referencing or work in the tokenizer, we do have to do some extra work
when walking the tree to resolve the "simple expression" tokens.

I'm proposing landing the current implementation as I don't think using
a `struct` would be strictly better, though in hindsight it probably
would have been a bit quicker to implement and so I plan to explore how
much simpler (or complex) it might be as a follow up.

Currently this is focused on `AND` and `OR` support, as I believe those
are the two primary operators that are relevant to the scanner, though
we still might want to have richer handling for the `WITH` and `+`
operators; currently both of those just get treated as being part of the
license expression (though it's not actually possible right now to allow
a license with `WITH` as the CLI expects license values to not have any
spaces - this too will be a follow-up for me).

Finally, I've purposely not put any caching in place even though that
should be easy, due to wanting to get this landed and as I don't expect
it to actually have a significant impact on the scanner performance
(ultimately most complex expressions in the real-world will be made up
of a single operator, and chopping+looping over strings in memory is
extremely fast)

Resolves #1299
  • Loading branch information
@G-Rath
G-Rath authored Nov 11, 2024
1 parent 581d1a3 commit 73fe113
Show file tree
Hide file tree
Showing 9 changed files with 817 additions and 12 deletions.
77 changes: 73 additions & 4 deletions cmd/osv-scanner/__snapshots__/main_test.snap
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1126,6 +1126,16 @@ Warning: `scan` exists as both a subcommand of OSV-Scanner and as a file on the
"licenses": [
"MIT"
]
},
{
"package": {
"name": "type-fest",
"version": "4.26.1",
"ecosystem": "npm"
},
"licenses": [
"CC0-1.0 OR MIT"
]
}
]
}
Expand All @@ -1142,7 +1152,46 @@ Warning: `scan` exists as both a subcommand of OSV-Scanner and as a file on the

[TestRun_Licenses/Licenses_in_summary_mode_json - 2]
Scanning dir ./fixtures/locks-licenses/package-lock.json
Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 3 packages
Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 4 packages

---

[TestRun_Licenses/Licenses_with_expressions - 1]
Scanning dir ./fixtures/locks-licenses/package-lock.json
Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 4 packages
overriding license for package npm/babel/6.23.0 with MIT AND (LGPL-2.1-or-later OR BSD-3-Clause)
overriding license for package npm/human-signals/5.0.0 with LGPL-2.1-only OR MIT OR BSD-3-Clause
overriding license for package npm/ms/2.1.3 with MIT WITH Bison-exception-2.2
+------------------------------+-----------+---------+---------+-------------------------------------------+
| LICENSE VIOLATION | ECOSYSTEM | PACKAGE | VERSION | SOURCE |
+------------------------------+-----------+---------+---------+-------------------------------------------+
| MIT WITH Bison-exception-2.2 | npm | ms | 2.1.3 | fixtures/locks-licenses/package-lock.json |
+------------------------------+-----------+---------+---------+-------------------------------------------+

---

[TestRun_Licenses/Licenses_with_expressions - 2]

---

[TestRun_Licenses/Licenses_with_invalid_expression - 1]
Scanning dir ./fixtures/locks-licenses/package-lock.json
Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 4 packages
overriding license for package npm/babel/6.23.0 with MIT AND (LGPL-2.1-or-later OR BSD-3-Clause))
overriding license for package npm/human-signals/5.0.0 with LGPL-2.1-only OR OR BSD-3-Clause
overriding license for package npm/ms/2.1.3 with MIT WITH (Bison-exception-2.2 AND somethingelse)
+--------------------------------------------------+-----------+---------------+---------+-------------------------------------------+
| LICENSE VIOLATION | ECOSYSTEM | PACKAGE | VERSION | SOURCE |
+--------------------------------------------------+-----------+---------------+---------+-------------------------------------------+
| LGPL-2.1-only OR OR BSD-3-Clause | npm | human-signals | 5.0.0 | fixtures/locks-licenses/package-lock.json |
| MIT WITH (Bison-exception-2.2 AND somethingelse) | npm | ms | 2.1.3 | fixtures/locks-licenses/package-lock.json |
+--------------------------------------------------+-----------+---------------+---------+-------------------------------------------+

---

[TestRun_Licenses/Licenses_with_invalid_expression - 2]
license LGPL-2.1-only OR OR BSD-3-Clause for package npm/human-signals/5.0.0 is invalid: unexpected OR after OR
license MIT WITH (Bison-exception-2.2 AND somethingelse) for package npm/ms/2.1.3 is invalid: unexpected ( after WITH

---

Expand Down Expand Up @@ -1184,6 +1233,16 @@ Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 3 pac
"licenses": [
"MIT"
]
},
{
"package": {
"name": "type-fest",
"version": "4.26.1",
"ecosystem": "npm"
},
"licenses": [
"CC0-1.0 OR MIT"
]
}
]
}
Expand All @@ -1203,7 +1262,7 @@ Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 3 pac

[TestRun_Licenses/No_license_violations_and_show-all-packages_in_json - 2]
Scanning dir ./fixtures/locks-licenses/package-lock.json
Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 3 packages
Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 4 packages

---

Expand Down Expand Up @@ -1342,6 +1401,16 @@ overriding license for package Packagist/league/flysystem/1.0.8 with 0BSD
"licenses": [
"MIT"
]
},
{
"package": {
"name": "type-fest",
"version": "4.26.1",
"ecosystem": "npm"
},
"licenses": [
"CC0-1.0 OR MIT"
]
}
]
}
Expand All @@ -1360,7 +1429,7 @@ overriding license for package Packagist/league/flysystem/1.0.8 with 0BSD

[TestRun_Licenses/Some_packages_with_license_violations_and_show-all-packages_in_json - 2]
Scanning dir ./fixtures/locks-licenses/package-lock.json
Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 3 packages
Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 4 packages

---

Expand Down Expand Up @@ -1403,7 +1472,7 @@ Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 3 pac

[TestRun_Licenses/Some_packages_with_license_violations_in_json - 2]
Scanning dir ./fixtures/locks-licenses/package-lock.json
Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 3 packages
Scanned <rootdir>/fixtures/locks-licenses/package-lock.json file and found 4 packages

---

Expand Down
22 changes: 20 additions & 2 deletions cmd/osv-scanner/fixtures/locks-licenses/package-lock.json
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion cmd/osv-scanner/fixtures/locks-licenses/package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
"dependencies": {
"babel": "^6.23.0",
"human-signals": "^5.0.0",
"ms": "^2.1.3"
"ms": "^2.1.3",
"type-fest": "^4.26.1"
}
}
11 changes: 11 additions & 0 deletions cmd/osv-scanner/fixtures/osv-scanner-expressive-licenses-config.toml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[[PackageOverrides]]
name = "babel"
license.override = ["MIT AND (LGPL-2.1-or-later OR BSD-3-Clause)"]

[[PackageOverrides]]
name = "human-signals"
license.override = ["LGPL-2.1-only OR MIT OR BSD-3-Clause"]

[[PackageOverrides]]
name = "ms"
license.override = ["MIT WITH Bison-exception-2.2"]
11 changes: 11 additions & 0 deletions cmd/osv-scanner/fixtures/osv-scanner-invalid-licenses-config.toml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[[PackageOverrides]]
name = "babel"
license.override = ["MIT AND (LGPL-2.1-or-later OR BSD-3-Clause))"]

[[PackageOverrides]]
name = "human-signals"
license.override = ["LGPL-2.1-only OR OR BSD-3-Clause"]

[[PackageOverrides]]
name = "ms"
license.override = ["MIT WITH (Bison-exception-2.2 AND somethingelse)"]
10 changes: 10 additions & 0 deletions cmd/osv-scanner/main_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -718,6 +718,16 @@ func TestRun_Licenses(t *testing.T) {
args: []string{"", "--format=json", "--experimental-licenses-summary", "./fixtures/locks-licenses/package-lock.json"},
exit: 0,
},
{
name: "Licenses with expressions",
args: []string{"", "--config=./fixtures/osv-scanner-expressive-licenses-config.toml", "--experimental-licenses", "MIT,BSD-3-Clause", "./fixtures/locks-licenses/package-lock.json"},
exit: 1,
},
{
name: "Licenses with invalid expression",
args: []string{"", "--config=./fixtures/osv-scanner-invalid-licenses-config.toml", "--experimental-licenses", "MIT,BSD-3-Clause", "./fixtures/locks-licenses/package-lock.json"},
exit: 1,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
Expand Down
Loading

0 comments on commit 73fe113

Please sign in to comment.