v31.0.0rc2

This is a release candidate for the upcoming 31 release.

v31 is a major release with many new features, and several bug fixes and
improvements including major updates to the package and dependency collection and to the license detection.

Several bugs have been fixed when compared with 31.0.0rc1.

See https://github.com/nexB/scancode-toolkit/blob/v31.0.0rc2/CHANGELOG.rst for an overview of the changes in v31 compared to v30.
Please try this release and report any installation issues so we can work towards a stable 31.
Thank you!

What's Changed

• Improve npm package processing by @pombredanne in #2997
• Update license detection by @pombredanne in #2998
• Add new license rules and license - Early summer 2022 by @pombredanne in #2999
• Bump version to 31.0.0rc2 by @JonoYang in #3000

Full Changelog: v31.0.0rc1...v31.0.0rc2

v31.0.0rc1

This is a release candidate for the upcoming 31 release.

v31 is a major release with many new features, and several bug fixes and
improvements including major updates to the package and dependency collection and to the license detection.

Several bugs have been fixed when compared with 31.0.0b5.

See https://github.com/nexB/scancode-toolkit/blob/v31.0.0rc1/CHANGELOG.rst for an overview of the changes in v31 compared to v30.
Please try this release and report any installation issues so we can work towards a stable 31.
Thank you!

What's Changed

• Add black and isort as testing dependencies #2969 by @johnmhoran in #2970
• Rename precise_license_detection field #2967 by @JonoYang in #2968
• Convert package data dict to PackageData #2971 by @JonoYang in #2973
• Update extractcode --shallow option description by @lf32 in #2959
• Support shortcut flags for cli by @lf32 in #2951
• Consider only copyrights in summry #2972 by @JonoYang in #2974
• Reimplement get installed packages by @JonoYang in #2988
• Report extracted_requirement correctly by @TG1999 in #2984
• Improve packagecode and other release prep by @pombredanne in #2992

New Contributors

• @lf32 made their first contribution in #2959

Full Changelog: v31.0.0b5...v31.0.0rc1

v31.0.0b5

This is a beta release for the upcoming 31 release.

v31 is a major release with many new features, and several bug fixes and
improvements including major updates to the package and dependency collection and to the license detection.

Several bugs have been fixed when compared by b4.

See https://github.com/nexB/scancode-toolkit/blob/v31.0.0b5/CHANGELOG.rst for an overview of the changes in v31 compared to v30.
Please try this release and report any installation issues so we can work towards a stable 31.
Thank you!

What's Changed

• Add link to scancode-toolkit-reference-scans by @AyanSinhaMahapatra in #2952
• Modify pypi PKG-INFO parse by @AyanSinhaMahapatra in #2953
• Prepare Release 31.b5 by @pombredanne in #2962

Full Changelog: v31.0.0b4...v31.0.0b5

v31.0.0b4

This is a beta release for the upcoming 31 release.

v31 is a major release with many new features, and several bug fixes and
improvements including major updates to the package and dependency collection and to the license detection.

Several bugs have been fixed when compared by b3.

See https://github.com/nexB/scancode-toolkit/blob/v31.0.0b4/CHANGELOG.rst for an overview of the changes.
Please try this release and report any installation issues so we can work towards a stable 31.
Thank you!

What's Changed

• Populate for packages field correctly #2929 by @JonoYang in #2939
• Prepare Release 31b4 by @pombredanne in #2941
• Duplicated dependencies package results by @JonoYang in #2944
• Prepare Release 31b4 by @pombredanne in #2947

Full Changelog: v31.0.0b3...v31.0.0b4

v31.0.0b3 - 2022-04-30

This is a beta release for the upcoming 31 release.

v31 is a major release with many new features, and several bug fixes and
improvements including major updates to the package and dependency collection and to the license detection.

See https://github.com/nexB/scancode-toolkit/blob/v31.0.0b3/CHANGELOG.rst for an overview of the changes.
Please try this release and report any installation issues so we can work towards a stable 31.
Thank you!

What's Changed

• Report packages at top level with file level package_manifests by @AyanSinhaMahapatra in #2710
• Updated install.rst by @beastrun12j in #2722
• Omnibus fall license improvements by @pombredanne in #2706
• Improve license detection by @pombredanne in #2737
• api.get_licenses: clarify and improve docstring for "min_score" argument by @zacchiro in #2763
• rules with "unqualified" license names are references, not notices by @petergardfjall in #2759
• Fix invalid license yaml files by resolving duplicated keys by @fangxlmr in #2776
• Fix azure pipeline vmimage deprecations by @AyanSinhaMahapatra in #2775
• Allow license rules to require the presence of certain defining keywords by @mrombout in #2773
• Add first draft ROADMAP by @pombredanne in #2736
• Add CycloneDx output option by @agschrei in #2698
• Remove regular expression futurewarning by @soimkim in #2788
• fix docstring in debian_copyright.py by @adii21-Ux in #2786
• fixes missing whitespace in prerequisites list by @altsalt in #2778
• Add PackageManifest Class by @AyanSinhaMahapatra in #2748
• Add new licenses and new detection rules by @pombredanne in #2765
• Rename first column of csv output to "path" by @JRavi2 in #2016
• Detect unknown licenses #1675 by @akugarg in #2592
• Improve copyright handling #2350 by @pombredanne in #2791
• Fixing OSI identifier for BSD-3-Clause; see also SPDX license metadata by @karsten-klein in #2797
• Fix GPL license detection false positive #2793 by @KevinJi22 in #2799
• 2789 inconsistent doc html app by @kunalchhabra37 in #2795
• Fixed inconsistency in --html-app FILE in cli-reference by @maynaS in #2790
• Replace freenode references with libera chat by @purna135 in #2816
• Adopt nexB/skeleton and bump dependencies by @pombredanne in #2818
• Fix bug recognizing license as license_notice instead of license_text by @adii21-Ux in #2817
• Fix incorrect license detection #2777 by @KevinJi22 in #2811
• Remove skeleton from docs by @AyanSinhaMahapatra in #2830
• Detect SPDX-FileContributor tags as authors by @pombredanne in #2838
• New license and copyright rule by @adii21-Ux in #2837
• Add key phrase tags to GPL detection rule by @pombredanne in #2821
• Make --version output valid YAML for parsing #2856 by @KevinJi22 in #2858
• Add Direct Note for Windows Users (New Comers) by @OsmiumOP in #2857
• Fixed Typo in Documentation by @OsmiumOP in #2862
• Remove version check locally by @adii21-Ux in #2860
• License improvement winter 2022 by @pombredanne in #2828
• Update link to documentation by @AyanSinhaMahapatra in #2867
• Improve license detection by @pombredanne in #2871
• Detect dependencies from build.gradle files by @pombredanne in #2822
• Fix small typo inside notes snippet by @Harshil-Jani in #2829
• Add Package Instances #2691 by @AyanSinhaMahapatra in #2825
• Improve license clarity scoring by @pombredanne in #2875
• Do not raise exception on package data mismatch #2886 by @AyanSinhaMahapatra in #2887
• Release 31 by @pombredanne in #2888
• Add primary license in summary by @JonoYang in #2884
• Remove usage of get_terminal_size in click by @AyanSinhaMahapatra in #2916
• Fix doc builds by @AyanSinhaMahapatra in #2896
• Update summary plugin by @JonoYang in #2914
• Shorten long file names by @pombredanne in #2918
• Added new copyright test cases by @abhishak3 in #2891
• Add system packages support in the new packages model by @AyanSinhaMahapatra in #2909
• Fix typo in summary: ambigous->ambiguous by @pombredanne in #2922
• Add system environment to scan headers by @pombredanne in #2923
• Update METADATA.bzl parser by @JonoYang in #2924
• Spring 2022 license updates by @pombredanne in #2921
• Process single package data file correctly by @pombredanne in #2933
• Fix package/dependency creation bugs by @AyanSinhaMahapatra in #2932

New Contributors

• @beastrun12j made their first contribution in #2722
• @zacchiro made their first contribution in #2763
• @fangxlmr made their first contribution in #2776
• @mrombout made their first contribution in #2773
• @agschrei made their first contribution in #2698
• @soimkim made their first contribution in #2788
• @adii21-Ux made their first contribution in #2786
• @altsalt made their first contribution in #2778
• @karsten-klein made their first contribution in #2797
• @KevinJi22 made their first contribution in #2799
• @kunalchhabra37 made their first contribution in #2795
• @maynaS made their first contribution in #2790
• @purna135 made their first contribution in #2816
• @OsmiumOP made their first contribution in #2857
• @Harshil-Jani made their first contribution in #2829
• @abhishak3 made their first contribution in #2891

Full Changelog: v30.1.0...v31.0.0b3

v30.1.0 - 2021-09-25

This is a bug fix release for these bugs:

• #2717

We now return the package in the summaries as before.

There is also a minor API change: we no longer return a count of "null" empty
values in the summaries for license, copyrights, etc.

Thank you to:

• Thomas Druez @tdruez

See also https://github.com/nexB/scancode-toolkit/tree/v30.0.0 for details on the main changes in v30.0.x

What's Changed

• Prepare bugfix release 30.0.1 #2713 by @pombredanne in #2715
• Return package details in summary #2717 by @pombredanne in #2718

Full Changelog: v30.0.1...v30.1.0

v30.0.1 - 2021-09-24

This is a minor bug fix release for these bugs:

• #2713
• #2713

We now correctly work with all supported Click versions.

Thank you to:

• Konstantin Kochin @vznncv
• Thomas Druez @tdruez

See also https://github.com/nexB/scancode-toolkit/tree/v30.0.0 for details on the main changes in v30.0.x

Full Changelog: v30.0.0...v30.0.1

v30.0.0 - 2021-09-23

This is a major release with new features, and several bug fixes and
improvements including major updates to the license detection.

We have dropped using calendar-based versions and are now switched back to semver
versioning. To ensure that there is no ambiguity, the new major version has been
updated from 21 to 30. The primary reason is that calver was not helping
integrators to track major version changes like semver does.

We also have introduced a new JSON output format version based on semver to
version the JSON output format data structure and have documented the new
versioning approach.

Here are the key changes for each area:

Package detection:

• The Debian packages declared license detection in machine readable copyright
  files and unstructured copyright has been significantly improved with the
  tracking of the detection start and end line of a license match. This is not
  yet exposed outside of tests but has been essential to help improve detection.

• Debian copyright license detection has been significantly improved with new
  license detection rules.

• Support for Windows packages has been improved (and in particular the handling
  of Windows packages detection in the Windows registry).

• Support for Cocoapod packages has been significantly revamped and is now
  working as expected.

• Support for PyPI packages has been refined, in particular package descriptions.

Copyright detection:

• The copyright detection accuracy has been improved and several bugs have been
  fixed.

License detection:

There have been some significant updates in license detection. We now track
34,164 license and license notices:

• 84 new licenses have been added,

• 34 existing license metadata have been updated,

• 2765 new license detection rules have been added, and

• 2041 existing license rules have been updated.

• Several license detection bugs have fixed.

• The SPDX license list 3.14 is now supported and has been synced with the
  licensedb. We also include the version of the SPDX license list in the
  ScanCode YAML, JSON and the SPDX outputs, as well as display it with the
  "--version" command line option.

• Unknown licenses have a new flag "is_unknown" in their metadata to identify
  them explicitly. Before that we were just relying on the naming convention of
  having "unknown" as part of a license key.

• Rules that match at least one unknown license have a flag "has_unknown" set
  and returned in the match results.

• Experimental: License detection can now "follow" license mentions that
  reference another file such as "see license in COPYING" where we can relate
  this mention to the actual license detected in the COPYING file. Use the new
  "--unknown-licenses" command line option to test this new feature.
  This feature will evolve significantly in the next version(s).

Outputs:

• The SPDX output now has the mandatory ids attribute per SPDX spec. And we
  support SPDX 2.2 and SPDX license list 3.14.

Miscellaneous

• There is a new "--no-check-version" CLI option to scancode to bypass live,
  remote outdated version check on PyPI

• The scan results and the CLI now display an outdated version warning when
  the installed ScanCode version is older than 90 days. This is to warn users
  that they are relying on outdated, likely buggy, insecure and inaccurate scan
  results and encourage them to update to a newer version. This is made entirely
  locally based on date comparisons.

• We now display again the command line progressbar counters correctly.

• A bug has been fixed in summarization.

• Generated code detection has been improved with several new keywords.

Thank you!

Many thanks to the many contributors that made this release possible and in
particular:

• Akanksha Garg @akugarg
• Armijn Hemel @armijnhemel
• Ayan Sinha Mahapatra @AyanSinhaMahapatra
• Bryan Sutula @sutula
• Chin-Yeung Li @chinyeungli
• Dennis Clark @DennisClark
• dyh @yunhua-deng
• Dr. Frank Heimes @FrankHeimes
• gunaztar @gunaztar
• Helio Chissini de Castro @heliocastro
• Henrik Sandklef @hesa
• Jiyeong Seok @dd-jy
• John M. Horan @johnmhoran
• Jono Yang @JonoYang
• Joseph Heck @heckj
• Luis Villa @tieguy
• Konrad Weihmann @priv-kweihmann
• mapelpapel @mapelpapel
• Maximilian Huber @maxhbr
• Michael Herzog @mjherzog
• MMarwedel @MMarwedel
• Mikko Murto @mmurto
• Nishchith Shetty @inishchith
• Peter Gardfjäll @petergardfjall
• Philippe Ombredanne @pombredanne
• Rainer Bieniek @rbieniek
• Roshan Thomas @Thomshan
• Sadhana @s4-2
• Sarita Singh @itssingh
• Sebastian Schuberth @sschuberth
• Siddhant Khare @Siddhant-K-code
• Soim Kim @soimkim
• Thorsten Godau @tgodau
• Yunus Rahbar @yns88

What's Changed

• Collect InstalledWindowsProgram installed files #2615 by @JonoYang in #2623
• Improve release creation speed by @pombredanne in #2627
• Omnibus license updates July/Aug 21 by @pombredanne in #2626
• Add new flag in License Data Model definition by @akugarg in #2548
• Update Contributing: Development setup-instructions by @mapelpapel in #2631
• Referenced_filenames should be returned by API function by @akugarg in #2632
• Add emails and urls to HTML output by @sritasngh in #2539
• Avoid misinterpreting MIT license notice as Apache-2.0, issue #2635 by @petergardfjall in #2636
• Add final report for GSoC'21 by @akugarg in #2648
• Add "--no-check-version" CLI option to scancode by @yns88 in #2662
• Align tests for pubspec with latest code by @pombredanne in #2628
• Add new licenses by @akugarg in #2625
• Add podspec.json and podfile.lock parsers by @AyanSinhaMahapatra in #2638
• Add new license Anti-Capitalist Software License #2362 by @sritasngh in #2364
• Do not mistake path for copyright year by @pombredanne in #2666
• Follow license reference to another file by @akugarg in #2616
• Bump commoncode #2583 by @pombredanne in #2676
• Detect only mit license, not boost #2675 by @pombredanne in #2678
• Detect ocb correctly license #2670 by @pombredanne in #2677
• Improve license referenced_filenames handling #1364 by @pombredanne in #2681
• Add script to report rules by @AyanSinhaMahapatra in #2685
• Update Azure CI to not use ubuntu-16.04 images by @AyanSinhaMahapatra in #2688
• Introduce output data format versioning #2653 by @AyanSinhaMahapatra in #2682
• Release preparation for 2021.08 by @pombredanne in #2680
• Improve license detection accuracy by @pombredanne in #2667
• Improve copyright detection by @pombredanne in #2701
• Add CI for Docs and ABOUT files by @AyanSinhaMahapatra in #2695
• Adopt SPDX v2.2 and fix SPDX TV correctness by @pombredanne in #2704
• Improve Copyright detection by @pombredanne in #2707
• Prepare new release by @pombredanne in #2705

New Contributors

• @mapelpapel made their first contribution in #2631
• @yns88 made their first contribution in #2662

Full Changelog: v21.8.4...v30.0.0

v21.8.4

This is a minor bug fix release primarily for Windows installation.
There is no feature change.

Installation:

• Application installation on Windows works again. This fixes #2610
• We now build and test app bundles on all supported Python versions: 3.6 to 3.9

Thank you to @gunaztar for reporting the #2610 bug

Documentation:

• Documentation is updated to reference supported Python versions 3.6 to 3.9

v21.7.30

This is a minor release with several bug fixes, major performance improvements
and support for new and improved package formats

WARNING: the app installatiom does not work on Windows. Everything else is fine. Use the next version v21.8.1 and later. See #2610

Many thanks to every contributors that made this possible and in particular:

• Abhigya Verma @abhi27-web
• Ayan Sinha Mahapatra @AyanSinhaMahapatra
• Dennis Clark @DennisClark
• Jono Yang @JonoYang
• Mayur Agarwal @mrmayurgithub
• Philippe Ombredanne @pombredanne
• Pierre Tardy @tardyp

Key changes:

Outputs:

• Add new YAML-formatted output. This is exactly the same data structure as for
  the JSON output
• Add new Debian machine readable copyright output.
• The CSV output "Resource" column has been renamed to "path".
• The SPDX output now has the mandatory DocumentNamespace attribute per SPDX specs #2344

Copyright detection:

• The copyright detection speed has been significantly improved with the tests
  taking roughly 1/2 of the time to run. This is achieved mostly by replacing
  NLTK with a the minimal and simplified subset we need in a new library named
  pygmars.

License detection:

• Add new licenses: now tracking 1763 licenses
• Add new license detection rules: now tracking 29475 license detection rules
• We have also improved license expression parsing and processing

Package detection:

• The Debian packages declared license detection has been significantly improved.
• The Alpine packages declared license detection has been significantly improved.
• There is new support for shell parsing and Alpine packages APKBUILD data collection.
• There is new support for various Windows packages detection using multiple
  techniques including MSI, Windows registry and several more.
• There is new support for Distroless Debian-like installed packages.
• There is new support for Dart Pub package manifests.