Improve --cyclonedx output option

[DennisClark]

Running scancode-toolkit-31.0.0b5 on MacOS 11.6.6
The --json-pp output option works fine, but
the --cyclonedx output option fails.

scancode-toolkit-31.0.0b5 % ./scancode --license --copyright --cyclonedx scancode_results_cyclonedx.json samples
Setup plugins...
Collect file inventory...
Scan files for: licenses, copyrights with 1 process(es)...
[####################] 66
ERROR: failed to run output plugin: cyclonedx:
Traceback (most recent call last):
File "//scancode-toolkit-31.0.0b5/src/scancode/cli.py", line 1067, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "//scancode-toolkit-31.0.0b5/src/formattedcode/output_cyclonedx.py", line 735, in process_codebase
bom = CycloneDxBom.from_codebase(codebase)
File "//scancode-toolkit-31.0.0b5/src/formattedcode/output_cyclonedx.py", line 633, in from_codebase
packages = codebase.attributes.packages
AttributeError: 'CodebaseAttributes' object has no attribute 'packages'

Scanning done.
Summary: licenses, copyrights with 1 process(es)
Errors count: 0
Scan Speed: 1.12 files/sec.
Initial counts: 44 resource(s): 33 file(s) and 11 directorie(s)
Final counts: 44 resource(s): 33 file(s) and 11 directorie(s)
Timings:
scan_start: 2022-06-08T180404.780531
scan_end: 2022-06-08T180436.257832
setup_scan:licenses: 2.00s
setup: 2.00s
scan: 29.42s
total: 31.49s
Removing temporary files...done.

[pombredanne]

@DennisClark can you try with "--package" ? It should not crash anyway.

[DennisClark]

Running it with
./scancode --license --copyright --package --cyclonedx scancode_results_cyclonedx.json samples
works without failing. About to review the results.

[DennisClark]

I got beyond the crash by specifying the --package option. Now there is a rather different problem which might be called "lost in translation". The samples provided for testing with sctk did not produce really interesting results, so I extracted a folder from `libX11-1.7.2.tar.bz2' and scanned that. The results using the --json-pp output option are extensive and detailed, but running the same scan with the --cyclonedx output option produces a very limited file with hardly any content. Associated files attached. Here are my commands:

./scancode --license --copyright --package --cyclonedx libX11-1.7.2-scan-cyclonedx.json /Users/me/Downloads/libX11-1.7.2
and
./scancode --license --copyright --package --json-pp libX11-1.7.2-scan-json-pp.json /Users/me/Downloads/libX11-1.7.2

libX11-1.7.2.tar.bz2.zip

libX11-1.7.2-scan-cyclonedx.json.zip

libX11-1.7.2-scan-json-pp.json.zip

[pombredanne]

@DennisClark Thanks... that's super useful to track this issue.

[AyanSinhaMahapatra]

@DennisClark The above PR addresses the crash.

1. We do not crash anymore
2. We show the following CLI warning:

CycloneDxPluginNoPackagesWarning: The --cyclonedx-xml option will not output any component/dependency data as there are no package data in the present scan. To get package data please rerun the scan with --package or --system-package CLI options enabled.

3. We also add the same message to the BOM metadata in warnings.

Please review the warning text also btw.

As for the did not produce really interesting results and produces a very limited file with hardly any content part of the issue, we need some more work and investigation. So let's keep this issue open.

[DennisClark]

@AyanSinhaMahapatra The warning text provided in your comment is good and very informative, thanks. No changes suggested.

[DennisClark]

a helpful link here:
https://cyclonedx.org/schema/bom-1.4.schema.json

[DennisClark]

another helpful link:
https://cyclonedx.org/use-cases/

[pombredanne]

The bug is fixed and we are keeping this open to ensure we can report more useful data.

[AyanSinhaMahapatra]

From @pombredanne at this comment:

please add an issue for follow up work to:

• support newer/newest versions of the spec
• add support for dependencies and more details

[AyanSinhaMahapatra]

See a sample output at #3016 (comment) for reference

[eastmadc]

CycloneDX v1.5 was released last month. @pombredanne do you think it would be worthwhile to add this support? Would you maintain legacy version support as well?
https://cyclonedx.org/news/cyclonedx-v1.5-released/

[pombredanne]

@eastmadc sure! I think the work will consist in migrating to this library https://gitlab.com/hoppr/hoppr-cyclonedx-models/-/tree/main/hoppr_cyclonedx_models that we already use in ScanCode.io and already has support for CycloneDX 1.5 (@jhlmco and team Thank you BTW.. ! )

Would you maintain legacy version support as well?

I think we should.

[mjherzog]

We need to add output options for CDX v1.4, 1.5 and v1.6