Update content to use OSCAL v1.1.1 models #204

aj-stein-nist · 2023-08-03T20:32:25Z

Committer Notes

This updates oscal-content make-based orchestration and touches up catalogs for #116. Guidance on these changes, future development, and future governance are further detailed in usnistgov/OSCAL#1947.

Example content with the updated tooling work and simulating this branch being merged can be found on my work (the last commit changing the HOME_REPO variable to work on this branch has been dropped, but was used to complete this final testing).

aj-stein-nist/oscal-content@c8a5374#diff-ecf37eddbbe732cfc8cb391c407ae4ad8f00500d1590cc65bb08569c80697ff2

All Submissions:

Have you followed the guidelines in our Contributing document?
Have you checked to ensure there aren't other open Pull Requests for the same update/change?
Have you squashed any non-relevant commits and commit messages? [instructions] (NOTE: Will do after review to aid review process and properly document many changes, big and small, potentially will not squash.) In this case, I believe it to be important that we keep all the commits for team understanding of significant changes in the repo. I will not squash these.
Do all automated CI/CD checks pass?

Changes to Core Features:

~~Have you added an explanation of what your changes do and why you'd like us to include them?~~
~~Have you written new tests for your core changes, as applicable?~~
~~Have you included examples of how to use your new feature(s)?~~

iMichaela · 2023-08-03T21:23:08Z

@aj-stein-nist Is the title of the PR wrong? It lists OSCAL 1.0.0

aj-stein-nist · 2023-08-03T21:37:44Z

@aj-stein-nist Is the title of the PR wrong? It lists OSCAL 1.0.0

Woops, fixed. :-)

Add this back as part of sprint work to build new Makefile(s) and recreate profile resolution tooling for local and GHA CI/CD for usnistgov/oscal-content#204.

This was removed in site migration work and did not appear to be relocated, as determined during the course of usnistgov/oscal-content#204. We can rely on the metaschema-xslt project, but given OSCAL has the profile resolver tool committed in its repo tree, it stands to reason to rely on its own independent Maven POM file, even if now redundant, as they may diverge in the future.

* Correct relative path to previous Maven POM dep file Add this back as part of sprint work to build new Makefile(s) and recreate profile resolution tooling for local and GHA CI/CD for usnistgov/oscal-content#204. * Add back pom.xml for Maven deps This was removed in site migration work and did not appear to be relocated, as determined during the course of usnistgov/oscal-content#204. We can rely on the metaschema-xslt project, but given OSCAL has the profile resolver tool committed in its repo tree, it stands to reason to rely on its own independent Maven POM file, even if now redundant, as they may diverge in the future.

This script is being added to facilitate running XSLT stylesheets, such as the XML/JSON JSON/XML conversion utilities that will be used in our oscal-content and related pipelines. This script will be committed to coincide with testing and review of the Makefile in usnistgov/oscal-content#204.

aj-stein-nist · 2023-08-14T17:53:46Z

This is important and still needs to be finished. I will continue work on this in the first half of Sprint 74 and intend to finish it as it is an important prerequisite for many upcoming changes given the tentative roadmap in usnistgov/OSCAL#1867.

* Correct relative path to previous Maven POM dep file Add this back as part of sprint work to build new Makefile(s) and recreate profile resolution tooling for local and GHA CI/CD for usnistgov/oscal-content#204. * Add back pom.xml for Maven deps This was removed in site migration work and did not appear to be relocated, as determined during the course of usnistgov/oscal-content#204. We can rely on the metaschema-xslt project, but given OSCAL has the profile resolver tool committed in its repo tree, it stands to reason to rely on its own independent Maven POM file, even if now redundant, as they may diverge in the future.

This script is being added to facilitate running XSLT stylesheets, such as the XML/JSON JSON/XML conversion utilities that will be used in our oscal-content and related pipelines. This script will be committed to coincide with testing and review of the Makefile in usnistgov/oscal-content#204.

* Correct relative path to previous Maven POM dep file Add this back as part of sprint work to build new Makefile(s) and recreate profile resolution tooling for local and GHA CI/CD for usnistgov/oscal-content#204. * Add back pom.xml for Maven deps This was removed in site migration work and did not appear to be relocated, as determined during the course of usnistgov/oscal-content#204. We can rely on the metaschema-xslt project, but given OSCAL has the profile resolver tool committed in its repo tree, it stands to reason to rely on its own independent Maven POM file, even if now redundant, as they may diverge in the future.

This script is being added to facilitate running XSLT stylesheets, such as the XML/JSON JSON/XML conversion utilities that will be used in our oscal-content and related pipelines. This script will be committed to coincide with testing and review of the Makefile in usnistgov/oscal-content#204.

As picked up in review with Nikita, we are converting the source XML files to JSON and YAML, but not all. We need to adjust the make vars that collect the source files for conversion to minified JSON to Include the resolved catalogs from profiles from the previous step. Reviewing the auto-committed test content confirm these files were being missed in conversion to JSON and YAML and we obviously need those.

Per discussion with Nikita during PR review and pairing, we should try to adjust the jq and yq dep install and management approach to be a little more ergonomic. As for xmllint, it was requested we align that to work like the dep mgmt in the usnistgov/OSCAL repo, since that is a compiled dep that is not offered statically and is best handled as a concern by the developer in their operating system, or by a dedicated install step in GHA workflows. Additionally, for these reasons, the clean wrapper target will not include deleting it as a default, but can be added later by a developer by explicitly requesting it.

Per PR review and consultation from Nikita, add in additional wrapper targets for artifacts and checks for better organization and to make a cleaner hierarchy for the make all target.

As discussed in the branch under review in #204, we now have absolute paths in the resolved catalogs based on the full paths to the profile, which is less than idea for published catalogs. Comment with explanation: #204 (comment) Example: https://github.com/aj-stein-nist/oscal-content/blob/dc5500ed9371b485fc21cb095d2fb9db6e2a1fd3/nist.gov/SP800-53/rev5/xml/NIST_SP-800-53_rev5_LOW-baseline-resolved-profile_catalog.xml#L11 To work around this for the time being, we are using sed in the Makefile to clean up once profile resolution is done and before the XML source is converted to the target JSON and YAML.

@d

Per PR review with Nikita, cite documentation for `make` and `Makefile` on how `$(@d)` is an alias to show to the the target wildcard file without the filename, useful for creating nested directories of content before copying.

@nikitawootten-nist

After debugging with @nikitawootten-nist, we determined during pairing that the SECONDEXPANSION calls, that are necessary to avoid some more hacky kludges with macros and functions. As it stands, when running with concurrent jobs (`-jN` with N>=2) it fails. We determined the cause of the failures for `convert-min-json-content` and `convert-yaml-content` targets are because those target prerequisites are flawed. This change will be greedy and presume all respective XML files for JSON targets and all JSON files for YAML targets. This will limit the upper bound of concurrent jobs for these targets, but makes it correct and safe.

aj-stein-nist · 2023-10-20T21:57:41Z

Doing some more testing but the concurrency fix as I committed is still not working perfectly.

https://github.com/aj-stein-nist/oscal-content/actions/runs/6592140253/job/17912204423

Catching up on Makefile docs and tutorials while debugging locally and GHA to figure out if I did it in the wrong target location or this is not sufficient as discussed during pairing. 😭

aj-stein-nist · 2023-10-24T14:25:08Z

It seems with c881404 we are very close (thanks to @nikitawootten-nist for #219) but I still see a regression in the sed path transformation that is not being handled in the latest testing on my fork as of aj-stein-nist/oscal-content@7392f08. Once that is wrapped up, this is ready.

* Rearranged targets, fixed build race conditions * Added a check for MacOS/Linux semantics

aj-stein-nist · 2023-10-24T18:21:41Z

I removed the testing commit to change the HOME_REPO, but this is ready for final review. Sample from my fork simiulating the merge to main is in aj-stein-nist/oscal-content@a45245e.

/cc @usnistgov/itl-oscal

nikitawootten-nist

All works, all approved. ~~Small nitpick below but it can be safely ignored~~

Edit: I deleted the nit as it may cause confusion down the line if oscal content includes resolved profiles with multiple inputs.

As discussed in the branch under review in #204, we now have absolute paths in the resolved catalogs based on the full paths to the profile, which is less than idea for published catalogs. Comment with explanation: #204 (comment) Example: https://github.com/aj-stein-nist/oscal-content/blob/dc5500ed9371b485fc21cb095d2fb9db6e2a1fd3/nist.gov/SP800-53/rev5/xml/NIST_SP-800-53_rev5_LOW-baseline-resolved-profile_catalog.xml#L11 To work around this for the time being, we are using sed in the Makefile to clean up once profile resolution is done and before the XML source is converted to the target JSON and YAML.

As discussed in the branch under review in usnistgov#204, we now have absolute paths in the resolved catalogs based on the full paths to the profile, which is less than idea for published catalogs. Comment with explanation: usnistgov#204 (comment) Example: https://github.com/aj-stein-nist/oscal-content/blob/dc5500ed9371b485fc21cb095d2fb9db6e2a1fd3/nist.gov/SP800-53/rev5/xml/NIST_SP-800-53_rev5_LOW-baseline-resolved-profile_catalog.xml#L11 To work around this for the time being, we are using sed in the Makefile to clean up once profile resolution is done and before the XML source is converted to the target JSON and YAML.

aj-stein-nist self-assigned this Aug 3, 2023

aj-stein-nist mentioned this pull request Aug 3, 2023

Update content with OSCAL 1.1.0 models #200

Closed

7 tasks

aj-stein-nist changed the title ~~Update content to use OSCAL v1.0.0 models~~ Update content to use OSCAL v1.1.0 models Aug 3, 2023

aj-stein-nist mentioned this pull request Aug 6, 2023

Script and dependency fixes for v1.1.0 content updates usnistgov/OSCAL#1874

Merged

6 tasks

aj-stein-nist mentioned this pull request Aug 7, 2023

Add XSLT runner utility usnistgov/OSCAL#1879

Merged

6 tasks

This was referenced Aug 14, 2023

Remove Example Component Definition with Obsolete Configuration #205

Closed

Updated MongoDB component definition example #82

Merged

aj-stein-nist linked an issue Aug 24, 2023 that may be closed by this pull request

Upgrade GitHub CI/CD Workflows to Current Stable Version #116

Closed

6 tasks

aj-stein-nist mentioned this pull request Aug 25, 2023

Jk/issue 10 #208

Merged

5 tasks

aj-stein-nist force-pushed the chore-submodule-v1.1.0-release branch 2 times, most recently from 65c7bc2 to 56a8e5e Compare September 8, 2023 23:23

aj-stein-nist linked an issue Sep 21, 2023 that may be closed by this pull request

Published OSCAL JSON Content Isn't Minified as Intended #119

Closed

aj-stein-nist force-pushed the chore-submodule-v1.1.0-release branch 4 times, most recently from 3b14a16 to 74c7398 Compare October 6, 2023 22:02

aj-stein-nist changed the title ~~Update content to use OSCAL v1.1.0 models~~ Update content to use OSCAL v1.1.1 models Oct 6, 2023

aj-stein-nist requested a review from a team October 6, 2023 22:04

aj-stein-nist added 11 commits October 20, 2023 15:58

Fix clean targets and reorganize Makefile for #116.

c8aebe4

Update top-level README and include it in archive for #116.

4108ccf

Add build instructions README for #116.

7c460cd

Do not multi-line all and clean targets for #116.

7fdb10c

Update targets and docs for #116

7a657e6

Per PR review and consultation from Nikita, add in additional wrapper targets for artifacts and checks for better organization and to make a cleaner hierarchy for the make all target.

Explain $(@d) make alias usage in #116

f0b1bd3

Per PR review with Nikita, cite documentation for `make` and `Makefile` on how `$(@d)` is an alias to show to the the target wildcard file without the filename, useful for creating nested directories of content before copying.

Update per @nikitawootten-nist's review feedback.

fb1b771

aj-stein-nist force-pushed the chore-submodule-v1.1.0-release branch from 99669b4 to bbdcac7 Compare October 20, 2023 19:59

aj-stein-nist force-pushed the chore-submodule-v1.1.0-release branch from 65292a1 to 5a07896 Compare October 24, 2023 01:31

Rearranged targets, fixed build race conditions (#219)

e96e2b5

* Rearranged targets, fixed build race conditions * Added a check for MacOS/Linux semantics

aj-stein-nist force-pushed the chore-submodule-v1.1.0-release branch 2 times, most recently from f95c749 to e96e2b5 Compare October 24, 2023 18:21

aj-stein-nist requested a review from nikitawootten-nist October 24, 2023 18:21

nikitawootten-nist approved these changes Oct 24, 2023

View reviewed changes

aj-stein-nist merged commit 1cad230 into develop Oct 24, 2023

aj-stein-nist deleted the chore-submodule-v1.1.0-release branch October 24, 2023 19:13

This was referenced Oct 24, 2023

Patch PR for cleanups / enhancements in SP800-53 OSCAL including 'implemented by' and 'assurance' props #174

Closed

Published OSCAL JSON Content Isn't Minified as Intended #119

Closed

github-actions bot mentioned this pull request Dec 21, 2023

Scheduled Check of Markdown Documents Found Bad Hyperlinks usnistgov/OSCAL#1971

Closed

github-actions bot mentioned this pull request Feb 15, 2024

Scheduled Check of Markdown Documents Found Bad Hyperlinks usnistgov/OSCAL#1981

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update content to use OSCAL v1.1.1 models #204

Update content to use OSCAL v1.1.1 models #204

aj-stein-nist commented Aug 3, 2023 •

edited

Loading

iMichaela commented Aug 3, 2023

aj-stein-nist commented Aug 3, 2023

aj-stein-nist commented Aug 14, 2023

aj-stein-nist commented Oct 20, 2023

aj-stein-nist commented Oct 24, 2023

aj-stein-nist commented Oct 24, 2023 •

edited

Loading

nikitawootten-nist left a comment •

edited

Loading

Update content to use OSCAL v1.1.1 models #204

Update content to use OSCAL v1.1.1 models #204

Conversation

aj-stein-nist commented Aug 3, 2023 • edited Loading

Committer Notes

All Submissions:

Changes to Core Features:

iMichaela commented Aug 3, 2023

aj-stein-nist commented Aug 3, 2023

aj-stein-nist commented Aug 14, 2023

aj-stein-nist commented Oct 20, 2023

aj-stein-nist commented Oct 24, 2023

aj-stein-nist commented Oct 24, 2023 • edited Loading

nikitawootten-nist left a comment • edited Loading

Choose a reason for hiding this comment

aj-stein-nist commented Aug 3, 2023 •

edited

Loading

aj-stein-nist commented Oct 24, 2023 •

edited

Loading

nikitawootten-nist left a comment •

edited

Loading