DXCDT-388: Enabling preservation of array replace markers in YAML #760
Conversation
…T-375-rename-load-to-loadAssetsFromAuth0
…ort' into fix-make-keyword-preservation-more-flexible
…to DXCDT-382-standardize-resource-identifiers
| @@ -86,7 +86,8 @@ async function dump(context: YAMLContext): Promise<ParsedActions> { | |||
| runtime: action.runtime, | |||
| dependencies: action.dependencies || [], | |||
| status: action.status, | |||
| secrets: mapSecrets(action.secrets || []), | |||
| secrets: | |||
| typeof action.secrets === 'string' ? action.secrets : mapSecrets(action.secrets || []), //Enables keyword preservation to operate on action secrets | |||
There was a problem hiding this comment.
Most array preservation will work as-is for all YAML handlers, but there is this one instance where a string value for action.secrets would trigger a runtime error when applying a map function.
src/context/yaml/index.ts
Outdated
| @@ -104,7 +109,7 @@ export default class YAMLContext { | |||
|
|
|||
| // Run initial schema check to ensure valid YAML | |||
| const auth0 = new Auth0(this.mgmtClient, this.assets, toConfigFn(this.config)); | |||
| await auth0.validate(); | |||
| if (!opts.disableKeywordReplacement) await auth0.validate(); | |||
There was a problem hiding this comment.
The schema validation needs to be disabled during keyword-preserved export because a field may be enforced as an array but will be expressed with an array replace marker (string).
Just now realizing that this should be a comment.
| //YAML format supports both single and double quotes for strings | ||
| const patternWithSingleQuotes = `'${pattern}'`; | ||
| const patternWithDoubleQuotes = `"${pattern}"`; | ||
|
|
||
| return new RegExp(`${patternWithQuotes}|${pattern}`, 'g'); | ||
| return new RegExp(`${patternWithSingleQuotes}|${patternWithDoubleQuotes}|${pattern}`, 'g'); |
There was a problem hiding this comment.
Array replace markers could be wrapped in single quotes, double quotes or not wrapped at all. When performing this replacement, it is necessary to check for all three.
src/tools/utils.ts
Outdated
| // wrapArrayReplaceMarkersInStrings will wrap array replacement markers in quotes. | ||
| // This is necessary for YAML format in the context of keyword replacement | ||
| // to preserve the keyword markers while also maintaining valid YAML syntax. | ||
| export function wrapArrayReplaceMarkersInStrings(body: string, mappings: KeywordMappings): string { |
There was a problem hiding this comment.
This function does the heavy lifting in this PR. It solves the problem outlined in Problem 1 of the description.
| const localAssets = ` | ||
| tenant: | ||
| friendly_name: "##ENV## Tenant" | ||
| enabled_locales: @@LANGUAGES@@ | ||
| connections: | ||
| - name: connection-1 | ||
| strategy: waad | ||
| options: | ||
| tenant_domain: "##DOMAIN##" | ||
| `; | ||
|
|
||
| fs.writeFileSync(tenantFile, localAssets); |
There was a problem hiding this comment.
Not possible to test this scenario with JSON, needs to be converted to YAML. Note the unwrapped @@LANGUAGES@@ array replace instance.
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## master #760 +/- ##
==========================================
+ Coverage 84.08% 84.13% +0.04%
==========================================
Files 115 115
Lines 3512 3523 +11
Branches 670 671 +1
==========================================
+ Hits 2953 2964 +11
Misses 322 322
Partials 237 237
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
🔧 Changes
One of the final pieces for delivery a MVP deliverable for keyword preservation on export is enabling the preservation of array replace keywords (ex:
@@LOGOUT_URLS@@) in YAML configuration formats. The hurdle with this limitation can be broken down into two major problems:Problem Statement 1:
When using array replace syntax in a YAML configuration file, the markers can be expressed as-is without quotes, which is invalid YAML. There is a built-in YAML to JSON conversion step that will fail. Example of an error-causing configuration:
Solution:
Introduce an intermediate step before the YAML parsing that looks for unquoted array replace markers and upon finding them, wrap in quotes. Array replace markers that are already wrapped in quotes should be left as-is.
Problem Statement 2:
When using array replace syntax in a YAML configuration file, the markers can be expressed as-is without quotes, wrapped with double quotes or wrapped in single quotes. While technically always true, the keyword replacement process preserves these array values wrapped in single-quotes.
For example, all of the following are valid configuration files:
Solution:
Appropriately handle each scenario during the keyword replacement step. In practice, this means introducing a third regex replacement step for single-quoted strings in addition to the double-quoted and unquoted array replace markers.
Together, the combination of these solutions enabled a seamless experience for users needing to preserve array replace keywords in the YAML format.
📚 References
Related RFC: #688
🔬 Testing
Added new unit tests for any functions that were altered. Additionally, added a couple assertions to E2E tests. Finally, did quite a bit of manual testing for another layer of verification.
📝 Checklist