diff --git a/README.md b/README.md index 7f377b7..a9d7cd5 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ # Test Message Generator -This small tool helps to generate (test) messages based on a template. It can act as a simple replacement for message source systems and allows to send on regular time intervals some data, based on a template that gets altered before sending based on a mapping. +This small tool helps to generate (test) messages based on a [Mustache](https://mustache.github.io/) template. It can act as a simple replacement for message source systems and allows to send on regular time intervals some data, based on a template that gets altered before sending. ## Docker The generator can be run as a Docker container, after creating a Docker image for it. The Docker container will keep running until stopped. @@ -11,14 +11,14 @@ docker build --tag vsds/test-message-generator . To run the generator, you can use: ```bash -docker run -v $(pwd)/data:/tmp/data -e TEMPLATEFILE=/tmp/data/other.template.json -e MAPPINGFILE=/tmp/data/other.mapping.json vsds/test-message-generator +docker run -v $(pwd)/data:/tmp/data -e TEMPLATEFILE=/tmp/data/template.json vsds/test-message-generator ``` You can also pass the following arguments when running the container: -* `TARGETURL=` to POST the output to the target URI instead of to the console * `SILENT=true` to display no logging to the console -* `MIMETYPE=` to use a different mime-type +* `TARGETURL=` to POST the output to the target URI instead of to the console +* `MIMETYPE=` to specify a mime-type when POSTing -Alternatively, you can also pass the template and mapping as string instead of as files, use `TEMPLATE` respectively `MAPPING`. +Alternatively, you can also pass the template as string instead of as file, use `TEMPLATE`. ## Build the Generator The generator is implemented as a [Node.js](https://nodejs.org/en/) application. @@ -29,11 +29,11 @@ npm run build ``` ## Run the Generator -The generator works based on a JSON template, defining the structure to use for each generated item, and a JSON mapping file, defining the transformations that need to be performed on the template. It can send the generated JSON data to a target URL or simply send it to the console. +The generator works based on a template, defining the structure to use for each generated item. It can send the generated data to a target URL or simply send it to the console. The generator takes the following command line arguments: * `--silent=` prevents any console debug output if true, defaults to false (not silent, logging all debug info) -* `--targetUrl` defines the target URL to where the generated JSON is POST'ed as `application/json`, no default (if not provided, sends output to console independant of `--silent`) +* `--targetUrl` defines the target URL to where the generated message is POST'ed as the configured mime-type, no default (if not provided, sends output to console independant of `--silent`) > **Note**: alternatively, you can provide the target URL as a plain text in a file named `TARGETURL` (located in the current working directory) allowing to change the target URL at runtime as the file is read at cron schedule time (see below), e.g.: > ```bash > echo http://example.org/my-ingest-endpoint > ./TARGETURL @@ -43,35 +43,28 @@ The generator takes the following command line arguments: > ```bash > echo https://webhook.site/f140204a-9514-4bfa-8d3e-fd18ba325ee3 > ./TARGETURL > ``` -* `--mimeType=` mime-type of message send to target URL, defaults to `application/json` +* `--mimeType=` mime-type of message send to target URL, no default * `--cron` defines the time schedule, defaults to `* * * * * * ` (every second) -* `--template=''` allows to provide the JSON template on the command line, no default (if not provided, you MUST provide `--templateFile`) -* `--templateFile=` allows to provide the JSON template in a file, no default (if not provided, you MUST provide `--template`) -* `--mapping=''` allows to provide the JSON mapping on the command line, no default (if not provided, you MUST provide `--mappingFile`) -* `--mappingFile=` allows to provide the JSON mapping in a file, no default (if not provided, you MUST provide `--mapping`) +* `--template=''` allows to provide the template on the command line, no default (if not provided, you MUST provide `--templateFile`) +* `--templateFile=` allows to provide the template in a file, no default (if not provided, you MUST provide `--template`) -The template or template file should simply contain a valid JSON structure (with one or more JSON objects). E.g.: +The template or template file should simply contain a message with mustache variables (between `{{` and `}}`). E.g.: ```json [ - { "id": "my-id", "type": "Something", "modifiedAt": "2022-09-09T09:10:00.000Z" }, - { "id": "my-other-id", "type": "SomethingElse", "modifiedAt": "2022-09-09T09:10:00.000Z" } + { "id": "my-id-{{index}}", "type": "Something", "modifiedAt": "{{timestamp}}" }, + { "id": "my-other-id-{{index}}", "type": "SomethingElse", "modifiedAt": "{{timestamp}}" } ] ``` -The mapping file is also a JSON file but uses a key/value mapping where the key conforms the [JSON path specifications](https://datatracker.ietf.org/doc/id/draft-goessner-dispatch-jsonpath-00.html) and the value conforms a syntax allowing to change the value matched by the JSON path to a new value obtained by replacing the variables specified in the value part, e.g.: -```json -{ "$.id": "${@}-${nextCounter}", "$.modifiedAt": "${currentTimestamp}" } -``` - -The `${@}` will be replaced by the currently match value of the JSON path `$.id` (e.g. `my-id`) while any other `${}` will use a property of the generator itself. Currently the only allowed properties are: -* `nextCounter`: increasing integer value, starting from 1 -* `currentTimestamp`: current date and time formatted as [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) in UTC (e.g. `2007-04-05T14:30:00.000Z`) +Currently the only allowed variables are: +* `index`: increasing integer value, starting from 1 +* `timestamp`: current date and time formatted as [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) in UTC (e.g. `2007-04-05T14:30:00.000Z`) You can run the generator after building it, e.g.: -Using this [template](./data/other.template.json) and this [mapping](./data//other.mapping.json) and with silent output to console: +Using this [template](./data/template.json) and with silent output to console: ```bash -node ./dist/index.js --templateFile ./data/other.template.json --mappingFile ./data/other.mapping.json --silent +node ./dist/index.js --templateFile ./data/template.json --silent ``` This results in something like the following: ``` @@ -80,9 +73,10 @@ This results in something like the following: {"id":"my-id-3","type":"Something","modifiedAt":"2022-09-12T13:15:44.003Z"} ... ``` + By specifying the template (containing multiple objects) and mapping on the command file: ```bash -node ./dist/index.js --template '[{"id": "my-id", "type": "Something", "modifiedAt": "2022-09-09T09:10:00.000Z" },{ "id": "my-other-id", "type": "SomethingElse", "modifiedAt": "2022-09-09T09:10:00.000Z" }]' --mapping '{ "$..id": "${@}-${nextCounter}", "$..modifiedAt": "${currentTimestamp}" }' --silent +node ./dist/index.js --template '[{"id": "my-id-{{index}}", "type": "Something", "modifiedAt": "{{timestamp}}" },{ "id": "my-other-id-{{index}}", "type": "SomethingElse", "modifiedAt": "{{timestamp}}" }]' --silent ``` This results in something like: ```json @@ -94,14 +88,13 @@ This results in something like: Alternatively you can generate the output using a different time schedule (e.g. every 2 seconds) to a [dummy HTTP server](https://docs.webhook.site/) (including debugging to the console): ```bash -node ./dist/index.js --templateFile ./data/other.template.json --mappingFile ./data/other.mapping.json --cron '*/2 * * * * *' --targetUrl https://webhook.site/ce3065f5-2f0b-49d8-8856-330ae3c6e737 +node ./dist/index.js --templateFile ./data/template.json --cron '*/2 * * * * *' --targetUrl https://webhook.site/ce3065f5-2f0b-49d8-8856-330ae3c6e737 --mimeType application/json ``` This results in: ``` Arguments: { _: [], - templateFile: './data/other.template.json', - mappingFile: './data/other.mapping.json', + templateFile: './data/template.json', cron: '*/2 * * * * *', targetUrl: 'https://webhook.site/ce3065f5-2f0b-49d8-8856-330ae3c6e737' } diff --git a/data/other.mapping.json b/data/other.mapping.json deleted file mode 100644 index d3ab498..0000000 --- a/data/other.mapping.json +++ /dev/null @@ -1 +0,0 @@ -{ "$.id": "${@}-${nextCounter}", "$.modifiedAt": "${currentTimestamp}" } \ No newline at end of file diff --git a/data/other.template.json b/data/other.template.json deleted file mode 100644 index fd448c8..0000000 --- a/data/other.template.json +++ /dev/null @@ -1 +0,0 @@ -{ "id": "my-id", "type": "Something", "modifiedAt": "2022-09-09T09:10:00.000Z" } \ No newline at end of file diff --git a/data/template.json b/data/template.json new file mode 100644 index 0000000..8ef0ee0 --- /dev/null +++ b/data/template.json @@ -0,0 +1 @@ +{ "id": "my-id-{{index}}", "type": "Something", "modifiedAt": "{{timestamp}}" } \ No newline at end of file diff --git a/package-lock.json b/package-lock.json index c7d2146..cf579ad 100644 --- a/package-lock.json +++ b/package-lock.json @@ -10,14 +10,14 @@ "license": "EUPL-1.2", "dependencies": { "cron": "^2.3.0", - "jsonpath": "^1.1.1", "minimist": "^1.2.8", + "mustache": "^4.2.0", "node-fetch": "^3.3.1" }, "devDependencies": { "@types/cron": "^2.0.1", - "@types/jsonpath": "^0.2.0", "@types/minimist": "^1.2.2", + "@types/mustache": "^4.2.2", "@types/node": "^18.15.11", "@types/node-fetch": "^2.6.3", "typescript": "^4.6.4" @@ -33,12 +33,6 @@ "@types/node": "*" } }, - "node_modules/@types/jsonpath": { - "version": "0.2.0", - "resolved": "https://registry.npmjs.org/@types/jsonpath/-/jsonpath-0.2.0.tgz", - "integrity": "sha512-v7qlPA0VpKUlEdhghbDqRoKMxFB3h3Ch688TApBJ6v+XLDdvWCGLJIYiPKGZnS6MAOie+IorCfNYVHOPIHSWwQ==", - "dev": true - }, "node_modules/@types/luxon": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/@types/luxon/-/luxon-3.2.0.tgz", @@ -51,6 +45,12 @@ "integrity": "sha512-jhuKLIRrhvCPLqwPcx6INqmKeiA5EWrsCOPhrlFSrbrmU4ZMPjj5Ul/oLCMDO98XRUIwVm78xICz4EPCektzeQ==", "dev": true }, + "node_modules/@types/mustache": { + "version": "4.2.2", + "resolved": "https://registry.npmjs.org/@types/mustache/-/mustache-4.2.2.tgz", + "integrity": "sha512-MUSpfpW0yZbTgjekDbH0shMYBUD+X/uJJJMm9LXN1d5yjl5lCY1vN/eWKD6D1tOtjA6206K0zcIPnUaFMurdNA==", + "dev": true + }, "node_modules/@types/node": { "version": "18.15.11", "resolved": "https://registry.npmjs.org/@types/node/-/node-18.15.11.tgz", @@ -101,11 +101,6 @@ "node": ">= 12" } }, - "node_modules/deep-is": { - "version": "0.1.4", - "resolved": "https://registry.npmjs.org/deep-is/-/deep-is-0.1.4.tgz", - "integrity": "sha512-oIPzksmTg4/MriiaYGO+okXDT7ztn/w3Eptv/+gSIdMdKsJo0u4CfYNFJPy+4SKMuCqGw2wxnA+URMg3t8a/bQ==" - }, "node_modules/delayed-stream": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz", @@ -115,72 +110,6 @@ "node": ">=0.4.0" } }, - "node_modules/escodegen": { - "version": "1.14.3", - "resolved": "https://registry.npmjs.org/escodegen/-/escodegen-1.14.3.tgz", - "integrity": "sha512-qFcX0XJkdg+PB3xjZZG/wKSuT1PnQWx57+TVSjIMmILd2yC/6ByYElPwJnslDsuWuSAp4AwJGumarAAmJch5Kw==", - "dependencies": { - "esprima": "^4.0.1", - "estraverse": "^4.2.0", - "esutils": "^2.0.2", - "optionator": "^0.8.1" - }, - "bin": { - "escodegen": "bin/escodegen.js", - "esgenerate": "bin/esgenerate.js" - }, - "engines": { - "node": ">=4.0" - }, - "optionalDependencies": { - "source-map": "~0.6.1" - } - }, - "node_modules/escodegen/node_modules/esprima": { - "version": "4.0.1", - "resolved": "https://registry.npmjs.org/esprima/-/esprima-4.0.1.tgz", - "integrity": "sha512-eGuFFw7Upda+g4p+QHvnW0RyTX/SVeJBDM/gCtMARO0cLuT2HcEKnTPvhjV6aGeqrCB/sbNop0Kszm0jsaWU4A==", - "bin": { - "esparse": "bin/esparse.js", - "esvalidate": "bin/esvalidate.js" - }, - "engines": { - "node": ">=4" - } - }, - "node_modules/esprima": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/esprima/-/esprima-1.2.2.tgz", - "integrity": "sha512-+JpPZam9w5DuJ3Q67SqsMGtiHKENSMRVoxvArfJZK01/BfLEObtZ6orJa/MtoGNR/rfMgp5837T41PAmTwAv/A==", - "bin": { - "esparse": "bin/esparse.js", - "esvalidate": "bin/esvalidate.js" - }, - "engines": { - "node": ">=0.4.0" - } - }, - "node_modules/estraverse": { - "version": "4.3.0", - "resolved": "https://registry.npmjs.org/estraverse/-/estraverse-4.3.0.tgz", - "integrity": "sha512-39nnKffWz8xN1BU/2c79n9nB9HDzo0niYUqx6xyqUnyoAnQyyWpOTdZEeiCch8BBu515t4wp9ZmgVfVhn9EBpw==", - "engines": { - "node": ">=4.0" - } - }, - "node_modules/esutils": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/esutils/-/esutils-2.0.3.tgz", - "integrity": "sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g==", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/fast-levenshtein": { - "version": "2.0.6", - "resolved": "https://registry.npmjs.org/fast-levenshtein/-/fast-levenshtein-2.0.6.tgz", - "integrity": "sha512-DCXu6Ifhqcks7TZKY3Hxp3y6qphY5SJZmrWMDrKcERSOXWQdMhU9Ig/PYrzyw/ul9jOIyh0N4M0tbC5hodg8dw==" - }, "node_modules/fetch-blob": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/fetch-blob/-/fetch-blob-3.2.0.tgz", @@ -228,28 +157,6 @@ "node": ">=12.20.0" } }, - "node_modules/jsonpath": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/jsonpath/-/jsonpath-1.1.1.tgz", - "integrity": "sha512-l6Cg7jRpixfbgoWgkrl77dgEj8RPvND0wMH6TwQmi9Qs4TFfS9u5cUFnbeKTwj5ga5Y3BTGGNI28k117LJ009w==", - "dependencies": { - "esprima": "1.2.2", - "static-eval": "2.0.2", - "underscore": "1.12.1" - } - }, - "node_modules/levn": { - "version": "0.3.0", - "resolved": "https://registry.npmjs.org/levn/-/levn-0.3.0.tgz", - "integrity": "sha512-0OO4y2iOHix2W6ujICbKIaEQXvFQHue65vUG3pb5EUomzPI90z9hsA1VsO/dbIIpC53J8gxM9Q4Oho0jrCM/yA==", - "dependencies": { - "prelude-ls": "~1.1.2", - "type-check": "~0.3.2" - }, - "engines": { - "node": ">= 0.8.0" - } - }, "node_modules/luxon": { "version": "3.2.1", "resolved": "https://registry.npmjs.org/luxon/-/luxon-3.2.1.tgz", @@ -287,6 +194,14 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/mustache": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/mustache/-/mustache-4.2.0.tgz", + "integrity": "sha512-71ippSywq5Yb7/tVYyGbkBggbU8H3u5Rz56fH60jGFgr8uHwxs+aSKeqmluIVzM0m0kB7xQjKS6qPfd0b2ZoqQ==", + "bin": { + "mustache": "bin/mustache" + } + }, "node_modules/node-domexception": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/node-domexception/-/node-domexception-1.0.0.tgz", @@ -322,58 +237,6 @@ "url": "https://opencollective.com/node-fetch" } }, - "node_modules/optionator": { - "version": "0.8.3", - "resolved": "https://registry.npmjs.org/optionator/-/optionator-0.8.3.tgz", - "integrity": "sha512-+IW9pACdk3XWmmTXG8m3upGUJst5XRGzxMRjXzAuJ1XnIFNvfhjjIuYkDvysnPQ7qzqVzLt78BCruntqRhWQbA==", - "dependencies": { - "deep-is": "~0.1.3", - "fast-levenshtein": "~2.0.6", - "levn": "~0.3.0", - "prelude-ls": "~1.1.2", - "type-check": "~0.3.2", - "word-wrap": "~1.2.3" - }, - "engines": { - "node": ">= 0.8.0" - } - }, - "node_modules/prelude-ls": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/prelude-ls/-/prelude-ls-1.1.2.tgz", - "integrity": "sha512-ESF23V4SKG6lVSGZgYNpbsiaAkdab6ZgOxe52p7+Kid3W3u3bxR4Vfd/o21dmN7jSt0IwgZ4v5MUd26FEtXE9w==", - "engines": { - "node": ">= 0.8.0" - } - }, - "node_modules/source-map": { - "version": "0.6.1", - "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz", - "integrity": "sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==", - "optional": true, - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/static-eval": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/static-eval/-/static-eval-2.0.2.tgz", - "integrity": "sha512-N/D219Hcr2bPjLxPiV+TQE++Tsmrady7TqAJugLy7Xk1EumfDWS/f5dtBbkRCGE7wKKXuYockQoj8Rm2/pVKyg==", - "dependencies": { - "escodegen": "^1.8.1" - } - }, - "node_modules/type-check": { - "version": "0.3.2", - "resolved": "https://registry.npmjs.org/type-check/-/type-check-0.3.2.tgz", - "integrity": "sha512-ZCmOJdvOWDBYJlzAoFkC+Q0+bUyEOS1ltgp1MGU03fqHG+dbi9tBFU2Rd9QKiDZFAYrhPh2JUf7rZRIuHRKtOg==", - "dependencies": { - "prelude-ls": "~1.1.2" - }, - "engines": { - "node": ">= 0.8.0" - } - }, "node_modules/typescript": { "version": "4.9.4", "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.9.4.tgz", @@ -387,11 +250,6 @@ "node": ">=4.2.0" } }, - "node_modules/underscore": { - "version": "1.12.1", - "resolved": "https://registry.npmjs.org/underscore/-/underscore-1.12.1.tgz", - "integrity": "sha512-hEQt0+ZLDVUMhebKxL4x1BTtDY7bavVofhZ9KZ4aI26X9SRaE+Y3m83XUL1UP2jn8ynjndwCCpEHdUG+9pP1Tw==" - }, "node_modules/web-streams-polyfill": { "version": "3.2.1", "resolved": "https://registry.npmjs.org/web-streams-polyfill/-/web-streams-polyfill-3.2.1.tgz", @@ -399,14 +257,6 @@ "engines": { "node": ">= 8" } - }, - "node_modules/word-wrap": { - "version": "1.2.3", - "resolved": "https://registry.npmjs.org/word-wrap/-/word-wrap-1.2.3.tgz", - "integrity": "sha512-Hz/mrNwitNRh/HUAtM/VT/5VH+ygD6DV7mYKZAtHOrbs8U7lvPS6xf7EJKMF0uW1KJCl0H701g3ZGus+muE5vQ==", - "engines": { - "node": ">=0.10.0" - } } } } diff --git a/package.json b/package.json index a1ffb4f..a0a5999 100644 --- a/package.json +++ b/package.json @@ -12,7 +12,7 @@ "license": "EUPL-1.2", "devDependencies": { "@types/cron": "^2.0.1", - "@types/jsonpath": "^0.2.0", + "@types/mustache": "^4.2.2", "@types/minimist": "^1.2.2", "@types/node": "^18.15.11", "@types/node-fetch": "^2.6.3", @@ -20,7 +20,7 @@ }, "dependencies": { "cron": "^2.3.0", - "jsonpath": "^1.1.1", + "mustache": "^4.2.0", "minimist": "^1.2.8", "node-fetch": "^3.3.1" } diff --git a/src/generator.ts b/src/generator.ts index c9c32b8..c97bf8f 100644 --- a/src/generator.ts +++ b/src/generator.ts @@ -1,36 +1,21 @@ -import jp from 'jsonpath'; +import Mustache from 'mustache'; -interface JsonMapping { - path: string, - value: (x: any) => any, -} - -export class JsonGenerator { +export class Generator { private _counter: number = 0; - private _mapping: JsonMapping[]; - - private substituteVariables(format: string, value: any): string { - var self: any = this; - return format.replace(/[$]{([^}]+)}/g, (_, variable) => variable == '@' ? value : self[variable] ? self[variable](value) : ''); - } - constructor(private _template: string, mapping: { [key: string]: string }) { - this._mapping = Object.keys(mapping).map(key => - ({ path: key, value: (x: any) => - this.substituteVariables(mapping[key] as string, x) } as JsonMapping)); - } + constructor(private _template: string) {} - public nextCounter(): Number { + public index(): Number { return ++this._counter; } - public currentTimestamp(): string { + public timestamp(): string { return new Date().toISOString(); } public createNext(): any { - var next = JSON.parse(this._template); - this._mapping.forEach(item => { jp.apply(next, item.path, item.value); }); + var data = { index: this.index(), timestamp: this.timestamp() }; + var next = Mustache.render(this._template, data); return next; } -} \ No newline at end of file +} diff --git a/src/index.ts b/src/index.ts index ae57463..97cd712 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,30 +1,25 @@ import { CronJob } from 'cron'; import { existsSync, readFileSync } from 'fs'; import minimist from 'minimist'; -import { JsonGenerator } from './generator.js'; +import { Generator } from './generator.js'; import fetch from 'node-fetch'; const args = minimist(process.argv.slice(2)); const silent: boolean = (/true/i).test(args['silent']); - -const cron = args['cron'] || '* * * * * *'; -const mimeType = args['mimeType'] || 'application/json'; if (!silent) console.debug("Arguments: ", args); +const cron = args['cron'] || '* * * * * *'; +const mimeType = args['mimeType']; let template: string = args['template'] || (args['templateFile'] && readFileSync(args['templateFile'], {encoding: 'utf8'})); if (!template) throw new Error('Missing template or templateFile'); -let mapping = args['mapping'] || (args['mappingFile'] && readFileSync(args['mappingFile'], {encoding: 'utf8'})); -if (!mapping) throw new Error('Missing mapping or mappingFile'); -mapping = JSON.parse(mapping); - -const generator: JsonGenerator = new JsonGenerator(template, mapping); +const generator: Generator = new Generator(template); const job = new CronJob(cron, async () => { - const next = generator.createNext(); - const body = JSON.stringify(next); + const body = generator.createNext(); const targetUrl = args['targetUrl'] || (existsSync('./TARGETURL') && readFileSync('./TARGETURL', 'utf-8').trimEnd()); if (targetUrl) { + if (!mimeType) throw new Error('Missing mimeType'); if (!silent) console.debug(`Sending to '${targetUrl}':`, body); const response = await fetch(targetUrl, { method: 'post',