Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add strict mode to parser #74

Merged
merged 40 commits into from
Apr 13, 2022
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
697493f
feat: Restructure options API
aaronccasanova Feb 9, 2022
9e42db0
chore: Remove debug comments
aaronccasanova Feb 9, 2022
36ef68c
chore: Remove debug comments
aaronccasanova Feb 9, 2022
67c0b03
chore: Alias args to argv to introduce less changes
aaronccasanova Feb 9, 2022
082bec5
feat: Replace option with
Feb 12, 2022
0909008
docs: Update README to reflect updated implementation
Feb 12, 2022
042480b
chore: Revert args options to argv
Feb 12, 2022
10df671
feat: Add strict mode to parser
aaronccasanova Feb 19, 2022
e44c46c
docs: Update README from PR feedback
aaronccasanova Feb 24, 2022
4063751
Merge branch 'main' of https://github.com/aaronccasanova/parseargs in…
aaronccasanova Feb 26, 2022
a50e940
chore: Reduce changes
aaronccasanova Feb 26, 2022
062bdc9
chore: Tidy up naming conventions
aaronccasanova Feb 26, 2022
28ae7b8
Merge branch 'main' into feat/restructure-options-api
bcoe Feb 27, 2022
e6fea72
Merge branch 'feat/restructure-options-api' of https://github.com/aar…
aaronccasanova Feb 27, 2022
ef06099
fix: Guard against prototype member access
aaronccasanova Feb 27, 2022
c228484
Merge branch 'feat/restructure-options-api' of https://github.com/aar…
aaronccasanova Mar 3, 2022
dd4f718
feat: Add strict mode type validation
aaronccasanova Mar 3, 2022
26df81c
chore: Remove custom unknown option error
aaronccasanova Mar 3, 2022
a04d11c
chore: Remove unknown option error from test
aaronccasanova Mar 3, 2022
ca05b43
fix: Update value validation to check against undefined
aaronccasanova Mar 3, 2022
a1c3544
Merge branch 'main' of https://github.com/aaronccasanova/parseargs in…
aaronccasanova Apr 9, 2022
fa0775b
fix: Update error conditionals to check for null or undefined values
aaronccasanova Apr 9, 2022
5b89108
Merge branch 'main' of https://github.com/aaronccasanova/parseargs in…
aaronccasanova Apr 10, 2022
1d412da
feat: Add custom error classes for unknown and invalid options
aaronccasanova Apr 10, 2022
a16fc7b
Adjust unknown option message to be agnostic of long or short options
aaronccasanova Apr 10, 2022
c4271a0
feat: Pass shortOption to storeOption util for better error messages
aaronccasanova Apr 10, 2022
b6107bb
Merge branch 'main' of https://github.com/aaronccasanova/parseargs in…
aaronccasanova Apr 10, 2022
948ca9e
fix(WIP): Update checks to use the ERR_INVALID_ARG_VALUE class
aaronccasanova Apr 10, 2022
a2ea48d
Merge branch 'main' of https://github.com/aaronccasanova/parseargs in…
aaronccasanova Apr 10, 2022
38575cd
Merge branch 'main' of https://github.com/aaronccasanova/parseargs in…
aaronccasanova Apr 11, 2022
99355dd
fix: Revert back to custom error classes and improve error messages
aaronccasanova Apr 12, 2022
e66a8b3
WIP: Default parseArgs to strict:true and update failing tests to str…
aaronccasanova Apr 12, 2022
a4fc92a
Update README to reflect strict:true behavior
aaronccasanova Apr 12, 2022
12fd0e6
fix: Improve robustness of the short option error message
aaronccasanova Apr 12, 2022
870cffe
fix: Improve error message for type:string options missing an argument
aaronccasanova Apr 12, 2022
694c75d
fix: Namespace unique parseArgs error classes
aaronccasanova Apr 12, 2022
9b76d55
docs: Update README examples
aaronccasanova Apr 12, 2022
fcb8c8d
Merge branch 'main' of https://github.com/aaronccasanova/parseargs in…
aaronccasanova Apr 12, 2022
e56246c
docs: Update remaing examples for consistency
aaronccasanova Apr 12, 2022
b4e0daf
Update index.js
aaronccasanova Apr 13, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 16 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ process.mainArgs = process.argv.slice(process._exec ? 1 : 2)
* `type` {'string'|'boolean'} (Required) Type of known option
* `multiple` {boolean} (Optional) If true, when appearing one or more times in `args`, results are collected in an `Array`
* `short` {string} (Optional) A single character alias for an option; When appearing one or more times in `args`; Respects the `multiple` configuration
* `strict` {Boolean} (Optional) A `Boolean` on wheather or not to throw an error when unknown args are encountered
* `strict` {Boolean} (Optional) A `Boolean` for whether or not to throw an error when unknown options are encountered, `type:'string'` options are missing an options-argument, or `type:'boolean'` options are passed an options-argument; defaults to `true`
* Returns: {Object} An object having properties:
* `values` {Object}, key:value for each option found. Value is a string for string options, or `true` for boolean options, or an array (of strings or booleans) for options configured as `multiple:true`.
* `positionals` {string[]}, containing [Positionals][]
Expand All @@ -97,24 +97,14 @@ process.mainArgs = process.argv.slice(process._exec ? 1 : 2)
const { parseArgs } = require('@pkgjs/parseargs');
```

```js
// unconfigured
const { parseArgs } = require('@pkgjs/parseargs');
const args = ['-f', '--foo=a', '--bar', 'b'];
const options = {};
const { values, positionals } = parseArgs({ args, options });
// values = { f: true, foo: 'a', bar: true }
// positionals = ['b']
```

```js
const { parseArgs } = require('@pkgjs/parseargs');
// type:string
const args = ['-f', '--foo=a', '--bar', 'b'];
const options = {
bar: {
type: 'string',
},
f: { type: 'boolean' },
foo: { type: 'string'},
bar: { type: 'string' },
};
const { values, positionals } = parseArgs({ args, options });
// values = { f: true, foo: 'a', bar: 'b' }
aaronccasanova marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -126,6 +116,7 @@ const { parseArgs } = require('@pkgjs/parseargs');
// type:string & multiple
const args = ['-f', '--foo=a', '--foo', 'b'];
const options = {
f: { type: 'boolean' },
foo: {
type: 'string',
multiple: true,
aaronccasanova marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -151,6 +142,17 @@ const { values, positionals } = parseArgs({ args, options });
// positionals = ['b']
```

```js
// unconfigured
const { parseArgs } = require('@pkgjs/parseargs');
const args = ['-f', '--foo=a', '--bar', 'b'];
const options = {};
const { values, positionals } = parseArgs({ strict: false, args, options });
// values = { f: true, foo: 'a', bar: true }
// positionals = ['b']
```


### F.A.Qs

- Is `cmd --foo=bar baz` the same as `cmd baz --foo=bar`?
Expand Down
18 changes: 17 additions & 1 deletion errors.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,25 @@ class ERR_INVALID_ARG_VALUE extends TypeError {
}
}

class ERR_INVALID_OPTION_VALUE extends Error {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As @cjihrig suggests elsewhere, could se use:

> new errors.codes.ERR_INVALID_ARG_VALUE('options.foo.short', '--bar', 'must be a single character');

To avoid needing to introduce a new error to errors.js?

Copy link
Collaborator Author

@aaronccasanova aaronccasanova Apr 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's an early WIP with all the strict error cases updated to use ERR_INVALID_ARG_VALUE: 948ca9e

I could use some suggestions for the error messages as the new message constraint of ERR_INVALID_ARG_VALUE and my current implementation doesn't read entirely accurate:

// node invalid-string-option.js --foo
parseArgs({
  strict: true,
  options: { foo: { type: 'string' } },
})
// TypeError: The property --foo must be provided a value, Received 'undefined'
// node invalid-boolean-option.js --foo=bar
parseArgs({
  strict: true,
  options: { foo: { type: 'boolean' } },
})
// TypeError: The property --foo does not expect a value, Received 'bar'

Really I'm hung up on the word property over option (e.g. The option --foo must...), but not sure if I'm overthinking it. Just want the message to be super clear to users.

Copy link
Collaborator Author

@aaronccasanova aaronccasanova Apr 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, the current unknown option message isn't entirely helpful to users e.g.

$ node unknown-option.js --foo

TypeError: The property option.foo must be configured, Received 'undefined'

Any suggestions to make this message more user-facing? e.g.

TypeError: The property --foo must be a known option, Received '--foo'

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any suggestions to make this message more user-facing?

As a user I would expect the message to be "Unknown option --foo".

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that was the original message before the rework with the addition of dynamism for long and short options e.g. Unknown long option --foo and Unknown short option -f. In an effort to reduce friction merging into Node I'm trying to find an equally helpful message within the constraints of ERR_INVALID_ARG_VALUE.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the format for ERR_INVALID_ARG_VALUE is a particularly good match for our use here where we are referring to options on the command line, rather than arguments to a javascript function. (Valiant try though!)

I see the node 18.x branch has updated code for the error which inserts property/argument depending on the name, but still not a good match for our option errors.

https://github.com/nodejs/node/blob/ec5a359ffd38b6f56a456a0b6a482b5167fbfce1/lib/internal/errors.js#L1246

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted to custom error classes and updated the error messages per: #74 (comment)

Examples based on the PR description prompts:

  • Unknown option encountered
Error: Unknown option '--foo'
Error: Unknown option '-f'
  • Option of type:'string' used like a boolean option e.g. lone --string
Error: Option '--foo <foo>' argument missing
Error: `Option '-f, --foo <foo>' argument missing
  • Option of type:'boolean' used like a string option e.g. --boolean=foo
Error: Option '--foo' does not take an argument
Error: `Option '-f, --foo' does not take an argument

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

My only minor wording musing is for the "Option of type:'string' used like a boolean option". With the option I used, I was briefly confusing by the long option name reused as the option argument name. I wonder about value or arg? But haven't convinced myself they are better, unless you really like one, stick with what you have!

Error: `Option '-f, --foo <foo>' argument missing

Error: `Option '-f, --foo <value>' argument missing

Error: `Option '-f, --foo <arg>' argument missing

The reuse does tend to work semantically. I picked a long one out of the node options to try:

Error: `Option '--trace-event-file-pattern <trace-event-file-pattern>' argument missing

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! That example is excessively long. I think I'll use value for consistency with the parseArgs results object.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

constructor(message) {
super(message);
this.code = 'ERR_INVALID_OPTION_VALUE';
}
}

class ERR_UNKNOWN_OPTION extends Error {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise, if we can avoid adding any new errors it simplifies merging into Node.js.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see: #97

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

@aaronccasanova aaronccasanova Apr 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, is it possible to namespace our error codes so we can provide the most accurate messages to users?
e.g.

  • ERR_PARGS_*
  • ERR_PARGS_UNKNOWN_OPTION
    • Unknown long option --foo
    • Unknown short option -f
  • ERR_PARGS_INVALID_OPTION_VALUE
    • Missing value for long option --foo with type:'string'
    • Unexpected value bar for short option -f with type:'boolean'

Copy link
Collaborator Author

@aaronccasanova aaronccasanova Apr 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There appear to be error namespaces in node core:

@bcoe @shadowspawn @ljharb What are your thoughts on using a namespace for parseArgs errors. For example ERR_PARGS_* or ERR_PARSEARGS_*

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preferences would be ERR_PARSE_ARGS, rather than the short hand.

But yes, I can't see any reason not to use a namespace.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated unique parseArgs error classes to ERR_PARSE_ARGS_*: 694c75d

constructor(option) {
super(`Unknown option '${option}'`);
this.code = 'ERR_UNKNOWN_OPTION';
}
}

module.exports = {
codes: {
ERR_INVALID_ARG_TYPE,
ERR_INVALID_ARG_VALUE
ERR_INVALID_ARG_VALUE,
ERR_INVALID_OPTION_VALUE,
ERR_UNKNOWN_OPTION,
}
};
60 changes: 52 additions & 8 deletions index.js
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ const {
const {
codes: {
ERR_INVALID_ARG_VALUE,
ERR_INVALID_OPTION_VALUE,
ERR_UNKNOWN_OPTION,
},
} = require('./errors');

Expand Down Expand Up @@ -73,16 +75,42 @@ function getMainArgs() {
}

const protoKey = '__proto__';
function storeOptionValue(options, longOption, value, result) {
const optionConfig = options[longOption] || {};

function storeOption({
strict,
options,
result,
longOption,
shortOption,
optionValue,
}) {
const hasOptionConfig = ObjectHasOwn(options, longOption);

aaronccasanova marked this conversation as resolved.
Show resolved Hide resolved
const optionConfig = hasOptionConfig ? options[longOption] : {};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might simplify this to options[longOption] ?? {}.


if (strict) {
if (!hasOptionConfig) {
throw new ERR_UNKNOWN_OPTION(shortOption == null ? `--${longOption}` : `-${shortOption}`);
}

const shortOptionErr = optionConfig.short ? `-${optionConfig.short}, ` : '';
Copy link
Collaborator Author

@aaronccasanova aaronccasanova Apr 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still getting the hang of primordials. Do I need to do ObjectHasOwn(optionConfig, 'short') ? '...' : ''; here?

Copy link
Collaborator

@shadowspawn shadowspawn Apr 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was validated if present as a single character on entry, so no need for extra checks here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The initial validation only checks own properties of the option config. Couldn't immediatly think of a way to confirm, but came up with this demo:
image

Not sure if this opens up any bugs or areas for malicious behavior, but might as well update the condition to:

const shortOptionErr = ObjectHasOwn(optionConfig, 'short') ? `-${optionConfig.short}, ` : '';

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator

@shadowspawn shadowspawn Apr 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arg, you are right! (Sorry, the validation wasn't doing what I thought.)

I assume the high level goal to behave the same as if the prototype pollution was not present?

That makes me question lots more of the code though. The options bag has introduced lots of property access:

  • same issue for multiple
  • and type, lots including utils. Although that might be ok if the next issue was fixed.
  • the validation of type at the top of parseArgs does not test it is not from prototype (from my recent change to make type required!)

Also, the destructing function parameters of both parseArgs and (now) storeOption receive properties from prototype if not specified by caller.

We could copy the input arguments into a safe(r) object before using them, by specifying all expected properties. Is this a pattern used in other node code? I suspect the usual pattern is just be paranoid everywhere because easier to be confident local code is being paranoid enough?!

I suggest only fix up the code introduced in this PR though. Get strict functional and separately have an more-paranoia-more-robust revisit!

Copy link
Collaborator

@shadowspawn shadowspawn Apr 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bcoe @ljharb
If my reading of this is correct, we are not currently robust against prototype pollution. Can we improve this after first version of an experimental feature?

(I have some ideas about how to improve it without littering the code and can have a go tonight, but aware clock is ticking.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, this is not a new problem. I looked back through major refactors and we have been vulnerable to prototype pollution in every iteration. Not causing pollution, but affected by prototype pollution. Opened #104 with demo.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the bigger concern for prototype pollution is untrusted input to the CLI modifying the prototype, vs., the user having mucked with proto in their application, and this affecting our parse.

This makes me think that the bigger security concern would be if we ever supported dot properties; as @shadowspawn has mentioned.


if (options[longOption].type === 'string' && optionValue == null) {
throw new ERR_INVALID_OPTION_VALUE(`Option '${shortOptionErr}--${longOption} <${longOption}>' argument missing`);
}

if (options[longOption].type === 'boolean' && optionValue != null) {
throw new ERR_INVALID_OPTION_VALUE(`Option '${shortOptionErr}--${longOption}' does not take an argument`);
}
}

if (longOption === protoKey) {
return;
}

// Values
const usedAsFlag = value === undefined;
const newValue = usedAsFlag ? true : value;
const usedAsFlag = optionValue === undefined;
const newValue = usedAsFlag ? true : optionValue;
if (optionConfig.multiple) {
// Always store value in array, including for flags.
// result.values[longOption] starts out not present,
Expand All @@ -99,9 +127,11 @@ function storeOptionValue(options, longOption, value, result) {

const parseArgs = ({
args = getMainArgs(),
strict = true,
options = {}
} = {}) => {
validateArray(args, 'args');
validateBoolean(strict, 'strict');
validateObject(options, 'options');
ArrayPrototypeForEach(
ObjectEntries(options),
Expand Down Expand Up @@ -158,7 +188,14 @@ const parseArgs = ({
// e.g. '-f', 'bar'
optionValue = ArrayPrototypeShift(remainingArgs);
}
storeOptionValue(options, longOption, optionValue, result);
storeOption({
strict,
options,
result,
longOption,
shortOption,
optionValue,
});
continue;
}

Expand Down Expand Up @@ -188,7 +225,14 @@ const parseArgs = ({
const shortOption = StringPrototypeCharAt(arg, 1);
const longOption = findLongOptionForShort(shortOption, options);
const optionValue = StringPrototypeSlice(arg, 2);
storeOptionValue(options, longOption, optionValue, result);
storeOption({
strict,
options,
result,
longOption,
shortOption,
optionValue,
});
continue;
}

Expand All @@ -200,7 +244,7 @@ const parseArgs = ({
// e.g. '--foo', 'bar'
optionValue = ArrayPrototypeShift(remainingArgs);
}
storeOptionValue(options, longOption, optionValue, result);
storeOption({ strict, options, result, longOption, optionValue });
continue;
}

Expand All @@ -209,7 +253,7 @@ const parseArgs = ({
const index = StringPrototypeIndexOf(arg, '=');
const longOption = StringPrototypeSlice(arg, 2, index);
const optionValue = StringPrototypeSlice(arg, index + 1);
storeOptionValue(options, longOption, optionValue, result);
storeOption({ strict, options, result, longOption, optionValue });
continue;
}

Expand Down
Loading