Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What makes a valid ObjectId #11227

Closed
vkarpov15 opened this issue Jan 16, 2022 Discussed in #11221 · 4 comments
Closed

What makes a valid ObjectId #11227

vkarpov15 opened this issue Jan 16, 2022 Discussed in #11221 · 4 comments
Milestone

Comments

@vkarpov15
Copy link
Collaborator

Discussed in #11221

Originally posted by Maximusya January 13, 2022
Right now I am migrating from Mongoose 5.x to 6.x and got about half of integration tests fail just because of undocumented changes in ObjectId validation.
These two mongoose validation methods differ in behavior both between the two in one version of mongoose and between mongoose versions (+dependencies):

  • mongoose.Types.ObjectId.isValid
  • mongoose.isValidObjectId

I get the expectation that once a variable has passed any of these validations, it can be safely specified both as new mongoose.Type.ObjectId(...) ctor arg, and any of MyModel.findById(..) et al.
Another kind of expectation (but probably far fetched): if a variable has passed any of these validations IT IS an id, but not a full document loaded from DB. Imagine a helper method that loads missing data if the argument passed in has properties that are not yet populated/loaded.

Example:

const mongoose = require('mongoose');

const testSubjects = {
  'null': null,
  'undefined': undefined,
  'new ObjectId()': new mongoose.Types.ObjectId(),
  'ObjectId().valueOf()': (new mongoose.Types.ObjectId()).valueOf(),
  'ObjectId().toString()': (new mongoose.Types.ObjectId()).toString(),
  'ObjectId().toHexString()': (new mongoose.Types.ObjectId()).toHexString(),
  '\'Z\'.repeat(12)': 'Z'.repeat(12),
  '\'F\'.repeat(24)': 'F'.repeat(24),
  '\'580e0797bb495f0a200e91ad\'': '580e0797bb495f0a200e91ad',
  '{ _id: new ObjectId() }': { _id: new mongoose.Types.ObjectId() },
  '{ _id: \'Z\'.repeat(12) }': { _id: 'Z'.repeat(12) },
  '{ _id: \'F\'.repeat(24) }': { _id: 'F'.repeat(24) },
  '{ _id: \'580e0797bb495f0a200e91ad\' }': { _id: '580e0797bb495f0a200e91ad' },
  '{ id: new ObjectId() }': { id: new mongoose.Types.ObjectId() },
  '{ id: \'Z\'.repeat(12) }': { id: 'Z'.repeat(12) },
  '{ id: \'F\'.repeat(24) }': { id: 'F'.repeat(24) },
  '{ id: \'580e0797bb495f0a200e91ad\' }': { id: '580e0797bb495f0a200e91ad' },
  '{ id: null }': { id: null },
  '{ id: undefined }': { id: undefined },
  '{ _id: null }': { _id: null },
  '{ _id: undefined }': { _id: undefined },
};

console.table(Object.entries(testSubjects).map(([key, value]) => ({
  test: key,
  'isValidObjectId': mongoose.isValidObjectId(value),
  'ObjectId.isValid': mongoose.Types.ObjectId.isValid(value),
  'new ObjectId': newObjectId(value),
})));

function newObjectId(value) {
  try {
    return new mongoose.Types.ObjectId(value);
  } catch (err) {
    return 'ERROR';
  }
}

[email protected] (same for @6.1.5) + [email protected]:

┌─────────┬───────────────────────────────────────┬─────────────────┬──────────────────┬──────────────────────────────────────────┐
│ (index) │                 test                  │ isValidObjectId │ ObjectId.isValid │               new ObjectId               │
├─────────┼───────────────────────────────────────┼─────────────────┼──────────────────┼──────────────────────────────────────────┤
│    0    │                'null'                 │      true       │      false       │ new ObjectId("61e117de7d6a06233f0cace6") │
│    1    │              'undefined'              │      true       │      false       │ new ObjectId("61e117de7d6a06233f0cace7") │
│    2    │           'new ObjectId()'            │      true       │       true       │ new ObjectId("61e117de7d6a06233f0cace0") │
│    3    │        'ObjectId().valueOf()'         │      true       │       true       │ new ObjectId("61e117de7d6a06233f0cace1") │
│    4    │        'ObjectId().toString()'        │      true       │       true       │ new ObjectId("61e117de7d6a06233f0cace2") │
│    5    │      'ObjectId().toHexString()'       │      true       │       true       │ new ObjectId("61e117de7d6a06233f0cace3") │
│    6    │           "'Z'.repeat(12)"            │      true       │       true       │ new ObjectId("5a5a5a5a5a5a5a5a5a5a5a5a") │
│    7    │           "'F'.repeat(24)"            │      true       │       true       │ new ObjectId("ffffffffffffffffffffffff") │
│    8    │     "'580e0797bb495f0a200e91ad'"      │      true       │       true       │ new ObjectId("580e0797bb495f0a200e91ad") │
│    9    │       '{ _id: new ObjectId() }'       │      true       │      false       │                 'ERROR'                  │
│   10    │       "{ _id: 'Z'.repeat(12) }"       │      true       │      false       │                 'ERROR'                  │
│   11    │       "{ _id: 'F'.repeat(24) }"       │      true       │      false       │                 'ERROR'                  │
│   12    │ "{ _id: '580e0797bb495f0a200e91ad' }" │      true       │      false       │                 'ERROR'                  │
│   13    │       '{ id: new ObjectId() }'        │      false      │      false       │                 'ERROR'                  │
│   14    │       "{ id: 'Z'.repeat(12) }"        │      false      │       true       │ new ObjectId("5a5a5a5a5a5a5a5a5a5a5a5a") │
│   15    │       "{ id: 'F'.repeat(24) }"        │      false      │       true       │ new ObjectId("ffffffffffffffffffffffff") │
│   16    │ "{ id: '580e0797bb495f0a200e91ad' }"  │      false      │       true       │ new ObjectId("580e0797bb495f0a200e91ad") │
│   17    │            '{ id: null }'             │      false      │      false       │                 'ERROR'                  │
│   18    │          '{ id: undefined }'          │      false      │      false       │                 'ERROR'                  │
│   19    │            '{ _id: null }'            │      false      │      false       │                 'ERROR'                  │
│   20    │         '{ _id: undefined }'          │      false      │      false       │                 'ERROR'                  │
└─────────┴───────────────────────────────────────┴─────────────────┴──────────────────┴──────────────────────────────────────────┘
$ npm ls bson
[email protected] /home/max/dev/mongoose616
└─┬ [email protected]
  ├── [email protected]
  └─┬ [email protected]
    └── [email protected] deduped

[email protected] + [email protected]

Note: [email protected] is demonstrated because since [email protected] { id: '580e0797bb495f0a200e91ad' } is considered a valid ObjectId. See #11209.
You can install this version today using version override in package.json:

  "overrides": {
    "bson": "4.6.0"
  }

┌─────────┬───────────────────────────────────────┬─────────────────┬──────────────────┬──────────────────────────────────────────┐
│ (index) │                 test                  │ isValidObjectId │ ObjectId.isValid │               new ObjectId               │
├─────────┼───────────────────────────────────────┼─────────────────┼──────────────────┼──────────────────────────────────────────┤
│    0    │                'null'                 │      true       │      false       │ new ObjectId("61e131d6bee71a108f689f2f") │
│    1    │              'undefined'              │      true       │      false       │ new ObjectId("61e131d6bee71a108f689f30") │
│    2    │           'new ObjectId()'            │      true       │       true       │ new ObjectId("61e131d6bee71a108f689f29") │
│    3    │        'ObjectId().valueOf()'         │      true       │       true       │ new ObjectId("61e131d6bee71a108f689f2a") │
│    4    │        'ObjectId().toString()'        │      true       │       true       │ new ObjectId("61e131d6bee71a108f689f2b") │
│    5    │      'ObjectId().toHexString()'       │      true       │       true       │ new ObjectId("61e131d6bee71a108f689f2c") │
│    6    │           "'Z'.repeat(12)"            │      true       │       true       │ new ObjectId("5a5a5a5a5a5a5a5a5a5a5a5a") │
│    7    │           "'F'.repeat(24)"            │      true       │       true       │ new ObjectId("ffffffffffffffffffffffff") │
│    8    │     "'580e0797bb495f0a200e91ad'"      │      true       │       true       │ new ObjectId("580e0797bb495f0a200e91ad") │
│    9    │       '{ _id: new ObjectId() }'       │      true       │      false       │                 'ERROR'                  │
│   10    │       "{ _id: 'Z'.repeat(12) }"       │      true       │      false       │                 'ERROR'                  │
│   11    │       "{ _id: 'F'.repeat(24) }"       │      true       │      false       │                 'ERROR'                  │
│   12    │ "{ _id: '580e0797bb495f0a200e91ad' }" │      true       │      false       │                 'ERROR'                  │
│   13    │       '{ id: new ObjectId() }'        │      false      │      false       │                 'ERROR'                  │
│   14    │       "{ id: 'Z'.repeat(12) }"        │      false      │      false       │ new ObjectId("5a5a5a5a5a5a5a5a5a5a5a5a") │
│   15    │       "{ id: 'F'.repeat(24) }"        │      false      │      false       │ new ObjectId("ffffffffffffffffffffffff") │
│   16    │ "{ id: '580e0797bb495f0a200e91ad' }"  │      false      │      false       │ new ObjectId("580e0797bb495f0a200e91ad") │
│   17    │            '{ id: null }'             │      false      │      false       │                 'ERROR'                  │
│   18    │          '{ id: undefined }'          │      false      │      false       │                 'ERROR'                  │
│   19    │            '{ _id: null }'            │      false      │      false       │                 'ERROR'                  │
│   20    │         '{ _id: undefined }'          │      false      │      false       │                 'ERROR'                  │
└─────────┴───────────────────────────────────────┴─────────────────┴──────────────────┴──────────────────────────────────────────┘
$ npm ls bson
[email protected] /home/max/dev/mongoose615bson460
└─┬ [email protected]
  ├── [email protected]
  └─┬ [email protected]
    └── [email protected] deduped

[email protected] + [email protected]

And some old versions (precisely from the codebase I am migrating now)

┌─────────┬───────────────────────────────────────┬─────────────────┬──────────────────┬──────────────────────────┐
│ (index) │                 test                  │ isValidObjectId │ ObjectId.isValid │       new ObjectId       │
├─────────┼───────────────────────────────────────┼─────────────────┼──────────────────┼──────────────────────────┤
│    0    │                'null'                 │      true       │      false       │ 61e13246a66c1202500f2c67 │
│    1    │              'undefined'              │      true       │      false       │ 61e13246a66c1202500f2c68 │
│    2    │           'new ObjectId()'            │      true       │       true       │ 61e13246a66c1202500f2c61 │
│    3    │        'ObjectId().valueOf()'         │      true       │       true       │ 61e13246a66c1202500f2c62 │
│    4    │        'ObjectId().toString()'        │      true       │       true       │ 61e13246a66c1202500f2c63 │
│    5    │      'ObjectId().toHexString()'       │      true       │       true       │ 61e13246a66c1202500f2c64 │
│    6    │           "'Z'.repeat(12)"            │      true       │       true       │ 5a5a5a5a5a5a5a5a5a5a5a5a │
│    7    │           "'F'.repeat(24)"            │      false      │       true       │ ffffffffffffffffffffffff │
│    8    │     "'580e0797bb495f0a200e91ad'"      │      true       │       true       │ 580e0797bb495f0a200e91ad │
│    9    │       '{ _id: new ObjectId() }'       │      true       │      false       │         'ERROR'          │
│   10    │       "{ _id: 'Z'.repeat(12) }"       │      true       │      false       │         'ERROR'          │
│   11    │       "{ _id: 'F'.repeat(24) }"       │      false      │      false       │         'ERROR'          │
│   12    │ "{ _id: '580e0797bb495f0a200e91ad' }" │      true       │      false       │         'ERROR'          │
│   13    │       '{ id: new ObjectId() }'        │      false      │      false       │         'ERROR'          │
│   14    │       "{ id: 'Z'.repeat(12) }"        │      false      │      false       │         'ERROR'          │
│   15    │       "{ id: 'F'.repeat(24) }"        │      false      │      false       │         'ERROR'          │
│   16    │ "{ id: '580e0797bb495f0a200e91ad' }"  │      false      │      false       │         'ERROR'          │
│   17    │            '{ id: null }'             │      false      │      false       │         'ERROR'          │
│   18    │          '{ id: undefined }'          │      false      │      false       │         'ERROR'          │
│   19    │            '{ _id: null }'            │      false      │      false       │         'ERROR'          │
│   20    │         '{ _id: undefined }'          │      false      │      false       │         'ERROR'          │
└─────────┴───────────────────────────────────────┴─────────────────┴──────────────────┴──────────────────────────┘
$ npm ls bson
[email protected] /home/max/dev/mongoose5123bson114
└─┬ [email protected]
  ├── [email protected] deduped
  └─┬ [email protected]
    └── [email protected] deduped

You can see how much has changed from those "old times".

Notable changes between versions

┌─────────┬───────────────────────────────────────┬─────────────────┬──────────────────┬──────────────────────────────────────────┐
│ (index) │                 test                  │ isValidObjectId │ ObjectId.isValid │               new ObjectId               │
├─────────┼───────────────────────────────────────┼─────────────────┼──────────────────┼──────────────────────────────────────────┤
[email protected] [email protected]:
│   16    │ "{ id: '580e0797bb495f0a200e91ad' }"  │      false      │       true       │ new ObjectId("580e0797bb495f0a200e91ad") │
[email protected] [email protected]:
│   16    │ "{ id: '580e0797bb495f0a200e91ad' }"  │      false      │      false       │ new ObjectId("580e0797bb495f0a200e91ad") │
[email protected] [email protected]:
│   16    │ "{ id: '580e0797bb495f0a200e91ad' }"  │      false      │      false       │         'ERROR'          │

┌─────────┬───────────────────────────────────────┬─────────────────┬──────────────────┬──────────────────────────────────────────┐
│ (index) │                 test                  │ isValidObjectId │ ObjectId.isValid │               new ObjectId               │
├─────────┼───────────────────────────────────────┼─────────────────┼──────────────────┼──────────────────────────────────────────┤
[email protected] [email protected], 4.6.1:
│    7    │           "'F'.repeat(24)"            │      true       │       true       │ new ObjectId("ffffffffffffffffffffffff") │
│   11    │       "{ _id: 'F'.repeat(24) }"       │      true       │      false       │                 'ERROR'                  │
[email protected] [email protected]:
│    7    │           "'F'.repeat(24)"            │      false      │       true       │ ffffffffffffffffffffffff │
│   11    │       "{ _id: 'F'.repeat(24) }"       │      false      │      false       │         'ERROR'          │

I can't help quoting a famous anecdote (translated):

A letter to the Balabanov Match Factory:
"I've been counting matches in your boxes for 11 years - there are 59, then 60, and sometimes 58. Are you all crazy there???"

MongoDB and mongoose seem to be mature products. And having a long history of changes/bugs with such base entity as ObjectId (regarding what is a valid ObjectId) - is appalling to say the least.

@vkarpov15 vkarpov15 added this to the 6.1.8 milestone Jan 16, 2022
@vkarpov15 vkarpov15 added priority Automatically set for Mongoose Pro subscribers and removed priority Automatically set for Mongoose Pro subscribers labels Jan 16, 2022
@vkarpov15 vkarpov15 modified the milestones: 6.1.8, 6.1.9 Jan 24, 2022
@vkarpov15
Copy link
Collaborator Author

See #11209 (comment). We made isValidObjectId() and isValid() consistent going forward. I don't see how the { id: <string> } change is breaking, but I'm open to hearing what issues it causes.

@Maximusya
Copy link

Would you prefer to continue in discussion #11221 or right here, in a closed issue?

@vkarpov15
Copy link
Collaborator Author

@Maximusya here is fine.

@Maximusya
Copy link

Well, new bugs and issues are being added as we speak.

Well, you state that We made isValidObjectId() and isValid() consistent going forward., but the change you delivered at 61d01fa is not what you stated.
With this change you've made isValidObjectId consistent with ObjectId constructor - not the ObjectId.isValid() check.

Here https://github.com/mongodb/js-bson/blob/master/src/objectid.ts#L295 in the first line if (id == null) return false nullish values are deemed INVALID, yet ObjectId ctor can happily accept such nullish value and the previous isValidObjectId implementation explicitly considered nullish value as VALID. There are no test of such values at test/index.test.js

Meanwhile in our business code we resorted to our own very narrow guard:

export const isValidObjectId = value => (
  value != null
  && (
    value instanceof mongoose.Types.ObjectId
    || (typeof value === 'string' && value.length === 24 && /^[0-9A-Fa-f]{24}$/.test(value))
  )
);

And as a precausion we added this set of tests to guard against future unexpected behavior (which just happened haha):

import mongoose from 'mongoose';
import { isValidObjectId } from '../app/validation';

describe('isValidObjectId', () => {
  const isValid = value => isValidObjectId(value);

  const valid = [
    new mongoose.Types.ObjectId(),
    '580e0797bb495f0a200e91ad',
    '580E0797BB495F0A200E91AD',
  ];
  const invald = [
    undefined,
    null,
    7,
    { },
    '7 chars',
    '12 charsssss',
    '15 charssssssss',
    '24 non-hex charsssssssss',
    { _id: new mongoose.Types.ObjectId() },
    { id: new mongoose.Types.ObjectId() },
    { _id: '580e0797bb495f0a200e91ad' },
    { id: '580e0797bb495f0a200e91ad' },
    { _id: '12 charsssss' },
    { id: '12 charsssss' },
    { _id: null },
    { id: null },
    { _id: undefined },
    { id: undefined },
    Buffer.alloc(3),
    Buffer.alloc(12),
    Buffer.alloc(15),
    { _id: Buffer.alloc(3) },
    { _id: Buffer.alloc(12) },
    { _id: Buffer.alloc(15) },
    { id: Buffer.alloc(3) },
    { id: Buffer.alloc(12) },
    { id: Buffer.alloc(15) },
  ];

  it('returns true for valid inputs', () => {
    valid.forEach(value => expect(value).toSatisfy(isValid));
  });

  it('returns false for invalid inputs', () => {
    invald.forEach(value => expect(value).not.toSatisfy(isValid));
  });
});

describe('mongoose.isValidObjectId', () => {
  const isValid = value => mongoose.isValidObjectId(value);

  const valid = [
    undefined,
    null,
    new mongoose.Types.ObjectId(),
    '580e0797bb495f0a200e91ad',
    '580E0797BB495F0A200E91AD',
    '12 charsssss',
    { _id: new mongoose.Types.ObjectId() },
    { _id: '580e0797bb495f0a200e91ad' },
    { _id: '12 charsssss' },
  ];
  const invald = [
    { },
    7,
    '7 chars',
    '15 charssssssss',
    { id: new mongoose.Types.ObjectId() },
    { id: '580e0797bb495f0a200e91ad' },
    { id: '12 charsssss' },
    { _id: null },
    { id: null },
    { _id: undefined },
    { id: undefined },
    Buffer.alloc(3),
    Buffer.alloc(12),
    Buffer.alloc(15),
    { _id: Buffer.alloc(3) },
    { _id: Buffer.alloc(12) },
    { _id: Buffer.alloc(15) },
    { id: Buffer.alloc(3) },
    { id: Buffer.alloc(12) },
    { id: Buffer.alloc(15) },
  ];

  it('returns true for valid inputs', () => {
    valid.forEach(value => expect(value).toSatisfy(isValid));
  });

  it('returns false for invalid inputs', () => {
    invald.forEach(value => expect(value).not.toSatisfy(isValid));
  });
});

describe('mongoose.Types.ObjectId.isValid', () => {
  const isValid = value => mongoose.Types.ObjectId.isValid(value);

  const valid = [
    7,
    new mongoose.Types.ObjectId(),
    '580e0797bb495f0a200e91ad',
    '580E0797BB495F0A200E91AD',
    '12 charsssss',
    { id: '580e0797bb495f0a200e91ad' },
    { id: '12 charsssss' },
    Buffer.alloc(12),
    { id: Buffer.alloc(12) },
  ];
  const invald = [
    undefined,
    null,
    { },
    '7 chars',
    '15 charssssssss',
    { _id: new mongoose.Types.ObjectId() },
    { id: new mongoose.Types.ObjectId() },
    { _id: '580e0797bb495f0a200e91ad' },
    { _id: '12 charsssss' },
    { _id: null },
    { id: null },
    { _id: undefined },
    { id: undefined },
    Buffer.alloc(3),
    Buffer.alloc(15),
    { _id: Buffer.alloc(3) },
    { _id: Buffer.alloc(12) },
    { _id: Buffer.alloc(15) },
    { id: Buffer.alloc(3) },
    { id: Buffer.alloc(15) },
  ];

  it('returns true for valid inputs', () => {
    valid.forEach(value => expect(value).toSatisfy(isValid));
  });

  it('returns false for invalid inputs', () => {
    invald.forEach(value => expect(value).not.toSatisfy(isValid));
  });
});

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants