Feature request: Improve performance when compiling enormous files

[navndn]

This is probably a bug, compilation hangs for HL7 FHIR Json-schema.

FHIR schema is quite large, around 60k lines

Official link to download fhir.schema.json :
https://www.hl7.org/fhir/fhir.schema.json.zip

It hangs on both Windows & Ubuntu.
Any clue on this?

CLI:
json2ts -i fhir.schema.json -o fhir.d.ts

No luck with program too,

const schema = JSON.parse(
  fs.readFileSync(path.resolve(__dirname, 'fhir.schema.json'), 'utf8')
);

compile(schema, 'FHIRSchema')
  .then(ts => fs.writeFileSync('fhir.d.ts', ts))
  .catch(console.error);

There is already a related issue #222 , but the original tittle doesn't specify HL7 FHIR schema.

[saulide]

Any update? Due to COVID, this is kind of helpfull

[dinbtechit]

Having same issue here! unable to generate the .ts files. Is it possible to break this into smaller files?

[bcherny]

3.3mb it pretty hefty. 600kb -- the biggest test case so far -- seems to work fine. Even when disabling formatting (running Prettier is by far the slowest part of compilation), JSTT hangs when run as part of a test case.

Can you try disabling formatting --format=false and trying again?

If it still hangs: I merged a test case for this issue, and disabled it for now. Whoever wants to take a shot at speeding up the compiler:

1. Clone this repo

2. Rename /test/e2e/realWorld.fhir.ignore.ts to test/e2e/realWorld.fhir.ts

3. Edit test/e2e/realWorld.fhir.ts, and add the following line to tell the test runner to only run this file:

   export const only = true

4. Edit src/parser.ts, and comment out the couple of log() calls to avoid terminal spam

5. Run the tests: yarn test (or, VERBOSE=true yarn test to also get perf telemetry)

[bcherny]

Hey @navndn @saulide @dinbtechit, mind trying again with the latest version (v10.1.2)? @G-Rath just landed a big perf improvement in #361.

[G-Rath]

I had a play around with this locally, and it seems the issue is with the amount of dereferencing that needs to be done:

(I killed the process after it'd been running for around 3 minutes).

Our schema.json (that is now compiling in ~3 seconds instead of ~120 seconds) is 1.1mb in size.

[bcherny]

@G-Rath If that's right, could be worth filing an issue for json-schema-ref-parser?

[G-Rath]

Yeah, probably - I'll look to do that if I have the time and a good reproduction case (I don't mind doing it, but also I've not got a lot context on the underlying issue other than hacking around, so might not be able to dedicate enough time to the issue to provide the info that's required for the maintainers).

I hacked memorize into crawl and that seems to unblock that phase of the pipeline - parsing then takes a couple of seconds but is very fast (probably because of the new memorization happening there).

So then the blocker becomes optimize - this schema results in a union of 146 members, which means lodash.uniqBy has to do a lot of work via stringify:

I'm toying around with the code in those areas to see if they can be improved with caching, but it's really not looking promising :/

[bcherny]

@G-Rath Thanks for digging in, and for documenting what you're finding. Even if you don't manage to get to the bottom of it, these will be great breadcrumbs for the next person to dig in!

[G-Rath]

Apparently there's going to be a new kid on the block that'll replace json-schema-ref-parser so will have to wait for that to come out before making an issue to improve performance, as it sounds like json-schema-ref-parser is in maintenance mode.

[bcherny]

Published as part of v11.0.0.