Add builtin optimize/bundle support (powered by esbuild!) #1615

FredKSchott · 2020-11-15T23:04:11Z

Background

Snowpack has always supported bundling via a third-party plugin. We have plugins for Webpack and (thanks to @ParamagicDev) Rollup.
@drwpow has been experimenting with new, interesting unbundled optimizations in https://www.npmjs.com/package/@snowpack/plugin-optimize, like adding preload tags to flatten deep import graphs and extracting/bundling CSS from your app without bundling the JS.
For a while now, users have been asking for bundling/optimizing to be built in to Snowpack by default.
We created Built-in production optimizations (incl. bundling) #1276 to flesh this out.

Changes

Resolves Built-in production optimizations (incl. bundling) #1276 - Adds a built-in optimize pipeline, powered by esbuild!
Adds a new experiments.optimize config that implements the following options:

experiments.optimize: {
  /** Your application file entrypoints: HTML if they exist in your build, but JS also supported for Rails, server-side, etc. */
  entrypoints: 'auto' | string[] | (({files: string[]}) => string[]);
  /** Add preload tags to flatten deep import waterfalls (entrypoints must be HTML) */
  preload: boolean;
  /** Bundle your application JS & CSS */
  bundle: boolean; 
  /** Minify the final build (JS, CSS, and HTML (todo)) */
  minify: boolean;
  /** Transpile the final build to a specific target, for older browsers */
  target: 'es2020' | 'es2019' | 'es2018' | 'es2017';
  /** Add a build-manifest.json file to your final build */
  manifest: boolean;
}

This support will be experimental! It will hopefully leave the experiments scope as a part of v3.0.
This isn't replacing our Webpack or Rollup plugins: this will still be experimental as esbuild continues to undergo development. If you need something more stable, we'll still recommended more stable bundlers.
Some of the more aspirational features of Built-in production optimizations (incl. bundling) #1276 are left out of this PR, for future enhancements.

`snowpack build` Results

Before (2.9sec load over 3G)

`preload: true` (2.3sec load over 3G, no observable impact on build speed)

(Notice the waterfall is gone)

`bundle: true` (2.0sec load over 3G, no observable impact on build speed)

Thoughts on esbuild

TODO: want to write out some thoughts here about betting on esbuild for our first class bundling support (vs. rollup or webpack)

Testing

TODO: Want to get some early feedback before testing.

Docs

TODO

vercel · 2020-11-15T23:04:13Z

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/pikapkg/snowpack/5d62dq1c8
✅ Preview: https://snowpack-git-wip-optimize.pikapkg.vercel.app

FredKSchott · 2020-11-16T00:28:44Z

Snowpack builds all assets (including npm packages) to ESM JS ahead of time, so we take care of a lot of the ecosystem weirdness before the code ever gets to esbuild. We only need esbuild to bundle valid ESM (all import URLs fully resolved) and CSS. Only the js and css loaders are needed/used on our end.

@evanw would love to get your thoughts on this & if you think esbuild is ready for this, stability- & timing-wise. This would land in Snowpack as an experiment at first but the plan would be to fully support in Snowpack v3.0 in a couple of months.

I have some feedback from the integration that I'd love to share as well, will create a couple follow up issues on your repo tonight/tomorrow.

evanw · 2020-11-16T02:43:54Z

@evanw would love to get your thoughts on this & if you think esbuild is ready for this, stability- & timing-wise. This would land in Snowpack as an experiment at first but the plan would be to fully support in Snowpack v3.0 in a couple of months.

Here's what I'd call out:

The JS bundler has matured a lot and should be good to use for something like this.
CSS support is much newer and still a work in progress. Basic use of @import to join CSS files together should be fine though, and should continue to work. The CSS/JS integration points are primitive and buggy and will change in the future, but it sounds like you don't need those? There also isn't yet support for down-leveling CSS to the target browser environment but I'm guessing you don't need that either. What do you need out of esbuild for CSS, if anything?
If you're feeling conservative, I'd recommend pinning the entire version of esbuild to make sure nothing breaks unless you explicitly upgrade esbuild and it passes your test suite. Otherwise please pin both the major and minor versions so that just the patch version varies. I'm using minor versions for breaking API changes because esbuild is pre-1.0.0.
Code splitting is still a work in progress. I wouldn't rely on it yet. It only works for ESM output and there is a known ordering bug with cross-chunk imports that I haven't fixed yet.
I'm sure you're aware of this but just in case: esbuild can't yet lower most ES6+ syntax to ES5. So you'll either have to post-process or pre-process that syntax away yourself.

I have some feedback from the integration that I'd love to share as well, will create a couple follow up issues on your repo tonight/tomorrow.

Sounds great! Thanks.

FredKSchott · 2020-11-16T07:16:15Z

Sounds good to all of the above! Appreciate the response.

What do you need out of esbuild for CSS, if anything?

I could have clarified: we do allow import './foo.css' so there is an interop between JS & CSS that we'll need to rely on. No need for anything more than what's supported today, but if there are any relevant issues to keep an eye on please lmk.

Code splitting is still a work in progress.

Ah right, I saw a mention of that. Will keep an eye out, but yup that's fine for now given the experimental status of things on our end. Webpack & Rollup plugins will still exist for more production-ready use.

lewisl9029 · 2020-11-16T23:26:03Z

I have no opinion on the bundling feature itself as I generally prefer the unbundled preloading approach, but I noticed in the preload benchmark you have some gray bars in the network inspector, which probably means you're running into the tcp connection limit and are not on http2.

If you run the same test using http2 I think the preload version would match/beat the bundled version (my bet is on beat 🙂).

FredKSchott · 2020-11-17T05:48:11Z

you're right! I just spun that up on local, we should replace with some real-world examples (H2/H3 + more files).

drwpow · 2020-11-18T20:08:35Z

snowpack/package.json

    "cacache": "^15.0.0",
    "cachedir": "^2.3.0",
+    "cheerio": "^1.0.0-rc.3",


Loose suggestion: if you‘re only using cheerio to find specific elements like <script> or <link>, hypertag works well for this (and even parses attributes). It could be a faster & much lighter-weight alternative. But if you need cheerio for something heavier in the future, it‘s fine

yea, the only reason I switched it out for cheerio was because we do a lot more with it. You can see in the file we're querying, modifying, adding and removing different elements as we optimize.

drwpow · 2020-11-18T20:25:27Z

snowpack/src/build/optimize.ts

+/**
+ * Given any set of user-input as entrypoints,
+ */
+async function resolveEntrypoints(


More of a DX observation than anything: for my project, I have src/index.js. So in my config, I specified entrypoints: ['./src/index.js']. But it threw a “not found“ error. I guess I have to specify ./_dist_/index.js instead here?

Ah I see—it wants HTML entrypoints. Hm, similar thing, though—if I specify ['./public/index.html'] or ['./index.html'] both throw a “not found“ error, so I’m not sure how to use this

Error: ENOENT: no such file or directory, open './public/index.html'

It should work with both JS & HTML entrypoints, but it looks like there's some bug here. since the 'auto' scanner returns absolute paths, i guess it needs absolute file paths to the final build/ output files? Either way, will fix to support relative files 👍

It should work with both JS & HTML entrypoints, but it looks like there's some bug here. since the 'auto' scanner returns absolute paths, i guess it needs absolute file paths to the final build/ output files? Either way, will fix to support relative files 👍

Gotcha. Yeah if I specify .js I get a “only HTML is supported” message right now

Also: need to fix the "seeing modulepreloads when only trying to bundle" issue

drwpow · 2020-11-18T20:38:39Z

In this PR we aren‘t pulling styles out of css.proxy.js files anymore, which could lead to FOUC and just a little bit of delayed TTI (not significant for small sites, but might be with enough CSS files). What are your thoughts on that issue?

drwpow · 2020-11-18T20:39:09Z

snowpack/src/build/optimize.ts

+
+interface ESBuildMetaInput {
+  bytes: number;
+  imports: {path: string}[];


Love this build manifest format 👍🏻 . Seeing the imports and size is really cool

drwpow · 2020-11-18T20:40:47Z

snowpack/src/build/optimize.ts

+    platform: 'browser',
+    metafile: path.join(config.buildOptions.out, 'build-manifest.json'),
+    publicPath: config.buildOptions.baseUrl,
+    minify: config.experiments.optimize!.minify,


Just a note: I’m seeing minify work for JS and CSS, but not HTML. Is that intentional?

intentional for this PR, but agreed that we should tackle eventually. My hope is that esbuild supports HTML one day, and then we'd get it for free. But, until then we may want to ship some popular html minifier.

I couldn't find one on NPM that I really liked, @drwpow any favorites that you've used in the past?

Not really. Fine for waiting to see if esbuild supports it as you said

drwpow · 2020-11-18T20:43:44Z

snowpack/src/build/optimize.ts

+    return allBuildFiles.filter((f) => f.endsWith('.html'));
+  }
+  if (Array.isArray(entrypoints)) {
+    return entrypoints;


WDYT about something like this?

if (Array.isArray(entrypoints)) { - return entrypoints; + return entrypoints.map(file => file.startsWith(path.sep) ? file : path.join(cwd, file))

Basically, use an absolute path if provided by the user, or assume it‘s relative to the project directory (see above comment for more context)

Done, added handling for relative paths to support both final build directory and source directory.

FredKSchott force-pushed the wip-optimize branch from fa8be4c to 352f6c7 Compare November 15, 2020 23:36

vercel bot deployed to Preview November 15, 2020 23:36 View deployment

FredKSchott force-pushed the wip-optimize branch from 352f6c7 to f66da96 Compare November 15, 2020 23:49

vercel bot deployed to Preview November 15, 2020 23:49 View deployment

FredKSchott mentioned this pull request Nov 16, 2020

Better support for github pages #1517

Closed

FredKSchott force-pushed the wip-optimize branch from f66da96 to 1555aec Compare November 16, 2020 07:22

vercel bot deployed to Preview November 16, 2020 07:23 View deployment

FredKSchott mentioned this pull request Nov 17, 2020

remove seperate sass, postcss build pipelines #1629

Merged

drwpow reviewed Nov 18, 2020

View reviewed changes

drwpow approved these changes Nov 18, 2020

View reviewed changes

FredKSchott changed the base branch from master to main November 18, 2020 23:20

FredKSchott force-pushed the wip-optimize branch from 1555aec to 80219af Compare November 19, 2020 00:20

vercel bot deployed to Preview November 19, 2020 00:20 View deployment

FredKSchott force-pushed the wip-optimize branch from 80219af to 0dc1e6c Compare November 19, 2020 01:03

vercel bot deployed to Preview November 19, 2020 01:03 View deployment

FredKSchott force-pushed the wip-optimize branch from 0dc1e6c to 76da9d9 Compare November 19, 2020 01:04

vercel bot deployed to Preview November 19, 2020 01:04 View deployment

FredKSchott force-pushed the wip-optimize branch from 76da9d9 to a5339f0 Compare November 19, 2020 01:06

vercel bot deployed to Preview November 19, 2020 01:06 View deployment

FredKSchott force-pushed the wip-optimize branch from a5339f0 to d0243a8 Compare November 19, 2020 01:18

vercel bot deployed to Preview November 19, 2020 01:18 View deployment

FredKSchott force-pushed the wip-optimize branch from d0243a8 to 2a8e9db Compare November 19, 2020 01:21

vercel bot deployed to Preview November 19, 2020 01:21 View deployment

add builtin optimize support

cea48ab

FredKSchott force-pushed the wip-optimize branch from 2a8e9db to cea48ab Compare November 19, 2020 02:15

vercel bot deployed to Preview November 19, 2020 02:15 View deployment

FredKSchott merged commit 7ea9709 into main Nov 19, 2020

FredKSchott deleted the wip-optimize branch November 19, 2020 02:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add builtin optimize/bundle support (powered by esbuild!) #1615

Add builtin optimize/bundle support (powered by esbuild!) #1615

FredKSchott commented Nov 15, 2020 •

edited

Loading

vercel bot commented Nov 15, 2020 •

edited

Loading

FredKSchott commented Nov 16, 2020

evanw commented Nov 16, 2020

FredKSchott commented Nov 16, 2020 •

edited

Loading

lewisl9029 commented Nov 16, 2020

FredKSchott commented Nov 17, 2020

drwpow Nov 18, 2020

FredKSchott Nov 18, 2020

drwpow Nov 18, 2020

drwpow Nov 18, 2020 •

edited

Loading

FredKSchott Nov 18, 2020

drwpow Nov 18, 2020

FredKSchott Nov 18, 2020

drwpow commented Nov 18, 2020

drwpow Nov 18, 2020

drwpow Nov 18, 2020

FredKSchott Nov 18, 2020

drwpow Nov 19, 2020

drwpow Nov 18, 2020

FredKSchott Nov 19, 2020

Add builtin optimize/bundle support (powered by esbuild!) #1615

Add builtin optimize/bundle support (powered by esbuild!) #1615

Conversation

FredKSchott commented Nov 15, 2020 • edited Loading

Background

Changes

snowpack build Results

Before (2.9sec load over 3G)

preload: true (2.3sec load over 3G, no observable impact on build speed)

bundle: true (2.0sec load over 3G, no observable impact on build speed)

Thoughts on esbuild

Testing

Docs

vercel bot commented Nov 15, 2020 • edited Loading

FredKSchott commented Nov 16, 2020

evanw commented Nov 16, 2020

FredKSchott commented Nov 16, 2020 • edited Loading

lewisl9029 commented Nov 16, 2020

FredKSchott commented Nov 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drwpow Nov 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drwpow commented Nov 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FredKSchott commented Nov 15, 2020 •

edited

Loading

`snowpack build` Results

`preload: true` (2.3sec load over 3G, no observable impact on build speed)

`bundle: true` (2.0sec load over 3G, no observable impact on build speed)

vercel bot commented Nov 15, 2020 •

edited

Loading

FredKSchott commented Nov 16, 2020 •

edited

Loading

drwpow Nov 18, 2020 •

edited

Loading