Web build performance: Tracking issue

[valerybugakov]

Description

To improve the front-end developer experience, we'd like to focus on improving build times for Storybook and the main web app bundle.

Tracked issues

@valerybugakov

• https://github.com/sourcegraph/sourcegraph/issues/22261 (PRs: #22580)
• https://github.com/sourcegraph/sourcegraph/issues/22312 (PRs: #22395)

Completed

• (🏁 905 days ago) https://github.com/sourcegraph/sourcegraph/issues/19555
• (🏁 587 days ago) https://github.com/sourcegraph/sourcegraph/issues/27714
• (🏁 587 days ago) https://github.com/sourcegraph/sourcegraph/issues/27702
• (🏁 587 days ago) https://github.com/sourcegraph/sourcegraph/issues/27682
• (🏁 561 days ago) https://github.com/sourcegraph/sourcegraph/issues/27354

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🎩 Quality of life
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• Pull Request

[github-actions]

Heads up @AlicjaSuska @umpox @valerybugakov @5h1rU @pdubroy - the "team/frontend-platform" label was applied to this issue.

[umpox]

Adding some general thoughts:

• Thought: It'd be great to be able to use esbuild and I think we'd see massive performance gains, but we might have a reliance on certain Webpack plugins that will not be supported through esbuild
• Note: We have a branch that integrates esbuild-loader for use instead of terser and babel. Findings: faster build perf, slightly bigger bundle size, no hot reloading (not easily supported afaik). Still could use Webpack plugins
• Thought: I know there have been a few discussions over using native ES modules through Vite, I believe this would require us to migrate to using Rollup for our builds. If we decide to go down this route, it might be worth investigating using Snowpack which I believe should still support our existing Webpack config (and plugins)

[valerybugakov]

The current state

For future reference based on the Slack discussion.

Did we try/consider esbuild?

Yes, @sourcegraph/frontend-platform considered Esbuild and tried using esbuild-loader in our Webpack config.

Esbuild issues

Esbuild doesn't support hot-module-replacement yet. This is one of the main blockers. Esbuild performance is awesome, but it's important for DX to bypass page load time and API requests when working on the web app UI. Ryan Florence left a great comment about it. It seems that Vite got the best from both worlds but still has many small inconveniences compared to Webpack (based on the GH issues and hacker news discussions), which are the result of an immature plugins ecosystem.

Another important consideration: apart from the web app, we have Storybook and browser extension powered by Webpack. We want to keep one tool for all our web applications because it results in a predictable behavior, and it's easier to support one instrument. Storybook just started working on the experimental API for custom builders like Vite but doesn't support raw Esbuild yet.

Last but not least, complete migration requires quite a bit of time. We didn't have a chance to explore all the available options yet properly. Even Webpack 5 upgrade got stuck twice because of various issues. As @sourcegraph/frontend-platform, we allocated time in Q3 to dig into available build tools and investigate tradeoffs properly.

Webpack improvements overview

Webpack 5 brings persistent caching, which significantly improves build performance and recompilation. With additional tweaks like DLL Plugin, it can be improved further. Also, we can consider module federation to avoid building the whole web application in development but split it up into multiple sub-applications via Webpack 5 plugins.

Current direction

Keeping Webpack (at least for now) seems like a more stable/simple way of improving our web build performance. We're definitely keeping an eye on Esbuild and other modern build tools available. Looking forward to exploring them properly in Q3.

[sqs]

@valerybugakov Thank you! This is very helpful to understand. I personally found the slow recompilation so bad that I prefer fast esbuild to anything else, even if that means lack of HMR, consistency (w/storybook), etc. Maybe it's just worse on my machine, though? It takes ~5-10s to recompile and reload, and I'm on an AMD Epyc 64-core workstation with 128GB RAM.

Also, the benefits of speeding up our frontend dev seem so large that it makes sense to prioritize them really highly above other frontend work. If you agree but weren't able to convince others of the importance of this kind of work, I'd be curious to hear that.

Anyway, with all this said, I trust you to make the right call here, I just want to give my input as a drive-by dev (not as CEO).

[valerybugakov]

@sqs, definitely, we see the issue with recompilation. Webpack 5 upgrade improves JS recompilation and multiple other things. It's in the main branch now. Check it out! Recompilation time will be improved further by the DLL plugin integration into the web app.

SCSS recompilation is slow too. Right now, ~70% of stylesheets are combined into one giant file, which means that changing one rule results in bundling it again from scratch. This aspect will be addressed by completely migrating to CSS modules, which will allow rebuilding SCSS files independently from each other.

I agree that speeding our frontend dev tooling is important; that's why we allocated a significant time to such improvements on our roadmap and landed a couple of important ones already. After completing current goals, we will revisit modern build tools alternatives once again.

[github-actions]

Heads up @pdubroy - the "team/frontend-platform" label was applied to this issue.