☂️ Syntax highlighter improvement ideas

[varungandhi-src]

This is an umbrella issue meant for collecting actionable ideas for improving the performance, reliability and debuggability of syntax highlighter. We can spin off individual points into separate issues which can be tracked.

Historical context

• RFC on maintenance of syntect-based highlighting code (Feb 2022)
• Performance report for syntax highlighting (Jan 2022)
• Description of tech debt in the highlighter (June 2021) (some of it has been fixed since then)

Development velocity

• https://github.com/sourcegraph/sourcegraph/issues/60690
• https://github.com/sourcegraph/sourcegraph/issues/60769

Observability

• Improved logging: Right now, the code in http-server-stabilizer has some logging, but the log messages aren't terribly useful. The Rust code has minimal logging, mostly during startup. We should be able to reuse the same log level infrastructure via env variables or similar so that we can enable more detailed logging in the Rust code. Also, we should add log statements at key points in the Rust code.
• Tracing support: Right now, we don't have visibility into what is taking more time in the Rust code (parsing? generating SCIP data? something else?). We already have some traces in Honeycomb measuring highlighting latency, it would be good to have Rust spans in those traces.

Reliability

• Root cause Incident 258: It's unclear why the server started crashing under moderate load, we should investigate and fix that. It's unclear if we're applying backpressure correctly or not; if not, we should consider updating Rocket to a newer version or replacing it with a more popular server.
• Fuzz testing: For file formats that are common + tend to be large + we're using syntect, let's fuzz the grammars to see if we can identify pathological inputs and fix the corresponding grammars (or move to Tree-sitter grammars for those languages, since Syntect still uses Sublime 3 grammars which are not updated anymore)
• Better handling for timeouts: Since we have our own fork of syntect, it might make sense to modify some of the APIs to explicitly check for timeouts and return early.
• https://github.com/sourcegraph/sourcegraph/issues/60558
• https://github.com/sourcegraph/sourcegraph/issues/62689

Performance

• Load testing: We should stress test the syntax highlighter in the standard configuration with http-server-stabilizer to measure performance as a function of RPS. In the past, we've done some testing with synhtml, which is a CLI tool that generates HTML directly using standard colors, but we should attempt to exercise the code paths similar to production to actually measure perf better.
• Investigate migration to Tree-sitter grammars for more languages.

Dev / Ops documentation

Document the following:

• Overall architecture
• Links to various tools with data about syntax highlighting (Grafana, Honeycomb etc.)
• Typical performance characteristics
• Typical resource consumption patterns
• Knobs for tuning performance/scaling