bug/panic caused by Vue.js syntax definition

[emidoots]

When trying to highlight Vue.js files using the same syntax file that works well in Sublime, Syntect panics:

thread '<unnamed>' panicked at 'begin <= end (13 <= 7) when slicing `<style lang="stylus" scoped>`', libcore/str/mod.rs:2098:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:511
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:426
   6: rust_begin_unwind
             at libstd/panicking.rs:337
   7: core::panicking::panic_fmt
             at libcore/panicking.rs:92
   8: core::str::slice_error_fail
             at libcore/str/mod.rs:0
   9: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::Range<usize>>::index::{{closure}}
  10: <syntect::highlighting::highlighter::HighlightIterator<'a, 'b> as core::iter::iterator::Iterator>::next
  11: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::from_iter
  12: syntect::html::highlighted_snippet_for_string
  13: <std::thread::local::LocalKey<
T>>::with
  14: syntect_server::rocket_route_fn_index
  15: rocket::rocket::Rocket::dispatch
  16: <rocket::rocket::Rocket as hyper::server::Handler>::handle
  17: <hyper::server::Worker<H>>::handle_connection
  18: hyper::server::listener::spawn_with::{{closure}}

(only the bits above syntect::html::highlighted_snippet_for_string are relevant)

I believe the above is pointing to

syntect/src/highlighting/highlighter.rs

Line 119 in 231eafc

let text = &self.text[self.pos..end];

as the culprit, but I haven't investigated what would cause this.

The syntax file I am using is here: vue.sublime-syntax (tmLanguage file here)

This file produces the panic (but highlights fine in ST3 with the same syntax file): https://raw.githubusercontent.com/vuejs/vue-cli/0ba111eed859aad02156e807ef71d0052b2e7f17/packages/%40vue/cli-ui-addon-webpack/src/components/ModuleListItem.vue

This file highlights fine in both Syntect and ST3 with the same syntax file: https://raw.githubusercontent.com/vuejs/vue-cli/0ba111eed859aad02156e807ef71d0052b2e7f17/packages/%40vue/cli-ui-addon-webpack/src/components/ModuleList.vue

[keith-hall]

have you tried with the sublime-syntax from Vue's new branch? https://github.com/vuejs/vue-syntax-highlight/blob/new/Vue%20Component.sublime-syntax

[emidoots]

@keith-hall Yes, however that produces a different error (which I admittedly haven't looked into very far).

Regardless, the syntax mentioned in the issue description does work in Sublime but not in Syntect, so it must be a bug if we aim for 1:1 compatibility, at least.

I'll try with the new branch again soon here

[keith-hall]

Regardless, the syntax mentioned in the issue description does work in Sublime but not in Syntect, so it must be a bug if we aim for 1:1 compatibility, at least.

oh I completely agree, I just saw that there was a more modern syntax that you could try in the interim, until the bug gets fixed :)

[emidoots]

makes perfect sense :) I will dig further soon and let you know what I find!

[emidoots]

have you tried with the sublime-syntax from Vue's new branch? [...]

Good news! I've just tried with this again, and the error I initially saw:

thread '<unnamed>' panicked at 'Can only call resolve on linked references: ByScope { scope: <source.stylus>, sub_context: None }', /Users/stephen/drive/godev/src/github.com/slimsag/syntect/src/parsing/syntax_definition.rs:205:18

It turned out I misunderstood this (I thought it was referencing the yaml macros), and in reality it was just complaining about referencing the Stylus syntax which I needed to convert and add. So the new branch here does indeed work with Syntect.

---

Of course, this is still a valid issue for the reason I mentioned before (1:1 compatibility with Sublime)

[sharkdp]

A similar (the same?) bug was reported here for OCaml files. This can be reproduced with syncat and the current master of syntect:

test.ml

let line_count filename =
   let f = open_in filename in
   let rec loop count =
     match try Some (input_line f) with End_of_file -> None
     with
       | Some(_) -> loop (count+1)
       | None -> count in
   loop 0

or minimal.ml:

with a -> b

The stack trace is the same as above:

▶ RUST_BACKTRACE=1 cargo run --example syncat --  /home/shark/Dropbox/Informatik/rust/bat-test/errors/test.ml
let line_count filename =
   let f = open_in filename in
   let rec loop count =
thread 'main' panicked at 'begin <= end (54 <= 40) when slicing `     match try Some (input_line f) with End_of_file -> None
`', libcore/str/mod.rs:2095:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:475
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:390
   6: rust_begin_unwind
             at libstd/panicking.rs:325
   7: core::panicking::panic_fmt
             at libcore/panicking.rs:77
   8: core::str::slice_error_fail
             at libcore/str/mod.rs:0
   9: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::Range<usize>>::index::{{closure}}
             at /checkout/src/libcore/str/mod.rs:1891
  10: <core::option::Option<T>>::unwrap_or_else
             at /checkout/src/libcore/option.rs:386
  11: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::Range<usize>>::index
             at /checkout/src/libcore/str/mod.rs:1891
  12: core::str::traits::<impl core::ops::index::Index<core::ops::range::Range<usize>> for str>::index
             at /checkout/src/libcore/str/mod.rs:1667
  13: <syntect::highlighting::highlighter::HighlightIterator<'a, 'b> as core::iter::iterator::Iterator>::next
             at src/highlighting/highlighter.rs:123
  14: <alloc::vec::Vec<T>>::extend_desugared
             at /checkout/src/liballoc/vec.rs:1969
  15: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::spec_extend
             at /checkout/src/liballoc/vec.rs:1866
  16: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::from_iter
             at /checkout/src/liballoc/vec.rs:1861
  17: <alloc::vec::Vec<T> as core::iter::traits::FromIterator<T>>::from_iter
             at /checkout/src/liballoc/vec.rs:1761
  18: core::iter::iterator::Iterator::collect
             at /checkout/src/libcore/iter/iterator.rs:1415
  19: syntect::easy::HighlightLines::highlight
             at src/easy.rs:65
  20: syncat::main
             at examples/syncat.rs:112
  21: std::rt::lang_start::{{closure}}
             at /checkout/src/libstd/rt.rs:74
  22: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:310
  23: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105
  24: std::rt::lang_start_internal
             at libstd/panicking.rs:289
             at libstd/panic.rs:392
             at libstd/rt.rs:58
  25: std::rt::lang_start
             at /checkout/src/libstd/rt.rs:74
  26: main
  27: __libc_start_main
  28: _start

[sharkdp]

I managed to create this minimal (?) example that triggers the same error:

Consider this Sublime Syntax:

%YAML 1.2
---
# http://www.sublimetext.com/docs/3/syntax.html
name: Dummy
file_extensions:
  - dummy
scope: source.dummy
contexts:
  main:
    - match: (?=(foo))
      captures:
        1: keyword
      push:
        - match: (foo)
          captures:
            1: keyword

and the following test.dummy file:

foo

This will cause syntect to panic with:

thread 'main' panicked at 'begin <= end (3 <= 0) when slicing `foo

This is the content of HighlightIterators changes field:

[(0, Push(<source.dummy>)), (0, Push(<keyword>)), (3, Pop(1)), (0, Push(<keyword>)), (3, Pop(1))]

I don't understand enough about the internals of syntect and sublime syntax definitions, but I believe this is somehow related to the negative lookahead in the (?=(foo)) regex. For some reason, this causes syntect to create the additional (0, Push(<keyword>)) entry.

If someone can give a hint / point me in the right direction, I can try to look into this.

[keith-hall]

I would guess the problem relates to Oniguruma returning the actual positions of the captured groups inside lookarounds - Sublime Text just ignores them (doesn't scope any text from a capture group inside a lookaround). It may be easier (if Oniguruma doesn't expose this information for us to use at

syntect/src/parsing/parser.rs

Lines 464 to 482 in 703e679

	// captures could appear in an arbitrary order, have to produce ops in right order
	// ex: ((bob)|(hi))* could match hibob in wrong order, and outer has to push first
	// we don't have to handle a capture matching multiple times, Sublime doesn't
	let mut map: Vec<((usize, i32), ScopeStackOp)> = Vec::new();
	for &(cap_index, ref scopes) in capture_map.iter() {
	if let Some((cap_start, cap_end)) = reg_match.regions.pos(cap_index) {
	// marking up empty captures causes pops to be sorted wrong
	if cap_start == cap_end {
	continue;
	}
	// println!("capture {:?} at {:?}-{:?}", scopes[0], cap_start, cap_end);
	for scope in scopes.iter() {
	map.push(((cap_start, -((cap_end - cap_start) as i32)),
	ScopeStackOp::Push(*scope)));
	}
	map.push(((cap_end, i32::MIN), ScopeStackOp::Pop(scopes.len())));
	}
	}
	map.sort_by(|a, b| a.0.cmp(&b.0));

) to just rewrite all regex patterns that contain captures in lookarounds to change them to non capture groups, but then it may be necessary to fixup the numbering of the scopes (captures: in the YAML) that should apply to the remaining matched groups, and I guess it wouldn't work if the capture group was used in a backreference... Perhaps just removing duplicate operations would help the concrete case given above?

[sharkdp]

@trishume (Tentatively coming back to a kind offer of you regarding requests like this): It would be really great to have a bugfix release of syntect with the PR by @keith-hall included - just in case you find the time. Most issues are easy to workaround somehow, but a panic is hard to handle. Thank you very much!

[trishume]

@sharkdp done, v3.0.1 has been pushed to crates.io with this change and the syntax precedence fix.

[sharkdp]

@trishume Thank you very much!

[pksunkara]

I am using Zola and am trying to vue syntax highlighting work. I used the new branch as described above. I have no idea why source.js is failing to be resolved. Would really appreciate some direction here.

thread '<unnamed>' panicked at 'Can only call resolve on linked references: ByScope { scope: <source.js>, sub_context: None }', /Users/brew/Library/Caches/Homebrew/cargo_cache/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/syntect-3.2.0/src/parsing/syntax_definition.rs:193:18

Syntect version is 3.2.0 in zola 0.10.1

[keith-hall]

Hmm, I'm not able to reproduce with adding https://github.com/vuejs/vue-syntax-highlight/blob/6eb71bc6bba5e6a284b6d1d3154484da6f366e21/Vue%20Component.sublime-syntax to the testdata/Packages folder, running makepacks and then downloading https://github.com/vuejs/vue-syntax-highlight/blob/new/samples/basic.vue and running cargo run --example syncat ./testdata/basic.vue - it correctly displays highlighted output without panicking.

Just to check, do you have a sublime-syntax file which defines a base scope of source.js somewhere? Normally it would be Packages/JavaScript/JavaScript.sublime-syntax

[pksunkara]

I am using zola which uses syntect underneath. Since they don't have a Vue sublime syntax, I am using the extra_syntaxes option from this doc page.

Vue syntax is in a different place and the JS syntax is loaded by zola, and therefore that might be causing the issue. But, I am not sure since I don't know how zola works.

[keith-hall]

Presumably it works for other people using zola and extra syntaxes. I will try when I get chance. I wonder if it could be useful to add a method to the SyntaxSet which will check if any references couldn't be resolved, and if so, list the ones loaded and the unresolved references, for easier debugging as the current panic message doesn't make it obvious where the problem is.

[pksunkara]

I have the site open sourced at https://github.com/pksunkara/pksunkara.github.io. You can easily reproduce the bug if you add vue to the code block in this section, https://github.com/pksunkara/pksunkara.github.io/blob/develop/content/posts/complex-vuejs-app-structure.md#srccomponentshellovue.

[keith-hall]

I was able to replicate this bug with Zola v0.10.1. It seems like Zola stores the "extra_syntaxes" separately from the ones it includes by default, so your custom Vue syntax can't find the default JavaScript syntax. The solution is to copy Zola's sublime_syntaxes/Packages/JavaScript folder to your vendor/syntax/ folder and reference it in your Cargo.toml extra_syntaxes, then zola build works successfully.
FWIW, it looks like the next version of zola will include the Vue syntax by default:
getzola/zola@bc496e6#diff-8903239df476d7401cf9e76af0252622R61

So the problem really seems to be #278

[pksunkara]

Thanks for looking into this. I assume zola didn't do it because there is no merging possible with syntect easily. I agree that fixing #278 should fix it. I know for a fact that @sharkdp already did that merging in bat.

Maybe we can simply port his code (with his permission)?

[pksunkara]

So, zola 0.11.0 is released and I was trying to see if them including the vue syntax by default fixes the issue. Looks like it does not. There is no error but there is no highlighting either. Does the sublime_syntaxes/Packages/JavaScript folder have to be beside Vue folder? Currently, Packages folder is beside Vue.

[keith-hall]

It looks like the JavaScript syntax definition Zola is currently using isn't compatible with syntect atm, because it is using branch points, which aren't supported by this library yet. See #271. Probably worth raising an issue on the Zola repo to downgrade the JS syntax definition for now.