[Feature] Support llguidance for constrained decoding

[JC1DA]

Motivation

This pull request integrates llguidance backend to extend guided decoding capabilities for sglang
llguidance backend supports regex, json and grammar (lark or ebnf)

We have just released a large JSON Schema benchmark and a paper. Of particular interest might be isolated mask-generation benchmarks - comparing LLGuidance, Outlines, XGrammar and llama.cpp grammars.

Modifications

• Added llguidance constrained backend to support masked decoding & json/ebnf/regex testcases.
• We have plan to further support fast-forward decoding in the near future

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.

[JC1DA]

Hi @Ying1123 @merrymercy @zhyncs
Can you guys help take a look at this PR?
We'd love to bring more choices for constraint decoding to sglang community :)

[zhyncs]

ok I'll help take a look asap. Thanks for your contribution!

[JC1DA]

ok I'll help take a look asap. Thanks for your contribution!

thanks @zhyncs, really appreciate it

[Harsha-Nori]

Hi @zhyncs, any chance we could get a brief review here? We'd like to deploy guidance + sglang for some of our users, and hopefully also deliver benefits to the sglang community! Just some pointers on things you'd like to see changed or improved would help us make sure we're working in the right spirit 🙂

[zhyncs]

Hi @zhyncs, any chance we could get a brief review here? We'd like to deploy guidance + sglang for some of our users, and hopefully also deliver benefits to the sglang community! Just some pointers on things you'd like to see changed or improved would help us make sure we're working in the right spirit 🙂

@Harsha-Nori Ah Sorry for the delayed response. I've been busy lately. We will review soon. Thank you for your understanding! BTW @JC1DA, can you help resolve the conflicts? Thanks!

[zhaochenyang20]

@JC1DA @zhyncs sure. love to help!

[zhaochenyang20]

@JC1DA Could you fix the conflicts first?

[JC1DA]

@JC1DA Could you fix the conflicts first?

hi @zhyncs @zhaochenyang20, fixed :) thanks for taking a look

[zhaochenyang20]

@JC1DA Hey. Why should we keep a fixed x-grammar verison? We will use x-grammar as default backend in the next PR.

[mmoskal]

@zhaochenyang20 this PR doesn't seem to change xgrammar version used, was the comment for a different PR?

[zhaochenyang20]

@mmoskal @JC1DA I see the change. Sorry for misunderstanding. cc @shuaills for review. Thanks!

[JC1DA]

@mmoskal @JC1DA I see the change. Sorry for misunderstanding. cc @shuaills for review. Thanks!

thanks @zhaochenyang20, just fixed the recent conflict

[zhaochenyang20]

Important dependies like transformers should not be fixed in this PR. Should we use a fixed transformers verison?

[JC1DA]

Important dependies like transformers should not be fixed in this PR. Should we use a fixed transformers verison?

I merged it from the main branch, we didn't add transformers in. We only added llguidance dependency.

@zhaochenyang20 Can you help rerun the workflows. Just reran pre-commit on pyproject.toml

[mmoskal]

@shuaills there is no cache; the grammars are compiled every time; it takes ~2ms on avarage on a large JSON schema test suite with p99.9 of under 40ms (single-threaded), see https://github.com/guidance-ai/jsonschemabench/tree/main/maskbench

[shuaills]

@shuaills there is no cache; the grammars are compiled every time; it takes ~2ms on avarage on a large JSON schema test suite with p99.9 of under 40ms (single-threaded), see https://github.com/guidance-ai/jsonschemabench/tree/main/maskbench

Why we don't need a cache here? Can you share some insights, thanks. @mmoskal

[mmoskal]

@shuaills there is some pre-computation, but it's per-tokenizer and let's call it grammar type (in this case JSON) not per grammar; anyway it's not very heavy

there are more details here https://github.com/guidance-ai/llguidance/blob/main/docs/optimizations.md

[shuaills]

LGTM @zhaochenyang20

[JC1DA]

Can you also rebase the pr?

Sure, I just did the rebase

[zhaochenyang20]

@shuaills shuai., if you feel good. Give an approval. I will merge it today.

[JC1DA]

@shuaills shuai., if you feel good. Give an approval. I will merge it today.

Hi @zhaochenyang20 , are we able to merge now or is they anything I should do to help?