Implement grammar sampling for function and parameters names #74

jeffreymeetkai · 2023-12-11T05:15:12Z

No description provided.

khai-meetkai

I think basically, we need to handle to logic to detect if the current step is at generating tokens for function or parameters. Can we implement this more elegantly using the approach as I did for streaming ? we have a state (a dictionary containing: current_tokens, current_text, current_function, current_param) and update state --> output a new token.
Then we implement function:
def update_state(current_state, tokens_by_probs, sampled_token) --> new_state, sampled_token

jeffreymeetkai

Main points

Stateful mechanism where the stage in gen_state consists of ["pre-function", "function", "pre-parameter", "parameter-name", "parameter-value"]
The mechanism works like a FSM, going through the different stages
In step_async, self.sample will be called first followed by self.update_gen_state

Additional pointers

Refactored update_grammar_sampling_gen_state by using template method design pattern instead of abstract method to reduce code repetition. Will change back to abstract method if future prompt template versions do not allow template method pattern.

khai-meetkai · 2023-12-13T02:19:02Z

functionary/vllm_monkey_patch/async_llm_engine.py

+            tokenizer=self.tokenizer,
+        )
+
+    def sample(


Can we implement this in prompt_template ? So in the future, we can re-use this for other framework such as: llama_cpp, HF instead of binding to vllm
def sample_gramma_token(self, tokenizer, state, delta_token_ids, model_sampled_token_id)

khai-meetkai · 2023-12-13T02:25:11Z

functionary/vllm_monkey_patch/async_llm_engine.py

+            output[i].samples[-1].output_token = grammar_sampled_token_id
+
+            # Update gen_state
+            self.update_gen_state(


Can we also include update_gen_state inside def sample, I think it would be more convenient, no need to duplicate the this chunk of code:

if gen_state["stage"] in ["pre-function", "function"]: options = [ tool_or_func["name"] for tool_or_func in self.tools_or_functions[request_id] ] else: func_name = gen_state["func_name"] for tool_or_func in self.tools_or_functions[request_id]: if tool_or_func["name"] == func_name: options = list(tool_or_func["parameters"]["properties"].keys())

I will put this chunk into step_async before calling prompt_template.grammar_sample().

khai-meetkai · 2023-12-13T02:28:35Z

functionary/prompt_template/base_template.py

+        raise NotImplementedError
+
+    @abstractmethod
+    def get_stopping_token(self, stage: Literal["function", "parameter"]) -> int:


Can we rename get_stopping_token to avoid confusion with: def get_stop_tokens_for_generation?

khai-meetkai · 2023-12-13T02:36:56Z

functionary/prompt_template/prompt_template_v2.py

+        return self.recipient_token
+
+    def get_stopping_token(self, stage: Literal["function", "parameter"]) -> int:
+        if stage == "function":


Can we return string, then use tokenizer to get token_id, so we won't depend on the Model. In the future we might use another model, not Mistral

The same for v1

khai-meetkai · 2023-12-13T02:40:54Z

functionary/vllm_monkey_patch/async_llm_engine.py

+                tool_or_func["name"]
+                for tool_or_func in self.tools_or_functions[request_id]
+            ]
+            if self.prompt_templates[request_id].version not in ["v1"]:


Can we add another method in prompt template for getting list of additional predefined function name with default implementation is return []

def get_predefined_function_names() --> List[str]
in base_template:
def get_predefined_function_names():
return []
in template_v2:
def get_predefined_function_names()
return ["all"]

khai-meetkai · 2023-12-13T02:57:38Z

functionary/vllm_monkey_patch/async_llm_engine.py

+
+                # Form the parameter name with the current sampled token id
+                if len(wellformed_params) == 0:
+                    curr_text = gen_state["curr_text"][


Can we use curr_text with removing previous components, so we don't have to handle this.
For example, when we enter state=function; curr_text = function_name in progress
when we enter state=parameter --> curr_text=parameter_name in progress
when we enter state=parameter-value --> curr_text=parameter-value in progress

khai-meetkai · 2023-12-13T08:38:20Z

functionary/vllm_monkey_patch/async_llm_engine.py

+            gen_state = self.gen_states[request_id]
+
+            # Form the functions/parameters options
+            if gen_state["stage"] in ["pre-function", "function"]:


Actually, the chunk of code to get options should be inside: grammar_sample, right ?
we should replace: options --> tools_or_functions["request_id"]
So the logic is totally inside prompt template, inference engine only provides the sorted list of tokens

Then we pass options to update_grammar_sampling_gen_state

This should be in async_llm_engine.py because the prompt_template doesn't store the list of tools/functions mapped from request_id.

We can pass: tools_or_funtions to grammar_sample, replacing options

def grammar_sample( self, gen_state: Dict, tools_or_funtions: List, delta_token_ids: List, model_sampled_token_id: int, tokenizer: Any, )

khai-meetkai · 2023-12-13T08:42:42Z

functionary/prompt_template/base_template.py

+
+    def update_grammar_sampling_gen_state(
+        self,
+        gen_state: Dict,


Maybe should describe fields inside gen_state and their values. For example for stage: what are values and the meaning

Sure, I will move the explanation from async_llm_engine to here.

khai-meetkai · 2023-12-13T08:53:36Z

functionary/prompt_template/base_template.py

+                    wellformed_params = gen_state["param_names"]
+
+                    # Form the parameter name with the current sampled token id
+                    new_curr_tokens = gen_state["curr_text"] + tokenizer.decode(


why not use this like line 203 ?

new_curr_tokens_id = gen_state["curr_tokens"] + [sampled_token_ind] new_curr_tokens = tokenizer.decode(new_curr_tokens_id)

khai-meetkai · 2023-12-13T09:00:59Z

functionary/prompt_template/prompt_template_v1.py

+        if stage == "function":
+            return ":"  # 28747
+        else:
+            return '":'  # 1264


Oh I think it is only: '"' instead of '":' ?
Because in your check:
sampled_token== self.get_stop_token_for_function_parameter(stage="parameter")
I assume that: '=:' is 2 tokens?

This is for parameter names. The parameters are generated in json format so the models always generate '":' right after it completes a parameter name.

khai-meetkai · 2023-12-13T09:02:38Z

functionary/vllm_monkey_patch/async_llm_engine.py

+        # Future versions are assumed to begin directly with "function" stage
+        self.gen_states[request_id] = {
+            "stage": "function"
+            if self.prompt_templates[request_id].version not in ["v1"]


Oh I think at first, it is always pre-function?

By the way, I think should initialize by an empty dict({}), by this vllm doesn't need to know the structure of state
we will check and initialize inside def grammar_sample
if len(gen_state) == 0:
gen_state = {...}

The start of function token <|recipient|> is alr provided in v2 prompt so we basically start from "function" stage instead. I will move this initialization into prompt_template too.

khai-meetkai · 2023-12-13T09:13:34Z

functionary/prompt_template/base_template.py

+                _ = json.loads(
+                    '{"'
+                    + gen_state["param_names"][-1]
+                    + gen_state["curr_text"].removesuffix(', "')


I just wonder if:
{"x": 123456}
and 123456 is splitted into: "123", "456", then it will stop at: {"x": 123 ? and remove to new state ?

gotcha. Will add somemore checks here.

if gen_state["curr_text"].endswith(', "'): """Conduct the json.loads operation"""

…ammar-sampling-parameters

khai-meetkai · 2023-12-27T04:15:20Z

functionary/prompt_template/base_template.py

+            ):
+                curr_text = gen_state["curr_text"].rstrip()
+                while True:
+                    if any([curr_text == option for option in options]):


this while loop can be forever? if the function name is not in the list even with grammar sampling? can this happen?

This while loop will not be forever because the grammar_sample function already forces the function name to be built towards one of the provided function names including "all". This part just loops from the back to remove unnecessary suffixes until we have the complete function name to put into gen_state["func_name"]

khai-meetkai · 2023-12-27T06:50:11Z

functionary/prompt_template/base_template.py

                pattern = stop_token + r".*$"
-                match_res = re.search(pattern, latest_param_str)
+                match_res = re.search(pattern, latest_param_str, re.DOTALL)


This one is not popular but: stop_token='":' is not general enough; can also be '" :' ? right ?
for example:
{"a" : 10}

For this part, it is ok because a parameter name will always be string. So it will always be "parameter-name": {parameter-value} ONLY

Once "parameter-name": is detected, we will go to parameter-value stage already.

khai-meetkai · 2023-12-27T06:52:41Z

functionary/prompt_template/base_template.py

                gen_state["stage"] = "pre-function"
            except:
                pass

            # Check if the current state can be converted to json, it means the
            # new state is back to "parameter-name" stage
-            pattern = r',.*"$'
-            if bool(re.search(pattern, gen_state["curr_text"])):
+            pattern = r',[\n\s]*"'


I think \s already includes: "\n"

jeffreymeetkai added 7 commits December 7, 2023 08:03

set up grammar sampling for functions, stress test batched inference

9043291

add detailed documentation

05faca3

make vllm mkpatch backward-compatible with prompt template v1

0909ae7

extend grammar sampling to params wip

5aed290

fix check_to_sample for params

8ffebd7

extend grammar sampling to params

dd635f2

pull from main

bd0d8b0

jeffreymeetkai requested review from musab-mk and khai-meetkai December 11, 2023 05:15

khai-meetkai reviewed Dec 11, 2023

View reviewed changes

jeffreymeetkai changed the title ~~Extend grammar sampling to parameters names~~ Implement grammar sampling for function and parameters names Dec 11, 2023

jeffreymeetkai added 6 commits December 11, 2023 09:56

generalize prompt_template versions to integrate new versions easier

1c71f61

revamp implementation for template v2

eced962

fixes to v2

0e410c0

revamp implementation foor template v1

8439346

Add documentation

bbd0bb5

refactor update_grammar_sampling_gen_state

5a13a80

jeffreymeetkai commented Dec 12, 2023

View reviewed changes

khai-meetkai reviewed Dec 13, 2023

View reviewed changes

refactor based on comments

36cfaa2

khai-meetkai reviewed Dec 13, 2023

View reviewed changes

jeffreymeetkai added 4 commits December 13, 2023 10:13

refactor based on comments

2e0faab

minor edit based on comment

d039411

fix delta_token_ids_by_logprobs

5e39ae2

resolve merge conflict

c824b17

jeffreymeetkai added 13 commits December 14, 2023 10:38

minor edit

6efcd19

grammar sampling for pre-parameter stage

3d5c5c9

add no-function-call stage

c7a98d6

handle no args

76b80ed

minor edit

7b51c49

fixes

856e415

Merge branch 'main' of https://github.com/MeetKai/functionary into gr…

6274640

…ammar-sampling-parameters

make parameter-name and value parsing compatible to all formats

543508c

resolve merge conflict

70ba0ef

edits

762b7f1

upgrade vllm dependency

1911833

resolve merge conflict

174de56

make grammar sampling more flexible

58f49d0

khai-meetkai reviewed Dec 27, 2023

View reviewed changes

minor edit based on comments

26e71ea

musab-mk approved these changes Dec 27, 2023

View reviewed changes

jeffreymeetkai merged commit 1a4fd68 into main Dec 27, 2023
3 checks passed

devidw mentioned this pull request May 6, 2024

status/support for grammar sampling NousResearch/Hermes-Function-Calling#22

Open

jeffreymeetkai deleted the grammar-sampling-parameters branch November 6, 2024 03:11

Implement grammar sampling for function and parameters names #74

Implement grammar sampling for function and parameters names #74

Conversation

jeffreymeetkai commented Dec 11, 2023

khai-meetkai left a comment

Choose a reason for hiding this comment

jeffreymeetkai left a comment

Choose a reason for hiding this comment

khai-meetkai Dec 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffreymeetkai Dec 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffreymeetkai Dec 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khai-meetkai Dec 13, 2023 •

edited

Loading

jeffreymeetkai Dec 13, 2023 •

edited

Loading

jeffreymeetkai Dec 13, 2023 •

edited

Loading