vLLMLocalInterfaceLayer returns results in incorrect format

[Permafacture]

Digging into an issue I was having, I have an idea that this might be the root of it.

model_name = "google/flan-t5-base"
prompt_node = PromptNode(model_name, max_length=256)
result = prompt_node("what is the capital of germany?")
print(len(result), result[0])

this returns (1, 'Berlin'). One prompt, one answer

When I try this with the vLLMLocalInterfaceLayer

model_id = "TheBloke/dolphin-2.2.1-mistral-7B-AWQ"

prompt_model = PromptModel(model_name_or_path=model_id,
                           invocation_layer_class=vLLMLocalInvocationLayer,
                           model_kwargs={
                               "maximum_context_length": 2**12,
                               "vLLM_kwargs": {
                                   "max_model_len": 2**12,
                                   "quantization": "AWQ"}
                               })

prompt_node = PromptNode(model_name_or_path=prompt_model, max_length=512)
result = prompt_node("what is the capital of germany?")
print(len(result), result[0])

The result is a list of length 400 where every element is a single character. Concatenated together the response is legitimate. Since the output has a different shape, I expect this is causing problems.

The real Issue I'm having is truing to run a ConversationalAgent.

prompt_node = PromptNode(model_name_or_path=prompt_model, max_length=512)
summary_memory = ConversationSummaryMemory(prompt_node)
conversational_agent = ConversationalAgent(
        prompt_node=prompt_node,
    )
response = conversational_agent.run("What is the capital of Germany?")

With T5 this works and with the vLLM model I get no response and the warning "Maximum number of iterations (2) reached for query (What is the capital of Germany?). Increase max_steps or no answer can be provided for this query." (increasing max steps to 10 gives the same response.)

[LLukas22]

The result is a list of length 400 where every element is a single character. Concatenated together the response is legitimate. Since the output has a different shape, I expect this is causing problems.

Yeah, this could cause the issue. Feel free to PR up the changes needed to format the output correctly. I have never used haystacks conversational agent, but i guess it just accepts a normal string as a return value.

[Permafacture]

fixed by #7

Can we get a pypi release that includes this?

[LLukas22]

Should be included in 0.1.2.