Add ability to test chatbots using AI
SketchingDev
released this
12 Nov 16:30
·
39 commits
to main
since this release
Adds the ability to test chatbots using ChatGPT.
Once OPENAI_API_KEY
is set with the API key for OpenAI then tests are started with the ai
command:
web-messaging-tester ai example.yml -id $DEPLOYMENT_ID -r $REGION
AI test definitions take the form of a prompt with phrases used for whether the test is considered a success or failure:
scenarios:
"Accept survey":
setup:
prompt: |
I want you to play the role of a customer talking to a company's online chatbot. You must not
break from this role, and all of your responses must be based on how a customer would realistically talk to a company's chatbot.
To help you play the role of a customer consider the following points when writing a response:
* Respond to questions with as few words as possible
* Answer with the exact word when given options e.g. if asked to answer with either 'yes' or 'no' answer with either 'yes' or 'no' without punctuation, such as full stops
As a customer you would like to leave feedback of a recent purchase of a light bulb you made where a customer service
rep was very helpful in finding the bulb with the correct fitting.
If at any point in the company's chatbot repeats itself then say the word 'FAIL'.
If you have understood your role and the purpose of your conversation with the company's chatbot then say the word 'Hello'
and nothing else.
terminatingPhrases:
pass: ["PASS"]
fail: ["FAIL"]
Full Changelog: genesys-web-messaging-tester-cli-v1.0.14...genesys-web-messaging-tester-cli-v2.0.0