AiChat for iTwinUI Docs

[r100-stack]

This is the AI Chat feature that is powered by OpenAI's embedding and text completion APIs.

Video35.webm

Logic/workflow

• features/search/scripts/create_chunks.js reads all .mdx files in apps/website/src/pages/docs and all .tsx files in /examples and creates docs_chunks.json and example_chunks.json respectively.
  • Docs chunks is an array of chunks where each chunk represents a new heading level.
  • Example chunks is an array of chunks where each chunk represents a code example.
  • This step can run in the CI/CD after any change to any docs mdx file.
  • Sample docs chunks:

    • // Sample input:
      lines = [
       'Alert is an element',
       '## Appearance',
       'Alert should be concise',
       '### Informational',
       'Default style',
       '## Placement',
       'Either inline or sticky',
      ]
      title = 'Alert'
      
      // Sample output:
      [
       {
         "header": ['Alert'],
         "content": 'Alert is an element',
       },
       {
         "header": ['Alert', 'Appearance'],
         "content": 'Alert should be concise',
       },
       {
         "header": ['Alert', 'Appearance', 'Informational'],
         "content": 'Default style',
       },
       {
         "header": ['Alert', 'Appearance', 'Placement'],
         "content": 'Either inline or sticky',
       },
      ]

  • Sample example chunks:

    • {
          "codeExampleTitle": "Alert.informational",
          "codeExample": "...code..."
      }

• We then generate embeddings for each chunk (info)
• When the user types a query, we encode the query too.
  • We then compare the query vector with all the saved vectors for each chunk.
  • We give the N (some number) most similar chunks to Azure OpenAI's completion API to summarize the chunks to answer the user's query.

Code structure

• apps/website/api/search.json.ts: A backend endpoint in astro to do the vector comparison to get the top chunks and then send and receive a response from the OpenAI completions API. This was done as an alternative to a Python server.
• features/search/scripts/create_chunks.mjs: This script is used to create the text chunks and also the code example chunks for the search index. I believe this needs to run in the CI step (or some place similar) to create the new chunks every time the docs site is updated.

Features

• Each query-response pair is currently logged locally in a JSON file called apps/website/src/_data/query_history.json
  • This is mainly for testing -- to see what were the behind the scenes contents for a particular query and thus try improving the search results.
• There is a radio button group. This tries to help providing more specific types of results: "code" or "text", in case the "balanced" mode does not give good results.
  • 
  • Upon hovering over the question mark:

Testing

Tested a few example prompts. I hope to have a more structured way of testing and evaluating soon.

Docs

On the docs website's /docs/... pages, I added an AI Chat button to the header.

[mayank99]

Spent some time playing around with this. Here are my initial thoughts.

UX feedback

• The dialog feels too cramped with multiple scrollbars, because the answers often need more space. I think using a separate page would work better, as there is no reason for this to be overlaid on the same page.
• The API responses take 5-7 seconds for me locally. A loading indicator would help during this time.
• The chat interface is a bit hard to read. Probably would be better to use chat bubbles to clearly show which parts are part of the AI response. Inside the bubbles, code snippets should be separated properly so that they don't bleed into the surrounding text.
• The radio group is confusing; why would I want "better code" or "better text"? Why can't I have both code and text be the best/most correct version possible?
• It would help to gather feedback through simple 👍/👎 buttons after each response.

The feature itself

The AI responses are very questionable. Often full of nonsensical code/text yet answered with such great confidence that an unsuspecting user would be easily fooled and get confused later. Here are my logs: query_history.json.

• I can understand that it has no way of answering some of the specific questions, but I would rather it said "I don't know, please contact the team" rather than spewing unhelpful garbage.
• A lot of times, it links to our docs but gets the url wrong, pointing to some itwin.ui domain. This is unacceptable.
• Even the few times it does answer the question somewhat correctly, it keeps going and adds unrelated fluff to the answer.

...which begs the question - what problem are we even trying to solve with this feature? Docs should be the place to find objectively correct information. When the "official" chatbot is so unhelpful, it reduces overall confidence in the documentation site.

[gretanausedaite]

I agree with Mayanks comments about UX and feature itself. It needs some improvements before releasing it to the users. My query_history.json.txt
It's a great start and amazing tool to demo.

Couple additional comments:

• I have my OS in light mode so Algolia search button is light (probably does not belong in this PR but adding it anyways)
  
• Some screenshots from code samples that are not correct:
  
  This one is bad because it refers to package that does not exist.
  

[adamhock]

I agree with Mayank's comments as well about this feature. It may become more useful as we add more complex components, but I don't think it's needed for the documentation site at the moment. It's still a really cool tool though!

When the AI doesn't know the specific answer to my question, it will just rewrite whatever is written on the documentation site (explaining the component, its features, and its variants). At that point, it might as well just link me to the documentation pages so I can read the same text in a better format.

Code examples are good to display, but I also think there needs to be an output displayed as well or at least a link to open the code in codesandbox.

Here is my query history: query_history.txt

[LostABike]

Echoing what everyone else has already said. So I'm just going to add other things:

• One idea for UX I have is make it like a support chat bot that can be toggled at bottom right of pages, which you usually see on business or shopping sites to answer common questions. I know those are different in that they are more hard-coded type with pre-determined question options, but one UX benefit of this is that you can still browse other documentation pages while still seeing the chat response/result for quick referencing. Currently old chat is wiped out after following a provided link so I lose reference of what it was telling me.
• It should have an ability to distinguish changes from different versions and adjust the response accordingly. Or maybe take version into account when being asked by users. This might be ambitious though since our documentation itself only reflects the latest change (in this case v3). But so v2 or even v1 users might not benefit from this at all, especially when asking about breaking changes. (you will see in my query that I ask about Tile usage and it doesn't know about the latest composition change yet)
• Sometimes I want to ask follow-up question but currently can only ask one question at a time since last question and response is wiped (also just opinion, maybe current way is better to prevent it from following up with random answers)
• This is great, should be the response for unconfident answers related to the API too
  

query: query_history.txt

[mayank99]

Since there has been no activity on this in a while, I'm closing this, but leaving the branch intact. PR can be reopened if/when we want to come back to it.