markdown-processor.ts error in case of some AI responses.

[xhejtman]

Hello, I got strange errors when AI (e.g. llama) returns specific markdown. E.g.:

To create a Docker image, you can follow these steps:

First, you need to create a `Dockerfile` that contains the commands to build your image. The `Dockerfile` should include the base image, copy files, install dependencies, and set environment variables as needed.

Here is an example of a simple `Dockerfile`:
` ` `docker
FROM ubuntu
RUN apt update && apt install -y wget
` ` `
Next, you need to build your Docker image using the `docker build` command. You can do this by running the following command in the directory where your `Dockerfile` is located:
` ` `bash
docker build -t myimage .
` ` `
This will create a Docker image with the name `myimage`.

You can then push your image to a registry like Harbor or Docker Hub using the `docker push` command.

For more information on creating and managing Docker images, you can refer to the official Docker documentation or other resources available online.

Source: [Dockerfile](/en/docs/docker/dockerfile)

(those ``` are actually without spaces, just here to not to mess with formating).

Rendering of such answer stops at the first Docker codeblock. The render looks like this:

and sometimes even this error is shown:

I use unmodified version of the markdown-processor.ts.

Is there anything I can do with this?

[xhejtman]

it seems, that if I remove this part:

.use(() => {
      return async (tree: Root, file) => {
        const transformer = await rehypeShiki;

        return transformer(tree, file, () => {
          // do nothing
        }) as Root;
      };
    });

the markdown is rendered correctly, it is not highlighted of course.

[xhejtman]

so the issue here seems to be, that shiki does not have fallback to plaintext, if language is not supported. first thing is, that original markdown-processor.ts includes only web bundle. Full bundle is needed for docker language, but it may still fail in some situations.

[fuma-nama]

Will see if I can send a PR to Shiki next month

[xhejtman]

another solution is to set lazy: false and fallbackLanguage: 'plaintext', in return rehypeShikiFromHighlighter(highlighter, {

[fuma-nama]

Done, rehype Shiki now works with the fallback language option even on lazy mode