Orama search is not supporting korean language

[icepeng]

Since Fumadocs 14, the default search implementation has been changed from Flexsearch to Orama. However, Orama currently does not support a Korean tokenizer (see supported languages), which presents a challenge for users building documentation sites in Korean.

While I am currently falling back to Algolia, a built-in search solution is preferred. Could we re-implement flexsearch client? Alternatively, could there be additional resources or documentation provided that would help guide to implement my own search client?

[fuma-nama]

You are right, I can ask the Orama team later. Algolia could be a temporary solution.

Or you can check https://fumadocs.vercel.app/docs/headless/search/orama#custom-algorithm to implement your own search API endpoint

[fuma-nama]

Not sure how you implemented it, you can do it like https://fumadocs.vercel.app/docs/ui/static-export#built-in-search

If you don't use the built-in methods, you can modify your api endpoint to return a JSON object from db.save()

[icepeng]

const defaultTokenizer = await tokenizer.createTokenizer({
  language: "english",
});

const normalizeToken = tokenizer.normalizeToken.bind(defaultTokenizer, "");

function trim(text: string[]): string[] {
  while (text[text.length - 1] === "") {
    text.pop();
  }
  while (text[0] === "") {
    text.shift();
  }
  return text;
}

function tokenize(input: string): string[] {
  if (typeof input !== "string") {
    return [input];
  }

  const splitRule = /[^ㄱ-ㅎ|ㅏ-ㅣ|가-힣|A-Za-zàèéìòóù0-9_'-]+/gim;
  const tokens = input
    .toLowerCase()
    .split(splitRule)
    .map((t) => normalizeToken(t))
    .filter(Boolean);

  const trimTokens = trim(tokens);

  return Array.from(new Set(trimTokens));
}

export const { staticGET: GET } = createSearchAPI("advanced", {
  indexes: () =>
    Promise.all(
      source.getPages().map(async (page) => {
        const { structuredData } = await page.data.load();

        return {
          id: page.url,
          title: page.data.title,
          description: page.data.description,
          structuredData,
          tag: page.slugs[0],
          url: page.url,
        } satisfies AdvancedIndex;
      }),
    ),
  tokenizer: {
    language: "english",
    tokenize: (raw) => tokenize(raw),
  },
});

I passed tokenize function to split both korean and english words.

        <RootProvider
          search={{
            options: {
              type: "static",
            },
          }}
        >

If I change staticGET to GET, and type: "fetch", korean search is working.

[fuma-nama]

Hmm I guess it’s because the query has to be tokenized too, and your custom tokenizer is not included in client bundle. Have to take a look tonight

[fuma-nama]

Yep my assumption was right, will add option for this later this week

[icepeng]

d95c21f 👍

[fuma-nama]

Yep you will be able to make your own init function, so custom options will be possible