Search not working with Jp queries

[mittei]

Steps to reproduce

1. Search "ペン"
2. It says "No search results for 'ペン'." (But ペン does appear in Lesson 1)

Description

This seems to be a known issue with mdbook:

• https://qiita.com/moonWalf/items/c236763965e85210c47e
• https://qiita.com/dalance/items/0a435d66e29f505faf6b

[Morgawr]

Yup, this sucks unfortunately... there seems to be a WIP plugin/patch for Chinese that replaces the search backend to read non-English symbols but I'm not sure if we can integrate that in the guide. If someone wants to give it a go I'm all for it, but it might be overcomplicated until they fix it upstream in the mdbook project (low chance of that though since the issue has been open for years).

See: rust-lang/mdBook#1081

[mittei]

It's kind of insane given Rust is so popular and mdbook is its de facto standard documentation generator 😂

[mittei]

I tried this and it sort of worked: rust-lang/mdBook#1081 (comment)

But it's awkward because normally searcher.js loads the search index (searchindex.json) pre-built by mdbook during build, but now with this workaround you're going to throw that away and rebuild the index on browser.

[Morgawr]

It's nice if you managed to get it to work! I feel like we might be able to use it if it works, even though it being client side might introduce some lag to some users (like those reading on a phone, not sure). I'm not a web dev, I wonder if it's even possible to use one backend or the other depending on the client being a mobile or a desktop PC. Probably unnecessary.

Anyways, do you have a PR or commit to share so I could take a look at it?

[mittei]

So the reason I wrote "sort of worked" was because it kinda broke English search. Normally "making" would match "make" (thanks to the "stemmer" process) but it no longer does.

I found a workaround so I tried it but it didn't work. It turns out the this.tokenizer isn't even called at all, you have to modify the global elasticlunr.tokenizer instead. But doing that makes JP search not work at all for some reason. And I tried a bunch of things but nothing worked so I gave up 😂

And here's what I have now: main...mittei:yokubi:fix-jp-search . It doesn't include the stuff in the second paragraph.

[Morgawr]

Ah, I see.. bummer. Thanks for the effort though! I'll take a look at it eventually and see if I can work around it, or if anyone else wants to give it a stab it would be nice. But still, this is relatively low priority given the amount of effort required (but I wouldn't say no if someone did it :))

[mittei]

Relevant articles I found:

• https://tbsmcd.net/post/added_search/
• https://ytyaru.hatenablog.com/entry/2023/02/21/000000

It seems like a pretty complicated issue 😑

[mittei]

I MADE IT WORK YOOOO #8

[mittei]

Actually I found a bug, let me fix it

[mittei]

Alright fixed