ttu-whispersync

A Violentmonkey script / Chrome extension for listening to audiobooks with ttu ebook-reader

Getting started

1. Install Anki and configure deck and note type
2. Install AnkiConnect
3. Add https://reader.ttsu.app (and http://localhost:5173 for local development) to the webCorsOriginList of AnkiConnect
4. You can install the extension as Violentmonkey script by clicking on this url (you need to be online at installation time in order to download required dependencies) or as unpackaged extension in a chromium browser by downloading the latest major version from releases or by building the current version by yourself
5. Open the reader / reload the tab and click on the icon in the bottom left corner to open the menu and load subtitle/audio files
6. Switch to "Match" Tab, match the subtitle against your book and click on "Save & reload page" to enable full functionality like reader highlighting, reader menu, autoscroll etc. after the page was reloaded
7. Configure your experience to your preferences by going to the settings tab

Development / Building

# Install pnpm
npm install --global pnpm

# Install dependencies
pnpm install

# Start Violentmonkey script development file watcher and auto rebuild
pnpm run dev

# Start extension development
pnpm run dev:ext

# Build prod version of Violentmonkey script
pnpm run build

# Build dev version of Violentmonkey script
pnpm run build:dev

# Build prod version of extension
pnpm run build:ext

Keybindings

Keybind	Description
Cmd + Space	Toggle playback
Alt + j	Toggle playback (alt keybind)
Cmd + d	Restart playback for active line
Alt + d	Toggle play and pause for active line
Cmd + l	Toggle playback loop for active line
Cmd + b	Toggle bookmark for active line
Cmd + m	Toggle for merge for active line
Cmd + e	Create new card for active line
Alt + e	Update last created card for active line
Cmd + o	Open last exported card in Anki Browser
Alt + g	Edit subtitle
Alt + z	Copy active subtitle
Cmd + q	Go to previous subtitle
Alt + q	Go to next subtitle
Cmd + k	Decrease playbackrate
Alt + k	Increase playbackrate
Cmd + Arrow Left	Rewind
Cmd + Arrow Down	Rewind #2
Cmd + Arrow Right	Fast-Forward
Cmd + Arrow Up	Fast-Forward #2

FAQ

How to create subtitles for audio files

• Follow instructions for SubPlz

What formats are supported

• .srt / .vtt for subtitle and .m4a / .m4b / .mp3 (with constant bitrate) for audio files (support may depends on used browser / actual audio codec / container)

Are mobile devices supported

• The functionality was briefly tested on mobile (android 10, kiwi browser 124.0.6327.4, AnkiconnectAndroid 1.13) but the main focus and support is currently on desktop systems and you may face issues on other devices

Known issues / missing features are e. g. :

• worse performance
• no filesystem api support
• no "open in anki browser" functionality
• no tag update functionality
• no duplicate check for creating cards
• various iOS quirks

How to use ttu-whispersync on mobile devices

Android

• Install and follow instructions for AnkiconnectAndroid (if you need multiple hosts for cors use a '*' character to whitelist all sources)
• Install a browser which supports ViolentMonkey or chrome extensions (e. g. Kiwi)
• Install ttu-whispersync as described in Getting started

iOS

• Install a browser which supports ViolentMonkey or chrome extensions (e. g. Orion)
• Install ttu-whispersync as described in Getting started

Note: iOS may block the ability to select certain files. For subtitles you therefore may need to store the content of a srt subtitle as txt file instead, while for audio you may need to convert the file to a common format like mp3 in advance. In case the loading spinner after selecting an audio file/editing a subtitle is not disappearing, tap a couple of times on the screen to force an user action event, wait 10 seconds and/or disable the cover + chapter options via settings. The active subtitle display can be slightly delayed due to an text track issue on iOS. There is also no equivalent to AnkiconnectAndroid meaning any export functionality is basically missing. As with ttu reader continous mode has several issues and should not be used

What are the differences between the Violentmonkey script and Chrome extension

• There are no functional differences, they only differ in the way of installation / storage location. Violentmonkey includes an auto update functionality while updates for the chrome extension need to be loaded manually by downloading the latest version from releases tab

What are the differences between desktop/mobile Chrome and/or other browsers

• The Filesystem API is only supported on desktop chromium browsers (some browsers like brave may need to have extra options enabled for it). On other devices/browsers you need to select your files every time you open or refresh the tab. Subtitles have a local storage option, audio files are never stored locally
• "Enable subtitle copy" will only work in chromium browsers as other browsers block clipboard actions without user interaction

The menu icon in the bottom left corner is not visible or duplicated

• It may happen that the footer element was not loaded at script injection time or that you are missing a required site update - connecting to the internet and hard refreshing the tab couple of times should fix this. If you are using the browser extension make sure it has the respective site permissions enabled

What is the "Invalid ttu version" error

• You are missing a required site update - connecting to the internet and hard refreshing the tab couple of times should fix this

I get a very low line match rate

• Try to change some of the matching options and see if different combinations may increase the match rate. You can also try to hint the script from where to start by clicking on the bullseye icon and clicking on the respective element afterwards

The matching speed is very low

• This can give you a hint that the script has problems to find a proper first match against your subtitle. You can try out to tell the script from where to start by clicking on the bullseye icon and clicking on the respective element afterwards or alternatively increase the number of match attempts

After matching the book it looks different

• By matching the book the text content is wrapped by helper elements to identify the related subtitle. Depending on the browser this may lead to some layout shifting afterwards and is expected / not really fixable. Whole missing characters or elements can be considered a bug which can be reported under "Issues". You can also try out the "Original" line space mode for matching in case you used a different one before

Audio file is not loading / loading spinner remains visible

• Some browser require user interaction in order to fire the required audio load events - so try to click/tap a couple of times on the loading spinner. If you have extensions like uBlock or use browser like brave (with their shield) try to disable them/whitelist the reader and check if the file is now loading

What are the differences between the recorder and ffmpeg backend

• The recorder backend will create audio clips by recording the system audio. Therefore it needs to play the audio parts in real time and without muted sound. To avoid issues the main controls of the script will be disabled during this duration. The recorder only supports mp3 as export format. Editing subtitles is only availabe as basic functionality. On browsers like Firefox you may encounter unresponsiveness during the mp3 conversion compared to Chrome
• The ffmpeg backend uses a WebAssembly compiled version of FFMPEG and therefore has more options for the audio export format. Editing subtitles is available with the advanced editor. The ffmpeg backend can process the audio in the background which allows you to continuing using most of the script functionality during export. Furthermore it can provide a data fallback for audio chapters. The ffmpeg backend is limited to files with a maximum size around 1.8 GB. As it needs to copy the file into memory and executes actions inside of it you may encounter unresponsiveness or site crashes on lower spec devices / big files - in this case refresh the tab and consider using the recorder backend instead

"Failed to fetch" error for exports

• Make sure your Anki and AnkiConnect are running (on mobile the service needs to be started) and that the respective urls were added to the webCorsOriginList list

Anki field requirements

• Anki cards require a non empty key field (first field of your card template). Trying to create a card without configuring the sentence or sound field (from which one of them need to be the key field) will therefore fail. In case you have the duplicate check disabled you can enable the "Allow empty key field" setting as workaround but note that this is considered bad practice and can also be confusing in the card browser

"Export(s) failed" details

• You can find more details by opening the developer tools of your browser (often F12 keybind on desktop) and checking the console tab / output

Card updates are not working on android

• Make sure to have AnkiconnectAndroid 1.14 (or higher) installed - previous versions are not supporting the required apis for updates

What is the bookmark functionality

• You can bookmark subtitles to "remember" them in order to listen to important lines again or do a bulk export later to not interrupt your session. You can find them by opening the subtitle list menu (main icon in the bottom left corner) and enabling the bookmark filter in the top controls. Playing / Looping via respective top control icon will play all bookmarked lines while skipping the ones in between and skipping the update to your current playback position. You can do a bulk export by clicking on the respective icon in the top controls - this will create a card with respective audio for every single subtitle (and is therefore limited to the create action)

What is the merge functionality

• You can flag subtitles for merge to "remember" them in order to listen to important lines again or do a merged export to combine multiple lines into one card. You can find them by opening the subtitle list menu (main icon in the bottom left corner) and enabling the merge filter in the top controls. Playing / Looping via respective top control icon will play all flagged lines while skipping the ones in between and skipping the update to your current playback position. You can do a merge export by clicking on the respective icon in the top controls - this will either create one single new card or update your last card with the content of all flagged lines and their respective audio

Is it possible to queue / execute exports in parallel

• No - there is no export queue implemented. If you e.g. execute an export keybind while an other export is still running it will be simply ignored. If you often want to export close lines it is better to use the bookmark functionality and do a bulk export in the end of your reading/listening session

Keybind 'X' is not executed

• Note that most of the keybinds are only executed for the current most recent active subtitle. If none exist no action will be executed but you can still do so with the reader / subtitle actions. For exports see also the previous point. Alternatively you can enable the "Enable time fallback" setting under keybindings which search through the whole list for the closest subtitle on the current playback position

The reader menu / action list is not opening - only my popup dictionary

• The popop dictionary is probably capturing the necessary events and therefore prevents the script functionality. Check available scan options for your dictionary (e. g. Yomitan will not work with enabled touch press option)

The audio gets out of sync after seeking / playing for a while

• You may use a mp3 file with variable bitrate which is not supported. Consider changing the bitrate to a constant value e. g. via Audacity export or FFMPEG:

Audacity

File => Export => Export as mp3

FFMPEG example command

ffmpeg.exe -y -i "path_to_input" -c:v copy -c:a libmp3lame -b:a 128 -map_metadata 0 -id3v2_version 3 -write_xing 0 "output_path"

Is it possible to sync data to other devices/browsers

• The playback position and persisted subtitles can be synced via default "Auto Import/Export" functionality. Matched book content can be synced via manual "Book Data" export - other data is currently not syncable

Is it possible to read/load books from an external storage source like gdrive?

• Currently only books loaded via browser storage source / local browser db are supported - otherwise you will get an error like 'required data for id x not found'. You can still sync the progress etc. to those sources

Is it possible to change the order of actions?

• You can configure the list of actions and their order for the reader menu, subtitle list and footer as per your preference via settings. To show the action check the respective checkbox. To change the order hold down your mouse/pointer on the respective action box and move it to the item with which you want to swap the position

Limitation for autoscroll in pagination mode

• In pagination mode it can happen that a subtitle line extends multiple pages in which case the autoscroll behavior is not working properly. This probably can't be fixed but you can try to minimize those occurrences by enabling the "Avoid Page Break" option of ttu
• In case you are using the "jpdb-browser-reader" extension you may face the issue of delayed scrolling as this extensions "pushes" the elements out of their original page / order

Potential enhancements

• Native ttu reader implementation
• finer granularity on progress updates during export / recording
• PiP / separate window for audiobook menu
• Custom/Extended duplicate checks for updates / mobile
• Text-to-speech support (e. g. VOICEVOX, Web Speech API or similar)
• External file sources

Not planned

• Listening statistics / goals
• Profiles / Per file settings (if then only after native setting sync / profile implementation on ttu)
• EPUB3 Media Overlays