Technologies Used

• VoiceVox Docker or VoiceVox Colab
• DeepL
• Deeplx
• Whisper OpenAI
• VB-Cable

Installation

1. Install the dependencies

pip install -r requirements.txt

2. Choose which TTS you want to use, VoiceVox or Silero. Uncomment and Comment to switch between them

# Choose between the available TTS engines
# Japanese TTS
voicevox_tts(tts)

# Silero TTS, Silero TTS can generate English, Russian, French, Hindi, Spanish, German, etc. Uncomment the line below. Make sure the input is in that language
# silero_tts(tts_en, "en", "v3_en", "en_21")

If you want to use VoiceVox, you need to run VoiceVox Engine first. You can run them on local using VoiceVox Docker or on Google Colab using VoiceVox Colab. If you use the Colab one, change voicevox_url on utils\TTS.py using the link you get from Colab.

voicevox_url = 'http://localhost:50021'

if you want to see the voice list of VoiceVox you can check this VoiceVox and see the speaker id on speaker.json then change it on utils/TTS.py. For Seliro Voice sample you can check this Seliro Samples

3. Choose which translator you want to use depends on your use case (optional if you need translation for the answers). Choose between google translate or deeplx. You need to convert the answer to Japanese if you want to use VoiceVox, because VoiceVox only accepts input in Japanese.

tts = translate_deeplx(text, f"{detect}", "JA")
tts = translate_google(text, f"{detect}", "JA")

DeepLx is free version of DeepL (No API Key Required). You can run Deeplx on docker, or if you want to use the normal version of deepl, you can make the function on utils\translate.py. I use DeepLx because i can't register on DeepL from my country. The translate result from DeepL is more accurate and casual than Google Translate. But if you want the simple way, just use Google Translate.

4. If you want to use the audio output from the program as an input for your Vtubestudio. You will need to capture your desktop audio using Virtual Cable and use it as input on VtubeStudio microphone.

5. If you planning to use this program for live streaming Use en.txt , jap.txt and output.txt as an input on OBS Text for Realtime Caption/Subtitles

FAQ

1. Mecab Error

this library is a little bit tricky to install. If you facing this problem, you can just delete and don't use the katakana_converter on utils/TTS.py. That function is optional, you can run the program without it. Delete this two line on utils/TTS.py

from utils.katakana import *
katakana_text = katakana_converter(tts)

and just pass the tts to next line of the code

params_encoded = urllib.parse.urlencode({'text': tts, 'speaker': 46})

Credits

This project is inspired by the work of shioridotdev. Special thanks to the creators of the technologies used in this project including VoiceVox Engine, DeepL, Whisper OpenAI, and VtubeStudio.