[BUG🐛]

[stachuman]

err_execute_model_input_20250223-185713.zip

Bug Description

I'm using my custom ebook conversion routine, which process file after file. This error happens irregular, not sure, how to narrow down to the exact source of problem. Can share my custom script, from which SOMETIMES I get the error.

2025-02-23 18:57:13,207 [ERROR] Engine background task failed
2025-02-23 18:57:13,207 [ERROR] Exception in callback _log_task_completion(error_callback=>)(<Task finishe...5713.pkl): ')>) at /home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py:45
handle: <Handle _log_task_completion(error_callback=>)(<Task finishe...5713.pkl): ')>) at /home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py:45>
Traceback (most recent call last):
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
return func(*args, **kwargs)
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1654, in execute_model
hidden_or_intermediate_states = model_executable(
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/auralis/models/xttsv2/components/vllm_mm_gpt.py", line 640, in forward
starting_sequence_start_ids, input_ids, positions = self._apply_op_to_seq_in_batch(input_ids,
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/auralis/models/xttsv2/components/vllm_mm_gpt.py", line 609, in _apply_op_to_seq_in_batch
assert (modified_positions >= 0).all()
AssertionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 55, in _log_task_completion
return_value = task.result()
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 872, in run_engine_loop
result = task.result()
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 795, in engine_step
request_outputs = await self.engine.step_async(virtual_engine)
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 347, in step_async
outputs = await self.model_executor.execute_model_async(
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 180, in execute_model_async
output = await make_async(self.driver_worker.execute_model
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 343, in execute_model
output = self.model_runner.execute_model(
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/worker/model_runner_base.py", line 152, in _wrapper
raise type(err)(
AssertionError: Error in model execution (input dumped to /tmp/err_execute_model_input_20250223-185713.pkl):

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/home/staszek/miniconda3/envs/auralis/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 67, in _log_task_completion
raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.

Minimal Reproducible Example

This is part of my script:

...

# Initialize
tts = TTS(scheduler_max_concurrency=20 ,vllm_logging_level=logging.WARNING).from_pretrained("AstraMindAI/xttsv2", gpt_model='AstraMindAI/xtts2-gpt')

# Configure logging
LOG_FILE = "/home/staszek/book/log/tts_conversion.log"
logging.basicConfig(
    level=logging.ERROR,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[
        logging.FileHandler(LOG_FILE, encoding="utf-8"),
        logging.StreamHandler(sys.stdout)
    ]
)
...
def convert_txt(input_file, voice, lang, output_file):
    try:
        with open(input_file, 'r', encoding='utf-8') as f:
            ready_text = f.read()

        request = TTSRequest(
            text=ready_text,
            speaker_files=[voice],
            audio_config=AudioPreprocessingConfig(
                normalize=True,
                trim_silence=False,
                enhance_speech=True
            )
        )
        output = tts.generate_speech(request)
        output_wav = output_file + ".wav"
        output.save(output_wav)

        convert_wav_to_mp3(output_wav, output_file)

        logging.info(f"✅ Output file created: {output_file}")
        return True
    except Exception as e:
        logging.error(f"❌ Error in convert_txt: {e}")
        return False

def convert_wav_to_mp3(output_wav, output_mp3):
    if os.path.isfile(output_wav):
        try:
            logging.info(f"🎵 Converting WAV to MP3: {output_mp3}")
            subprocess.run([
                "/usr/bin/ffmpeg", "-y", "-i", output_wav, "-ab", "192k", output_mp3
            ], check=True)
            os.remove(output_wav)
        except subprocess.CalledProcessError:
            logging.error("❌ Error: Failed to convert WAV to MP3")

Expected Behavior

[Describe what you expected to happen]

Actual Behavior

[Describe what actually happened]

Error Logs

[Paste relevant error logs here, ensuring the logging level is set to DEBUG]

Environment

Please run the following commands and include the output:

# OS Information
uname -a

Linux duzy 6.1.0-31-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07) x86_64 GNU/Linux

# Python version
python --version
Python 3.10.16


# Installed Python packages
pip list
Package                           Version
--------------------------------- -------------
aiofiles                          24.1.0
aiohappyeyeballs                  2.4.6
aiohttp                           3.11.12
aiosignal                         1.3.2
annotated-types                   0.7.0
anyio                             4.8.0
asttokens                         3.0.0
async-timeout                     5.0.1
attrs                             25.1.0
audioread                         3.0.1
auralis                           0.2.8.post2
beautifulsoup4                    4.13.3
blinker                           1.9.0
blis                              0.7.11
bs4                               0.0.2
cachetools                        5.5.2
catalogue                         2.0.10
certifi                           2025.1.31
cffi                              1.17.1
charset-normalizer                3.4.1
click                             8.1.8
cloudpathlib                      0.20.0
cloudpickle                       3.1.1
colorama                          0.4.6
compressed-tensors                0.8.0
confection                        0.1.5
cutlet                            0.5.0
cymem                             2.0.11
datasets                          3.3.2
decorator                         5.2.0
dill                              0.3.8
diskcache                         5.6.3
distro                            1.9.0
docopt                            0.6.2
EbookLib                          0.18
einops                            0.8.1
exceptiongroup                    1.2.2
executing                         2.2.0
fastapi                           0.115.8
ffmpeg                            1.4
filelock                          3.17.0
Flask                             3.1.0
frozenlist                        1.5.0
fsspec                            2024.12.0
fugashi                           1.4.0
future                            1.0.0
gguf                              0.10.0
h11                               0.14.0
hangul-romanize                   0.1.0
httpcore                          1.0.7
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.29.1
idna                              3.10
importlib_metadata                8.6.1
iniconfig                         2.0.0
interegular                       0.3.3
ipython                           8.32.0
itsdangerous                      2.2.0
jaconv                            0.4.0
jedi                              0.19.2
Jinja2                            3.1.5
jiter                             0.8.2
joblib                            1.4.2
jsonschema                        4.23.0
jsonschema-specifications         2024.10.1
langcodes                         3.5.0
langid                            1.1.6
language_data                     1.3.0
lark                              1.2.2
lazy_loader                       0.4
librosa                           0.10.2.post1
llvmlite                          0.44.0
lm-format-enforcer                0.10.10
lxml                              5.3.1
marisa-trie                       1.2.1
markdown-it-py                    3.0.0
MarkupSafe                        3.0.2
matplotlib-inline                 0.1.7
mdurl                             0.1.2
mistral_common                    1.5.3
mojimoji                          0.0.13
morfeusz2                         1.99.9
mpmath                            1.3.0
msgpack                           1.1.0
msgspec                           0.19.0
multidict                         6.1.0
multiprocess                      0.70.16
murmurhash                        1.0.12
mutagen                           1.47.0
nest-asyncio                      1.6.0
networkx                          3.4.2
nltk                              3.9.1
num2words                         0.5.14
numba                             0.61.0
numpy                             1.26.4
nvidia-cublas-cu12                12.4.5.8
nvidia-cuda-cupti-cu12            12.4.127
nvidia-cuda-nvrtc-cu12            12.4.127
nvidia-cuda-runtime-cu12          12.4.127
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cufft-cu12                 11.2.1.3
nvidia-curand-cu12                10.3.5.147
nvidia-cusolver-cu12              11.6.1.9
nvidia-cusparse-cu12              12.3.1.170
nvidia-ml-py                      12.570.86
nvidia-nccl-cu12                  2.21.5
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.4.127
openai                            1.63.2
OpenCC                            1.1.9
opencv-python-headless            4.11.0.86
outlines                          0.0.46
packaging                         24.2
pandas                            2.2.3
parso                             0.8.4
partial-json-parser               0.2.1.1.post5
pexpect                           4.9.0
pillow                            11.1.0
pip                               25.0
platformdirs                      4.3.6
pluggy                            1.5.0
pooch                             1.8.2
preshed                           3.0.9
prometheus_client                 0.21.1
prometheus-fastapi-instrumentator 7.0.2
prompt_toolkit                    3.0.50
propcache                         0.3.0
protobuf                          5.29.3
psutil                            7.0.0
ptyprocess                        0.7.0
pure_eval                         0.2.3
py-cpuinfo                        9.0.0
pyairports                        2.1.1
pyarrow                           19.0.1
pycountry                         24.6.1
pycparser                         2.22
pydantic                          2.10.6
pydantic_core                     2.27.2
Pygments                          2.19.1
pyloudnorm                        0.1.1
pypinyin                          0.53.0
pytest                            8.3.4
python-dateutil                   2.9.0.post0
python-dotenv                     1.0.1
pytz                              2025.1
PyYAML                            6.0.2
pyzmq                             26.2.1
ray                               2.42.1
referencing                       0.36.2
regex                             2024.11.6
requests                          2.32.3
rich                              13.9.4
rpds-py                           0.23.1
safetensors                       0.5.2
scikit-learn                      1.6.1
scipy                             1.15.2
sentencepiece                     0.2.0
setuptools                        75.8.0
shellingham                       1.5.4
six                               1.17.0
smart-open                        7.1.0
sniffio                           1.3.1
sounddevice                       0.5.1
soundfile                         0.13.1
soupsieve                         2.6
soxr                              0.5.0.post1
spacy                             3.7.5
spacy-legacy                      3.0.12
spacy-loggers                     1.0.5
srsly                             2.5.1
stack-data                        0.6.3
starlette                         0.45.3
sympy                             1.13.1
thinc                             8.2.5
threadpoolctl                     3.5.0
tiktoken                          0.9.0
tokenizers                        0.21.0
tomli                             2.2.1
torch                             2.5.1
torchaudio                        2.5.1
torchvision                       0.20.1
tqdm                              4.67.1
traitlets                         5.14.3
transformers                      4.49.0
triton                            3.1.0
typer                             0.15.1
typing_extensions                 4.12.2
tzdata                            2025.1
urllib3                           2.3.0
uvicorn                           0.34.0
uvloop                            0.21.0
vllm                              0.6.4.post1
wasabi                            1.1.3
watchfiles                        1.0.4
wcwidth                           0.2.13
weasel                            0.4.1
websockets                        15.0
Werkzeug                          3.1.3
wheel                             0.45.1
wrapt                             1.17.2
xformers                          0.0.28.post3
xxhash                            3.5.0
yarl                              1.18.3
zipp                              3.21.0

# GPU Information (if applicable)
nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.15              Driver Version: 570.86.15      CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        On  |   00000000:01:00.0  On |                  Off |
|  0%   39C    P5             23W /  450W |    3101MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            1243      G   /usr/lib/xorg/Xorg                      271MiB |
|    0   N/A  N/A            1449      G   /usr/bin/gnome-shell                     45MiB |
|    0   N/A  N/A           25952      C   ...iniconda3/envs/ocr/bin/python        566MiB |
|    0   N/A  N/A           30073      G   /usr/bin/nautilus                        39MiB |
|    0   N/A  N/A           45289      C   python3                                1898MiB |
|    0   N/A  N/A           46902      G   /usr/lib/firefox-esr/firefox-esr        176MiB |
+-----------------------------------------------------------------------------------------+


# CUDA version (if applicable)
nvcc --version

Possible Solutions

[If you have ideas on how to solve the issue, include them here]

Additional Information

? not sure what else :(