text-generation-inference:3.0.1 docker container timeout on image fetching from fastapi static files.

[dinoelT]

System Info

We are running the tgi container and a fastapi app that queries the model.
I will refer to them as "tgi" and "llm-api".
Both docker containers are on the same bridge network.

The "llm-api" app receives an image from the frontend, saves it to the FastApi StaticFiles folder and then passes the image_url to the LLM container.
I tried to pass the image as base64, but the body is too big and I got an error.

I tried passing the url relative to the container, for example, the "tgi" container would receive the image with the url http://llm-api:5050/images/image.png.
In this case, requesting the image times out.

Our "llm-api" container is also accessible through https.
The url looks something like https://www.our-api.com/images/image.png.
The image is accessible in the browser, but still times out when TGI tries to request it.

I also tested png vs jpeg(to ensure that the size of the image is not the problem), but both fail.

When I enter the "tgi" container using docker exec, and curl the image url(both container relative url and internet accessible url), it downloads successfully.

I also tested using an s3 presigned url for the image, and it works without no problems.

I think it has something to do with how the image is served, maybe headers.
When making the request in python for two images, first one is the "rabbit.png" from the examples on huggingface and the other one being the image served through fastapi, I get these headers:
rabbit.png:
{'Content-Type': 'image/png', 'Content-Length': '421149', 'Connection': 'keep-alive', 'Last-Modified': 'Tue, 09 May 2023 19:50:38 GMT', 'x-amz-storage-class': 'INTELLIGENT_TIERING', 'x-amz-server-side-encryption': 'AES256', 'x-amz-version-id': 'izTd4OuYRPRtTQE2vn_5MAIwzslFPXtp', 'Content-Disposition': 'inline; filename*=UTF-8''rabbit.png; filename="rabbit.png";', 'Accept-Ranges': 'bytes', 'Server': 'AmazonS3', 'Date': 'Tue, 21 Jan 2025 09:33:38 GMT', 'ETag': '"5a1a38d7128e43c310b6b3e7aac9b368"', 'X-Cache': 'Hit from cloudfront', 'Via': '1.1 75368b7bf20cbbab33ae6a3ed6416920.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'VIE50-P3', 'X-Amz-Cf-Id': 'qbnX2L_TnkQ8g97zTpEwH0d-1lF4vQizAcCDsfxk5jN5suN56UnYWQ==', 'Age': '320', 'Content-Security-Policy': "default-src 'none'; sandbox", 'Vary': 'Origin'}

Our image:
{'Date': 'Tue, 21 Jan 2025 09:33:39 GMT', 'Content-Type': 'image/png', 'Content-Length': '1938871', 'Connection': 'keep-alive', 'server': 'uvicorn', 'last-modified': 'Tue, 21 Jan 2025 08:06:37 GMT', 'etag': '26b32cc22f24763220da7f8507d64ca0', 'accept-ranges': 'bytes', 'content-disposition': 'inline; filename="yourfilename.jpg"', 'content-security-policy': "default-src 'none'; sandbox", 'vary': 'Origin'}

Does someone know why this is happening? Is it a bug or is there a reason why this limitation is imposed?
Thank you very much for your time.

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Steps to reproduce:

• Serve images through fastapi Static files.
• Call TGI with the url of the image

Expected behavior

Images to load when being served by fastapi StaticFiles