POST with UploadFile returns 422 when called by client application

[aretasg]

Example

import os
import shutil
from pathlib import Path
from tempfile import TemporaryDirectory

from fastapi import FastAPI, File, UploadFile

app = FastAPI()

def save_upload_file(upload_file: UploadFile, destination: Path) -> None:

    # https://github.com/tiangolo/fastapi/issues/426#issuecomment-542828790

    try:
        with open(destination, "wb") as buffer:
            shutil.copyfileobj(upload_file.file, buffer)
    finally:
        upload_file.file.close()


@app.post("/upload_two_files", summary="Upload and save two files")
def upload_sequences(
    first_file: UploadFile = File(...),
    second_file: UploadFile = File(...)
    ):

    app_root = os.path.dirname(os.path.abspath(__file__))
    upload_folder = os.path.join(app_root, 'upload-folder')
    prefix = os.path.abspath(upload_folder)

    temp_dir = TemporaryDirectory(prefix=prefix)

    first_file_filename = first_file.filename
    attributes_file_location = os.path.join(temp_dir.name, first_file_filename)
    save_upload_file(first_file, first_file_file_location)

    second_file_filename = second_file.filename
    second_file_file_location = os.path.join(temp_dir.name, second_file_filename)
    save_upload_file(second_file, second_file_file_location)

    response = {"first_file": first_file_filename, "second_file": second_file_filename}

    return response

Description

I am trying to upload two files and process them.
It works from OpenAPI/Swagger UI (/docs) and from Postman with 'Content-Type: multipart/form-data'.
However, when a KNIME workflow I have inherited is making a call the code generates 422.
I have checked the KNIME node and it also contains 'Content-Type: multipart/form-data'. You can read more about the node here.
I have tried changing headers for the past few days and about to give up but would really prefer to keep FastAPI instead of turning back to Flask. I realize it's a very KNIME specific question but I would appreciate any advice at this point.

Environment

• OS: Linux
• FastAPI Version: 0.60.1
• Python version: 3.8

Additional context

Before I had this piece of Flask code which did work with the client code:

import os
from tempfile import TemporaryDirectory

from werkzeug.utils import secure_filename
from flask import Flask, request

app = Flask(__name__)

@app.route('/upload_two_files', methods=['POST'])
def upload_sequences():

    first_file = request.files['first_file']
    second_file = request.files['second_file']

    app_root = os.path.dirname(os.path.abspath(__file__))
    upload_folder = os.path.join(app_root, 'upload-folder')
    prefix = os.path.abspath(upload_folder)

    temp_dir = TemporaryDirectory(prefix=prefix)

    first_file_filename = secure_filename(first_file.filename)
    first_file_location = os.path.join(prefix, first_file_filename)
    first_file.save(first_file_location)

    second_file_filename = secure_filename(second_file.filename)
    second_file_location = os.path.join(prefix, second_file_filename)
    second_file_file.save(second_file_location)

    response = {"first_file": first_file_filename, "second_file": second_file_filename}

[ycd]

Do you have python-multipart installed on your current environment?

[zachbellay]

@ycd I am encountering this issue as well and do have python-multipart installed.

[zachbellay]

I was using curl to hit the Fast API endpoint, and as it turns out my curl request was incorrect. After opening the Swagger docs, a suggested curl was given, which does work in this case:
curl -X POST "http://localhost:8000/inference/masks" -H "accept: application/json" -H "Content-Type: multipart/form-data" -F "[email protected];type=image/jpeg" . Sorry for hijacking, hopefully this helps anyone else with my problem.

[aretasg]

@ycd yes, I do have it installed.

@zachbellay this is not much different from what Swagger docs cURL example (which does work for me) except you have added type?

I am more hoping to know whether there is a way to get a more detailed stack trace from Uvicorn on my 422 besides 422 Unprocessable Entity

[ycd]

@aretasg can you try sending a curl request with Swaggers suggested curl?

[aretasg]

@ycd Sure, just did that and it worked fine I have executed the following cURL which is a copy/paste from Swagger.

curl -X POST "http://10.0.5.13:5000/upload_two_files" -H  "accept: application/json" -H  "Content-Type: multipart/form-data" -F "[email protected]" -F "[email protected]"

It seems I have no issues making a call via Swagger/curl/Postman but I do from KNIME... Bizzare. The content-type is set to multipart in KNIME so I am wondering what else could I be missing? Perhaps some other header...

[aretasg]

With the help of middleware, I have managed to get a bit more detailed logs. It seems the client is not sending a second file however I know it does because of the working Flask example (see my initial message).

2020-08-19 21:29:17,600 : [DEBUG] : app : logmy422 :: that failed
2020-08-19 21:29:17,600 : [DEBUG] : app : logmy422 :: [b'{"detail":[{"loc":["body","second_file"],"msg":"field required","type":"value_error.missing"}]}']
INFO:     10.0.5.13:35478 - "POST /upload_two_files HTTP/1.1" 422 Unprocessable Entity
2020-08-19 21:29:17,601 : [DEBUG] : multipart : callback :: Calling on_part_begin with no data
2020-08-19 21:29:17,601 : [DEBUG] : multipart : callback :: Calling on_header_field with data[38:57]
2020-08-19 21:29:17,601 : [DEBUG] : multipart : callback :: Calling on_header_value with data[59:110]
2020-08-19 21:29:17,601 : [DEBUG] : multipart : callback :: Calling on_header_end with no data
2020-08-19 21:29:17,601 : [DEBUG] : multipart : callback :: Calling on_header_field with data[112:124]
2020-08-19 21:29:17,601 : [DEBUG] : multipart : callback :: Calling on_header_value with data[126:145]
2020-08-19 21:29:17,601 : [DEBUG] : multipart : callback :: Calling on_header_end with no data
2020-08-19 21:29:17,601 : [DEBUG] : multipart : callback :: Calling on_headers_finished with no data
2020-08-19 21:29:17,602 : [DEBUG] : multipart : callback :: Calling on_part_data with data[149:4096]
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 386, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/fastapi/applications.py", line 181, in __call__
    await super().__call__(scope, receive, send)
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 26, in __call__
    await response(scope, receive, send)
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/starlette/responses.py", line 228, in __call__
    await run_until_first_complete(
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/starlette/concurrency.py", line 18, in run_until_first_complete
    [task.result() for task in done]
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/starlette/concurrency.py", line 18, in <listcomp>
    [task.result() for task in done]
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/starlette/responses.py", line 225, in stream_response
    await send({"type": "http.response.body", "body": b"", "more_body": False})
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 156, in _send
    await send(message)
  File "/home/aretas/test_deployment/service-test/venv/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 516, in send
    raise RuntimeError("Response content shorter than Content-Length")
RuntimeError: Response content shorter than Content-Length

[aretasg]

Using Request directly instead of UploadFile, I can confirm that the endpoint does not receive the second file from the client - could be a parsing issue by Multipartparser class in Starlette due to unusual request formatting by the client.

I don't think this a FastAPI issue any longer, so I will close with this comment.

[tiangolo]

Thanks for the help here everyone! 👏 🙇

Thanks for reporting back and closing the issue @aretasg 👍

If you are still having issues, one way to check if the problem is related to Starlette would be to make a simple test with Quart, which is mostly compatible with Flask, so you should be able to re-use the Flask code. And as Quart uses the same ASGI spec, you could run it with Uvicorn, to also discard a problem with Uvicorn (or alternatively, try Hypercorn instead of Uvicorn).

[aretasg]

I have opened an issue on Starlette's repo and identified the problem and a workaround. It is due to non-standard line breaks used in the request body which is now I think it an issue of python-multipart library.
encode/starlette#1059