Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG][PYTHON] Do not set Content-Type for GET, HEAD or DELETE requests #9852

Merged
merged 3 commits into from
Jul 7, 2021

Conversation

johnboyes
Copy link
Contributor

@johnboyes johnboyes commented Jun 27, 2021

This commit ensures that the Python generator no longer sets a default
Content-Type ofapplication/json for GET, HEAD and DELETE
requests.

Fixes #9831

Related issues/PRs: #476, #1061, #5941, #6167

Having the Content-Type set for these requests was causing issues with
other tools which insist that GET, HEAD and DELETE requests do not have
a Content-Type (as per the OpenAPI 3 specification).

An example of the problem that this commit fixes is when using
Prism as a validation proxy.

Prism rejects any GET request that has a Content-Type.

Here is an example of the problem manifesting itself.

To validate the fix in this commit:

  1. Start with any OpenAPI3 spec e.g. the Petstore example at
    https://editor.swagger.io/
  2. Generate Python client code for the spec
  3. Look at the generated rest.py e.g. in the standard sample in this
    repo
    and see that the Content-Type defaults to application/json
    for all HTTP methods (including GET, HEAD and DELETE), rather than
    there being no Content-Type for GET, HEAD and DELETE.

cc @taxpon, @frol, @mbohlool, @cbornet, @kenjones-cisco, @tomplus, @Jyhess, @arun-nalla, @spacether

PR checklist

  • Read the contribution guidelines.
  • Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
  • Run the following to build the project and update samples:
    ./mvnw clean package 
    ./bin/generate-samples.sh
    ./bin/utils/export_docs_generators.sh
    
    Commit all changed files.
    This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
    These must match the expectations made by your contribution.
    You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
    For Windows users, please run the script in Git BASH.
  • File the PR against the correct branch: master, 5.1.x, 6.0.x
  • If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

The Python generator no longer sets a default `Content-Type` of
`application/json` for `GET`, `HEAD` and `DELETE` requests.

Having the `Content-Type` set for these requests was causing issues with
other tools which insist that GET, HEAD and DELETE requests do not have
a Content-Type (as per the OpenAPI 3 specification).

An example of the problem that this commit fixes is when using
[Prism][1] as a [validation proxy][2].

[Prism rejects any GET request that has a Content-Type][3].

Here is [an example of the problem manifesting itself][4].

To validate the fix in this commit:

1. Start with any OpenAPI3 spec e.g. the Petstore example at
https://editor.swagger.io/
2. Generate Python client code for the spec
3. Look at the generated `rest.py` e.g. in the [standard sample in this
repo][5] and see that the `Content-Type` defaults to `application/json`
for all HTTP methods (including `GET`, `HEAD` and `DELETE`), rather than
there being no `Content-Type` for `GET`, `HEAD` and `DELETE`.

Fixes OpenAPITools#9831

[1]: https://github.com/stoplightio/prism
[2]: https://meta.stoplight.io/docs/prism/docs/guides/03-validation-proxy.md
[3]: stoplightio/prism#1408 (comment)
[4]: https://github.com/agilepathway/gauge-openapi-example/pull/28/checks?check_run_id=2888606052#step:13:18
[5]: https://github.com/OpenAPITools/openapi-generator/blob/969cea8ce10cb9d012c3936fb377d631c0d044c9/samples/openapi3/client/petstore/python/petstore_api/rest.py#L141
@johnboyes
Copy link
Contributor Author

Hi, is anyone able to take a look at this, please? 🙏🏻
@taxpon, @frol, @mbohlool, @cbornet, @kenjones-cisco, @tomplus, @Jyhess, @arun-nalla, @spacether

@wing328
Copy link
Member

wing328 commented Jul 4, 2021

@johnboyes thanks for the PR. My understanding is that it shouldn't set a default content-type header at all. I think that's how other clients (e..g ruby, php, C#, etc) set the content-type at the moment: no default and set the content-type only if it's provided in the spec.

@johnboyes
Copy link
Contributor Author

Hi @wing328, thanks for the comment. If that's the case, I propose to treat that (not setting a default content-type header for any type of request) as a subsequent separate issue, if you agree? Reasoning being that all of the related issues/PRs to this one (#476, #1061, #5941, #6167) all specifically mention GET, HEAD and DELETE requests only, and I'm wary of this PR having unexpected side-effects if it grows in scope - especially as the PR as is fixes a genuine, current issue. Thoughts?

@wing328
Copy link
Member

wing328 commented Jul 5, 2021

Updated the samples (as mentioned in the 3rd item in the PR checklist). Let's see if the CI tests pass.

@wing328
Copy link
Member

wing328 commented Jul 6, 2021

Travis CI (including the python tests) was not run since this project has run out of free build credits.

I tested it locally but got the following errors:

        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                # Only set a default Content-Type for POST, PUT, PATCH and OPTIONS requests
                if (method != 'DELETE') and ('Content-Type' not in headers):
                    headers['Content-Type'] = 'application/json'
                if query_params:
                    url += '?' + urlencode(query_params)
>               if re.search('json', headers['Content-Type'], re.IGNORECASE):
E               KeyError: 'Content-Type'

petstore_api/rest.py:148: KeyError

To run the petstore server locally, please refer to https://github.com/OpenAPITools/openapi-generator/wiki/Integration-Tests#how-to-add-integration-tests-for-new-petstore-samples

The earlier commit 9dfe1f6 introduced a bug for `DELETE` requests on the
standard Python generator.  This commit fixes that bug and also includes
the updated samples.
@johnboyes
Copy link
Contributor Author

I tested it locally but got the following errors:

Thanks @wing328, I was able to recreate the error locally too (thanks to your helpful comments, which helped me work out how to update the samples properly and to run the integration tests locally). I have fixed the error and also updated the samples in e846404 - very grateful if you can take another look, please.

@wing328
Copy link
Member

wing328 commented Jul 7, 2021

Tested locally and the result is good:

============================= 159 passed in 4.65s ==============================
___________________________________ summary ____________________________________
  py3: commands succeeded
  congratulations :)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:22 min
[INFO] Finished at: 2021-07-07T11:01:49+08:00
[INFO] ------------------------------------------------------------------------

@wing328 wing328 merged commit 7084a79 into OpenAPITools:master Jul 7, 2021
johnboyes added a commit to agilepathway/available-pets-consumer-contract that referenced this pull request Nov 2, 2021
Removing because there is a bug whereby the Java OpenAPI Generator sends
GET requests with a content-type, which causes Prism to reject these
requests.

The bug is the same (in Java) as the [Python bug][1] which was fixed
earlier in 2021.  It may well be more difficult to fix in Java, however,
as Java is the core language which OpenAPI Generator is written in.

Using the Prism validation proxy mode is more important than using the
OpenAPI Generator, so we may well want to refactor this repo to
reinstate the validation proxy and to remove the Generator.  This would
mean that we would hand-roll our own client code, but that's ok - as the
[Thoughtworks Tech Radar says][2], this type of bug is one of the risks
of using generated code rather than rolling our own.

[1]: OpenAPITools/openapi-generator#9852
[2]: https://www.thoughtworks.com/radar/techniques/spec-based-codegen
johnboyes added a commit to agilepathway/available-pets-consumer-contract that referenced this pull request Nov 2, 2021
* Convert example from Python to Java

This repo now uses a Java client SDK, rather than
the Python one that was being used before.  We only used Python before
this commit because we based this repo on [it's Python sibling][1].

This commit also shows that the code change to use a different language
in this workflow is really not very big.

[1] https://github.com/agilepathway/gauge-openapi-example

* Remove validation proxy mode

Removing because there is a bug whereby the Java OpenAPI Generator sends
GET requests with a content-type, which causes Prism to reject these
requests.

The bug is the same (in Java) as the [Python bug][1] which was fixed
earlier in 2021.  It may well be more difficult to fix in Java, however,
as Java is the core language which OpenAPI Generator is written in.

Using the Prism validation proxy mode is more important than using the
OpenAPI Generator, so we may well want to refactor this repo to
reinstate the validation proxy and to remove the Generator.  This would
mean that we would hand-roll our own client code, but that's ok - as the
[Thoughtworks Tech Radar says][2], this type of bug is one of the risks
of using generated code rather than rolling our own.

[1]: OpenAPITools/openapi-generator#9852
[2]: https://www.thoughtworks.com/radar/techniques/spec-based-codegen

* Add the OpenAPITools config file to source control

As per the guidance in the [OpenAPITools README][1]

[1]: https://github.com/OpenAPITools/openapi-generator-cli
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG][PYTHON] Content-Type: application/json on GET(also HEAD,DELETE) requests
2 participants