Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Error processing OpenAPI json file for python client #11763

Closed
3 of 6 tasks
rpgoldman opened this issue Mar 1, 2022 · 16 comments · Fixed by #12135
Closed
3 of 6 tasks

[BUG] Error processing OpenAPI json file for python client #11763

rpgoldman opened this issue Mar 1, 2022 · 16 comments · Fixed by #12135

Comments

@rpgoldman
Copy link

Bug Report Checklist

  • Have you provided a full/minimal spec to reproduce the issue?
  • Have you validated the input using an OpenAPI validator (example)?
  • Have you tested with the latest master to confirm the issue still exists?
  • Have you searched for related issues/PRs?
  • What's the actual output vs expected output? (just errors out)
  • [Optional] Sponsorship to speed up the bug fix or feature request (example)
Description

Trying to process the OpenAPI spec from the Owlery system, https://github.com/phenoscape/owlery (spec posted below) with the python and python-experimental systems, I get errors processing the files.

Using the command-line locally I get:

Exception in thread "main" java.lang.RuntimeException: Could not process operation:
  Tag: class Tag {
    name: Knowledgebases
    description: null
    externalDocs: null
}
  Operation: null
  Resource: get /kbs
  Schemas: {}
  Exception: Cannot read field "isEnum" because "cp" is null

Using the docker image I get:

Exception in thread "main" java.lang.RuntimeException: Could not process operation:
  Tag: class Tag {
    name: Knowledgebases
    description: null
    externalDocs: null
}
  Operation: null
  Resource: get /kbs
  Schemas: {}
  Exception: null
	at org.openapitools.codegen.DefaultGenerator.processOperation(DefaultGenerator.java:1163)
	at org.openapitools.codegen.DefaultGenerator.processPaths(DefaultGenerator.java:1054)
	at org.openapitools.codegen.DefaultGenerator.generateApis(DefaultGenerator.java:549)
	at org.openapitools.codegen.DefaultGenerator.generate(DefaultGenerator.java:891)
	at org.openapitools.codegen.cmd.Generate.execute(Generate.java:441)
	at org.openapitools.codegen.cmd.OpenApiGeneratorCommand.run(OpenApiGeneratorCommand.java:32)
	at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:66)
Caused by: java.lang.NullPointerException
	at org.openapitools.codegen.languages.PythonClientCodegen.fromProperty(PythonClientCodegen.java:440)
	at org.openapitools.codegen.DefaultCodegen.getContent(DefaultCodegen.java:6755)
	at org.openapitools.codegen.DefaultCodegen.fromOperation(DefaultCodegen.java:4011)
	at org.openapitools.codegen.DefaultGenerator.processOperation(DefaultGenerator.java:1131)
	... 6 more
openapi-generator version

5.4.0

OpenAPI declaration file content or url

YAML is available here: https://gist.github.com/rpgoldman/a65d85f15a9f8b08298cb19446c1b7f0

Generation Details

I believe "steps to reproduce," below, has all the required information, but happy to collect more if needed.

Steps to reproduce

CLI:

 openapi-generator generate --package-name owlery_client -v --generator-name python --output owlery-client -i swagger.json

Docker:

docker run --rm -v "${PWD}:/local" openapitools/openapi-generator-cli generate -i /local/swagger.json -g python-experimental -o /local/owlery-client
Related issues/PRs
Suggest a fix
@rpgoldman rpgoldman changed the title [BUG] Error processing OpenAPI json file [BUG] Error processing OpenAPI json file for python client Mar 1, 2022
@spacether
Copy link
Contributor

spacether commented Mar 30, 2022

Your spec is missing a response schema definition in your endpoints:

paths:
  "/kbs":
    get:
      tags:
        - Knowledgebases
      summary: List available knowledgebases
      description: List names of available knowledgebases in this instance of Owlery
      responses:
        '200':
          description: knowledgebase names
          content:
            application/json:
              examples:
                response:
                  value:
                    - phenoscape
                    - obo-ontologies
                    - monarch

Please include the schema definition under application/json as the spec describes here:
https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.1.0.md#mediaTypeObject

@rpgoldman
Copy link
Author

rpgoldman commented Apr 11, 2022

@spacether Thank you for clarifying that.

I have some better thoughts in a comment I will add below; please ignore this for now.

I think there is still an argument to be made that

Exception in thread "main" java.lang.RuntimeException: Could not process operation:
  Tag: class Tag {
    name: Knowledgebases
    description: null
    externalDocs: null
}
  Operation: null
  Resource: get /kbs
  Schemas: {}
  Exception: null

...could be improved as an error message. Something seems to be going wrong in exception handling, because the lines generating the exception should be printing the exception:

} catch (Exception ex) {
String msg = "Could not process operation:\n" //
+ " Tag: " + tag + "\n"//
+ " Operation: " + operation.getOperationId() + "\n" //
+ " Resource: " + httpMethod + " " + resourcePath + "\n"//
+ " Schemas: " + openAPI.getComponents().getSchemas() + "\n" //
+ " Exception: " + ex.getMessage();
throw new RuntimeException(msg, ex);
}

But in the above printout, the string on the Exception is null. So is there an ill-formed or incomplete Exception being raised here?

I'm not sure why the output from the local generator is different; I suppose it's just more up-to-date?

Cannot read field "isEnum" because "cp" is null

I'm not enough of a Java programmer to fully understand the lines that check the tags, but if there are mandatory tags, wouldn't it make sense to raise an exception for empty values in that code block, instead of waiting till we loop over the tags and try to unpack values from a missing tag?

if (tagNames != null) {
if (swaggerTags == null) {
for (String tagName : tagNames) {
tags.add(new Tag().name(tagName));
}
} else {
for (String tagName : tagNames) {
boolean foundTag = false;
for (Tag tag : swaggerTags) {
if (tag.getName().equals(tagName)) {
tags.add(tag);
foundTag = true;
break;
}
}
if (!foundTag) {
tags.add(new Tag().name(tagName));
}
}
}
}
if (tags.isEmpty()) {
tags.add(new Tag().name("default"));
}

I wish I could help more with a solution, but the flow of control here is hard for me to understand, so I don't see where that exception was originally raised, and searching for "Cannot read field" in this repo on GitHub gives me nothing.

@rpgoldman
Copy link
Author

It occurs to me now that I have been thinking about this all wrong: the problem here was that the OpenAPI spec that I was working from was ill-formed, and the openapi-generator isn't telling me.

But this means that my process for generating the API is defective (in this case because I assumed I had been given a well-formed spec): instead it needs a step that checks to make sure that the specification is well-formed, and not feed to the openapi-generator unless the checker terminates successfully.

It would be great if the generator would do this for us, so users don't have to compose tools themselves, but I should be able to also set up a pipeline for myself, if I can find a good way to run the validator and get good error messages.

Given that framework, we can probably close this issue.

@rpgoldman
Copy link
Author

TL;DR

It appears that the online OpenAPI validators accept response specifications without a schema field, and the openapi-generator does not. The spec link to Media Type Object does not specify that the schema field is required. So if this field is required by the openapi-generator, the generator should emit a clearer warning, since users cannot rely on validation to help identify incorrect (from the PoV of the generator) spec documents.

Details

When I fix that API spec, and validate it using the online swagger editor and the online swagger validator I still get errors from the openapi-generator:

  Exception: Cannot read field "isEnum" because "cp" is null
	at org.openapitools.codegen.DefaultGenerator.processOperation(DefaultGenerator.java:1163)
	at org.openapitools.codegen.DefaultGenerator.processPaths(DefaultGenerator.java:1054)
	at org.openapitools.codegen.DefaultGenerator.generateApis(DefaultGenerator.java:549)
	at org.openapitools.codegen.DefaultGenerator.generate(DefaultGenerator.java:891)
	at org.openapitools.codegen.cmd.Generate.execute(Generate.java:441)
	at org.openapitools.codegen.cmd.OpenApiGeneratorCommand.run(OpenApiGeneratorCommand.java:32)
	at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:66)
Caused by: java.lang.NullPointerException: Cannot read field "isEnum" because "cp" is null
	at org.openapitools.codegen.languages.PythonClientCodegen.fromProperty(PythonClientCodegen.java:440)
	at org.openapitools.codegen.DefaultCodegen.getContent(DefaultCodegen.java:6751)
	at org.openapitools.codegen.DefaultCodegen.fromOperation(DefaultCodegen.java:4006)
	at org.openapitools.codegen.DefaultGenerator.processOperation(DefaultGenerator.java:1131)
	... 6 more

Testing with python-experimental gives different errors:

[main] WARN  o.o.codegen.utils.URLPathUtils - No value found for variable '{basePathToReplace' in server definition '{{basePathToReplace}}' and no user override specified, default to empty string.
[main] WARN  o.o.codegen.utils.URLPathUtils - 'scheme' not defined in the spec (2.0). Default to [http] for server URL [http://}]
[main] WARN  o.o.codegen.utils.URLPathUtils - No value found for variable '{basePathToReplace' in server definition '{{basePathToReplace}}' and no user override specified, default to empty string.
[main] WARN  o.o.codegen.utils.URLPathUtils - 'scheme' not defined in the spec (2.0). Default to [http] for server URL [http://}]
[main] WARN  o.o.c.l.AbstractPythonCodegen - Type null not handled properly in toExampleValue
Exception in thread "main" java.lang.RuntimeException: Could not generate model 'inline_response_200'
	at org.openapitools.codegen.DefaultGenerator.generateModels(DefaultGenerator.java:532)
	at org.openapitools.codegen.DefaultGenerator.generate(DefaultGenerator.java:888)
	at org.openapitools.codegen.cmd.Generate.execute(Generate.java:441)
	at org.openapitools.codegen.cmd.OpenApiGeneratorCommand.run(OpenApiGeneratorCommand.java:32)
	at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:66)
Caused by: com.github.jknack.handlebars.HandlebarsException: model.handlebars:3:3: java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.util.HashMap$Node[] java.util.HashMap.table accessible: module java.base does not "opens java.util" to unnamed module @689eb690
    model.handlebars:3:3
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at com.github.jknack.handlebars.context.FieldValueResolver$FieldMember.setAccessible(FieldValueResolver.java:150)
	at com.github.jknack.handlebars.context.MemberValueResolver.cache(MemberValueResolver.java:82)
	at com.github.jknack.handlebars.context.MemberValueResolver.resolve(MemberValueResolver.java:54)
	at com.github.jknack.handlebars.Context$CompositeValueResolver.resolve(Context.java:199)
	at com.github.jknack.handlebars.internal.path.PropertyPath.eval(PropertyPath.java:52)
	at com.github.jknack.handlebars.Context$PathExpressionChain.next(Context.java:361)
	at com.github.jknack.handlebars.Context$PathExpressionChain.eval(Context.java:381)
	at com.github.jknack.handlebars.Context.get(Context.java:618)
	at com.github.jknack.handlebars.internal.Partial.override(Partial.java:253)
	at com.github.jknack.handlebars.internal.Partial.merge(Partial.java:226)
	at com.github.jknack.handlebars.internal.BaseTemplate.apply(BaseTemplate.java:125)
	at com.github.jknack.handlebars.internal.TemplateList.merge(TemplateList.java:95)
	at com.github.jknack.handlebars.internal.BaseTemplate.apply(BaseTemplate.java:125)
	at com.github.jknack.handlebars.internal.BaseTemplate.apply(BaseTemplate.java:113)
	at org.openapitools.codegen.templating.HandlebarsEngineAdapter.compileTemplate(HandlebarsEngineAdapter.java:92)
	at org.openapitools.codegen.TemplateManager.write(TemplateManager.java:163)
	at org.openapitools.codegen.DefaultGenerator.processTemplateToFile(DefaultGenerator.java:1034)
	at org.openapitools.codegen.DefaultGenerator.processTemplateToFile(DefaultGenerator.java:1021)
	at org.openapitools.codegen.DefaultGenerator.generateModel(DefaultGenerator.java:388)
	at org.openapitools.codegen.DefaultGenerator.generateModels(DefaultGenerator.java:523)
	... 4 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.util.HashMap$Node[] java.util.HashMap.table accessible: module java.base does not "opens java.util" to unnamed module @689eb690
	... 28 more

Looking at the spec, it appears that the validators accept response specifications without a schema field, and the openapi-generator does not. I am not sufficiently sophisticated to know which of the two is correct. When I look at the swagger editor's copy of the JSON schema for oas, I don't see a required on the schema field: https://github.com/swagger-api/swagger-editor/blob/6a3ed91f2ec893d9df5e2e1cb3e62b86d18d74cc/src/plugins/json-schema-validator/oas3-schema.yaml#L465-L475

Media type objects in the textual specification doesn't annotate schema as REQUIRED

@rpgoldman
Copy link
Author

Sorry to keep on at this, but I have added schemas to all responses, and validated the openapi spec using the online editor.

The resulting file successfully generates a python client API using python, but fails for python-experimental with this error:

Exception in thread "main" java.lang.RuntimeException: Could not generate model 'inline_response_200'
	at org.openapitools.codegen.DefaultGenerator.generateModels(DefaultGenerator.java:532)
	at org.openapitools.codegen.DefaultGenerator.generate(DefaultGenerator.java:888)
	at org.openapitools.codegen.cmd.Generate.execute(Generate.java:441)
	at org.openapitools.codegen.cmd.OpenApiGeneratorCommand.run(OpenApiGeneratorCommand.java:32)
	at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:66)
Caused by: com.github.jknack.handlebars.HandlebarsException: model.handlebars:3:3: java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.util.HashMap$Node[] java.util.HashMap.table accessible: module java.base does not "opens java.util" to unnamed module @689eb690
    model.handlebars:3:3
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at com.github.jknack.handlebars.context.FieldValueResolver$FieldMember.setAccessible(FieldValueResolver.java:150)
	at com.github.jknack.handlebars.context.MemberValueResolver.cache(MemberValueResolver.java:82)
	at com.github.jknack.handlebars.context.MemberValueResolver.resolve(MemberValueResolver.java:54)
	at com.github.jknack.handlebars.Context$CompositeValueResolver.resolve(Context.java:199)
	at com.github.jknack.handlebars.internal.path.PropertyPath.eval(PropertyPath.java:52)
	at com.github.jknack.handlebars.Context$PathExpressionChain.next(Context.java:361)
	at com.github.jknack.handlebars.Context$PathExpressionChain.eval(Context.java:381)
	at com.github.jknack.handlebars.Context.get(Context.java:618)
	at com.github.jknack.handlebars.internal.Partial.override(Partial.java:253)
	at com.github.jknack.handlebars.internal.Partial.merge(Partial.java:226)
	at com.github.jknack.handlebars.internal.BaseTemplate.apply(BaseTemplate.java:125)
	at com.github.jknack.handlebars.internal.TemplateList.merge(TemplateList.java:95)
	at com.github.jknack.handlebars.internal.BaseTemplate.apply(BaseTemplate.java:125)
	at com.github.jknack.handlebars.internal.BaseTemplate.apply(BaseTemplate.java:113)
	at org.openapitools.codegen.templating.HandlebarsEngineAdapter.compileTemplate(HandlebarsEngineAdapter.java:92)
	at org.openapitools.codegen.TemplateManager.write(TemplateManager.java:163)
	at org.openapitools.codegen.DefaultGenerator.processTemplateToFile(DefaultGenerator.java:1034)
	at org.openapitools.codegen.DefaultGenerator.processTemplateToFile(DefaultGenerator.java:1021)
	at org.openapitools.codegen.DefaultGenerator.generateModel(DefaultGenerator.java:388)
	at org.openapitools.codegen.DefaultGenerator.generateModels(DefaultGenerator.java:523)
	... 4 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.util.HashMap$Node[] java.util.HashMap.table accessible: module java.base does not "opens java.util" to unnamed module @689eb690
	... 28 more

In case it's helpful, here are some relevant bits of the verbose output. First the schema that the error message says is untranslatable:

      "inline_response_200" : {
        "properties" : {
          "value" : {
            "$ref" : "#/components/schemas/_kbs__kb__subclasses_value"
          }
        },
        "type" : "object"
      },

then the referenced named schema:

      "_kbs__kb__subclasses_value" : {
        "properties" : {
          "@id" : {
            "format" : "uri",
            "type" : "string"
          },
          "superClassOf" : {
            "items" : {
              "format" : "uri",
              "type" : "string"
            },
            "type" : "array"
          },
          "value" : {
            "type" : "string"
          }
        },
        "required" : [ "@id", "superClassOf" ],
        "type" : "object"
      },

I see these warnings, but they do not appear to be relevant:

[main] WARN  o.o.codegen.utils.ModelUtils - Multiple schemas found in the OAS 'content' section, returning only the first one (application/sparql-query)
[main] WARN  o.o.codegen.utils.ModelUtils - Multiple schemas found in the OAS 'content' section, returning only the first one (application/sparql-query)

Perhaps this is related to the API in lines 608-623, which defines two alternative body formats, application/sparql-query or application/x-www-form-urlencoded

@spacether
Copy link
Contributor

I am working on allowing the python-experimental generator to ingest responses where schema is unset in:
#12135

@rpgoldman
Copy link
Author

Thanks, @spacether

A quick couple of notes:

  1. the issue of there being two different content specs might be a bug. According to the OpenAPI spec, content is of type Map, so I think this warning message might indicate a problem:

    [main] WARN o.o.codegen.utils.ModelUtils - Multiple schemas found in the OAS 'content' section, returning only the first one (application/sparql-query)

    Would you like me to open an issue for this?

  2. The exception in the backtrace seems to be the same as this issue which looks to be closed. Has the fix for this been added to openapi-generator? I don't know what the relationship is between swagger-codegen and openapi-generator. The issue discussion claims it was fixed in handlebars 4.3.0. Again, if it would help, I can open an issue for this. Unfortunately that issue does not explain what it is about a spec that triggers this issue and it does not happen in the legacy python code generator.

    Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.util.HashMap$Node[] java.util.HashMap.table accessible: module java.base does not "opens java.util" to unnamed module @689eb690

@rpgoldman
Copy link
Author

Aha! For my sub-issue 2 above, the magical answer is to prepend

_JAVA_OPTIONS="--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED"

to the invocation of openapi-generator. I can use the python-experimental generator as follows:

_JAVA_OPTIONS="--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED" openapi-generator generate --package-name owlery_client -vv --generator-name python-experimental --output owlery-client -i swagger.yaml 2>&1 | tee openapi-generator-experimental.out

Then it runs to completion successfully.

@spacether
Copy link
Contributor

spacether commented Apr 13, 2022

You can ignore the warning message. The multiple content types are stored and used with python-experimental.

There are two problems currently.

  • DefaultCodegen.java assumes that a response mediatype will have a schema
  • the python-experimental code assumes that also

The PR that I am working on will fix both of those issues and this use case for python-experimental.

You can see generation of your spec working in the commits iny PR.

@rpgoldman
Copy link
Author

rpgoldman commented Apr 13, 2022

Great, thanks. I have worked around that for now by adding schemas everywhere. I was having the error reported above when using python-experimental, but with the _JAVA_ARGS workaround now both of the code generators work for me!

@spacether
Copy link
Contributor

This now works on master branch in python-experimental
When schemas are omitted there is no class to deserialize to, so response.body is unset.

@homemdasneves
Copy link

@spacether I'm facing the same issue. Is there a quick way to test your fix with openapi-generator-cli?
I tried to openapi-generator-cli version-manager set 6.0.0 which installed the 6.0.0-beta.
Unfortunatelly it looks like your PR #12135 isn't in the 6.0.0-beta. Shouldn't it?

Alternatively, since I'm just looking for a functional handlebars based generator, is there any other I could use?

@spacether
Copy link
Contributor

spacether commented May 23, 2022

You need to pull down the latest master and build it using the instructions in the Readme to use it. Python-experimental is the only handlebars generator as far as I know.

Also what's your reason for using handlebars? I am curious about your use case.

@homemdasneves
Copy link

Thanks.

About my use case, I'm generating some react components for a backoffice. It's a "bring your own API" backoffice (react-admin). So per each model of the API I'd need some CRUD components for the backoffice, which I'd rather generate as much as possible. Most of the times I needed to generate something from an API spec, the available templates have sufficed. Now I want something a bit further from a simple customization and honestly I'd rather work it out on the template side, rather than going the more involved way of making a custom generator. Unfortunately mustache logicless templating won't let me even customize a model name. I feel handlebars could unlock a lot of possibilities in this project when it gets more support / generators. I've seen some comments aligned with that idea (like this one), but the reality is that generators aint't adopting handlebars (at least yet). Can't really understand why. Yours is the only one. Maybe I'm overlooking it but it almost feels to me that with handlebars and a single good generator, the Java part of the current generators could be ditched in favor of purely template-based ones.

Thanks for your work on this!

@spacether
Copy link
Contributor

spacether commented May 23, 2022

If you wanted you could convert another generator to use handlebars. I made a tool to convert from mustache to handlebar templates. Sadly our schema interfaces in the Java layer are not as simple and self similar as your comment implies. Through great effort I have made them close enough and added missing schema info. If you look at the python-experimental model templates you will see that they recursively call each other to render any schema and create model classes at any depth of schema inlinining. Using handlebars definitely allows one to use fancier if else not and or conditionals which I made use of.

@corrigancd
Copy link

Just leaving this here, mainly in case it helps someone or might provide a clue.

So, I was using Java 17 with the python generator with the petstore.json and got the java.lang.reflect.InaccessibleObjectException error.

I was able to run other generators, confirmed this by testing with the javascript, java, go generators.

Then I swapped to Java 11 and was able to generate a python SDK when using Java 11, haven't tried other versions of Java, but perhaps someone might be able to comment on the Java version compatibility..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants