Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkmst- Prokarma microservice toolkit for java #7023

Merged
merged 20 commits into from
Dec 19, 2017

Conversation

sanshuman
Copy link
Contributor

@sanshuman sanshuman commented Nov 22, 2017

Description of the PR

Features:

  1. Can generate springboot project dynamically from the swagger spec.
  2. Inbuilt integration with services like eureka registry, zipkin, spring boot admin, circuit breaker
  3. Configurable middle-ware handlers like trace audit correlation rate-limit
  4. provides cucumber templates and test steps
  5. Inbuilt configurable spring security
  6. Gatling tests template

Usage:

This is the sample Prokarma microservice template project generated from swagger codegen. Using swagger specification you can convert any definition to spring boot microservice.
It has the integration with the below services:
eureka registry, zipkin , spring boot admin, circuit breaker.

By default only the circuit breaker annotation is enabled. But one has to write the fallback method. The needed dependency for it is also been enabled. To generate the integration and
related configuration for eureka, zipkin, spring boot admin one has two options:

  1. When generating from UI click on pkmst under Generate Server drop-down. One has to provide vendor extensions inside the swagger specification as below:
swagger: "2.0"
info:
  description: "This is a sample Product Catalogue Server.\
    \  For this sample, you can use the api key `special-key` to test the authorization\
    \ filters."
  version: "1.0.0"
  x-codegen:
    eurekaUri: "http://localhost:8080"
    zipkinUri: "http://localhost:9411"
    springBootAdminUri: "http://localhost:8000"
    pkmstInterceptor: "true"

PLease note the vendor extensions are inside the info tag of the swagger specification. All the tags are case sensitive. Once given all the related configuration and the dependency
will be enabled.

  1. When generating from the maven plugin one has to provide configuration inside pom as below:
    inside the swagger codegen maven plugin under the configuration section
<configuration>
	<inputSpec>product-swagger.yaml</inputSpec>
	<language>pkmst</language>
	<output>${project.build.directory}/generated-sources</output>
<configOptions>
	  <groupId>com.prokarma</groupId>
	  <artifactId>product-catalogue</artifactId>
	  <artifactVersion>1.0</artifactVersion>
	  <basePackage>com.prokarma.pkmst</basePackage>
	  <serviceName>ProductCatalogue</serviceName>
	  <eurekaUri></eurekaUri>
	  <zipkinUri>http://localhost:9411</zipkinUri>
	  <springBootAdminUri>http://localhost:8000</springBootAdminUri>
	  <pkmstInterceptor>true</pkmstInterceptor>
</configOptions>
</configuration>

Generated project details

Genareted project will be Springboot project which contains three profiles local, dev, dev-config which can be configured accordingly. Once you have provided the uris you can see the necessary configurations generated inside the local and the dev
yml files. dev-config is for the config server if you want to use.(also enable the dependency for the config server inside the pom)

[Note: one has to run the zipkin, eureka, spring boot admin servers to be able to connect from the app. This project assumes that you have all the servers
up and running.]

Also provided are the middleware handlers:

  1. traceInterceptor: is an id passed in from client and will be unique with an application context. The id will be passed into the backend and return to the consumer for transaction tracing.

  2. correlationInterceptor: generates a UUID in the first API/service and pass it to all other APIs/services in the call tree for tracking purpose.

  3. rateLimitInterceptor: is a rate limiting handler to limit number of concurrent requests on the server. Once the limit is reached, subsequent requests will be queued for later execution. The size of the queue is configurable.

  4. auditInterceptor: audit logs most important info about request and response into audit.log in JSON format with config file that controls which fields to be logged

  5. bodyInterceptor:s a body parser middleware that is responsible for parsing the content of the request based on Content-Type in the request header.

To be able to generate the handlers and the necessary configurations one has to provide the pkmstInterceptor key inside the vendor extensions or through
the maven plugin.
Once provided all the handlers are registered in the interceptor registry inside the appconfig file and can be enabled or disabled through the configuration provided inside
the application yml as below:

interceptor:
   enable:
       audit: true
       body: true
       rateLimit: true
       traceability: true
       correlation: true

For testing we have placeholders for junit test class, integration test class, cucumber sample
feature file(implement according to your needs), gatling load test.

Ways to run the project:

  1. Normal spring boot application

  2. Using Dockerfile to run in the container:
    dockerfile will be generated inside the project by default. Image can be created using docker cli or through the maven plugin

<build>
  <plugins>
    ...
    <plugin>
      <groupId>com.spotify</groupId>
      <artifactId>docker-maven-plugin</artifactId>
      <version>VERSION GOES HERE</version>
      <configuration>
        <imageName>example</imageName>
        <dockerDirectory>docker</dockerDirectory>
        <resources>
           <resource>
             <targetPath>/</targetPath>
             <directory>${project.build.directory}</directory>
             <include>${project.build.finalName}.jar</include>
           </resource>
        </resources>
      </configuration>
    </plugin>
    ...
  </plugins>
</build>

Use manifest.yml file to push the application to the cloud.

HttpLogging filter is provided for logging in the request and response. Can be found inside the com.prokarma.pkmst.logging package.
Spring security is also provided to secure the resources. Please modify according to your needs.

First run:
Import the project in to the eclipse. Run the app as an spring boot application.The project will run on http://localhost:8008
Swagger ui available on:
http://localhost:8008/swagger-ui.html
If all the configurations have been enabled(depending on the port) below are some of the URls to access:
eureka: http://localhost:8080
zipkin: http://localhost:9411

@bbdouglas (2017/07) @JFCote (2017/08) @sreeshas (2017/08) @jfiala (2017/08) @lukoyanov (2017/09) @cbornet (2017/09)

io.swagger.codegen.languages.ZendExpressivePathHandlerServerCodegen
io.swagger.codegen.languages.PkmstServerCodegen
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you keep the alphabetical order ?

public class {{serviceName}}Steps {

@Given("^I want to write a step with precondition$")
public void i_want_to_write_a_step_with_precondition() throws Throwable {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the point of this ?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pkmst microservice project will
a)Have all dependencies for writing BDD test using cucumber in POM
b)we will generate a sample .feature file (BDD driven test)
b)We will also generate a sample {servicename}steps file ,making it easy for developer to go ahead and extend woth actual test code.

Following tests are autogenerated by pkmst.
a)Unit test via junit
b)Behaviour driven Test via cucumber
c)Performance Test autogenerated via gatling

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pkmst microservice project provides support for cucumber based BDD.
a)pkmst will generate a sample feature file called Pkmst.feature
b)generate all the needed dependency in POM
c)generate a step file (Actual implementation of the feature file) file .
The step file and feature file is a place holder which development can go and edit as needed

For now we are making sure BDD support via cucmber with needed files and configurations exist.but developer needs to edit it for exct behaviour.

Next steps : we are looking at generating the .feature file and step.java based on swagger. //in progress.

@cbornet
Copy link
Contributor

cbornet commented Nov 23, 2017

Can you give more details on what pkmst framework is (never heard of it) ? Is it Free and Open Source ? It looks like a lot of duplication from the existing Spring templates. It would probably be better to have it as a SpringCodegen library.

Arranged pkmstServerCodeGen and SymfonyServerCodegen in aphabatical order
@ninodpillai
Copy link

Sure ,
History : pkmst was fully developed by prokarma-coe (center of excellence) . we spent nearly an year doing initial research and finally building a design for our microservice toolkit. Initial release was based on Maven archetype. Once we successfuly used it we thaught its a good idea to generate the same project but now with swagger code gen. (Link provided below for additional documentation on maven archetype for pkmst)

pkmst : Release under open source Apache 2.0 . we are validating all the licenses pkmst depends on.(Whitesource plugin is used with our jenkins pipeline).

IMP : We have used some class to understand how swagger code generator architecture works, but pkmst is a complete design in itself and opiniated in what value it provides.

CloudFoundry Ready to be deployed to cloud foundry with defaul manifest.

Middleware handler : In order for microservice to depend on least libraries or other solutions like API Gateway we added lot of cool extenable interceptor like
traceInterceptor
correlationInterceptor
rateLimitInterceptor
auditInterceptor
bodyInterceptor:s

Idea here was to demonstrate microservice can be intelligent .
Example

  • We can bring in some of the core resposibility of api gateway into microservice
  • We can bring in core responsibility of ESB (Transform , audit etc) into microservice.

via iddleware handler into pkmst microservice. This make microservice intelligent and have its own life cycle. we extended middleware handler framework provided by light4j into pkmst spring boot request handler pipeline.

Security : OAuth2 configuration generated . easy to customize and enable /disable on needed REST Url.

Containerization : Default dependencies and dockerfile is generated . Developer can create Docker image of the generated microservice with few simple commands

Logmanagment : Cloud foundry based binding are generated that sends logs to respective log management server (Example ELK or Splunk) .Cloud foundry bindings are generated based on vendor extensions supplied via swagger file.

Service registry : Microservice generated can automatically register itself with Registry server configured (Example Eureka or consul etc)

Tests : All the needd dependency and default test classes are automatically generated . Most of the microservice testing from funcational and non functional apsects are addressed by providing following broad support on testing front.
unit Test : Junit test are automatically generated based on swagger.
cucumber test : pkmst highly advocates BDD and generates all default class and files . we are yet figuring out how to generate the complete step file based on swagger definition. current we generate all file and user need to modify them.
performance test : pkmst generated needed scala files and class to test the performance of microservice deployed via gatling.

Tracing : Pkmst default support tracing to enable and viaualize tracing across microservices. This feature enbles for easy trouble shooting of the service in the chain of services that has the issue. which otherwise would be a very difficult problem in production to trace. pkmst leverages zipkin to provide the tracing support.

API documentation : Pkmst provided swagger UI support.

**Example ** : We have good number of example that demonstrate basic examples + feature example of pkmst .(Currently all example sits within prokarma private repo we need to know more on where we can commit the same).

Summing up : No single technology which we looked into provided all the capability in one box . it was developers /designer /architect responsibility to understand each of them , validate pros/cons and leverage them in their projects. this took lot of time.

PKMST bring all the best practise , tool and stack in one box , that can then be modified extended and changed as needed by teams to support their own needs ,but we wanted them to start from where we left and not reinvent the wheels.

a)pkmst docuemtation for maven Archetype
https://mst4j-getting-started.mybluemix.net/#/core-eco

b)deployed swagger editor on prokarma bluemix clodu with pkmst support
http://swagger-editor.mybluemix.net/

We are glad that we are releasing this to swagger codegen project and we highly appreciate your valuable time in providing feedback.

Please let us know how can we improve and what can we do more.

regards

@wing328 wing328 added this to the v2.3.0 milestone Dec 1, 2017
@cbornet
Copy link
Contributor

cbornet commented Dec 1, 2017

I think you did a great work but I wonder if this repo is the good one to host such code. This is a very opiniated way of doing things and I think swagger-codegen should focus on much simpler and non-opiniated integration of frameworks. Or else there will be a multiplication of projects that then need to be maintained, increase the CI time, etc...
Have you tried doing it as a separate project and using the maven plugin to generate the swagger parts at compile time ? This is how it's done in JHipster (another opiniated server generator) and it works pretty well.

@wing328
Copy link
Contributor

wing328 commented Dec 1, 2017

I'll review this weekend and merge accordingly if no question from me.

@ninodpillai
Copy link

Thank you for the comment , was the hard work of the team. :-)

let me try to address the question regarding , if managing this generator will increase miantainance and cycle time.

IMP : Like all other generators , swagger codegen project will needs to only manage the generator java class and supporting mustache templates No other maintainance is needed.
it will be another awesome generator with capability to generate full grown enterprise grade microservice .

Few points revisited
a)Yes this framework brings to the table all advance features to build a microservice based on swagger yaml.

b)It does not forces anything by default . Example when a developer generates a microservice using this generator , he get only the basic features enabled. As the developer/team matures he get to choose and enable advance features as he needs. Also developer can go ahead and add/modify replace without fear of propriery code. (as everything is open sourced)

c)We have used all open source and standard tool used by community(Cucumber, gatling ,zipkin, eureka, spring, java etc) .

Jhispter : I am fan of jhispters and we have drawn some inspirtation with frameworks like jhispter, gokit etc before comming with our own.

Also my personell feeling is dev teams already know how to develop basic /hello world services and they want something they can invest timeand use in their respective enterprise project and extend as needed.

Once again thanks for the opportunity for sharing our thaughts.

Out team has created video that show veryhigh level view + swagger code gen integration .

https://drive.google.com/file/d/1y3uV4HWJfw2jcfBpYq7uVl_bGpt5b7KV/view?invite=CITsoQU&ts=5a214c93

regards

/**
* Created by prokarma on 04/09/17.
*/
public class PkmstServerCodegen extends DefaultCodegen implements CodegenConfig {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanshuman given that it's a Java generator, can it extend AbstractJavaCodegen instead?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes , makes more sense . Refactoring to AbstractJavaCodegen.

}

public String getName() {
return "pkmst";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about naming it as "java-pkmst" to follow the convention "{lang}-{framework}"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @wing328 We have changed the server codegen java file accordingly as the convention suggest from pkmstservercodegen to JavaPKMSTServerCodegen. Above name "pkmst" is what the user sees. Please let us know if its fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also added the missing typeInfoAnnotation.mustache file. Could you send the swagger yaml file used for testing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi William we have refactored and merged the code. Can you share some swagger yaml apart from pet store that we can use for testing. Also let us know if you have any queries. Our testing is in progress.
@wing328

@wing328
Copy link
Contributor

wing328 commented Dec 5, 2017

Btw, I got some errors when using the generator:

[main] ERROR io.swagger.codegen.AbstractGenerator - pkmst/typeInfoAnnotation.mustache (No such file or directory)
Exception in thread "main" java.lang.RuntimeException: Could not generate model 'Animal'
	at io.swagger.codegen.DefaultGenerator.generateModels(DefaultGenerator.java:409)
	at io.swagger.codegen.DefaultGenerator.generate(DefaultGenerator.java:735)
	at io.swagger.codegen.cmd.Generate.run(Generate.java:285)
	at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:35)
Caused by: java.lang.RuntimeException: can't load template pkmst/typeInfoAnnotation.mustache
	at io.swagger.codegen.AbstractGenerator.getTemplateReader(AbstractGenerator.java:64)
	at io.swagger.codegen.DefaultGenerator$4.getTemplate(DefaultGenerator.java:759)
	at com.samskivert.mustache.Mustache$IncludedTemplateSegment.execute(Mustache.java:744)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:827)
	at com.samskivert.mustache.Mustache$SectionSegment.execute(Mustache.java:863)
	at com.samskivert.mustache.Template.executeSegs(Template.java:114)
	at com.samskivert.mustache.Mustache$IncludedTemplateSegment.execute(Mustache.java:762)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:827)
	at com.samskivert.mustache.Mustache$InvertedSegment.execute(Mustache.java:887)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:827)
	at com.samskivert.mustache.Mustache$SectionSegment.execute(Mustache.java:863)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:827)
	at com.samskivert.mustache.Mustache$SectionSegment.execute(Mustache.java:848)
	at com.samskivert.mustache.Template.executeSegs(Template.java:114)
	at com.samskivert.mustache.Template.execute(Template.java:91)
	at com.samskivert.mustache.Template.execute(Template.java:82)
	at io.swagger.codegen.DefaultGenerator.processTemplateToFile(DefaultGenerator.java:765)
	at io.swagger.codegen.DefaultGenerator.generateModels(DefaultGenerator.java:396)
	... 3 more

You may have missed "pkmst/typeInfoAnnotation.mustache" when committing the changes.

@wing328
Copy link
Contributor

wing328 commented Dec 12, 2017

I did another test and found another missing template

[main] ERROR io.swagger.codegen.AbstractGenerator - pkmst/enumOuterClass.mustache (No such file or directory)
Exception in thread "main" java.lang.RuntimeException: Could not generate model 'EnumClass'
	at io.swagger.codegen.DefaultGenerator.generateModels(DefaultGenerator.java:409)
	at io.swagger.codegen.DefaultGenerator.generate(DefaultGenerator.java:735)
	at io.swagger.codegen.cmd.Generate.run(Generate.java:285)
	at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:35)
Caused by: java.lang.RuntimeException: can't load template pkmst/enumOuterClass.mustache

Please create ./bin/java-pkmst-petstore-server.sh and .\bin\windows\java-pkmst-petstore-server.bat to generate Petstore samples for pkmst generator.

return;
}
if (swagger.getTags() != null) {
System.out.println("Tags are::" + swagger.getTags());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use LOGGER.debug instead

@wing328
Copy link
Contributor

wing328 commented Dec 17, 2017

I did another test and got the following errors:

[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/pkmst/jersey2/src/main/java/com/prokarma/pkmst/controller/UserApi.java:[87,103] '}' expected
[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/pkmst/jersey2/src/main/java/com/prokarma/pkmst/controller/UserApi.java:[87,110] illegal start of type
[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/pkmst/jersey2/src/main/java/com/prokarma/pkmst/controller/UserApi.java:[87,111] ';' expected
[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/pkmst/jersey2/src/main/java/com/prokarma/pkmst/controller/UserApi.java:[97,132] illegal start of expression
[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/pkmst/jersey2/src/main/java/com/prokarma/pkmst/controller/UserApi.java:[97,138] <identifier> expected
[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/pkmst/jersey2/src/main/java/com/prokarma/pkmst/controller/UserApi.java:[97,144] <identifier> expected
[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/pkmst/jersey2/src/main/java/com/prokarma/pkmst/controller/UserApi.java:[97,156] ';' expected

Please take a look.

@wing328 wing328 merged commit cb59591 into swagger-api:master Dec 19, 2017
@wing328
Copy link
Contributor

wing328 commented Dec 19, 2017

Minor enhancements via #7209

@wing328
Copy link
Contributor

wing328 commented Dec 19, 2017

@sanshuman there are still issues with the Petstore sample, please pull the latest master to take a look. Here are some error messages:

[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/java-pkmst/src/main/java/com/prokarma/pkmst/controller/PetApi.java:[62,103] illegal start of expression
[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/java-pkmst/src/main/java/com/prokarma/pkmst/controller/PetApi.java:[62,109] <identifier> expected
[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/java-pkmst/src/main/java/com/prokarma/pkmst/controller/PetApi.java:[62,125] <identifier> expected
[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/java-pkmst/src/main/java/com/prokarma/pkmst/controller/PetApi.java:[67,5] illegal start of expression
[ERROR] /private/tmp/swagger-codegen/samples/server/petstore/java-pkmst/src/main/java/com/prokarma/pkmst/controller/PetApi.java:[67,13] illegal start of expression

I've commented out the java-pkmst test for the time being: 6898763

@wing328
Copy link
Contributor

wing328 commented Dec 19, 2017

As pointed out by the PKMST team, the resource folder is incorrect in the bin script. I've filed #7213 to fix it.

Please visit below links for more information on PKMST:

Getting started
https://pkmst-getting-started.mybluemix.net
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got the following when accessing https://pkmst-getting-started.mybluemix.net:

404 Not Found: Requested route ('pkmst-getting-started.mybluemix.net') does not exist.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wing328 It was under maintenence. now it's up and we are currently updating the content according to swagger codegen.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good. I'll send out a tweet to promote the new generator.

@wing328
Copy link
Contributor

wing328 commented Dec 20, 2017

Tweet to promote the new generator: https://twitter.com/wing328/status/943403399080329216

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants