Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New Backend #118

Closed
kstan79 opened this issue Sep 21, 2023 · 20 comments
Closed

New Backend #118

kstan79 opened this issue Sep 21, 2023 · 20 comments

Comments

@kstan79
Copy link
Contributor

kstan79 commented Sep 21, 2023 •
edited
Loading

I know that current backend for prototyping and not design for production use.

Since bpmn-server is pure backend engine I recommend to create new backend using nestjs. it offer benefit which included:

  1. More robust design
  2. bigger ecosystem and we able to use lot of nestjs plug-in like sso authentication, swagger-ui, middleware
  3. We have better way to manage complex workflow requirement
  4. Use openapi we no need maintain client library. It mean we can accept frontend request from Java,python, .net without headache

Others bpmn project like kogito, they did similar and are they use quarkus which can allow developer use plenty of quarkus plugin.

I use it sometimes but I can't stay with them cause it is too rigit

If you agree we may work on it
@bpmn-ts
Copy link

bpmn-ts commented Sep 21, 2023 via email

@kstan79
Copy link
Contributor Author

kstan79 commented Sep 21, 2023

We may start from easier way like:

  1. Create an kickstart git repo like "bpmn-nest-template"
  2. Inside content latest nestjs project and appropriate route/service controller
  3. Put some bpmn sample, delegates inside
  4. Write read me for how to kick start:
  • add more process definition
  • change configuration via .env
  • try api via swagger
  • turn on api key or sso jwt token (like keycloak)
  • how to build for production
  • and etc

One comfortable we may create cli executor and user install it via "npm install -g bpmn-server-cli"

Then setup and maintain the project using

  • bpmn-server-cli init my-proj
  • bpm-server-cli start/stop
  • bpmn-server-cli add-deligate uploads3 emailuser push-notification
  • etc

@ralphhanna
Copy link
Collaborator

ralphhanna commented Sep 21, 2023 via email

@ingpconci
Copy link

I tried to create a loopback extension to use with the loopback backend framework
https://github.com/ingpconci/loopback4-bpmn-server
I know the loopback project is not updated like Nestjs, but the project is hosted on the OpenJS foundation
https://openjsf.org/blog/2022/04/25/node-js-framework-loopback-graduates-to-at-large-project-at-openjs-foundation/
What do you think about it?

@kstan79
Copy link
Contributor Author

kstan79 commented Sep 27, 2023 •
edited
Loading

Sounds good, I will spend sometime check loop back is it good for me,

@ralphhanna
Copy link
Collaborator

Can we think of Bpmn-server as a microservice that can be used in either framework?

@kstan79
Copy link
Contributor Author

kstan79 commented Sep 28, 2023 •
edited
Loading

IMHO bpmn act as purely backend microservices, it doesn't effect much as long as it comply openapi api standard. All languages and framework can tan to openapi easily.

I not recommend bpmn server embed as library to support 2 or more framework cause it add burden to development for testing and documentation. Let's pick one and stick with it until perfect, than decide again.

In typescript I only have experience in nuxt and nestjs, never try loopback framework so I can't give comment about loopback good or bad yet.

Nest has build plenty of features I think suitable for industry grade application, bpmn-server adopt by industry most of the time so I feel we park inside good for us

@ralphhanna
Copy link
Collaborator

Hi
I have looked at https://github.com/ingpconci/loopback4-bpmn-server
and it seems to be a full port to loopback4 which will force everyone to use loopback4.

I am hoping that only the bpmn-client will be ported to lb4 with is minimal code while keeping bpmn-server as pure nodeJS

Thanks, Let us keep the discussion going

@ingpconci
Copy link

Hi
I use the loopback as a client for bpmn-server, because I want to use the loopback API framework to control authentication, filtering and connecting to the relational database PostgreSQL.
I imported the library, created the server instance and used it in the controllers.
I don't modify the code of bpmn-server and I change only the 4 customization files for the library

image

The main effort is focused on defining a flexible system user task assignment; to facilitate the acceptance of BPMN workflow, I use this approach based on association between Lane in diagram and Use Role

image

In the extension, I put the controller API endpoints to support the Role management.
In the picture there is the Html interface.
image

I would like to put in the loopback bpmn-server extension an API endpoint that returns the information about the User Task Form.
At the moment, I define the BPMN process with camunda modeler and use the camunda form definition.

image

<bpmn:process id="Process_07ii97u" isExecutable="true"> <bpmn:extensionElements> <zeebe:userTaskForm id="userTaskForm_08ui781">{ "components": [ { "text": "&lt;h2&gt;(1.1) New nonconformity&lt;/h2&gt;", "label": "Text view", ….. }</zeebe:userTaskForm>

In my Html App, I have developed a form engine that uses this information in the model definition to build the Form, but in the future I would like to facilitate the construction of the form at server side. The function must provide a form representation that permit to build a non Html form (e.g. a standalone C++ app form engine)

image

@bpmn-ts
Copy link

bpmn-ts commented Sep 29, 2023 via email

@kstan79
Copy link
Contributor Author

kstan79 commented Sep 29, 2023 •
edited
Loading

After view back your project seems u did extension for loopback only. Previously misunderstand and thought you host whole loopback application in repo.

@bpmn-ts you may need to decide is loopback a good framework and qualify as firstclass citizen for bpmn or not. if yes we may update documentation according that approach. Else, i will spend times to do in nestjs.

@ingpconci can you make your effort become npm package so it more portable? I'm excited actually because your may solve a big issue which can make bpmn-server use in production

@ingpconci
Copy link

I clone the project locally

git clone https://github.com/ralphhanna/bpmn-server.git

Then compile and pack

> npm run clean
> npm run build
> npm pack
...
npm notice === Tarball Details ===
npm notice name:          bpmn-server
npm notice version:       1.3.28
npm notice filename:      bpmn-server-1.3.28.tgz
npm notice package size:  10.2 MB
npm notice unpacked size: 49.3 MB
npm notice shasum:        7c52e0ac88d9b340516c8b484db5d7700af73c26
npm notice integrity:     sha512-DqC9SRCt3HlzT[...]mrIoswkvWnNjw==
npm notice total files:   851
npm notice
bpmn-server-1.3.28.tgz

After that I insert in the package.json

 "dependencies": {
    "bpmn-moddle": "^8.0.1",
    "bpmn-server": "file:bpmn-server-1.3.28.tgz",
    "camunda-bpmn-moddle": "^7.0.1",
    "tslib": "^2.0.0"
  },

@ingpconci
Copy link

To use in the main loopback application, I import these libraries into the application.ts

    "loopback-connector-postgresql": "^6.0.8",
    "loopback4-tenant-table-filter": "file:loopback4-tenant-table-filter-1.0.8.tgz",
    "bpmn-server": "file:bpmn-server-1.3.28.tgz",
    "loopback4-bpmn-server": "file:loopback4-bpmn-server-1.0.0.tgz"

and insert the initialization code

    //--------------------------------------------------------------------------
    // BpmnServerComponent
    //--------------------------------------------------------------------------
    const definitionsPath = path.join(__dirname, '../processes/')
    console.log('AppServerApplication:startup: bpmnProcessDefinitionDirectory=', definitionsPath);
    const componentOptions: Loopback4BpmnServerComponentOptions = {
      bpmnProcessDefinitionDirectory: definitionsPath
    }
    this.configure(Loopback4BpmnServerComponentBindings.COMPONENT).to(componentOptions);
    this.component(Loopback4BpmnServerComponent);

Remember that the extension uses a multitenant authorization extension
https://github.com/ingpconci/loopback4-tenant-table-filter
that need to be configured before the BpmnServerComponent

    //--------------------------------------------------------------------------
    // Multitenant Authorization from loopback4-tenant-table-filter
    // Authorization provider
    //--------------------------------------------------------------------------
    const authorizationOptions: AuthorizationOptions = {
      precedence: AuthorizationDecision.DENY,
      defaultDecision: AuthorizationDecision.DENY,
    };
    this.configure(AuthorizationBindings.COMPONENT).to(authorizationOptions);
    this.component(AuthorizationComponent);
    // bind the authorizer provider
    this.bind('authorizationProviders.multitenant-authorizer-provider')
      .toProvider(MultitenantAuthorizerProvider)
      .tag(AuthorizationTags.AUTHORIZER);

The loopback4-tenant-table-filter permit to have a unique DB but different isolated organization
For example, the bpmnProcessInstance table contains records of different tenant
image

When I start the process I set this instance variable in the process with the tenantId of the user

      inputData: {
        tenantId: this.authorizationService.currentTenantId,
        startProcessUserId: this.currentUser.id
      }
    const startEngineResult = await this.bpmnServer.engine.start(processName, inputData);

The bpmn server engine use this information for all operation in the DB in the custom datastore that uses the
bpmnProcessInstanceRepository.createNoTenantIdFilter

      
  // save instance to DB
  static seq = 0;
  private async saveInstance(instance, items) {
    //	this.logger.log("Saving...");
    console.log('CustomDataStore.saveInstance:');
......
......
      //-----------------------------------------------------------
      //Get tenantId from inputData
      //-----------------------------------------------------------
      const currentTenantId = instance.data.tenantId;
      bpmnProcessInstance.tenantId = currentTenantId;

      const result = await this.server.configuration.database.loopbackRepositories.bpmnProcessInstanceRepository.createNoTenantIdFilter(bpmnProcessInstance);
      debug('saveInstance: new instance inserted! result=', result);

@ralphhanna
Copy link
Collaborator

I believe the full App (Web UI, Server) can use any framework. but I would rather have bpmn-server not rely on any framework, because that would be restrictive.

In other words I like that bpmn-server to remain on pure nodeJS and only offer ExpressJS for demo purposes only?

So the question is what features in bpmn-server can be enhanced using these frameworks?

Any thoughts?

@kstan79
Copy link
Contributor Author

kstan79 commented Sep 30, 2023

I don't think it is good ideal to maintain as independent express due to below reason

  1. Pure express framework far behind production use, there is a lot of thing need to do and until up to production level to use. Such as cluster, cache, security, queue system and etc which need high experience developer assemble it in express. Use 3rd party framework provide reliable documentation and step to do that.

  2. I believe you wish to remain independent with good reason, but I think that make project getting far from provide easy to use bpmn solution. my thought is bpmn his kind of low code solution and everything include setup/database shall implement low code concept as well. I feel you pick mongodb is wise decision cause it also match low code philosophy concept.

Current project quality, completeness, reliability yet to achieve perfect. In this situation flexible is kind of no-no and drag the progress. Why don't Make everything static at the time being until near perfect, then decide again in future?

@kstan79
Copy link
Contributor Author

kstan79 commented Sep 30, 2023

Anyhow, it is my personal perspective, which I believe people wish to find robust bpmn framework rather than bpmn library.

I'm ok with using someone bpmn framework, as long as reliable and suitable

@ralphhanna
Copy link
Collaborator

Not sure I understand your recommendation, can you please clarify what do you recommend, keep in mind Express is only used for demo purposes

@kstan79
Copy link
Contributor Author

kstan79 commented Oct 1, 2023 •
edited
Loading

  1. Many kind users, some is programmer and some is not.
  2. Event though the specific user is programmer, he might not expert in typescript and express
  3. The user may feel interested in this project and wish to implement in their organisation
  4. They may not have capability to walk through complex setup process, and they wish to have something just work fine for production
  5. Most of the time, their concern is how to put bpmn inside, how to chain this microservice to their existing webapp. They not care what framework it is, they care is reliable and well documented step. If possible it can install and forget.

Technically most of time the users want something prepackaged for production use, included me. For production package, it need:

  1. quick start
  2. Documentation for implementation, configuration base on common needs:
  3. Authentication/authorisation
  4. Database setup
  5. Integration with existing frontend web app
  6. Some test case, some video guides or walk through

Some of above requirement will need the project to settle down best practice of implement bpmn, in this case flexibility may not a good ideal. We may rather put that as advance topic like how to use others framework (if it is necessary)

@ingpconci
Copy link

When I chose the bpmn-server library, I checked other libraries and full frameworks like Camunda Products.
The Camunda product meets all of your needs, but it is not open source.
I agree with @ralphhanna. It's better to keep the core as a pure Node.js library.
Now there is an Express client for demo purposes to verify the behavior of the library, but I agree that it should not be used for production.
My approach was to use the library in the main application service based on loopback.
Here are the advantages of my approach:

  • The BPMN data is saved in the same database as the main application server.
  • We don't have to manage any other services or Java applications on the server.
    However, there is one downside:
  • The main application cannot be scaled because it is currently not possible to have two instances of the engine that share process instance data.

I agree that a reference framework could benefit the development of the bpmn-server library. I am also open to discussing the possibility of making my main loopback application service, along with the HTML angular client, open source. This could be useful for managing and interacting with the bpmn server.

@ralphhanna
Copy link
Collaborator

@ingpconci Thanks very much for you comments and contribution.

I definitely appreciate any additional contribution to enhance the Client UI.

As for the server, would a package like Memcached help solve the scalability issue?

Thanks again

@bpmnServer bpmnServer locked and limited conversation to collaborators Oct 3, 2023
@ralphhanna ralphhanna converted this issue into discussion #123 Oct 3, 2023

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kstan79 @ingpconci @ralphhanna @bpmn-ts