Suggestion to compartmentalize data

[MaheshkumarSundaram]

Hi @ralphhanna,

I ran into a critical issue due to my below usecase.

One of my Service Tasks queries data files stored from MySQL (external db; I still use Mongo for bpmn-web) into the workflow instance. This is done to keep the data files unchanged throughout the span of the workflow instance.

Basically, the output of the service would be to inject the below sample structure into the instance data:

templates: {
fileName1: fileContent1,
fileName2: fileContent2,
...
...
fileNameN: fileContentN
}

In the above, fileContent can of type json, ini, text, md or html. The size of these file contents may range from 2MB to 5MB. I plan to use it like $item.data.templates.<fileName> in subsequent tasks.

But this causes the below error in the web server which makes sense due to the sheer amount of data in addition to all the other instance records for the Mongo query:

Exception: stack RangeError: The value of "offset" is out of range. It must be >= 0 && <= 17825792. Received 17825799
    at validateOffset (node:buffer:121:3)
    at Buffer.write (node:buffer:1092:5)
    at serializeNumber (F:\Programming\bpmn-web\node_modules\bson\lib\bson\parser\serializer.js:116:16)
    at serializeInto (F:\Programming\bpmn-web\node_modules\bson\lib\bson\parser\serializer.js:711:17)
    at serializeObject (F:\Programming\bpmn-web\node_modules\bson\lib\bson\parser\serializer.js:347:18)
    at serializeInto (F:\Programming\bpmn-web\node_modules\bson\lib\bson\parser\serializer.js:947:17)
    at serializeObject (F:\Programming\bpmn-web\node_modules\bson\lib\bson\parser\serializer.js:347:18)
    at serializeInto (F:\Programming\bpmn-web\node_modules\bson\lib\bson\parser\serializer.js:947:17)
    at serializeObject (F:\Programming\bpmn-web\node_modules\bson\lib\bson\parser\serializer.js:347:18)
    at serializeInto (F:\Programming\bpmn-web\node_modules\bson\lib\bson\parser\serializer.js:947:17)

I can't directly query MySQL everytime as the data are subject to change and I would like to keep those data files same throughout the lifespan of a workflow instance.

I am still exploring options but I am listing these two here:
Option 1:
I am thinking to create a separate collection called wf_data_templates with an index instId which is the id of wf_instances. But then accessing like $item.data.templates.<fileName> in the workflow would fail as they aren't native to the collection. But this complicates everything I suppose.

Option 2:
Write the file contents to the webserver's disk catgeorized into direcotries (directory name - instance id) and keep just the path in the Mongo's instance data. But then if there are 100 workflows ongoing, there can be many such data fetching service tasks in one workflow. May be lead to file write or access errors as well. Cleaning up these files when a workflow completes would be a hassle too.

I did think of Redis but it can't be used reliably as my OS is Windows.

Option 2 with file system could work if there is a reliable way to delete the folder when a workflow instance completes. Is there any way to call a Service Task automatically under the hood when an instance completes?

I am struggling to find an effective solution to resolve this!!!
Just thought to ask you whether you could suggest any feasible solution to deal with my usecase!?

Any help would be much appreciated. Thank you so much.

[ralphhanna]

Hi @MaheshkumarSundaram
I like option 1 more, but instead of $item.data.template.. you can create a method in $AppServices.getTemplate(item)

[MaheshkumarSundaram]

Sorry @ralphhanna. $item.data.template.. can be used anywhere no matter the type of BPMN element. But if I create a method getTemplate in appServices, I can only call this in a Service Task right?

Not sure whether I am understanding your point.

[ralphhanna]

No, it can be used anywhere as well Sent from my iPhone On Sep 11, 2024, at 1:10 PM, Maheshkumar Sundaram ***@***.***> wrote:  Sorry @ralphhanna<https://github.com/ralphhanna>. $item.data.template.. can be used anywhere no matter the type of BPMN element. But if I create a method getTemplate in appServices, I can only call this in a Service Task right? Not sure whether I am understanding your point. — Reply to this email directly, view it on GitHub<#226 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC2XVKEVRPX6B45WXCAYKJ3ZWB2N5AVCNFSM6AAAAABOBJXYVWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRRG4YTGNY>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[MaheshkumarSundaram]

@ralphhanna, ah, great! Never knew this despite me using this for more than a year, 😆. But thanks for the clarification

[MaheshkumarSundaram]

Hi @ralphhanna,

In relation to the above usecase, I ran into another issue.

In a Service task named VarsCheck, I used $appServices.GetTemplate(item, <docName>) as an Input Parameter.

async VarsCheck(input, context) {
        console.log('Task:', context.item.name);
        for (const [key, value] of Object.entries(input)) {
            console.log(key, ': ', value);
        }
    }

    async GetTemplate(item, docVarName) {
        const instId = item.token.execution.instance.id;
        try {
            if (!isAlphanumericString(docVarName)) {
                throw new Error("GetTemplate's document name can only contain numbers, letters and underscores.");
            }
            const template = await fetchWfDataFragmentTemplate(instId, docVarName);
               
            console.log('template:', template);
            return template;
        } catch (error) {
            return { bpmnError: error.message };
        }
    }

The template is fetched properly but I suppose the execution order differs.
In VarsCheck when I print the input, I get MyBMOD : Promise { <pending> }.
I did try to make GetTemplate sysnchronous in which case I get MyBMOD : undefined.

[ralphhanna]

Hi @MaheshkumarSundaram
Yes, I see the problem, input variables are not async calls.

But since you are using it in serviceTask why not include the fetch logic in the serviceTask instead of the input

input: item.BMOD2

async VarsCheck(input, context) {
        var template=await this.getTemplate(input)
        console.log('Task:', context.item.name);
        for (const [key, value] of Object.entries(templalte)) {
            console.log(key, ': ', value);
        }
    }

[MaheshkumarSundaram]

@ralphhanna

For Service Tasks, yes, definitely, I could integrate the fetch logic into the implementation.

The issue would be in other types of tasks. Say User Task or a Receive Task input/output variables?

Is there any work around for them?

[ralphhanna]

I am working in a new release to allow all scripts to execute async functions including input/output variables
Thanks

[MaheshkumarSundaram]

@ralphhanna

Great! Thanks for that. Would you be able to give an approximation of when this release can be expected?

[ralphhanna]

But for now, you can have a start trigger to load the data into item.data.templates and input value will just access them