Transition internal structs to Protocol Buffer Schema

[yottahmd]

Considering transitioning our structs for basic information like Dag, Step, and Status to a Protocol Buffer schema. This change is expected to offer several advantages in the future, including:

• Schema Versioning: Protocol Buffers support built-in schema versioning, making it easier to evolve our data structures over time without breaking existing applications.
• Simplified Database Integration: By utilizing Protocol Buffers, we can store data of those structure in any storage easily and efficiently compared to JSON.
• REST API Generation with gRPC Gateway: With Protocol Buffers and gRPC Gateway, we can automatically generate REST APIs from our schema and documents.

I think it is a good stepping stone towards a gRPC-based client-server architecture that will make it easier to implement features to manage remote servers from local machines.

[garunitule]

Hi, @yohamta

I am also in favor of the transition to Protocol Buffer and the gradual transition from Protocol Buffer to gRPC.

May I work on it?

[yottahmd]

Hi @garunitule , that would be really helpful! Please, go ahead and thank you so much for your support!

[garunitule]

@yohamta
May I discuss one thing with you?

I would like to discuss how to handle the OutputVariables field (*utils.SyncMap type) in the Step type.

I could only think of wrapping the type generated by protocol buffers and adding an OutputVariables field.
Is there any other way? From an operational standpoint, I would like to use the generated type as is.

[yottahmd]

Hi @garunitule 👋

Thank you for working on this!

I suggest changing the OutputVariables type to a regular map and using a sync.Mutex to lock the sections where the values are being read or modified. We can add a sync.Mutex field to the Node struct, which acts like a wrapper for Step. Also, we might need to make some minor adjustments to the related sections.

Is there any other way? From an operational standpoint, I would like to use the generated type as is.

Agreed 🙂

[garunitule]

@yohamta
Thank you for your response!
I will proceed with making some minor adjustments.

[yottahmd]

Sorry, let me close this issue for now. We’ll keep JSON format for now to prioritize feature implementation to go to cloud.