Connector and Pipeline lifecycle

[lyuboxa]

Hey everyone,

I'm doing some work to automate connector provisioning and while I understand the reasons behind the designs,
few bits are causing some friction which require some extra work on the client, particularly:

1. Connector cannot be updated on a started pipeline
2. Connector cannot be added to a started pipeline
3. Connector cannot be deleted on a started pipeline

In both of these cases, the operation expands to:

1. Stop pipeline
2. Wait for stop
3. Add/update/delete connector
4. Start pipeline
5. Ensure running

I understand and appreciate the explicitness and would like to know what are you thoughts in pushing these operations
further into the service?

Thanks!

[hariso]

I understand and appreciate the explicitness and would like to know what are you thoughts in pushing these operations
further into the service?

IMHO that makes sense. In the end, it's a matter of implementation which is, ideally, hidden from a client. We may find it possible to add connectors to a running pipeline, and the client should not have to care about when that's possible and when not.

By the way, what are your use cases? IIRC you were using some simple pipeline with just a source and destination, in which case it may be simpler to give you an option to create a pipeline and the connectors at the same time. We would then, at least for now, not have to care about the complexities in Conduit itself.

[lyuboxa]

Any use case will do here but I think pipelines with a more complex life cycle (adding/removing processors, connectors etc often)
could further exacerbate the complexity on the user side.

[lovromazgon]

I think we could allow these operations on running pipelines, in the first implementation we could simply stop and start the pipeline behind the scenes while we add/update/delete the connector. In the future we could make it a bit smarter and do some of these changes on the fly. It's only a matter of priorities, and right now I think we have a bunch of tasks that are higher on the list, since this one doesn't prevent you from adding/updating/deleting a connector, it only makes it a bit simpler.

Feel free to create an issue for this, if you didn't already, we will triage it and plan it in when we have time.

[lyuboxa]

I think we could allow these operations on running pipelines, in the first implementation we could simply stop and start the pipeline behind the scenes while we add/update/delete the connector.

I think this is pretty reasonable start. I filed an issue here #201