Add framework for adding new drivers at runtime, for instance for cloud

[afbjorklund]

The docker-machine program had a plugin system, so that you could provide new drivers besides the built-in ones.

This would be a good option to provide drivers for various proprietary hypervisors and cloud vendors, for instance.

They typically come with a big SDK, that might require certain hardware and large downloads when building.

So it would be nice if those were kept out of the Lima code base, but is still a nice addition to enable them...

---

https://github.com/machine-drivers/docker.github.io/blob/173d3c65f8e7df2a8c0323594419c18086fc3a30/machine/AVAILABLE_DRIVER_PLUGINS.md

https://github.com/machine-drivers/docker.github.io/blob/173d3c65f8e7df2a8c0323594419c18086fc3a30/machine/DRIVER_SPEC.md

[jandubois]

Having pluggable drivers might be useful, but they should be maintained in separate repos (like socket_vmnet).

Random memory: I think docker-machine-driver-hyperkit must be run with as root and has several easily exploitable vulnerabilities (e.g. because it spawns subprocesses that also run as root). So this might be something to look out for as you implement other drivers.

[afbjorklund]

I was thinking to only add the API, but keep all (both?) supported drivers in-tree

[afbjorklund]

Also for me I think that qemu is plenty, it was mostly for the cloud...

[jandubois]

but keep all (both?) supported drivers in-tree

Yes, the supported drivers need to stay in, off course, but things like adding a hyperkit driver, or cloud drivers, should be external. I don't actually see that anyone would actively maintain them, if the history of docker-machine drivers is any indication.

[afbjorklund]

What I meant was that I would not write another xhyve/hyperkit driver (especially not with vz already being there), I think we do need a hyperv driver but that can be done by exec to some PowerShell scripts rather than linking with a big SDK and thus it can stay in.

[afbjorklund]

I will do some kind of proof-of-concept interface next month, with one private cloud and one public cloud implementation.

[afbjorklund]

A simplified approach, for drivers that are internal to the program but have a lot of dependencies and bloat the binary, is to use the plugin interface instead of the net/rpc. The two major drawbacks is that it needs to be compiled at the same time as the program, and that it currently does not work on Windows (no .dll support yet, only .so). But it makes it easier to include something like libvirt, that has a lot of dependencies but allows for remote communication (the qemu:///session doesn't really add anything).

• https://pkg.go.dev/plugin

• https://pkg.go.dev/net/rpc

It would not be a solution for random cloud vendor drivers, that would be shipped independent of the limactl program.

Those would need some kind of RPC solution, perhaps wrapped in a framework like https://github.com/hashicorp/go-plugin

[AkihiroSuda]

Can we use REST or gRPC over stdio?

[afbjorklund]

I think that go-plugin can use both rpc and grpc.

[afbjorklund]

Having to define a public interface and a protocol for something like REST or gRPC is overkill for this use case, I think...

That was the benefit of using a "plugin" wrapper, over doing it ourselves. Just doing function or remote calls, like above.

[afbjorklund]

My attempt to load libvirt dynamically failed, due to it calling pkg-config (to find libvirt.pc) already from the go.mod...

The building and loading of the .so worked fine, but it was hard to make the plugin optional without a separate module.

[balajiv113]

Yes, during the initial driver interface as well the idea was to provide support via external plugin mechanism on future.

We should be easily able to expose a external plugin via https://pkg.go.dev/plugin with current code. Will just need driverutil to load the external drivers.

[afbjorklund]

If we use plugin, then the driver is not "external" anymore. It needs to be built at the same time, or it won't load later.

So if we want external plugins, then they will need to use RPC. Also, they were unable to module library dependencies.

[afbjorklund]

When doing the RPC driver, we should also consider how to forward configuration. This needs to be communicated as well.

We should not do it like minikube, when they either compile with the driver config or duplicate the struct without an API.

Note that we might not know (in the lima main program) what configuration that the plugin is added (in the plugin program).

For docker-machine, there was a rather hacky framework based on CLI flags which "worked" but json/yaml would be better...

[afbjorklund]

We also might have to add some new functions to the interface (=API), for instance the ability to upload images.

And how to send credentials to the driver, for instance passing on a token. Monitoring might also need revising.