Refactor : Use a sidecar for puppet things which cannot be unloaded #40

glennsarti · 2018-06-25T08:25:13Z

The language server is only able to load the default puppet modules that are outside of the thing being edited. This is due to Puppet not really being able to unload classes/types/facts etc. on the fly. So instead we need to call out to an external process to parse that information and then return it to the language server, thus the sidecar.

e.g.

To enumerate all of the Puppet custom types in a working directory;

Language server starts a sidecar process, instructing it to enumerate all classes/types/functions in the working directory; and return that in a structured data file (e.g. JSON)
Sidecar runs, and exports the relevant information to a file
Language server reads the file, and caches the information for use with autocomplete etc.

When the files are edited again, the process is repeated.

The sidecar would also be used for node graph generation, puppet strings
The sidecar would also be used to enumerate the default types/classes/functions and facts
The sidecar wouldn't be needed for linting as that doesn't require loading of puppet code (AFAIK)
The language server would need to throttle sidecar invocations otherwise max CPU and possibly memory could be reached pretty quickly.

Previously all of the Puppet helpers ran inside of a single instance of Puppet, however this causes issues because classes and types cannot be unloaded in a running instance. So instead, a sidecar executable is created which will run out of process, in a fresh Puppet configuration, to enable enumerating types, classes and functions etc. This commit creates the basic skeleton of this sidecar process.

This commit marks the puppet-language-sidecar ruby file as executable. This is mainly helpful for linux based developers.

The Language Server and the sidecar need an agreed upon protocol to communicate properly. The agreed upon encoding will be UTF8 JSON strings. This commit adds the required classes which can be used to serialise and deserialise objects to and from the sidecar process. This commit also includes tests to validate the serialisation process.

…tions and types This commit adds the ability for the sidecar process to enumerate classes, functions and types, either in the default configuration (i.e. like a real puppet agent) or only within the confines of a workspace directory (e.g. when editing a module or control-repo). Currently this only supports querying a puppet module workspace. This commit also adds the ability to pass in puppet options (e.g. --vardir) into the puppet configuration for the sidecar. This commit also uses a filesystem based caching mechanism to speed up the default enumerations. The majority of this code was extracted from the existing PuppetHelper in the Language Server and Puppet Monkey Patches.

…unctions and types This commit adds implementation and behavioural tests for the language sidecar process enumerating classes, functions and types.

…tions and types This commit adds the ability for the sidecar process to enumerate classes, functions and types, either in the default configuration (i.e. like a real puppet agent) or only within the confines of a workspace directory (e.g. when editing a module or control-repo). Currently this only supports querying a puppet module workspace. This commit also adds the ability to pass in puppet options (e.g. --vardir) into the puppet configuration for the sidecar. This commit also uses a filesystem based caching mechanism to speed up the default enumerations. The majority of this code was extracted from the existing PuppetHelper in the Language Server and Puppet Monkey Patches.

…unctions and types This commit adds implementation and behavioural tests for the language sidecar process enumerating classes, functions and types.

This commit adds the ability for the Sidecar to fetch the resources for a given typename and title via the Action Parameters command line argument. This is similar to a `puppet resource` call. This commit also adds the required sidecar protocol additions and tests for the behaviour.

Previously a sidecar process was introduced to query puppet for information, however the Language Server was yet to take advantage of it. This commit modifies the Puppet Helper in the Language Server to use the sidecar instead. This commit also uses a queueing mechanism to throttle the sidecar and ensure that if there are duplicate sidecar requests, only the most recent is processed. This ensures that that CPU and memory use is capped, but still maintains a good repsonse time and user experience.

With the removal of the Puppet Helper calling puppet directly, but using the sidecar, many of the test fixtures are no longer required. This commit removes the un-needed files.

Previously a sidecar process was introduced to query puppet for information, however the Language Server was yet to take advantage of it. This commit modifies the Puppet Helper in the Language Server to use the sidecar instead. This commit also uses a queueing mechanism to throttle the sidecar and ensure that if there are duplicate sidecar requests, only the most recent is processed. This ensures that that CPU and memory use is capped, but still maintains a good repsonse time and user experience.

Previously a sidecar process was introduced to query puppet for resource information, however the Language Server was yet to take advantage of it. This commit modifies the Puppet Helper in the Language Server to use the sidecar instead.

With the removal of the Puppet Helper calling puppet directly, but using the sidecar, many of the test fixtures are no longer required. This commit removes the un-needed files.

This commit adds the ability for the Sidecar to fetch the resources for a given typename and title via the Action Parameters command line argument. This is similar to a `puppet resource` call. This commit also adds the required sidecar protocol additions and tests for the behaviour.

This commit adds the ability for the sidecar to generate a node graph for a manifest file on disk. The path the manifest is passed via the source Action Parameter. This commit also adds the required sidecar protocol additions and tests for the behaviour.

Previously a sidecar process was introduced to query puppet for information, however the Language Server was yet to take advantage of it. This commit modifies the Puppet Helper in the Language Server to use the sidecar instead. This commit also uses a queueing mechanism to throttle the sidecar and ensure that if there are duplicate sidecar requests, only the most recent is processed. This ensures that that CPU and memory use is capped, but still maintains a good repsonse time and user experience.

Previously a sidecar process was introduced to query puppet for resource information, however the Language Server was yet to take advantage of it. This commit modifies the Puppet Helper in the Language Server to use the sidecar instead.

With the removal of the Puppet Helper calling puppet directly, but using the sidecar, many of the test fixtures are no longer required. This commit removes the un-needed files.

TODO