[Puppet4API Project Task 1-3] Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions #121

glennsarti · 2019-04-09T08:43:19Z

Task 1

Add a Puppet 4 API feature flag - As much as possible, all code should be hidden behind this flag

Task 2

Load Puppet 4 API Functions but present them as Puppet 3 functions in the Sidecar Protocol. This preserves existing behaviour of the loading, but requires no changes to the language server itself

Task 3

Use Puppet Strings (YARD) to load Puppet 4 documentation - Puppet 4 API does not use dedicated doc parameters, instead all documentation is provided by YARD based comments, interpretted by the Puppet-Strings gem. Puppet Strings extends TARD to understand Puppet syntax

The text was updated successfully, but these errors were encountered:

Previously the monkey patch to inject the editor workspace as a module did not extract the module name correctly. While this was ok for older Puppet 3 API, the Puppet 4 API namespaces objects based on the module name, so this TODO item needed to be completed. Now we extract the module name correctly.

* This commit adds and validates a pup4api feature flag in the sidecar. This feature flag will be used to toggle the loading of Puppet information to use the newer Puppet 4 API. * This commit copies the helper and monkey patch files for the pup4api flag in prepration for the work to modify them to use the pup4api * This commit also copies and modifies the integration tests to use the pup4api feature flag and ensure that the behaviour does not change when the flag is enabled * Note that the pup4api feature flag is gated on the Puppet Gem version being v6.0 or higher. This is due to Puppet 6 having all of the required helpers needed by the Sidecar, even though the API we will be using was introduced in Puppet 4.

Previously functions were loaded with the autoloader classes however in the new Puppet 4 API we use the environment loaders. In particular we use the discover method to discover all of the available "things", in this case functions, within the environment. `current_env.loaders.private_environment_loader.discover(:function)` * Note this commit only loads the Puppet 3 API functions using the Puppet 4 loaders, thus there are no test changes. Later commits will load Puppet 4 functions * Note we use the private loader, not the public one, because we want all of the available things, not just the ones that are publicly exposed. We still need intellisense on private objects. * Note that we monkey patch the ruby_legacy_function_instantiator (Puppet 3 API functions) and ruby_function_instantiator (Puppet 4 API functions) as in default Puppet any errors would terminate all function discovery. Instead we trap and log the error and move on. * Due to PUP-9509 we need to monkey patch the cached loader to also load Puppet 3 API functions. Previously these were missing * The Sidecar then uses the appropriate method to enumerate the available functions based on the presence or absence of the feature flag.

Previously only Puppet 3 API functions were loaded. This commit monkey patches and tracks the Puppet 4 API function creator. The function enumerator is then expanded to also resolve Puppet 4 API functions.

This commit adds the test fixtures, and tests, to assert that the sidecar does indeed load Puppet 4 API functions appropriately and that the feature flag does not affect the loading of other assets. Note that function documentation is not available yet and will come in later commits.

Previously the Puppet 4 API functions would be loaded however they would not have any documentation notes as this was removed in the newer API. Instead we need to use Puppet Strings and YARD to generate the documentation. This commit: Note that Puppet Strings and YARD are only available in the PDK, not Puppet Agent. Once this feature is no longer hidden behind a feature flag, they can be vendored into Editor Services * Adds the puppet-strings gem for development purpsoes * Sets up YARD via Puppet Strings which adds the Puppet Language parsing into YARD * Caches the result from YARD to speed up documentation resolution for multiple items in the same file * Post processes the function loader and adds the documetation for Puppet 4 API functions

Previously the Puppet 4 API loader would always _actually_ load the Puppet assets whereas the Puppet 3 loader had a caching layer to speed up the process. This commit adds same caching layer to the Puppet 4 API loaders, specifically for functions in this case: * Fixes minor typos in the filesystem cache object and adds a `clear` method which should only be used for testing * Move the caching object to live on the PuppetLanguageServerSideCar module. This is required as the monkey patches have no way of knowing how to access the cache without it. * Functions were modified to have an additional was_cached and was_preloaded property. was_cached indicates that the function was loaded from cache and should not be saved back to cache either. was_preloaded indicates that the object was loaded prior to the loading process. They should also not be saved to cache because they can never be loaded from cache. * The function creation methods were modified to load the function metadata from cache. Even if we load the metadata from the cache, we still need _actually_ load the function in Puppet as it keeps track of function loading and will attempt reloads if it's not seen. Fortunately loading functions is quick and the user won't really see any slow downs. The slow part of the process is the puppet string documentation which is not processed when `.was_cached` is set to true * Added tests to the function loading to ensure that the metadata from the function loading is that same whether it is loaded from cache or not.

Previously the file cache would use a SHA256 hash of the file content. While this works, it's just as easy to use the file modification time stamp. This reduces the the file IO to check the cache. It's unlikely that a file would have different content while the modification timestamp stays the same. Note that this only occurs when the pup4api feature flag is present.

Previously functions were loaded with the autoloader classes however in the new Puppet 4 API we use the environment loaders. In particular we use the discover method to discover all of the available "things", in this case functions, within the environment. `current_env.loaders.private_environment_loader.discover(:function)` * Note this commit only loads the Puppet 3 API functions using the Puppet 4 loaders, thus there are no test changes. Later commits will load Puppet 4 functions * Note we use the private loader, not the public one, because we want all of the available things, not just the ones that are publicly exposed. We still need intellisense on private objects. * Note that we monkey patch the ruby_legacy_function_instantiator (Puppet 3 API functions) and ruby_function_instantiator (Puppet 4 API functions) as in default Puppet any errors would terminate all function discovery. Instead we trap and log the error and move on. * Due to PUP-9509 we need to monkey patch the cached loader to also load Puppet 3 API functions. Previously these were missing * The Sidecar then uses the appropriate method to enumerate the available functions based on the presence or absence of the feature flag.

Previously only Puppet 3 API functions were loaded. This commit monkey patches and tracks the Puppet 4 API function creator. The function enumerator is then expanded to also resolve Puppet 4 API functions.

This commit adds the test fixtures, and tests, to assert that the sidecar does indeed load Puppet 4 API functions appropriately and that the feature flag does not affect the loading of other assets. Note that function documentation is not available yet and will come in later commits.

Previously the Puppet 4 API functions would be loaded however they would not have any documentation notes as this was removed in the newer API. Instead we need to use Puppet Strings and YARD to generate the documentation. This commit: Note that Puppet Strings and YARD are only available in the PDK, not Puppet Agent. Once this feature is no longer hidden behind a feature flag, they can be vendored into Editor Services * Adds the puppet-strings gem for development purpsoes * Sets up YARD via Puppet Strings which adds the Puppet Language parsing into YARD * Caches the result from YARD to speed up documentation resolution for multiple items in the same file * Post processes the function loader and adds the documetation for Puppet 4 API functions

Previously the Puppet 4 API loader would always _actually_ load the Puppet assets whereas the Puppet 3 loader had a caching layer to speed up the process. This commit adds same caching layer to the Puppet 4 API loaders, specifically for functions in this case: * Fixes minor typos in the filesystem cache object and adds a `clear` method which should only be used for testing * Move the caching object to live on the PuppetLanguageServerSideCar module. This is required as the monkey patches have no way of knowing how to access the cache without it. * Functions were modified to have an additional was_cached and was_preloaded property. was_cached indicates that the function was loaded from cache and should not be saved back to cache either. was_preloaded indicates that the object was loaded prior to the loading process. They should also not be saved to cache because they can never be loaded from cache. * The function creation methods were modified to load the function metadata from cache. Even if we load the metadata from the cache, we still need _actually_ load the function in Puppet as it keeps track of function loading and will attempt reloads if it's not seen. Fortunately loading functions is quick and the user won't really see any slow downs. The slow part of the process is the puppet string documentation which is not processed when `.was_cached` is set to true * Added tests to the function loading to ensure that the metadata from the function loading is that same whether it is loaded from cache or not.

Previously the file cache would use a SHA256 hash of the file content. While this works, it's just as easy to use the file modification time stamp. This reduces the the file IO to check the cache. It's unlikely that a file would have different content while the modification timestamp stays the same. Note that this only occurs when the pup4api feature flag is present.

Previously the monkey patch to inject the editor workspace as a module did not extract the module name correctly. While this was ok for older Puppet 3 API, the Puppet 4 API namespaces objects based on the module name, so this TODO item needed to be completed. Now we extract the module name correctly.

The Sidecar will be experimenting with extracting puppet metadata (classes, functions and types etc.) via the Puppet Strings gem. This is an experimental function and is to be hidden behind a feature flag so users can opt in to the feature if needed. This commit: * Adds a feature flag detection method so downstream code can query if certain flags are set * Verifies that the feature flag is valid. Puppet Strings, for now, is only available from the PDK ruby based environment, not the Agent ruby envuronment. * Duplicates the puppet_helper and puppet_monkey_patches files in prepartion of them being modified when the feature flag is set. This allows changes to code to be truly isolated depending on the flag status * Duplicates the integration tests so that behaviour can be verified that is has not changed when the flag is set * Updates the Gemfile to bring in Puppet Strings during development. Note that the Puppet Strings gem is NOT vendored, though that may change in the future

The Puppet Strings gem uses YARD to parse the relevant files, however this is not useful for the Sidecar as we need access to the ruby objects, not a markdown or JSON file being created. This commit: * Adds a PuppetStringsHelper which can configure and execute YARD in the same way Puppet Strings does and then allow the Sidecar to extract the information it needs later. * Later commits will modify the helper to understand the various metadata the Sidecar needs. * Adds a caching layer to the results of running YARD. This means that if a file is queried more than once, YARD will only be executed once as running YARD is an expensive exercise. * Monkey patches YARD to suppress ALL output. By default the command line parameters still emit text of STDOUT, STDERR which breaks the STDIO transport for the Sidecar.

Currently the Sidecar can only detect and load Puppet 3 API functions. The newer Puppet 4 API functions use a different loader and schema, and importantly have additional properties e.g. Puppet 4 API functions have one or more signatures, whereas Puppet 3 API functions use arity. In order to add the Puppet Strings loader, this commit will load Puppet 4 API functions but present them to the Language Server as if they were Puppet 3. A later commit will then change this behaviour so that all the metadata for Puppet 4 API functions will be known by the Language Server, and Puppet 3 API function metadata will be munged into the 4 API equivalent. This commit: * Extends the Sidecar protocol to add a function_version property to the Puppet Function schema. This can be used later by the Language Server to determine how to handle the metadata. * Adds a new method called retrieve_via_puppet_strings to the puppet_helper. This method queries the Puppet Loaders (vai Puppet-As-A-Library PAL) for all the files for particular puppet objects (functions in this case) and then gets the documentation about these files via the Puppet Strings helper * The PAL files are only available on Puppet Gem 6 and above, so the feature flag is modified to only be active on Puppet version 6+ * Adds in a new method called 'discover_paths' on all PAL loaders. The loaders themselves are normally used to load _something_ by name, however the Sidecar wants to load EVERYTHING. Generally this information is private to each loader. By adding this additional method, we can extract all of the discoverable paths, without needing to write our own loaders * Removes the old function loading and monkey patches

This commit adds test fixtures for Puppet 4 API style functions and modifies the integration tests to expect these new fixtures.

The Sidecar will be experimenting with extracting puppet metadata (classes, functions and types etc.) via the Puppet Strings gem. This is an experimental function and is to be hidden behind a feature flag so users can opt in to the feature if needed. This commit: * Adds a feature flag detection method so downstream code can query if certain flags are set * Verifies that the feature flag is valid. Puppet Strings, for now, is only available from the PDK ruby based environment, not the Agent ruby envuronment. * Duplicates the puppet_helper and puppet_monkey_patches files in prepartion of them being modified when the feature flag is set. This allows changes to code to be truly isolated depending on the flag status * Duplicates the integration tests so that behaviour can be verified that is has not changed when the flag is set * Updates the Gemfile to bring in Puppet Strings during development. Note that the Puppet Strings gem is NOT vendored, though that may change in the future

The Puppet Strings gem uses YARD to parse the relevant files, however this is not useful for the Sidecar as we need access to the ruby objects, not a markdown or JSON file being created. This commit: * Adds a PuppetStringsHelper which can configure and execute YARD in the same way Puppet Strings does and then allow the Sidecar to extract the information it needs later. * Later commits will modify the helper to understand the various metadata the Sidecar needs. * Adds a caching layer to the results of running YARD. This means that if a file is queried more than once, YARD will only be executed once as running YARD is an expensive exercise. * Monkey patches YARD to suppress ALL output. By default the command line parameters still emit text of STDOUT, STDERR which breaks the STDIO transport for the Sidecar.

Currently the Sidecar can only detect and load Puppet 3 API functions. The newer Puppet 4 API functions use a different loader and schema, and importantly have additional properties e.g. Puppet 4 API functions have one or more signatures, whereas Puppet 3 API functions use arity. In order to add the Puppet Strings loader, this commit will load Puppet 4 API functions but present them to the Language Server as if they were Puppet 3. A later commit will then change this behaviour so that all the metadata for Puppet 4 API functions will be known by the Language Server, and Puppet 3 API function metadata will be munged into the 4 API equivalent. This commit: * Extends the Sidecar protocol to add a function_version property to the Puppet Function schema. This can be used later by the Language Server to determine how to handle the metadata. * Adds a new method called retrieve_via_puppet_strings to the puppet_helper. This method queries the Puppet Loaders (vai Puppet-As-A-Library PAL) for all the files for particular puppet objects (functions in this case) and then gets the documentation about these files via the Puppet Strings helper * The PAL files are only available on Puppet Gem 6 and above, so the feature flag is modified to only be active on Puppet version 6+ * Adds in a new method called 'discover_paths' on all PAL loaders. The loaders themselves are normally used to load _something_ by name, however the Sidecar wants to load EVERYTHING. Generally this information is private to each loader. By adding this additional method, we can extract all of the discoverable paths, without needing to write our own loaders * Removes the old function loading and monkey patches

This commit adds test fixtures for Puppet 4 API style functions and modifies the integration tests to expect these new fixtures.

Currently the Sidecar can only detect and load Puppet 3 API functions. The newer Puppet 4 API functions use a different loader and schema, and importantly have additional properties e.g. Puppet 4 API functions have one or more signatures, whereas Puppet 3 API functions use arity. In order to add the Puppet Strings loader, this commit will load Puppet 4 API functions but present them to the Language Server as if they were Puppet 3. A later commit will then change this behaviour so that all the metadata for Puppet 4 API functions will be known by the Language Server, and Puppet 3 API function metadata will be munged into the 4 API equivalent. This commit: * Extends the Sidecar protocol to add a function_version property to the Puppet Function schema. This can be used later by the Language Server to determine how to handle the metadata. * Adds a new method called retrieve_via_puppet_strings to the puppet_helper. This method queries the Puppet Loaders (vai Puppet-As-A-Library PAL) for all the files for particular puppet objects (functions in this case) and then gets the documentation about these files via the Puppet Strings helper * The PAL files are only available on Puppet Gem 6 and above, so the feature flag is modified to only be active on Puppet version 6+ * Adds in a new method called 'discover_paths' on all PAL loaders. The loaders themselves are normally used to load _something_ by name, however the Sidecar wants to load EVERYTHING. Generally this information is private to each loader. By adding this additional method, we can extract all of the discoverable paths, without needing to write our own loaders * Removes the old function loading and monkey patches

This commit adds test fixtures for Puppet 4 API style functions and modifies the integration tests to expect these new fixtures.

Previously the results of extracting puppet metadata needed to be calculated whenever the Sidecar was run. This commit modifies the Puppet Strings helper to also use a caching mechanism to store the results and speed up metadata gathering. * Adds a clear! method to the cache, mainly for testing purposes. * Adds integration tests to ensure that objects read from cache mirror the original results * Adds serialisation and deserialisation methods to the FileDocumentation object which allows it to be cached and read back.

Previously the monkey patch to inject the editor workspace as a module did not extract the module name correctly. While this was ok for older Puppet 3 API, the Puppet 4 API namespaces objects based on the module name, so this TODO item needed to be completed. Now we extract the module name correctly.

The Sidecar will be experimenting with extracting puppet metadata (classes, functions and types etc.) via the Puppet Strings gem. This is an experimental function and is to be hidden behind a feature flag so users can opt in to the feature if needed. This commit: * Adds a feature flag detection method so downstream code can query if certain flags are set * Verifies that the feature flag is valid. Puppet Strings, for now, is only available from the PDK ruby based environment, not the Agent ruby envuronment. * Duplicates the puppet_helper and puppet_monkey_patches files in prepartion of them being modified when the feature flag is set. This allows changes to code to be truly isolated depending on the flag status * Duplicates the integration tests so that behaviour can be verified that is has not changed when the flag is set * Updates the Gemfile to bring in Puppet Strings during development. Note that the Puppet Strings gem is NOT vendored, though that may change in the future

The Puppet Strings gem uses YARD to parse the relevant files, however this is not useful for the Sidecar as we need access to the ruby objects, not a markdown or JSON file being created. This commit: * Adds a PuppetStringsHelper which can configure and execute YARD in the same way Puppet Strings does and then allow the Sidecar to extract the information it needs later. * Later commits will modify the helper to understand the various metadata the Sidecar needs. * Adds a caching layer to the results of running YARD. This means that if a file is queried more than once, YARD will only be executed once as running YARD is an expensive exercise. * Monkey patches YARD to suppress ALL output. By default the command line parameters still emit text of STDOUT, STDERR which breaks the STDIO transport for the Sidecar.

Currently the Sidecar can only detect and load Puppet 3 API functions. The newer Puppet 4 API functions use a different loader and schema, and importantly have additional properties e.g. Puppet 4 API functions have one or more signatures, whereas Puppet 3 API functions use arity. In order to add the Puppet Strings loader, this commit will load Puppet 4 API functions but present them to the Language Server as if they were Puppet 3. A later commit will then change this behaviour so that all the metadata for Puppet 4 API functions will be known by the Language Server, and Puppet 3 API function metadata will be munged into the 4 API equivalent. This commit: * Extends the Sidecar protocol to add a function_version property to the Puppet Function schema. This can be used later by the Language Server to determine how to handle the metadata. * Adds a new method called retrieve_via_puppet_strings to the puppet_helper. This method queries the Puppet Loaders (vai Puppet-As-A-Library PAL) for all the files for particular puppet objects (functions in this case) and then gets the documentation about these files via the Puppet Strings helper * The PAL files are only available on Puppet Gem 6 and above, so the feature flag is modified to only be active on Puppet version 6+ * Adds in a new method called 'discover_paths' on all PAL loaders. The loaders themselves are normally used to load _something_ by name, however the Sidecar wants to load EVERYTHING. Generally this information is private to each loader. By adding this additional method, we can extract all of the discoverable paths, without needing to write our own loaders * Removes the old function loading and monkey patches

This commit adds test fixtures for Puppet 4 API style functions and modifies the integration tests to expect these new fixtures.

Previously the results of extracting puppet metadata needed to be calculated whenever the Sidecar was run. This commit modifies the Puppet Strings helper to also use a caching mechanism to store the results and speed up metadata gathering. * Adds a clear! method to the cache, mainly for testing purposes. * Adds integration tests to ensure that objects read from cache mirror the original results * Adds serialisation and deserialisation methods to the FileDocumentation object which allows it to be cached and read back.

…t API v3 functions (#126) (GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions

glennsarti self-assigned this Apr 9, 2019

glennsarti mentioned this issue May 22, 2019

(GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions #126

Merged

1 task

glennsarti changed the title ~~[Puppet4API Project Task 1-4] Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions~~ [Puppet4API Project Task 1-3] Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions May 22, 2019

michaeltlombardi closed this as completed in #126 Jun 6, 2019

michaeltlombardi added a commit that referenced this issue Jun 6, 2019

(GH-121) Load Puppet Functions via Puppet API v4 and present as Puppe…

ca0175f

…t API v3 functions (#126) (GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Puppet4API Project Task 1-3] Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions #121

[Puppet4API Project Task 1-3] Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions #121

glennsarti commented Apr 9, 2019 •

edited

Loading

[Puppet4API Project Task 1-3] Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions #121

[Puppet4API Project Task 1-3] Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions #121

Comments

glennsarti commented Apr 9, 2019 • edited Loading

Task 1

Task 2

Task 3

glennsarti commented Apr 9, 2019 •

edited

Loading