Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions #126

Conversation

glennsarti
Copy link
Contributor

@glennsarti glennsarti commented May 22, 2019

  • Need to add caching back.

Fixes #121

Task 1-3 of the Puppet 4 API Project

Task 1

Add a Puppet Strings feature flag - As much as possible, all code should be hidden behind this flag

Task 2

Load Puppet 4 API Functions but present them as Puppet 3 functions in the Sidecar Protocol. This preserves existing behaviour of the loading, but requires no changes to the language server itself

Task 3

Use Puppet Strings (YARD) to load Puppet 4 documentation - Puppet 4 API does not use dedicated doc parameters, instead all documentation is provided by YARD based comments, interpretted by the Puppet-Strings gem. Puppet Strings extends YARD to understand Puppet syntax


Previously the monkey patch to inject the editor workspace as a module did not
extract the module name correctly. While this was ok for older Puppet 3 API,
the Puppet 4 API namespaces objects based on the module name, so this TODO
item needed to be completed. Now we extract the module name correctly.


he Sidecar will be experimenting with extracting puppet metadata (classes,
functions and types etc.) via the Puppet Strings gem. This is an experimental
function and is to be hidden behind a feature flag so users can opt in to the
feature if needed.

This commit:

  • Adds a feature flag detection method so downstream code can query if certain
    flags are set
  • Verifies that the feature flag is valid. Puppet Strings, for now, is only
    available from the PDK ruby based environment, not the Agent ruby envuronment.
  • Duplicates the puppet_helper and puppet_monkey_patches files in prepartion of
    them being modified when the feature flag is set. This allows changes to code
    to be truly isolated depending on the flag status
  • Duplicates the integration tests so that behaviour can be verified that is
    has not changed when the flag is set
  • Updates the Gemfile to bring in Puppet Strings during development. Note that
    the Puppet Strings gem is NOT vendored, though that may change in the future

The Puppet Strings gem uses YARD to parse the relevant files, however this is
not useful for the Sidecar as we need access to the ruby objects, not a markdown
or JSON file being created. This commit:

  • Adds a PuppetStringsHelper which can configure and execute YARD in the same
    way Puppet Strings does and then allow the Sidecar to extract the information
    it needs later.
  • Later commits will modify the helper to understand the various metadata the
    Sidecar needs.
  • Adds a caching layer to the results of running YARD. This means that if a
    file is queried more than once, YARD will only be executed once as running
    YARD is an expensive exercise.
  • Monkey patches YARD to suppress ALL output. By default the command line
    parameters still emit text of STDOUT, STDERR which breaks the STDIO transport
    for the Sidecar.

Currently the Sidecar can only detect and load Puppet 3 API functions. The
newer Puppet 4 API functions use a different loader and schema, and importantly
have additional properties e.g. Puppet 4 API functions have one or more
signatures, whereas Puppet 3 API functions use arity.

In order to add the Puppet Strings loader, this commit will load Puppet 4 API
functions but present them to the Language Server as if they were Puppet 3.
A later commit will then change this behaviour so that all the metadata for
Puppet 4 API functions will be known by the Language Server, and Puppet 3 API
function metadata will be munged into the 4 API equivalent.

This commit:

  • Extends the Sidecar protocol to add a function_version property to the Puppet
    Function schema. This can be used later by the Language Server to determine
    how to handle the metadata.
  • Adds a new method called retrieve_via_puppet_strings to the puppet_helper.
    This method queries the Puppet Loaders (vai Puppet-As-A-Library PAL) for all
    the files for particular puppet objects (functions in this case) and then gets
    the documentation about these files via the Puppet Strings helper
  • The PAL files are only available on Puppet Gem 6 and above, so the feature
    flag is modified to only be active on Puppet version 6+
  • Adds in a new method called 'discover_paths' on all PAL loaders. The loaders
    themselves are normally used to load something by name, however the Sidecar
    wants to load EVERYTHING. Generally this information is private to each
    loader. By adding this additional method, we can extract all of the
    discoverable paths, without needing to write our own loaders
  • Removes the old function loading and monkey patches

This commit adds test fixtures for Puppet 4 API style functions and modifies the
integration tests to expect these new fixtures.

@glennsarti glennsarti force-pushed the gh121-use-strings-to-extract-functions branch from a75ea8d to e28ce70 Compare May 22, 2019 06:24
@glennsarti glennsarti added enhancement New feature or request Language Server labels May 22, 2019
@glennsarti glennsarti added this to the 0.20.0 milestone May 22, 2019
@glennsarti glennsarti self-assigned this May 22, 2019
@glennsarti glennsarti changed the title {WP} (GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions {WIP} (GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions May 22, 2019
@glennsarti glennsarti force-pushed the gh121-use-strings-to-extract-functions branch 2 times, most recently from 02a5231 to b088bf0 Compare May 22, 2019 12:57
@glennsarti glennsarti changed the title {WIP} (GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions (GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions May 22, 2019
Previously the monkey patch to inject the editor workspace as a module did not
extract the module name correctly. While this was ok for older Puppet 3 API,
the Puppet 4 API namespaces objects based on the module name, so this TODO
item needed to be completed.  Now we extract the module name correctly.
The Sidecar will be experimenting with extracting puppet metadata (classes,
functions and types etc.) via the Puppet Strings gem.  This is an experimental
function and is to be hidden behind a feature flag so users can opt in to the
feature if needed.

This commit:
* Adds a feature flag detection method so downstream code can query if certain
  flags are set
* Verifies that the feature flag is valid.  Puppet Strings, for now, is only
  available from the PDK ruby based environment, not the Agent ruby envuronment.
* Duplicates the puppet_helper and puppet_monkey_patches files in prepartion of
  them being modified when the feature flag is set. This allows changes to code
  to be truly isolated depending on the flag status
* Duplicates the integration tests so that behaviour can be verified that is
  has not changed when the flag is set
* Updates the Gemfile to bring in Puppet Strings during development. Note that
  the Puppet Strings gem is NOT vendored, though that may change in the future
The Puppet Strings gem uses YARD to parse the relevant files, however this is
not useful for the Sidecar as we need access to the ruby objects, not a markdown
or JSON file being created.  This commit:

* Adds a PuppetStringsHelper which can configure and execute YARD in the same
  way Puppet Strings does and then allow the Sidecar to extract the information
  it needs later.
* Later commits will modify the helper to understand the various metadata the
  Sidecar needs.
* Adds a caching layer to the results of running YARD. This means that if a
  file is queried more than once, YARD will only be executed once as running
  YARD is an expensive exercise.
* Monkey patches YARD to suppress ALL output.  By default the command line
  parameters still emit text of STDOUT, STDERR which breaks the STDIO transport
  for the Sidecar.
Currently the Sidecar can only detect and load Puppet 3 API functions.  The
newer Puppet 4 API functions use a different loader and schema, and importantly
have additional properties e.g. Puppet 4 API functions have one or more
signatures, whereas Puppet 3 API functions use arity.

In order to add the Puppet Strings loader, this commit will load Puppet 4 API
functions but present them to the Language Server as if they were Puppet 3.
A later commit will then change this behaviour so that all the metadata for
Puppet 4 API functions will be known by the Language Server, and Puppet 3 API
function metadata will be munged into the 4 API equivalent.

This commit:
* Extends the Sidecar protocol to add a function_version property to the Puppet
  Function schema.  This can be used later by the Language Server to determine
  how to handle the metadata.
* Adds a new method called retrieve_via_puppet_strings to the puppet_helper.
  This method queries the Puppet Loaders (vai Puppet-As-A-Library PAL) for all
  the files for particular puppet objects (functions in this case) and then gets
  the documentation about these files via the Puppet Strings helper
* The PAL files are only available on Puppet Gem 6 and above, so the feature
  flag is modified to only be active on Puppet version 6+
* Adds in a new method called 'discover_paths' on all PAL loaders.  The loaders
  themselves are normally used to load _something_ by name, however the Sidecar
  wants to load EVERYTHING. Generally this information is private to each
  loader. By adding this additional method, we can extract all of the
  discoverable paths, without needing to write our own loaders
* Removes the old function loading and monkey patches
This commit adds test fixtures for Puppet 4 API style functions and modifies the
integration tests to expect these new fixtures.
This setting has no effect.  This commit removes the redundant code.
Previously the results of extracting puppet metadata needed to be calculated
whenever the Sidecar was run.  This commit modifies the Puppet Strings helper
to also use a caching mechanism to store the results and speed up metadata
gathering.

* Adds a clear! method to the cache, mainly for testing purposes.
* Adds integration tests to ensure that objects read from cache mirror the
  original results
* Adds serialisation and deserialisation methods to the FileDocumentation
  object which allows it to be cached and read back.
@michaeltlombardi michaeltlombardi merged commit ca0175f into puppetlabs:master Jun 6, 2019
@glennsarti glennsarti deleted the gh121-use-strings-to-extract-functions branch June 6, 2019 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Language Server
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Puppet4API Project Task 1-3] Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions
2 participants