-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions #126
Merged
michaeltlombardi
merged 7 commits into
puppetlabs:master
from
glennsarti:gh121-use-strings-to-extract-functions
Jun 6, 2019
Merged
(GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions #126
michaeltlombardi
merged 7 commits into
puppetlabs:master
from
glennsarti:gh121-use-strings-to-extract-functions
Jun 6, 2019
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
glennsarti
force-pushed
the
gh121-use-strings-to-extract-functions
branch
from
May 22, 2019 06:24
a75ea8d
to
e28ce70
Compare
glennsarti
changed the title
{WP} (GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions
{WIP} (GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions
May 22, 2019
glennsarti
commented
May 22, 2019
...on/puppet-languageserver-sidecar/featureflag_puppetstrings/featureflag_puppetstrings_spec.rb
Outdated
Show resolved
Hide resolved
glennsarti
force-pushed
the
gh121-use-strings-to-extract-functions
branch
2 times, most recently
from
May 22, 2019 12:57
02a5231
to
b088bf0
Compare
glennsarti
changed the title
{WIP} (GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions
(GH-121) Load Puppet Functions via Puppet API v4 and present as Puppet API v3 functions
May 22, 2019
Previously the monkey patch to inject the editor workspace as a module did not extract the module name correctly. While this was ok for older Puppet 3 API, the Puppet 4 API namespaces objects based on the module name, so this TODO item needed to be completed. Now we extract the module name correctly.
The Sidecar will be experimenting with extracting puppet metadata (classes, functions and types etc.) via the Puppet Strings gem. This is an experimental function and is to be hidden behind a feature flag so users can opt in to the feature if needed. This commit: * Adds a feature flag detection method so downstream code can query if certain flags are set * Verifies that the feature flag is valid. Puppet Strings, for now, is only available from the PDK ruby based environment, not the Agent ruby envuronment. * Duplicates the puppet_helper and puppet_monkey_patches files in prepartion of them being modified when the feature flag is set. This allows changes to code to be truly isolated depending on the flag status * Duplicates the integration tests so that behaviour can be verified that is has not changed when the flag is set * Updates the Gemfile to bring in Puppet Strings during development. Note that the Puppet Strings gem is NOT vendored, though that may change in the future
The Puppet Strings gem uses YARD to parse the relevant files, however this is not useful for the Sidecar as we need access to the ruby objects, not a markdown or JSON file being created. This commit: * Adds a PuppetStringsHelper which can configure and execute YARD in the same way Puppet Strings does and then allow the Sidecar to extract the information it needs later. * Later commits will modify the helper to understand the various metadata the Sidecar needs. * Adds a caching layer to the results of running YARD. This means that if a file is queried more than once, YARD will only be executed once as running YARD is an expensive exercise. * Monkey patches YARD to suppress ALL output. By default the command line parameters still emit text of STDOUT, STDERR which breaks the STDIO transport for the Sidecar.
Currently the Sidecar can only detect and load Puppet 3 API functions. The newer Puppet 4 API functions use a different loader and schema, and importantly have additional properties e.g. Puppet 4 API functions have one or more signatures, whereas Puppet 3 API functions use arity. In order to add the Puppet Strings loader, this commit will load Puppet 4 API functions but present them to the Language Server as if they were Puppet 3. A later commit will then change this behaviour so that all the metadata for Puppet 4 API functions will be known by the Language Server, and Puppet 3 API function metadata will be munged into the 4 API equivalent. This commit: * Extends the Sidecar protocol to add a function_version property to the Puppet Function schema. This can be used later by the Language Server to determine how to handle the metadata. * Adds a new method called retrieve_via_puppet_strings to the puppet_helper. This method queries the Puppet Loaders (vai Puppet-As-A-Library PAL) for all the files for particular puppet objects (functions in this case) and then gets the documentation about these files via the Puppet Strings helper * The PAL files are only available on Puppet Gem 6 and above, so the feature flag is modified to only be active on Puppet version 6+ * Adds in a new method called 'discover_paths' on all PAL loaders. The loaders themselves are normally used to load _something_ by name, however the Sidecar wants to load EVERYTHING. Generally this information is private to each loader. By adding this additional method, we can extract all of the discoverable paths, without needing to write our own loaders * Removes the old function loading and monkey patches
This commit adds test fixtures for Puppet 4 API style functions and modifies the integration tests to expect these new fixtures.
This setting has no effect. This commit removes the redundant code.
Previously the results of extracting puppet metadata needed to be calculated whenever the Sidecar was run. This commit modifies the Puppet Strings helper to also use a caching mechanism to store the results and speed up metadata gathering. * Adds a clear! method to the cache, mainly for testing purposes. * Adds integration tests to ensure that objects read from cache mirror the original results * Adds serialisation and deserialisation methods to the FileDocumentation object which allows it to be cached and read back.
glennsarti
force-pushed
the
gh121-use-strings-to-extract-functions
branch
from
May 31, 2019 04:28
b088bf0
to
3e13a8a
Compare
michaeltlombardi
approved these changes
Jun 6, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #121
Task 1-3 of the Puppet 4 API Project
Task 1
Add a Puppet Strings feature flag - As much as possible, all code should be hidden behind this flag
Task 2
Load Puppet 4 API Functions but present them as Puppet 3 functions in the Sidecar Protocol. This preserves existing behaviour of the loading, but requires no changes to the language server itself
Task 3
Use Puppet Strings (YARD) to load Puppet 4 documentation - Puppet 4 API does not use dedicated
doc
parameters, instead all documentation is provided by YARD based comments, interpretted by the Puppet-Strings gem. Puppet Strings extends YARD to understand Puppet syntaxPreviously the monkey patch to inject the editor workspace as a module did not
extract the module name correctly. While this was ok for older Puppet 3 API,
the Puppet 4 API namespaces objects based on the module name, so this TODO
item needed to be completed. Now we extract the module name correctly.
he Sidecar will be experimenting with extracting puppet metadata (classes,
functions and types etc.) via the Puppet Strings gem. This is an experimental
function and is to be hidden behind a feature flag so users can opt in to the
feature if needed.
This commit:
flags are set
available from the PDK ruby based environment, not the Agent ruby envuronment.
them being modified when the feature flag is set. This allows changes to code
to be truly isolated depending on the flag status
has not changed when the flag is set
the Puppet Strings gem is NOT vendored, though that may change in the future
The Puppet Strings gem uses YARD to parse the relevant files, however this is
not useful for the Sidecar as we need access to the ruby objects, not a markdown
or JSON file being created. This commit:
way Puppet Strings does and then allow the Sidecar to extract the information
it needs later.
Sidecar needs.
file is queried more than once, YARD will only be executed once as running
YARD is an expensive exercise.
parameters still emit text of STDOUT, STDERR which breaks the STDIO transport
for the Sidecar.
Currently the Sidecar can only detect and load Puppet 3 API functions. The
newer Puppet 4 API functions use a different loader and schema, and importantly
have additional properties e.g. Puppet 4 API functions have one or more
signatures, whereas Puppet 3 API functions use arity.
In order to add the Puppet Strings loader, this commit will load Puppet 4 API
functions but present them to the Language Server as if they were Puppet 3.
A later commit will then change this behaviour so that all the metadata for
Puppet 4 API functions will be known by the Language Server, and Puppet 3 API
function metadata will be munged into the 4 API equivalent.
This commit:
Function schema. This can be used later by the Language Server to determine
how to handle the metadata.
This method queries the Puppet Loaders (vai Puppet-As-A-Library PAL) for all
the files for particular puppet objects (functions in this case) and then gets
the documentation about these files via the Puppet Strings helper
flag is modified to only be active on Puppet version 6+
themselves are normally used to load something by name, however the Sidecar
wants to load EVERYTHING. Generally this information is private to each
loader. By adding this additional method, we can extract all of the
discoverable paths, without needing to write our own loaders
This commit adds test fixtures for Puppet 4 API style functions and modifies the
integration tests to expect these new fixtures.