Skip to content

Commit

Permalink
(puppetlabsGH-121) Add caching layer to Puppet 4 API functions
Browse files Browse the repository at this point in the history 
Previously the Puppet 4 API loader would always _actually_ load the Puppet
assets whereas the Puppet 3 loader had a caching layer to speed up the process.
This commit adds same caching layer to the Puppet 4 API loaders, specifically
for functions in this case:

* Fixes minor typos in the filesystem cache object and adds a `clear` method
which should only be used for testing

* Move the caching object to live on the PuppetLanguageServerSideCar module.
This is required as the monkey patches have no way of knowing how to access the
cache without it.

* Functions were modified to have an additional was_cached and was_preloaded
property.  was_cached indicates that the function was loaded from cache and
should not be saved back to cache either. was_preloaded indicates that the
object was loaded prior to the loading process. They should also not be saved
to cache because they can never be loaded from cache.

* The function creation methods were modified to load the function metadata
from cache.  Even if we load the metadata from the cache, we still need
_actually_ load the function in Puppet as it keeps track of function loading and
will attempt reloads if it's not seen. Fortunately loading functions is quick
and the user won't really see any slow downs. The slow part of the process is
the puppet string documentation which is not processed when `.was_cached` is set
to true

* Added tests to the function loading to ensure that the metadata from the
function loading is that same whether it is loaded from cache or not.
  • Loading branch information
@glennsarti
glennsarti committed Apr 10, 2019
1 parent aa31944 commit 47e4e90
Show file tree
Hide file tree
Showing 6 changed files with 185 additions and 52 deletions.
19 changes: 14 additions & 5 deletions lib/puppet-languageserver-sidecar/cache/filesystem.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ def initialize(_options = {})
begin
Dir.mkdir(@cache_dir) unless Dir.exist?(@cache_dir)
rescue Errno::ENOENT => e
PuppetLanguageServerSidecar.log_message(:error, "[PuppetLanguageServerSidecar::Cache::File] An error occured while creating file cache. Disabling cache: #{e}")
PuppetLanguageServerSidecar.log_message(:error, "[PuppetLanguageServerSidecar::Cache::FileSystem] An error occured while creating file cache. Disabling cache: #{e}")
@cache_dir = nil
end
end
Expand All @@ -35,20 +35,20 @@ def load(absolute_path, section)

# Check that this is from the same language server version
unless json_obj['sidecar_version'] == PuppetLanguageServerSidecar.version
PuppetLanguageServerSidecar.log_message(:debug, "[PuppetLanguageServerSidecar::load] Error loading #{absolute_path}: Expected sidecar_version version #{PuppetLanguageServerSidecar.version} but found #{json_obj['sidecar_version']}")
PuppetLanguageServerSidecar.log_message(:debug, "[PuppetLanguageServerSidecar::Cache::FileSystem.load] Error loading #{absolute_path}: Expected sidecar_version version #{PuppetLanguageServerSidecar.version} but found #{json_obj['sidecar_version']}")
return nil
end
# Check that the source file hash matches
content_hash = calculate_hash(absolute_path)
if json_obj['file_hash'] != content_hash
PuppetLanguageServerSidecar.log_message(:debug, "[PuppetLanguageServerSidecar::load] Error loading #{absolute_path}: Expected file_hash of #{content_hash} but found #{json_obj['file_hash']}")
PuppetLanguageServerSidecar.log_message(:debug, "[PuppetLanguageServerSidecar::Cache::FileSystem.load] Error loading #{absolute_path}: Expected file_hash of #{content_hash} but found #{json_obj['file_hash']}")
return nil
end
PuppetLanguageServerSidecar.log_message(:debug, "[PuppetLanguageServerSidecar::load] Loading #{absolute_path} from cache")
PuppetLanguageServerSidecar.log_message(:debug, "[PuppetLanguageServerSidecar::Cache::FileSystem.load] Loading #{absolute_path} from cache")

json_obj['data']
rescue RuntimeError => detail
PuppetLanguageServerSidecar.log_message(:debug, "[PuppetLanguageServerSidecar::load] Error loading #{absolute_path}: #{detail}")
PuppetLanguageServerSidecar.log_message(:debug, "[PuppetLanguageServerSidecar::Cache::FileSystem.load] Error loading #{absolute_path}: #{detail}")
raise
end

Expand All @@ -65,9 +65,18 @@ def save(absolute_path, section, content_string)
content['path'] = absolute_path
content['section'] = section

PuppetLanguageServerSidecar.log_message(:debug, "[PuppetLanguageServerSidecar::Cache::FileSystem.save] Saving #{absolute_path} to cache")
save_file(cache_file, content.to_json)
end

# WARNING - This method is only intended for testing the cache
# and should not be used for normal operations
def clear
return unless active?
PuppetLanguageServerSidecar.log_message(:warn, '[PuppetLanguageServerSidecar::Cache::FileSystem.clear] Filesystem based cache is being cleared')
FileUtils.rm(Dir.glob(File.join(cache_dir, '*')), :force => true)
end

private

def file_key(filepath, section)
Expand Down
39 changes: 25 additions & 14 deletions lib/puppet-languageserver-sidecar/puppet_helper_pup4api.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ def self.available_object_types
end

# Retrieve objects via the Puppet 4 API loaders
def self.retrieve_via_pup4_api(_cache, options = {})
def self.retrieve_via_pup4_api(options = {})
PuppetLanguageServerSidecar.log_message(:debug, '[PuppetHelper::retrieve_via_pup4_api] Starting')

object_types = options[:object_types].nil? ? available_object_types : options[:object_types]
Expand All @@ -174,29 +174,30 @@ def self.retrieve_via_pup4_api(_cache, options = {})
:rich_data => true
}

# Mark currently loaded functions as pre-loaded. This can happen because a handful of functions are pre-loaded by puppet as they
# are critical and loaded before our caching takes place e.g. debug or warn
Puppet::Parser::Functions.monkey_function_list.each { |_k, item| item.was_preloaded = true }
Puppet::Functions.monkey_function_list.each { |_k, item| item.was_preloaded = true }

# TODO: Needed? Puppet[:tasks] = true
Puppet.override(context_overrides, 'LanguageServer Sidecar') do
current_env.loaders.private_environment_loader.discover(:function) if object_types.include?(:function)
end

if object_types.include?(:function)
# Enumerate V3 Functions from the monkey patching
Puppet::Parser::Functions.monkey_function_list
.select { |_k, i| path_has_child?(options[:root_path], i[:source_location][:source]) }
.each do |name, item|
obj = PuppetLanguageServerSidecar::Protocol::PuppetFunction.from_puppet(name, item)
result[:functions] << obj
end
result[:functions].concat(Puppet::Parser::Functions.monkey_function_list.select { |_k, i| path_has_child?(options[:root_path], i.source) }.values)
# Enumerate V4 Functions from the monkey patching
Puppet::Functions.monkey_function_list
.select { |_k, i| path_has_child?(options[:root_path], i[:source_location][:source]) }
.each do |name, item|
file_doc = PuppetLanguageServerSidecar::PuppetStringsHelper.file_documentation(item[:source_location][:source])
result[:functions].concat(Puppet::Functions.monkey_function_list.select { |_k, i| path_has_child?(options[:root_path], i.source) }.values)

item[:doc] = file_doc[:functions][name][:doc] unless file_doc.nil? || file_doc[:functions].nil? || file_doc[:functions][name].nil?
obj = PuppetLanguageServerSidecar::Protocol::PuppetFunction.from_puppet(name, item)
result[:functions] << obj
# Generate the documentation for the function if it didn't come from cache and not already resolved
result[:functions].each do |item|
next if item.was_cached || !item.doc.nil?
file_doc = PuppetLanguageServerSidecar::PuppetStringsHelper.file_documentation(item.source)
item.doc = file_doc[:functions][item.key][:doc] unless file_doc.nil? || file_doc[:functions].nil? || file_doc[:functions][item.key].nil?
end

save_objects_to_cache(result[:functions], PuppetLanguageServerSidecar::Cache::FUNCTIONS_SECTION) if PuppetLanguageServerSidecar.cache.active? && !result[:functions].empty?
end
PuppetLanguageServerSidecar.log_message(:debug, "[PuppetHelper::retrieve_via_pup4_api] Finished loading #{result[:functions].count} functions") if object_types.include?(:function)

Expand All @@ -205,6 +206,16 @@ def self.retrieve_via_pup4_api(_cache, options = {})

# Private functions

# Assumes the cache is active and object_list is not empty
def self.save_objects_to_cache(object_list, cache_section)
cache = PuppetLanguageServerSidecar.cache

object_list.select { |item| !item.was_cached && !item.was_preloaded }.map(&:source).uniq.each do |source_file|
cache.save(source_file, cache_section, object_list.select { |item| !item.was_cached && !item.was_preloaded && item.source == source_file }.to_json)
end
end
private_class_method :save_objects_to_cache

def self.prune_resource_parameters(resources)
# From https://github.com/puppetlabs/puppet/blob/488661d84e54904124514ab9e4500e81b10f84d1/lib/puppet/application/resource.rb#L146-L148
if resources.is_a?(Array)
Expand Down
94 changes: 72 additions & 22 deletions lib/puppet-languageserver-sidecar/puppet_monkey_patches_pup4api.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,11 @@ def newfunction(name, options = {}, &block)
caller = Kernel.caller_locations[monkey_index + 1]
# Call the original new function method
result = original_newfunction(name, options, &block)
# Append the caller information
result[:source_location] = {
:source => caller.absolute_path,
:line => caller.lineno - 1 # Convert to a zero based line number system
}
monkey_append_function_info(name, result)
monkey_append_function_info(name.to_s, result,
:source_location => {
:source => caller.absolute_path,
:line => caller.lineno - 1 # Convert to a zero based line number system
})

result
end
Expand All @@ -27,20 +26,23 @@ def monkey_clear_function_info
@monkey_function_list = {}
end

def monkey_append_function_info(name, value)
def monkey_append_function_info(name, value, options = {})
return unless @monkey_function_list.nil? || @monkey_function_list[name].nil?
@monkey_function_list = {} if @monkey_function_list.nil?
@monkey_function_list[name] = {
:arity => value[:arity],
:name => value[:name],
:type => value[:type],
:doc => value[:doc],
:source_location => value[:source_location]
}
func_hash = {
:arity => value[:arity],
:name => name,
:type => value[:type],
:doc => value[:doc]
}.merge(options)

@monkey_function_list = {} if @monkey_function_list.nil?
@monkey_function_list[name] = PuppetLanguageServerSidecar::Protocol::PuppetFunction.from_puppet(name, func_hash)
end

def monkey_function_list
@monkey_function_list = {} if @monkey_function_list.nil?
@monkey_function_list.clone
@monkey_function_list
end
end
end
Expand Down Expand Up @@ -76,18 +78,20 @@ def self.monkey_clear_function_info
end

def self.monkey_append_function_info(name, func, options = {})
@monkey_function_list = {} if @monkey_function_list.nil?
@monkey_function_list[name] = {
return unless @monkey_function_list.nil? || @monkey_function_list[name].nil?
func_hash = {
:arity => func.signatures.empty? ? -1 : func.signatures[0].args_range[0], # Fake the arity parameter
:name => name,
:type => :rvalue, # All Puppet 4 functions return a value
:doc => nil # Docs are filled in post processing via Yard
}.merge(options)
@monkey_function_list = {} if @monkey_function_list.nil?
@monkey_function_list[name] = PuppetLanguageServerSidecar::Protocol::PuppetFunction.from_puppet(name, func_hash)
end

def self.monkey_function_list
@monkey_function_list = {} if @monkey_function_list.nil?
@monkey_function_list.clone
@monkey_function_list
end
end
end
Expand Down Expand Up @@ -123,8 +127,6 @@ def newtype(name, options = {}, &block)
end
end

# The Ruby Legacy Function Instantiator doesn't have any error catching upon loading and would normally cause the entire puppet
# run to fail. However as we're a bit special, we can wrap the loader in rescue block and just continue on
require 'puppet/pops/loader/ruby_legacy_function_instantiator'
module Puppet
module Pops
Expand All @@ -134,7 +136,32 @@ class << self
alias_method :original_create, :create
end

# The Ruby Legacy Function Instantiator doesn't have any error catching upon loading and would normally cause the entire puppet
# run to fail. However as we're a bit special, we can wrap the loader in rescue block and just continue on
def self.create(loader, typed_name, source_ref, ruby_code_string)
cache = PuppetLanguageServerSidecar.cache
if cache.active?
cached_result = cache.load(source_ref, PuppetLanguageServerSidecar::Cache::FUNCTIONS_SECTION)
unless cached_result.nil?
begin
funcs = PuppetLanguageServerSidecar::Protocol::PuppetFunctionList.new
funcs.from_json!(cached_result)

# Put the cached objects into the monkey patched list
funcs.each do |item|
item.was_cached = true
Puppet::Parser::Functions.monkey_function_list[item.key.to_s] = item unless Puppet::Parser::Functions.monkey_function_list.include?(item.key.to_s)
end
rescue StandardError => e
PuppetLanguageServerSidecar.log_message(:warn, "[MonkeyPatch::Puppet::Pops::Loader::RubyFunctionInstantiator] Error while deserializing #{source_ref} from cache: #{e}")
end
end
end

# Even if we load the metadata from the cache, we still need _actually_ load the function in Puppet as it keeps track
# of function loading and will attempt reloads if it's not seen. Fortunately loading functions is quick and the user won't
# really see any slow downs. The slow part of the process is the puppet string documentation which is not processed when
# `.was_cached` is set to true
original_create(loader, typed_name, source_ref, ruby_code_string)
rescue LoadError, StandardError => err
PuppetLanguageServerSidecar.log_message(:error, "[MonkeyPatch::Puppet::Pops::Loader::RubyLegacyFunctionInstantiator] Error loading legacy function #{typed_name.name}: #{err} #{err.backtrace}")
Expand All @@ -144,8 +171,6 @@ def self.create(loader, typed_name, source_ref, ruby_code_string)
end
end

# The Ruby Function Instantiator doesn't have any error catching upon loading and would normally cause the entire puppet
# run to fail. However as we're a bit special, we can wrap the loader in rescue block and just continue on
require 'puppet/pops/loader/ruby_function_instantiator'
module Puppet
module Pops
Expand All @@ -155,7 +180,32 @@ class << self
alias_method :original_create, :create
end

# The Ruby Function Instantiator doesn't have any error catching upon loading and would normally cause the entire puppet
# run to fail. However as we're a bit special, we can wrap the loader in rescue block and just continue on
def self.create(loader, typed_name, source_ref, ruby_code_string)
cache = PuppetLanguageServerSidecar.cache
if cache.active?
cached_result = cache.load(source_ref, PuppetLanguageServerSidecar::Cache::FUNCTIONS_SECTION)
unless cached_result.nil?
begin
funcs = PuppetLanguageServerSidecar::Protocol::PuppetFunctionList.new
funcs.from_json!(cached_result)

# Put the cached objects into the monkey patched list
funcs.each do |item|
item.was_cached = true
Puppet::Functions.monkey_function_list[item.key.to_s] = item unless Puppet::Functions.monkey_function_list.include?(item.key.to_s)
end
rescue StandardError => e
PuppetLanguageServerSidecar.log_message(:warn, "[MonkeyPatch::Puppet::Pops::Loader::RubyFunctionInstantiator] Error while deserializing #{source_ref} from cache: #{e}")
end
end
end

# Even if we load the metadata from the cache, we still need _actually_ load the function in Puppet as it keeps track
# of function loading and will attempt reloads if it's not seen. Fortunately loading functions is quick and the user won't
# really see any slow downs. The slow part of the process is the puppet string documentation which is not processed when
# `.was_cached` is set to true
original_create(loader, typed_name, source_ref, ruby_code_string)
rescue LoadError, StandardError => err
PuppetLanguageServerSidecar.log_message(:error, "[MonkeyPatch::Puppet::Pops::Loader::RubyLegacyFunctionInstantiator] Error loading function #{typed_name.name}: #{err} #{err.backtrace}")
Expand Down
13 changes: 12 additions & 1 deletion lib/puppet-languageserver-sidecar/sidecar_protocol_extensions.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,18 @@ def self.from_puppet(name, item, locator)
end
end

class PuppetFunctionList < PuppetLanguageServer::Sidecar::Protocol::PuppetFunctionList
def child_type
PuppetLanguageServerSidecar::Protocol::PuppetFunction
end
end

class PuppetFunction < PuppetLanguageServer::Sidecar::Protocol::PuppetFunction
attr_accessor :was_cached # Whether this function's metadata was loaded from the cache
attr_accessor :was_preloaded # Whether this function's was pre-loaded by puppet at startup, which avoids our caching layer

def self.from_puppet(name, item)
obj = PuppetLanguageServer::Sidecar::Protocol::PuppetFunction.new
obj = PuppetLanguageServerSidecar::Protocol::PuppetFunction.new
obj.key = name
obj.source = item[:source_location][:source]
obj.calling_source = obj.source
Expand All @@ -45,6 +54,8 @@ def self.from_puppet(name, item)
obj.doc = item[:doc]
obj.arity = item[:arity]
obj.type = item[:type]
obj.was_cached = false
obj.was_preloaded = false
obj
end
end
Expand Down
25 changes: 18 additions & 7 deletions lib/puppet_languageserver_sidecar.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,18 @@
end

module PuppetLanguageServerSidecar
attr_reader :sidecar_cache

def self.cache
@sidecar_cache = PuppetLanguageServerSidecar::Cache::Null.new if @sidecar_cache.nil?
@sidecar_cache
end

def self.cache=(value)
@sidecar_cache = value
end
private_class_method :cache=

def self.version
PuppetEditorServices.version
end
Expand Down Expand Up @@ -257,11 +269,11 @@ def self.execute(options)
PuppetLanguageServerSidecar::PuppetHelper.retrieve_classes(cache)

when 'default_functions'
cache = options[:disable_cache] ? PuppetLanguageServerSidecar::Cache::Null.new : PuppetLanguageServerSidecar::Cache::FileSystem.new
self.cache = options[:disable_cache] ? PuppetLanguageServerSidecar::Cache::Null.new : PuppetLanguageServerSidecar::Cache::FileSystem.new
if use_pup4api
PuppetLanguageServerSidecar::PuppetHelper.retrieve_via_pup4_api(cache, :object_types => [:function])[:functions]
PuppetLanguageServerSidecar::PuppetHelper.retrieve_via_pup4_api(:object_types => [:function])[:functions]
else
PuppetLanguageServerSidecar::PuppetHelper.retrieve_functions(cache)
PuppetLanguageServerSidecar::PuppetHelper.retrieve_functions(self.cache)
end

when 'default_types'
Expand Down Expand Up @@ -300,14 +312,13 @@ def self.execute(options)
:root_path => PuppetLanguageServerSidecar::Workspace.root_path)

when 'workspace_functions'
null_cache = PuppetLanguageServerSidecar::Cache::Null.new
self.cache = PuppetLanguageServerSidecar::Cache::Null.new
return nil unless inject_workspace_as_module || inject_workspace_as_environment
if use_pup4api
PuppetLanguageServerSidecar::PuppetHelper.retrieve_via_pup4_api(null_cache,
:root_path => PuppetLanguageServerSidecar::Workspace.root_path,
PuppetLanguageServerSidecar::PuppetHelper.retrieve_via_pup4_api(:root_path => PuppetLanguageServerSidecar::Workspace.root_path,
:object_types => [:function])[:functions]
else
PuppetLanguageServerSidecar::PuppetHelper.retrieve_functions(null_cache,
PuppetLanguageServerSidecar::PuppetHelper.retrieve_functions(self.cache,
:root_path => PuppetLanguageServerSidecar::Workspace.root_path)
end

Expand Down
Loading

0 comments on commit 47e4e90

Please sign in to comment.