InSpec loads whichever *.rb file it finds along the way causing itself to fail

[walterdolce]

Description

I started implementing an InSpec profile and I obviously wanted to do it by adopting a BDD/TDD workflow. I have setup the components required to test various scenarios for the profile. You can have a look at the overall structure here.

The problem is simple, when running inspec check . from the root of the project, it picks up whatever it finds starting from that position. This means it's going to assume every *.rb file from that directory position will be something related to InSpec itself, when in reality really is not.

The output produced is the following:

$ inspec check .

WARN: Unresolved specs during Gem::Specification.reset:
      ffi (>= 1.0.1)
      multi_json (~> 1.10)
      rspec (~> 3)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
./test/cookbooks/gcc-installer/metadata.rb:1:in `load_with_context': undefined method `name' for #<#<Class:0x007f93eaabed90>:0x007f93eaabe4a8> (NoMethodError)
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile_context.rb:146:in `instance_eval'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile_context.rb:146:in `load_with_context'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile_context.rb:130:in `load_control_file'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile.rb:144:in `block in collect_tests'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile.rb:140:in `each'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile.rb:140:in `collect_tests'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile.rb:417:in `load_checks_params'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile.rb:410:in `load_params'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile.rb:133:in `params'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile.rb:299:in `controls_count'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/profile.rb:270:in `check'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/lib/inspec/cli.rb:68:in `check'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
	from /Users/{{user}}/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/inspec-1.5.0/bin/inspec:12:in `<top (required)>'
	from /Users/{{user}}/.rbenv/versions/2.3.0/bin/inspec:23:in `load'
	from /Users/{{user}}/.rbenv/versions/2.3.0/bin/inspec:23:in `<main>'

InSpec and Platform Version

# InSpec version is 1.5.0

== Operating System Details

Name:      mac_os_x
Family:    darwin
Release:   10.10.5
Arch:      x86_64

Replication Case

• Fork and clone the repository linked above
• Checkout the WIP branch where I put the initial integration tests setup
• bundle install
• inspec check . or bundle exec inspec check . (depending on whether you have it installed already)

Possible Solutions

InSpec should take into consideration only the *.rb files which pertain to the work it needs to do (just whatever is in controls/ ? Please clarify).

Stacktrace

Included above.

[chris-rock]

@walterdolce I cannot reproduce that issue:

➜  tests git clone [email protected]:walterdolce/inspec-profile-telnet.git
Cloning into 'inspec-profile-telnet'...
remote: Counting objects: 68, done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 68 (delta 9), reused 65 (delta 6), pack-reused 0
Receiving objects: 100% (68/68), 13.65 KiB | 0 bytes/s, done.
Resolving deltas: 100% (9/9), done.
Checking connectivity... done.
➜  tests cd inspec-profile-telnet 
➜  inspec-profile-telnet git:(master) bundle install 
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Using builder 3.2.2
Using coderay 1.1.1
Using diff-lcs 1.2.5
Using excon 0.54.0
Using json 2.0.2
Using erubis 2.7.0
Using ffi 1.9.14
Using hashie 3.4.6
Using httpclient 2.8.2.4
Using method_source 0.8.2
Using mixlib-log 1.7.1
Installing parallel 1.10.0
Using slop 3.6.0
Using rainbow 2.1.0
Using rspec-support 3.5.0
Using rubyzip 1.2.0
Using sslshake 1.0.13
Using thor 0.19.1
Using mixlib-shellout 2.2.7
Using net-ssh 3.2.0
Using little-plugger 1.1.4
Using multi_json 1.12.1
Using nori 2.6.0
Using rubyntlm 0.6.1
Using bundler 1.12.5
Using gyoku 1.3.1
Using docker-api 1.32.1
Using gssapi 1.2.0
Using pry 0.10.4
Using rspec-core 3.5.4
Using rspec-expectations 3.5.0
Using rspec-mocks 3.5.0
Using net-scp 1.2.1
Using logging 2.1.0
Using rspec-its 1.2.0
Using rspec 3.5.0
Using winrm 2.1.0
Using winrm-fs 1.0.0
Using train 0.21.1
Using inspec 1.5.0
Bundle complete! 1 Gemfile dependency, 40 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
➜  inspec-profile-telnet git:(master) bundle exec inspec check .
Location:    .
Profile:     telnet
Controls:    2
Timestamp:   2016-11-25T12:57:38-07:00
Valid:       true

No errors or warnings
➜  inspec-profile-telnet git:(master) 

[walterdolce]

@chris-rock you didn't checkout the feature branch

[chris-rock]

@walterdolce I can confirm that issue now. The old versions of InSpec allowed placing tests in test directory instead of controls. PR #1327 is removing this behavior. Once this is merged, your code works just fine. Thank you for reporting this issue.

[walterdolce]

Thank you @chris-rock for the promptly PR. Will be waiting for it to be merged. 🍺

[chris-rock]

@walterdolce InSpec 1.6.0 is released now

[walterdolce]