SSH prompting password for an instance inside VPC

[nirmaljss]

I am trying to spin up an ec2 instance inside a VPC on a private subnet. Every time I run kitchen test, I am able to spin up the instance with the right security groups and in the right subnet range. When test-kitchen is trying to SSH on to the instance, it is asking for password. However, when I manually try to ssh (ssh <private_ip> -i <path_to_ssh_key> -l ubuntu) on to the machine I succeed without being prompted for a password.

The following is my .kitchen.yml file

---
driver:
  name: ec2
  aws_ssh_key_id: id-spanning
  security_group_ids: ['sg-9....5']
  region: us-east-1
  availability_zone: us-east-1a
  require_chef_omnibus: true
  subnet_id: subnet-5...0
  associate_public_ip: false
  instance_type: m3.medium
  interface: private

transport:
  ssh_key: ~/.ssh/id-spanning.pem
  connection_timeout: 10
  connection_retries: 5
  username: ubuntu

provisioner:
  name: chef_solo

platforms:
  - name: Ubuntu-14.04
    driver:
      image_id: ami-8821cae0

suites:
  - name: default
    run_list:
    attributes:

I have the aws credentials in place on the environment variables. The following is my output.

kitchen test
-----> Starting Kitchen (v1.4.0)
-----> Cleaning up any prior instances of <default-Ubuntu-1404>
-----> Destroying <default-Ubuntu-1404>...
       EC2 instance <i-16f468c6> destroyed.
       Finished destroying <default-Ubuntu-1404> (0m1.90s).
-----> Testing <default-Ubuntu-1404>
-----> Creating <default-Ubuntu-1404>...
       Creating <>...
If you are not using an account that qualifies under the AWS
free-tier, you may be charged to run these suites. The charge
should be minimal, but neither Test Kitchen nor its maintainers
are responsible for your incurred costs.

       Instance <i-8fad345f> requested.
       EC2 instance <i-8fad345f> created.
       Waited 0/300s for instance <i-8fad345f> to become ready.
       Waited 5/300s for instance <i-8fad345f> to become ready.
       Waited 10/300s for instance <i-8fad345f> to become ready.
       Waited 15/300s for instance <i-8fad345f> to become ready.
       Waited 20/300s for instance <i-8fad345f> to become ready.
       Waited 25/300s for instance <i-8fad345f> to become ready.
       EC2 instance <i-8fad345f> ready.
Password: 

I tried several times and haven't had any luck on bypassing the password to allow test-kitchen to ssh on to the instance. The following is my kitchen diagnose output.

---
timestamp: 2015-05-26 15:34:29 UTC
kitchen_version: 1.4.0
instances:
  default-Ubuntu-1404:
    platform:
      os_type: unix
      shell_type: bourne
    state_file:
      hostname: ''
      server_id: i-1.....6
    driver:
      associate_public_ip: false
      availability_zone: us-east-1a
      aws_access_key_id: 
      aws_secret_access_key: 
      aws_session_token: 
      aws_ssh_key_id: id-spanning
      block_device_mappings: 
      ebs_optimized: false
      flavor_id: 
      iam_profile_name: 
      image_id: ami-8821cae0
      instance_type: m3.medium
      interface: private
      kitchen_root: "/Users/jonnas2/Desktop/apache101"
      log_level: :info
      name: ec2
      price: 
      private_ip_address: 
      region: us-east-1
      retryable_sleep: 5
      retryable_tries: 60
      security_group_ids:
      - sg-9....5
      shared_credentials_profile: 
      subnet_id: subnet-5....0
      tags:
        created-by: test-kitchen
      test_base_path: "/Users/jonnas2/Desktop/apache101/test/integration"
      user_data: 
      username: 
    provisioner:
      attributes: {}
      chef_metadata_url: 
      chef_omnibus_install_options: 
      chef_omnibus_root: "/opt/chef"
      chef_omnibus_url: https://www.chef.io/chef/install.sh
      chef_solo_path: "/opt/chef/bin/chef-solo"
      clients_path: 
      cookbook_files_glob: README.*,metadata.{json,rb},attributes/**/*,definitions/**/*,files/**/*,libraries/**/*,providers/**/*,recipes/**/*,resources/**/*,templates/**/*
      data_bags_path: 
      data_path: 
      encrypted_data_bag_secret_key_path: 
      environments_path: 
      http_proxy: 
      https_proxy: 
      kitchen_root: "/Users/jonnas2/Desktop/apache101"
      log_file: 
      log_level: :info
      name: chef_solo
      nodes_path: 
      require_chef_omnibus: true
      roles_path: 
      root_path: "/tmp/kitchen"
      run_list: []
      solo_rb: {}
      sudo: true
      sudo_command: sudo -E
      test_base_path: "/Users/jonnas2/Desktop/apache101/test/integration"
    transport:
      compression: zlib
      compression_level: 6
      connection_retries: 5
      connection_retry_sleep: 1
      connection_timeout: 10
      keepalive: true
      keepalive_interval: 60
      kitchen_root: "/Users/jonnas2/Desktop/apache101"
      log_level: :info
      max_wait_until_ready: 600
      name: ssh
      port: 22
      ssh_key: "/Users/jonnas2/.ssh/id-spanning.pem"
      test_base_path: "/Users/jonnas2/Desktop/apache101/test/integration"
      username: ubuntu
    verifier:
      busser_bin: "/tmp/verifier/bin/busser"
      http_proxy: 
      https_proxy: 
      kitchen_root: "/Users/jonnas2/Desktop/apache101"
      log_level: :info
      name: busser
      root_path: "/tmp/verifier"
      ruby_bindir: "/opt/chef/embedded/bin"
      sudo: true
      sudo_command: sudo -E
      suite_name: default
      test_base_path: "/Users/jonnas2/Desktop/apache101/test/integration"
      version: busser

Any help would be greatly appreciated. Thanks.

[koertkuipers]

we upgraded recently from 0.8.0 and are also seeing this
(and after downgrade to 0.8.0 it works again)

[tyler-ball]

Hey @nirmaljss and @koertkuipers - we recently merged test-kitchen/test-kitchen#704 into core TK which disables password auth if an ssh_key is configured.

Can you try using TK master locally and seeing if this fixes your issue (or at least exposes why the key is not being accepted in the first place)? You can add a Gemfile to your cookbook with the following contents:

source 'https://rubygems.org'

gem "test-kitchen", git: "https://github.com/test-kitchen/test-kitchen"
gem "kitchen-ec2", "~> 0.9.2"
gem "berkshelf"

The run bundle install and bundle exec kitchen create.

[nirmaljss]

@tyler-ball You Sir made my day!!! It works like a charm with the TK master. This issue can be closed and I hope TK cuts a release soon. Thank you.

[tyler-ball]

Great! I'll leave this open so we can track the following TODOs:

• Release TK master
• Make sure the version constraints in kitchen-ec2 will pick up the newly released version in the ChefDK builds and via bundle update

[tyler-ball]

@nirmaljss Kitchen-ec2 0.10.0 (released yesterday) should include this change. I'm going to go make sure the ChefDK build has the latest Test Kitchen now, so I'm going to close this bug. Thanks for the report!

[kg4giy]

I am not sure this is fixed.

$ chef -v
Chef Development Kit Version: 0.9.0
chef-client version: 12.5.1
berks version: 4.0.1
kitchen version: 1.4.2

*** LOCAL GEMS ***

kitchen-ec2 (0.10.0)

driver:
name: ec2
security_group_ids: ["sec group"]
require_chef_omnibus: true
region: us-east-1
availability_zone: d
subnet_id: "subnet-subnet"
associate_public_ip: true
interface: private

transport:
ssh_key: "/home/ubuntu/.ssh/awskey.pem"
username: ["root"]

provisioner:
name: chef_solo

Still fails. I am able to log in with ssh -i ....

I have tried just about every combination and permutation to get this working - the instance boots, but you can't log in automagically. Is there a way to force the the auth by passwd off?

I should add that it times out:

Waiting for SSH service on 172.31.51.50:22, retrying in 3 seconds
Waiting for SSH service on 172.31.51.50:22, retrying in 3 seconds
Waiting for SSH service on 172.31.51.50:22, retrying in 3 seconds
Waiting for SSH service on 172.31.51.50:22, retrying in 3 seconds
Waiting for SSH service on 172.31.51.50:22, retrying in 3 seconds
Waiting for SSH service on 172.31.51.50:22, retrying in 3 seconds
Waiting for SSH service on 172.31.51.50:22, retrying in 3 seconds

[kg4giy]

Hmm...update - works with Ubuntu as the image, not CentOS.

Changes:

AMI to front page Ubuntu image
Username from "root" to "ubuntu"

No other changes.

centos-kitchen.yml

---

driver:
name: ec2
security_group_ids: ["sg-...f"]
require_chef_omnibus: true
region: us-east-1
availability_zone: d
subnet_id: "subnet-...c"
associate_public_ip: true
interface: private

transport:
ssh_key: "/home/ubuntu/.ssh/...a.pem"
username: ["root"]

provisioner:
name: chef_solo

platforms:

• name: centos-6.4
  driver:
  image_id: ami-26cc934e
  instance_type: t1.micro
  block_device_mappings:
  - ebs_device_name: /dev/sdb
  ebs_volume_type: gp2
  ebs_virtual_name: test
  ebs_volume_size: 8
  ebs_delete_on_termination: true

suites:

• name: default
  run_list:
  attributes:

ubuntu-kitchen.yml

---

driver:
name: ec2
security_group_ids: ["sg-...f"]
require_chef_omnibus: true
region: us-east-1
availability_zone: d
subnet_id: "subnet-...c"
associate_public_ip: true
interface: private

transport:
ssh_key: "/home/ubuntu/.ssh/...a.pem"
username: ["ubuntu"]

provisioner:
name: chef_solo

platforms:

• name: ubuntu-14.04
  driver:
  image_id: ami-d05e75b8
  instance_type: t2.micro
  block_device_mappings:
  - ebs_device_name: /dev/sdb
  ebs_volume_type: gp2
  ebs_virtual_name: test
  ebs_volume_size: 8
  ebs_delete_on_termination: true

suites:

• name: default
  run_list:
  attributes:

I can only conclude then that there is a problem with the AMI? for the CentOS 6.4 image on Amazon (and you have to hunt for it anyway).

Opinions?