TK Can't Connect to EC2 Instance via SSH #135

jcderose · 2015-05-28T20:37:19Z

In the process of creating an EC2 instance via TK, TK gets stuck attempting to SSH into the instance.

➜ kitchen create
-----> Starting Kitchen (v1.4.0)
-----> Creating <default-amazon>...
       Creating <>...
If you are not using an account that qualifies under the AWS
free-tier, you may be charged to run these suites. The charge
should be minimal, but neither Test Kitchen nor its maintainers
are responsible for your incurred costs.

       Instance <i-de580128> requested.
       EC2 instance <i-de580128> created.
       Waited 0/300s for instance <i-de580128> to become ready.
       Waited 5/300s for instance <i-de580128> to become ready.
       Waited 10/300s for instance <i-de580128> to become ready.
       Waited 15/300s for instance <i-de580128> to become ready.
       Waited 20/300s for instance <i-de580128> to become ready.
       EC2 instance <i-de580128> ready.
       Waiting for SSH service on ec2-52-25-230-10.us-west-2.compute.amazonaws.com:22, retrying in 3 seconds
       Waiting for SSH service on ec2-52-25-230-10.us-west-2.compute.amazonaws.com:22, retrying in 3 seconds
       Waiting for SSH service on ec2-52-25-230-10.us-west-2.compute.amazonaws.com:22, retrying in 3 seconds

This "Waiting" message continues endlessly until I quit the process (and then TK barks that the instance wasn't created successfully, even though it's up and running in AWS). While TK is attempting to SSH in during the kitchen create process, I'm able to SSH into the instance manually so I know the instance is accepting SSH connections correctly.

I'm running the latest version of TK and the kitchen-ec2 driver.

➜ chef gem list | grep kitchen
kitchen-ec2 (0.9.2)
test-kitchen (1.4.0)

Here's my .kitchen.yml file.


---
driver:
  name: ec2
  security_group_ids: ["sg-5d213838"]
  region: us-west-2
  availability_zone: us-west-2a
  require_chef_omnibus: true
  subnet_id: subnet-eb8f489c
  image_id: "ami-ff527ecf"
  associate_public_ip: true

transport:
  username: ec2-user

provisioner:
  name: chef_solo

platforms:
  - name: amazon

suites:
  - name: default
    run_list:
    attributes:

Here's my environment variables (for the SSH connection, set in my login script):

export AWS_ACCESS_KEY_ID="XXXXXXXX"
export AWS_SECRET_ACCESS_KEY="XXXXXXXX"
export AWS_SSH_KEY_ID="key1"
export AWS_SSH_KEY="$HOME/.ssh/$AWS_SSH_KEY_ID.pem"

The text was updated successfully, but these errors were encountered:

tyler-ball · 2015-05-28T20:43:32Z

Hey @jcderose - thanks for the bug report. Do you have any configs set in your ~/.ssh/config? Also, can you run Test Kitchen with debug enabled (kitchen create -l debug) and post the output to a gist? This should give us more information about what is being passed into net ssh.

jcderose · 2015-05-28T21:00:35Z

No AWS-related configs are set in my ~/.ssh/config file. Here's the output of running Test Kitchen with debug enabled (kitchen create -l debug):

https://gist.github.com/jcderose/00e612d024232e9e7e7f

It looks like the connection is refused, and then authentication fails. However, when I manually login from the same device, I have no problems:

➜ ssh -i "$HOME/.ssh/key1.pem" [email protected]

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2015.03-release-notes/
18 package(s) needed for security, out of 46 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-10-172-54-114 ~]$

tyler-ball · 2015-05-28T21:35:24Z

Hey @jcderose - I'm pretty sure I know the issue. You're correct - you can see in the debug log that its an authentication failure because the netssh configs don't have :ssh_key present.

In your .kitchen.yml set this for your transport:

transport:
  ssh_key: "$HOME/.ssh/key1.pem"

The aws_ssh_key_id config in .kitchen.yml only tells AWS which key to put on the instance it spins up. It does not tell TK which key to use to log into that instance (although thats probably something that should be improved).

tyler-ball · 2015-05-28T21:36:17Z

If that fix works for you, we should file another bug to default the transport ssh_key to aws_ssh_key if it is present as an ENV variable. Right now, we don't do anything with ENV['AWS_SSH_KEY']

jcderose · 2015-05-28T21:45:45Z

Interesting follow-up here: I added the :ssh_key in my transport section as you suggested but it looks like kitchen-ec2 is attempting to find the :ssh_key directory inside my current working directory:

➜ pwd
/Users/obrenrose/myapp

➜ kitchen create -l debug
-----> Starting Kitchen (v1.4.0)
-----> Creating <default-amazon>...
       Creating <>...
If you are not using an account that qualifies under the AWS
free-tier, you may be charged to run these suites. The charge
should be minimal, but neither Test Kitchen nor its maintainers
are responsible for your incurred costs.

D      Creating EC2 Instance..
       Instance <i-6f702999> requested.
       EC2 instance <i-6f702999> created.
       Waited 0/300s for instance <i-6f702999> to become ready.
       Waited 5/300s for instance <i-6f702999> to become ready.
       Waited 10/300s for instance <i-6f702999> to become ready.
       Waited 15/300s for instance <i-6f702999> to become ready.
       Waited 20/300s for instance <i-6f702999> to become ready.
       Waited 25/300s for instance <i-6f702999> to become ready.
       EC2 instance <i-6f702999> ready.
D      [SSH] opening connection to [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>22, :compression=>"zlib", :compression_level=>6, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/Users/obrenrose/myapp/$HOME/.ssh/key1.pem"]}>
D      [SSH] connection failed (#<Errno::ECONNREFUSED: Connection refused - connect(2) for "ec2-52-11-166-63.us-west-2.compute.amazonaws.com" port 22>)
       Waiting for SSH service on ec2-52-11-166-63.us-west-2.compute.amazonaws.com:22, retrying in 3 seconds

jcderose · 2015-05-28T21:49:55Z

Sorry, one follow-up note: it also looks like my $HOME variable isn't being translated into my actual $HOME path.

tyler-ball · 2015-05-29T17:44:19Z

@jcderose If you put your full path does it connect successfully? If so I'll file an issue to default the transport ssh_key to the expanded path from ENV['AWS_SSH_KEY'] (if it is present)

jcderose · 2015-05-29T17:47:30Z

@tyler-ball the full (explicit) path didn't work either.

➜ kitchen create -l debug
-----> Starting Kitchen (v1.4.0)
-----> Creating <default-amazon>...
       Creating <>...
If you are not using an account that qualifies under the AWS
free-tier, you may be charged to run these suites. The charge
should be minimal, but neither Test Kitchen nor its maintainers
are responsible for your incurred costs.

D      Creating EC2 Instance..
       Instance <i-71277c87> requested.
       EC2 instance <i-71277c87> created.
       Waited 0/300s for instance <i-71277c87> to become ready.
       Waited 5/300s for instance <i-71277c87> to become ready.
       Waited 10/300s for instance <i-71277c87> to become ready.
       Waited 15/300s for instance <i-71277c87> to become ready.
       Waited 20/300s for instance <i-71277c87> to become ready.
       Waited 25/300s for instance <i-71277c87> to become ready.
       EC2 instance <i-71277c87> ready.
D      [SSH] opening connection to [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>22, :compression=>"zlib", :compression_level=>6, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/Users/obrenrose/.ssh/key1.pem"]}>
D      [SSH] connection failed (#<Errno::ECONNREFUSED: Connection refused - connect(2) for "ec2-52-11-131-1.us-west-2.compute.amazonaws.com" port 22>)
       Waiting for SSH service on ec2-52-11-131-1.us-west-2.compute.amazonaws.com:22, retrying in 3 seconds

tyler-ball · 2015-05-29T17:55:21Z

What is the full ssh command you are running to connect manually? ssh -i /Users/obrenrose/.ssh/key1.pem ec2-user@...?

tyler-ball · 2015-05-29T17:56:04Z

Hmmmm - I don't see the username in the debug info... @fnichol is it supposed to be there?

jcderose · 2015-05-29T18:00:20Z

@tyler-ball yes the full ssh command I'm using to manually connect is:

ssh -i "/Users/obrenrose/.ssh/key1.pem" [email protected]

jcderose · 2015-06-01T16:29:29Z

@tyler-ball, I updated to kitchen-ec2 0.9.3 and am still experiencing the same issue.

➜  opsworks-refactor git:(master) ✗ kitchen create -l debug
-----> Starting Kitchen (v1.4.0)
-----> Creating <default-amazon>...
       Creating <>...
If you are not using an account that qualifies under the AWS
free-tier, you may be charged to run these suites. The charge
should be minimal, but neither Test Kitchen nor its maintainers
are responsible for your incurred costs.

D      Creating EC2 Instance..
       Instance <i-fb93b50d> requested.
       EC2 instance <i-fb93b50d> created.
       Waited 0/300s for instance <i-fb93b50d> to become ready.
       Waited 5/300s for instance <i-fb93b50d> to become ready.
       Waited 10/300s for instance <i-fb93b50d> to become ready.
       Waited 15/300s for instance <i-fb93b50d> to become ready.
       Waited 20/300s for instance <i-fb93b50d> to become ready.
       Waited 25/300s for instance <i-fb93b50d> to become ready.
       EC2 instance <i-fb93b50d> ready.
D      [SSH] opening connection to [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>22, :compression=>"zlib", :compression_level=>6, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/Users/obrenrose/.ssh/key1.pem"]}>
D      [SSH] connection failed (#<Errno::ECONNREFUSED: Connection refused - connect(2) for "ec2-52-25-25-159.us-west-2.compute.amazonaws.com" port 22>)
       Waiting for SSH service on ec2-52-25-25-159.us-west-2.compute.amazonaws.com:22, retrying in 3 seconds

➜  ~  ssh -i "/Users/obrenrose/.ssh/key1.pem" [email protected]

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2015.03-release-notes/
18 package(s) needed for security, out of 46 available
Run "sudo yum update" to apply all updates.
[ec2-user ~]$

gmiranda23 · 2015-06-01T16:54:58Z

@jcderose It looks like this may be a duplicate of #137. Can you follow the resolution (upgrade to latest branch via instructions in #129) and let us know if that resolves your issue?

jcderose · 2015-06-01T17:03:55Z

@gmiranda23, that did not resolve the issue. I updated my Gemfile to:

source 'https://rubygems.org'

gem "test-kitchen", git: "https://github.com/test-kitchen/test-kitchen"
gem "kitchen-ec2", "~> 0.9.2"
gem "berkshelf"

Then ran bundle install and bundle exec kitchen create.

➜ bundle exec kitchen create -l debug
-----> Starting Kitchen (v1.4.1.dev)
D      Berksfile found at /Users/obrenrose/Consensus/ghe.consensuscorpdev.com/jderose/opsworks-refactor/Berksfile, loading Berkshelf
D      Berkshelf 3.2.4 library loaded
-----> Creating <default-amazon>...
       Creating <>...
If you are not using an account that qualifies under the AWS
free-tier, you may be charged to run these suites. The charge
should be minimal, but neither Test Kitchen nor its maintainers
are responsible for your incurred costs.

D      Creating EC2 Instance..
       Instance <i-0e6041f8> requested.
       EC2 instance <i-0e6041f8> created.
       Waited 0/300s for instance <i-0e6041f8> to become ready.
       Waited 5/300s for instance <i-0e6041f8> to become ready.
       Waited 10/300s for instance <i-0e6041f8> to become ready.
       Waited 15/300s for instance <i-0e6041f8> to become ready.
       Waited 20/300s for instance <i-0e6041f8> to become ready.
       EC2 instance <i-0e6041f8> ready.
D      [SSH] opening connection to [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>22, :compression=>"zlib", :compression_level=>6, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/Users/obrenrose/.ssh/key1.pem"], :auth_methods=>["publickey"]}>
D      [SSH] connection failed (#<Timeout::Error: execution expired>)
       Waiting for SSH service on ec2-52-26-62-202.us-west-2.compute.amazonaws.com:22, retrying in 3 seconds
D      [SSH] opening connection to [email protected]<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>22, :compression=>"zlib", :compression_level=>6, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/Users/obrenrose/.ssh/key1.pem"], :auth_methods=>["publickey"], :user=>"ec2-user"}>
D      [SSH] connection failed (#<Errno::ECONNREFUSED: Connection refused - connect(2) for "ec2-52-26-62-202.us-west-2.compute.amazonaws.com" port 22>)
       Waiting for SSH service on ec2-52-26-62-202.us-west-2.compute.amazonaws.com:22, retrying in 3 seconds

tyler-ball · 2015-06-02T19:34:15Z

@jcderose I'm guessing you're limiting your output for brevity (which I appreciate) but just to make sure - are you giving it up to a minute to keep retrying and make sure the connection is still refused? Put another way, does TK keep trying to connect while you can open a different terminal and manually connect?

I'm asking because the debug output from your latest comment looks like it has all the correct connection information - the host is "ec2-52-26-62-202.us-west-2.compute.amazonaws.com", the port is 22, it has the correct key listed and the user is set.

I think the next debug step I can think of to try is to manually create the SSH connection from within ruby. Can you run bundle exec irb and then try the following:

require "net/ssh"
options = {:user_known_hosts_file=>"/dev/null", :keys=>["/Users/obrenrose/.ssh/key1.pem"], :auth_methods=>["publickey"], :user=>"ec2-user"}
c = Net::SSH.start("ec2-52-26-62-202.us-west-2.compute.amazonaws.com","ec2-user", options)
c.exec!("pwd")

jcderose · 2015-06-03T17:00:41Z

@tyler-ball it looks like I just needed to be patient. I re-ran the kitchen create command this morning and it connected successfully after roughly 30 - 60 seconds of waiting/retrying.

tyler-ball · 2015-06-03T18:36:15Z

@jcderose Yay! I'm glad we finally figured it out - thanks for your patience!

anitad · 2015-12-21T22:02:58Z

@tyler-ball Hi , I need to make changes to my kitchen.yml in order to connect to EC2 instance via bastion host (proxy). Below is the script to ssh into ec2 instance using proxy host -

#!/bin/bash
BASTION_HOST=52.11.158.2
KEY_PAIR=/Users/hgerritsen/pre-prod-west2-key.pem
ssh -i $KEY_PAIR -o "proxycommand ssh -W %h:%p -i $KEY_PAIR ec2-user@$BASTION_HOST" -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ec2-user@$1

is there a way to call the above script or insert the ssh command options(-o) mentioned above in the kitchen.yml file under the transport section?

Existing kitchen.yml -

platforms:
name: amazon
driver:
image_id: ami-1181b921
transport:
username: ec2-user
ssh_key: ~/pre-prod-west2-key.pem

tyler-ball added Bug Feedback Requested Triaged labels May 28, 2015

tyler-ball added Accepted and removed Triaged labels May 29, 2015

jcderose closed this as completed Jun 3, 2015

bookshelfdave removed the Accepted label Jun 3, 2015

jcderose mentioned this issue Jul 7, 2015

SSH Connection Expiring Upon Instance Creation #173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TK Can't Connect to EC2 Instance via SSH #135

TK Can't Connect to EC2 Instance via SSH #135

jcderose commented May 28, 2015

tyler-ball commented May 28, 2015

jcderose commented May 28, 2015

tyler-ball commented May 28, 2015

tyler-ball commented May 28, 2015

jcderose commented May 28, 2015

jcderose commented May 28, 2015

tyler-ball commented May 29, 2015

jcderose commented May 29, 2015

tyler-ball commented May 29, 2015

tyler-ball commented May 29, 2015

jcderose commented May 29, 2015

jcderose commented Jun 1, 2015

gmiranda23 commented Jun 1, 2015

jcderose commented Jun 1, 2015

tyler-ball commented Jun 2, 2015

jcderose commented Jun 3, 2015

tyler-ball commented Jun 3, 2015

anitad commented Dec 21, 2015

TK Can't Connect to EC2 Instance via SSH #135

TK Can't Connect to EC2 Instance via SSH #135

Comments

jcderose commented May 28, 2015

tyler-ball commented May 28, 2015

jcderose commented May 28, 2015

tyler-ball commented May 28, 2015

tyler-ball commented May 28, 2015

jcderose commented May 28, 2015

jcderose commented May 28, 2015

tyler-ball commented May 29, 2015

jcderose commented May 29, 2015

tyler-ball commented May 29, 2015

tyler-ball commented May 29, 2015

jcderose commented May 29, 2015

jcderose commented Jun 1, 2015

gmiranda23 commented Jun 1, 2015

jcderose commented Jun 1, 2015

tyler-ball commented Jun 2, 2015

jcderose commented Jun 3, 2015

tyler-ball commented Jun 3, 2015

anitad commented Dec 21, 2015

Existing kitchen.yml -