Support EC2 instance connect

[samjo-nyang]

**Issue number: #38 **

Description of changes:

• Install ec2-instance-connect package
• Update sshd_config file
• Make symbolic link of persistent ssh host key to /etc/ssh in the container
• Execute eic_harvest_hostkeys for one time

Testing done:
This patch is currently used in our internal Kubernetes cluster.

Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[zmrow]

Thanks for the PR! 🎉

Since additional setup is required to use EC2 Instance Connect, we'll also want to update the documentation to point at the AWS docs re: security group guidance and IAM permissions. (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html)

[jpculp]

Sorry for the delayed response. After thinking about it a bit more, it might make more sense to have the userdata structure look more like:

{
   "ssh":{
      "authorized-keys":[
         "ssh-rsa EXAMPLEAUTHORIZEDPUBLICKEYHERE my-key-pair"
      ],
      "trusted-user-ca-keys":[
         "ssh-rsa EXAMPLETRUSTEDCAPUBLICKEYHERE [email protected]"
      ],
      "authorized-keys-command": "...",
      "authorized-keys-command-user": "..."
}

You could parse the authorized keys command and user and output them to their own config under /etc/ssh/sshd_config.d/. If those values are for EC2 Instance Connect you could conditionally trigger the symlink and key harvest logic you have in the current PR. We are going to double-check on our end to see if the key harvest logic is required or whether it is safe to omit. Is this something you would be open to implementing? Or would you be more comfortable with us working on it?

[jpculp]

I've verified that eic_harvest_hostkeys is used for establishing trust, so we should not omit it.

[samjo-nyang]

@jpculp Thanks for your review!
I re-implemented the logic using "authorized-keys-command" and "authorized-keys-command-user" userdata. Is this fine?
Also, I force-pushed this branch to squash commits and fix author info.

[jpculp]

Thank you for the quick update @samjo-nyang! I think this is very close.

I'm afraid I led you astray by suggesting the switch to a separate config file. After some testing on my end, it seems that the OpenSSH in the admin container does not yet support the Include directive in the sshd_config file itself. It might be better to just append to sshd_config directly. You could have a variable like SSHD_CONFIG_FILE variable pointing to "${SSHD_CONFIG_DIR}/sshd_config".

[samjo-nyang]

@jpculp Updated! (and also force-pushed)

[willthames]

I currently get

[  321.634455] host-ctr[7917]: Failed to configure ssh authentication with user-data: {"ssh":{"authorized-keys-command":"/opt/aws/bin/eic_run_authorized_keys %u %f","authorized-keys-command-user":"ec2-instance-connect"}}
[  321.648738] host-ctr[7917]: rsa key already exists, will use existing key.
[  321.651485] host-ctr[7917]: ecdsa key already exists, will use existing key.
[  321.654770] host-ctr[7917]: ed25519 key already exists, will use existing key.
[  321.665726] host-ctr[7917]: /etc/ssh/sshd_config: line 22: Bad configuration option: Include
[  321.668964] host-ctr[7917]: /etc/ssh/sshd_config: terminating, 1 bad configuration options
[  321.708568] host-ctr[7917]: time="2021-07-09T02:56:48Z" level=fatal msg="Container admin exited with non-zero status"

I understand the Include problem is mentioned and fixed above, but the userdata problem would remain.

My reading is that this happens because setting authorized-keys-command and authorized-keys-command-user does not increment available_auth_methods

[samjo-nyang]

@willthames I just re-ordered the statements and increase available_auth_methods when use_eic=1. It should work now.

[willthames]

Thanks @samjo-nyang - I'll give it a go!

[willthames]

This works great for me now.

[jpculp]

This looks great to me!

Also, thank you for testing this out @willthames!

[samjo-nyang]

Force-pushed (remove use_eic=1 condition and symlink the host keys for all cases)