Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AWS and Azure source block to Packer config #53

Open
wants to merge 23 commits into
base: main
Choose a base branch
from

Conversation

EmmanuelNwa247
Copy link
Collaborator

@EmmanuelNwa247 EmmanuelNwa247 commented Mar 1, 2025

This PR includes and resolves #26 and #27

AWS

  • Refactoring post-install script to use sudo as a configurable variable for AWS build and

  • Updated the ubuntu.pkr.hcl config file to reflect changes made to main

  • Removed aws-post-install script to maintain single source of Post-install script

  • Updated packMachines.yml with AWS secrets to build AMI and deploy to Skylight's AWS Account via Github Actions

Testing

Testing Instructions:

  • Navigate to Actions
  • Click on the [pack virtual machine images](https://github.com/CDCgov/dibbs-vm/actions/workflows/packMachines.yml)
  • Run Workflow, then use Emmanuel/Cloud-source as Branch to test and build the AMI
  • This should successfully build the AMI
  • Log in to Skylight's AWS account
  • Navigate to the Ami section on the portal
  • You should see an AMI with the naming convention Ubuntu-2404-dibbs-ecr-viewer-main

AZURE

Summary

  • Added an Azure source block in ubuntu.pkr.hcl to maintain the AMD64 architecture with the Ubuntu LTS AMI.

  • Modified variables.pkr.hcl to include Azure credentials for authentication.

Changes

  • Updated ubuntu.pkr.hcl to support Azure Image build.

  • Added necessary variables for Azure deployment in variables.pkr.hcl.

  • Updated packMachines.yml with Azure secrets to build Image and deploy to Skylight Azure's Account via Github Actions

How to Test

  • Navigate to Actions
  • Click on the [pack virtual machine images](https://github.com/CDCgov/dibbs-vm/actions/workflows/packMachines.yml)
  • Run Workflow, then use Emmanuel/Cloud-source as Branch to test and build the Azure image
  • This should successfully build the image
  • Log in to Skylight's Azure account
  • Locate the resource group skylight-dibbs-vm1
  • You should see an Azure image with the naming convention Ubuntu-2404-dibbs-ecr-viewer-main

Copy link
Collaborator

@rin-skylight rin-skylight left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great start! Thank you for tackling the AWS side of this. I have some concerns, but a little cleanup is likely all that is necessary.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script does fundamentally the same thing as our existing post-install.sh. It seems that the only difference is the addition of the sudo command. Can you elaborate on this? I'd like to keep us from having to maintain multiple versions of essentially the same file, if possible.

Copy link
Collaborator Author

@EmmanuelNwa247 EmmanuelNwa247 Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for the review, @rin-skylight Our default post-install.sh fails when building the AWS Ubuntu VM because it does not have root permissions to perform package installations and file modifications. when I run with our default I usually run into this error message, hence the addition of sudo ,

==> iso.amazon-ebs.aws-ami: Provisioning with shell script: scripts/aws-post-install.sh
    iso.amazon-ebs.aws-ami: post-install
    iso.amazon-ebs.aws-ami: Reading package lists...
==> iso.amazon-ebs.aws-ami: E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)
==> iso.amazon-ebs.aws-ami: E: Unable to lock directory /var/lib/apt/lists/
==> iso.amazon-ebs.aws-ami: curl: (23) Failure writing output to destination
==> iso.amazon-ebs.aws-ami: chmod: cannot access '/etc/apt/keyrings/docker.asc': No such file or directory
==> iso.amazon-ebs.aws-ami: tee: /etc/apt/sources.list.d/docker.list: Permission denied
==> iso.amazon-ebs.aws-ami: E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)
    iso.amazon-ebs.aws-ami: Reading package lists...
==> iso.amazon-ebs.aws-ami: E: Unable to lock directory /var/lib/apt/lists/
==> iso.amazon-ebs.aws-ami: W: Problem unlinking the file /var/cache/apt/pkgcache.bin - RemoveCaches (13: Permission denied)
==> iso.amazon-ebs.aws-ami: W: Problem unlinking the file /var/cache/apt/srcpkgcache.bin - RemoveCaches (13: Permission denied)
==> iso.amazon-ebs.aws-ami: E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
==> iso.amazon-ebs.aws-ami: E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes total sense. Here's a question...is there a way for us to still use the singular script, so we aren't maintaining two separate versions, but be able to dynamically choose when to use sudo? Something like a variable called use_sudo that we can toggle on or off per build, so we don't maintain multiple copies. Just trying to reduce our workload down the road, so we aren't maintaining a separate post-install for every flavor of VM that we build.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes total sense. Here's a question...is there a way for us to still use the singular script, so we aren't maintaining two separate versions, but be able to dynamically choose when to use sudo? Something like a variable called use_sudo that we can toggle on or off per build, so we don't maintain multiple copies. Just trying to reduce our workload down the road, so we aren't maintaining a separate post-install for every flavor of VM that we build.

instance_type = var.aws_instance_type
region = var.aws_region

source_ami_filter {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get an image from the Ubuntu site like we do for the Qemu source? If we did that, AFAIK we wouldn't need to modify the scripts to account for the differences caused by using a different image.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be very interested in hearing the pros and cons of this approach. I'm still seeing AWS-specific lines in our code, which I would love to see disappear if building off the custom image works for us.

That said, GCP makes custom images a pain, and it's next on the list for Colorado, so we would either have to use a similar separation like what we've done here for AWS, or simply say "forget this" and create an OVF/OVA that GCP admins could manually import. (Since GCP provides that functionality)

@EmmanuelNwa247 EmmanuelNwa247 force-pushed the Emmanuel/Cloud-source branch from 38a75f9 to 97232fb Compare March 13, 2025 21:09
@EmmanuelNwa247 EmmanuelNwa247 requested a review from alismx March 13, 2025 23:08
@EmmanuelNwa247 EmmanuelNwa247 changed the title Add AWS source block to Packer config Add AWS and Azure source block to Packer config Mar 21, 2025
@EmmanuelNwa247 EmmanuelNwa247 marked this pull request as draft March 27, 2025 13:35
@EmmanuelNwa247 EmmanuelNwa247 force-pushed the Emmanuel/Cloud-source branch from 3c3856e to ec9cd4c Compare March 27, 2025 17:02
@EmmanuelNwa247 EmmanuelNwa247 marked this pull request as ready for review March 27, 2025 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Build out Amazon Machine Image provisioner
4 participants