Update VMSS to Mariner with FIPS enabled #3741
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
s-fairchild
changed the title
niontive
reviewed
Aug 2, 2024
s-fairchild
force-pushed
the
s-fairchild/ARO-8989-update-bootstrap
branch
6 times, most recently
from
36529ee to
3863fee
Compare
|
azp run ci,e2e |
niontive
reviewed
Aug 12, 2024
s-fairchild
requested review from
jewzaam,
bennerv,
hawkowl,
rogbas,
petrkotas,
jharrington22,
cblecker,
cadenmarchese,
UlrichSchlueter,
SudoBrendan,
yjst2012,
jaitaiwan,
anshulvermapatel,
hlipsig,
tiguelu,
SrinivasAtmakuri,
mociarain and
AldoFusterTurpin
as code owners
s-fairchild
force-pushed
the
s-fairchild/ARO-8989-update-bootstrap
branch
from
cceea5c to
53d5d25
Compare
kimorris27
reviewed
Aug 13, 2024
s-fairchild
added
release-blocker
next-release
tsatam
reviewed
Aug 15, 2024
There was a problem hiding this comment.
Changes mostly LGTM - @kimorris27 covered most of the suggestions I have, and I'd like to see a deployment of this change in a non-dev environment.
s-fairchild
force-pushed
the
s-fairchild/ARO-8989-update-bootstrap
branch
from
8e6db08 to
374af12
Compare
@cadenmarchese We could manually apply updates ourselves. I think that would require creating a pipeline to do so, which would give us more control. That's a topic I think is worth discussion amongst the team. For them to be automatically applied, we have no real insight or control. The OS updates are re-provisioning the instance with an updated OS image. I believe that means our custom script will have to run after this each time. That would mean we'd see two reboots before the update is completed. One for the updated OS re-image, our custom script runs, reboots post completion, then it's back to responding to requests. You can read their documentation here: Automatic Guest VM Patching |
|
/azp run e2e |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run e2e |
|
Azure Pipelines successfully started running 1 pipeline(s). |
s-fairchild
force-pushed
the
s-fairchild/ARO-8989-update-bootstrap
branch
2 times, most recently
from
b7cc458 to
08003b3
Compare
|
/azp run e2e |
|
Azure Pipelines successfully started running 1 pipeline(s). |
s-fairchild
force-pushed
the
s-fairchild/ARO-8989-update-bootstrap
branch
from
08003b3 to
f5debe1
Compare
…ly Configured FIPS Mode System Changes: Remove lvm disk resize, Mariner does not use lvm, the disk is automatically grown to the full size specified. Remove semanage, Mariner Linux does not have selinux configured. Remove gateway log rotation config Log rotation for the podman level driver log was not the correct approach. The podman log driver is now journald, so all logs will be shipped to journald rather than a ctr.log file. fips mode is manually configured following the example code at https://eng.ms/docs/products/azure-linux/features/security/fips SKU cbl-mariner-2-gen2-fips does not support Automatic OS Updates, therefore we are switching to cbl-mariner-2-gen2, manually configuring fips mode, to allow for Automatic OS Updates. Script Changes: Restructure VMSS bootstrap bash scripts for increased reliability, and easier debugging Move all shared code into a commonly shared file to be sourced by all bootstrapping scripts. This allows for code reuse, minimal duplication. Fix mdm mdsd certificate download script During mdm and mdsd setup, I've added wait steps for the download scripts to complete getting certificates. Without this, the download scripts run in a subshell and fixing up the certificates fails. Add firewalld configuration, required for podman networking Add podman aro network creation to isolate RP containers from possible interaction on the default podman network. Package Changes: Install Azure Security Monitor via VMSS Extension Remove RHUI and Microsoft repo configuration, add Mariner Extended repo config Increase rpm retry time to 30 minutes total, every 30 seconds.
This is to reduce the amount of type conversions needed.
s-fairchild
force-pushed
the
s-fairchild/ARO-8989-update-bootstrap
branch
from
f5debe1 to
169f42c
Compare
|
/azp run e2e |
|
Azure Pipelines successfully started running 1 pipeline(s). |
tsatam
approved these changes
Aug 22, 2024
cadenmarchese
approved these changes
Aug 22, 2024
kimorris27
approved these changes
Aug 22, 2024
Merged
edisonLcardenas
pushed a commit
that referenced
this pull request
Sep 16, 2024
* Update RP and Gateway vmss OS image to cbl-mariner-2-gen2 with Manually Configured FIPS Mode System Changes: Remove lvm disk resize, Mariner does not use lvm, the disk is automatically grown to the full size specified. Remove semanage, Mariner Linux does not have selinux configured. Remove gateway log rotation config Log rotation for the podman level driver log was not the correct approach. The podman log driver is now journald, so all logs will be shipped to journald rather than a ctr.log file. fips mode is manually configured following the example code at https://eng.ms/docs/products/azure-linux/features/security/fips SKU cbl-mariner-2-gen2-fips does not support Automatic OS Updates, therefore we are switching to cbl-mariner-2-gen2, manually configuring fips mode, to allow for Automatic OS Updates. Script Changes: Restructure VMSS bootstrap bash scripts for increased reliability, and easier debugging Move all shared code into a commonly shared file to be sourced by all bootstrapping scripts. This allows for code reuse, minimal duplication. Fix mdm mdsd certificate download script During mdm and mdsd setup, I've added wait steps for the download scripts to complete getting certificates. Without this, the download scripts run in a subshell and fixing up the certificates fails. Add firewalld configuration, required for podman networking Add podman aro network creation to isolate RP containers from possible interaction on the default podman network. Package Changes: Install Azure Security Monitor via VMSS Extension Remove RHUI and Microsoft repo configuration, add Mariner Extended repo config Increase rpm retry time to 30 minutes total, every 30 seconds. * Embed scripts as strings rather than []byte This is to reduce the amount of type conversions needed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue this PR addresses:
Covers improving bootstrap scripts in : ARO-6773
Covers move to Mariner OS for FIPS in: ARO-8989
Fixes
What this PR does / why we need it:
This moves our RP and Gateway VMSS instances to Mariner OS, which is compatible with FIPS.
This is also needed for a resilient to failure RP deployment.
Our current implementation has shown some faults over time with RP deployments, and need improvements in general.
Move all shared code into a commonly shared file to be sourced by all
bootstrapping scripts. This allows for code reuse, minimal duplication.
Change VMSS OS to Mariner with FIPS enabled by default.
Firewalls are configured at the Azure vnet level. I did not configure
firewalld, because in my testing Mariner OS does not supportfirewalldout of the box.Test plan for issue:
Local Full Service Dev RP Testing
Tested during development by deploying a full service development RP environment.
Further testing will be done by deploying to INT. I will update with the successful INT deployment once completed.
Deployment to Canary Sector
Successfully deployed to Canary sector in pipeline run
Example Log Output
Abort
Log
Is there any documentation that needs to be updated for this PR?
No
How do you know this will function as expected in production?
Per testing mentioned in the test plan.