Skip to content

Commit

Permalink
Remove data store version, use OS version instead
Browse files Browse the repository at this point in the history
We found confusion between data store versions and OS versions, so this
simplifies the system to only use OS version.  If there's been no change in the
data store, which is now determined by there being no migrations, then the new
data store will simply be a link to the last version that did have changes.

These are the changes at a high level:
* Remove the data_store_version library and its type, instead using semver::Version
  * Cascade this change through migrator, storewolf, update_metadata, and updog
* Remove OS version -> data store version mapping in updog
* Remove the data-store-version file, instead using OS version from bottlerocket-release
* In migrator, just symlink to previous data store if there are no migrations
* In migrator, flip symlinks for current, major, minor, and patch, since any could change
* In storewolf, also create the new patch-level symlink

One change deserves more description.  It's an extension of the problem from #644
that only showed up while testing this change, and was made easier to fix by
these changes, so it was done together.

Background: In migrator, we were giving each migration the same source and
target data store.  Migrations make whatever changes they want to a
MigrationData, each key of which is then written to the data store.

Problem: Let's say the system has settings A and B.  One migration adds setting
C.  This is written to the target data store.  Another migration tries to
remove setting B.  It does so in memory, but B has already been written to the
target by the first migration, so it's to no avail.

Solution: Chain the input and output of migrations in order.  The first
migration receives the source data store and adds setting C, writing out to a
*new* data store path it was given.  The second migration is given that new
data store as a source; it removes setting B and writes out to another new data
store path it was given.  At the end of the chain, the final data store is kept
and the intermediate ones are removed.
  • Loading branch information
tjkirch committed Feb 21, 2020
1 parent cac0bc9 commit f8a95cf
Show file tree
Hide file tree
Showing 43 changed files with 500 additions and 1,034 deletions.
1 change: 0 additions & 1 deletion Release.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
version = "0.2.2"
datastore_version = "0.3"

[migrations]
# Happy path - skip over 0.1, which had an issue with compressed migrations. Normal new migrations go here.
Expand Down
1 change: 0 additions & 1 deletion packages/workspaces/data-store-version

This file was deleted.

2 changes: 1 addition & 1 deletion packages/workspaces/migrator.service
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ Description=Bottlerocket data store migrator

[Service]
Type=oneshot
ExecStart=/usr/bin/migrator --datastore-path /var/lib/bottlerocket/datastore/current --migration-directories /var/lib/bottlerocket/datastore/migrations --migrate-to-version-from-file /usr/share/bottlerocket/data-store-version
ExecStart=/usr/bin/migrator --datastore-path /var/lib/bottlerocket/datastore/current --migration-directories /var/lib/bottlerocket/datastore/migrations --migrate-to-version-from-os-release
RemainAfterExit=true
StandardError=journal+console

Expand Down
3 changes: 0 additions & 3 deletions packages/workspaces/workspaces.spec
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ Summary: Bottlerocket's first-party code
License: Apache-2.0 OR MIT

# sources < 100: misc
Source1: data-store-version
Source2: api-sysusers.conf
# Taken from https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt
Source3: eni-max-pods
Expand Down Expand Up @@ -217,7 +216,6 @@ done
install -d %{buildroot}%{migration_dir}

install -d %{buildroot}%{_cross_datadir}/bottlerocket
install -p -m 0644 %{S:1} %{buildroot}%{_cross_datadir}/bottlerocket

install -d %{buildroot}%{_cross_sysusersdir}
install -p -m 0644 %{S:2} %{buildroot}%{_cross_sysusersdir}/api.conf
Expand Down Expand Up @@ -251,7 +249,6 @@ install -p -m 0644 %{S:201} %{buildroot}%{_cross_tmpfilesdir}/host-containers.co
%{_cross_bindir}/apiserver
%{_cross_unitdir}/apiserver.service
%{_cross_unitdir}/migrator.service
%{_cross_datadir}/bottlerocket/data-store-version
%{_cross_sysusersdir}/api.conf

%files -n %{_cross_os}apiclient
Expand Down
22 changes: 5 additions & 17 deletions workspaces/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion workspaces/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ members = [
"api/apiserver",
"api/apiclient",
"api/bork",
"api/data_store_version",
"api/moondog",
"api/netdog",
"api/sundog",
Expand Down
2 changes: 1 addition & 1 deletion workspaces/api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Further docs:
* [Migration system](migration/)

The migrator ensures the data store is up to date by running any applicable data store migrations.
The existing data store format version is found by looking at the symlink naming in `/var/lib/bottlerocket/datastore`, and the incoming data store format version is found by looking at `/usr/share/bottlerocket/data-store-version` in the booting image.
The existing data store format version is found by looking at the symlink naming in `/var/lib/bottlerocket/datastore`, and the incoming data store format version is found by looking at `/etc/os-release` in the booting image.

On first boot, [storewolf](#storewolf) hasn’t run yet, so there’s no data store, so the migrator has nothing to do.

Expand Down
2 changes: 1 addition & 1 deletion workspaces/api/apiserver/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,4 +79,4 @@ See `../../apiclient/README.md` for client examples.

## Colophon

This text was generated using [cargo-readme](https://crates.io/crates/cargo-readme), and includes the rustdoc from `src/lib.rs`.
This text was generated using [cargo-readme](https://crates.io/crates/cargo-readme), and includes the rustdoc from `src/lib.rs`.
19 changes: 0 additions & 19 deletions workspaces/api/data_store_version/Cargo.toml

This file was deleted.

14 changes: 0 additions & 14 deletions workspaces/api/data_store_version/README.md

This file was deleted.

9 changes: 0 additions & 9 deletions workspaces/api/data_store_version/README.tpl

This file was deleted.

32 changes: 0 additions & 32 deletions workspaces/api/data_store_version/build.rs

This file was deleted.

Loading

0 comments on commit f8a95cf

Please sign in to comment.