inventory: explain

inventory/README.md

@@ -1,17 +1,42 @@
-### .. magic inside
 
-this inventory consists of three inventories.
-
-
-# base_inventory
-the first one prints some ansible-compatible hostlist based on all the folders inside host_vars.
-
-
-# keyed_groups_stage_1.config
-this is a configuration for the "constructed" inventory plugin. it dynamically constructs group memberships based on the host_vars. as the first inventory doesnt output any host & group vars, the constructed inventory has to fetchit on its own. This pretty new feature is available since ansible 2.11 . Its controlled by the "use_vars_plugins" key. Unfortunately its searching for the host&group vars in the folder of the first inventory plugin. As its located here, we need to symlink 'em here in order to allow it accomplishing its job.
-
-.. pretty hacky, but less hacky than before :) ..lets hope ansible  will continue improving 
-
-
-# keyed_groups_stage_2.config
-this is yet another configuration for the "constructed" inventory plugin. in contrast to the first stage its purpose is to dynamically construct even more group memberships based on inherited vars from the previous stage.
+# Inventory construction
+
+Inventory data is used to generate host-specific OpenWrt config files,
+which are then combined into an image file that can be flashed to a device.
+
+Data model concepts are most easily explained by example,
+so please also check out the location files in the `locations/` directory.
+We have one file per network location, and one or more hosts in each location.
+Ansible had groups and hosts, one file per host, which wasn't practical for us.
+
+There are five (now four) stages that construct our inventory data:
+
+1. The base inventory script
+  - Handled by Ansible's "script" inventory plugin.
+  - Translates our location-centered data model to Ansible's data model.
+  - It collects the hostvars for all hosts, which consist of the host object
+    in a location file, merged with the surrounding location object.
+  - See documentation in `base_inventory` for more details.
+2. `host_vars/`
+  - In earlier times we used individual files in `host_vars/`.
+3. Keyed groups, pt. 1
+  - Handled by Ansible's "constructed" inventory plugin.
+  - Injects additional data based on certain property values.
+  - There are two parts so that new data from part 1 can set properties
+    that result in more new data in part 2 (e.g. `model` and OpenWrt version).
+  - This stage handles `target`, `model`, and `role`.
+4. Keyed groups, pt. 2
+  - Same, but handles `target` and `openwrt_version`.
+5. Merge vars
+  - Handled by Ansible's "merge_vars" action plugin.
+  - All hostvars construction so far was only able to overwrite properties,
+    but in some cases we need to merge with the existing property.
+  - For specific properties, a "merge var" can be set:
+    `packages: ["some-pkg"]` and `xxx__packages__to_merge: ["another-pkg"]`
+    where `xxx` is an arbitrary name to allow for multiple merge vars.
+    For this arbitrary name we usually pick something that describes the
+    scope we're currently in, e.g. `location__packages__to_merge`.
+  - These merge vars are merged together into one
+    before any templates or tasks make use of hostvars.
+  - Handles `ssh_keys`, `packages`, `sysctl`, `rclocal`,
+    `disabled_services`, `wireless_profiles`, `channel_assignments_*`.

inventory/base_inventory

@@ -1,18 +1,68 @@
 #!/usr/bin/env bash
 
+#
+# This script's output is a JSON object containing an array of all host names,
+# and an array with the initial hostvars for each host.
+# More information on inventory construction can be found at
+# https://docs.ansible.com/ansible/latest/dev_guide/developing_inventory.html
+#
+# The queries and conversions are done with the help of the `jq` and `yq` tools.
+#
+# We first grab the JSON representation of every location YAML definition,
+# then read `.hosts[].hostname` for each location to create the list of all hosts.
+#
+# Usually Ansible then calls this script with `--host <name>` for every host.
+# That does get very slow with hundreds of hosts,
+# which is why Ansible allows for constructing all hostvars objects in advance.
+#
+# To construct the hostvars object for a host,
+# we take as a base the full location object (without `.hosts`),
+# then merge the object from `.hosts[]` with a matching `hostname` value.
+# This way, host values overwrite location values.
+# (We actually merge host<-location<-host to preserve JSON ordering.)
+#
+# Example location file:
+#
+#   ---
+#   location: pktpls
+#   hosts:
+#     - hostname: pktpls-core
+#       role: corerouter
+#       string: host-var-has-precedence
+#       object: { two: 456 }
+#       array: [ bar ]
+#   string: will-be-overridden
+#   object: { one: 123 }
+#   array: [ foo ]
+#
+# Resulting hostvars object, before keyed groups being applied:
+#
+#   {
+#     "location": "pktpls",
+#     "hostname": "pktpls-core",
+#     "role": "corerouter",
+#     "string": "host-var-has-precedence",
+#     "object": {
+#       "two": 456
+#     },
+#     "array": [
+#       "bar"
+#     ]
+#   }
+#
+
 set -e
 # set -x
 
 case "$1" in
 --host)
-    # No op - won't be called by Ansible anymore because --list contains all data.
-    # See https://docs.ansible.com/ansible/latest/dev_guide/developing_inventory.html#tuning-the-external-inventory-script
+    # No op, only ever called with --list
     echo "{}"
     exit 0
     ;;
 --list)
     # Print all location files as consecutive JSON objects.
-    # Later jq -s/--slurp will read these as one top-level array of objects.
+    # Further down, jq -s/--slurp reads these as one top-level array of objects.
     locjson="$(yq '.' locations/*.yml)"
     cat <<EOF
 {