Implement 3rd party vendors' configuration files support (oamg#5)

* Add the initial implementation of third-party integration

* Add vendors' PES events parsing

* Add vendors dir existence checks

* Add a testing deployment script

* Remove references to target system type selector

* Correct the copying in deployment script

* Vendor list handling in PES actor

* Trace-level logging for vendor repomaps

* Repomap additional logging

* Deployment script touch-up: base on cloudlinux branch

* Make PES events scanner accept empty event lists

* Add a test case for empty PES file

* Add missing file to the deployment script

* Add information about the vendors mechanism to README

* Create an additional deployment script for non-CL cases

* Create the vendors.d folder with the generic deployment script

* Update deployment scripts

* Clarifications to the README file related to repositories

* Fix the deployment script not having the correct path

* Implement vendor package signature loading

* Additional logging for the vendor activation

* Accept multiple active vendor lists if present

* Fix the ActiveVendor loading in vendor signature scanner

* Add the new channel-swtich file to deployment script

* Produce custom repofile messages for vendor repofiles

* Remove unneeded FactsPhase sets

* Simplify the package signature check

* Reorder the FactsPhase to make sure signed package scanner receives data

* Target 3rd_parties' version of signed package scanner

* Fix vendor signature comparsion

* Allow for multiple RepositoriesMap messages to be processed

* Changed actor execution order to provide vendor repo data

* Fix the dnfplugin str encode errors

* Revert "Fix the dnfplugin str encode errors"

This reverts commit ddcbbf5.

* Fix the dnfplugin str encode errors (without introducing platform-dependent functions)

* Some clarifications in README about vendors.d

* Move a "successful read" message to debug category

* Switch encode handling to version independent of six version

* Add docstrings to new actors

* Remove script files from the repository

Co-authored-by: Aleksey Petryankin <[email protected]>

README.md

@@ -29,3 +29,144 @@ We’ll gladly answer your questions and lead you to through any troubles with t
 actor development.
 
 You can reach us at IRC: `#leapp` on freenode.
+
+
+## Third-party integration
+
+If you would like to add your **signed** 3rd party packages into the upgrade process, you can use the third-party integration mechanism.
+
+There are four components for adding your information to the elevation process:
+- <vendor_name>.csv: repository mapping file
+- <vendor_name>.repo: package repository information
+- <vendor_name>.sigs: list of package signatures of vendor repositories
+- <vendor_name>.json: package migration event list
+
+All these files **must** have the same <vendor_name> part.
+
+### Repository mapping file
+
+This CSV file provides information on mappings between source system repositories (repositories present on the system being upgraded) and target system repositories (package repositories to be used during the upgrade).
+
+The first line of the file, per CSV format, should contain the headers. Standard headers for vendor.csv files look like this:
+
+```CSV
+Source system repoid,Target system repoid in custom repo file,Target system repo name in PES,Source system minor versions,Target system minor versions,architecture,type (rpm/srpm/debuginfo),source product type (ga/beta,htb),target product type (ga/beta/htb)
+```
+
+Following lines should contain the repository map entries. As an example:
+
+```CSV
+Source system repoid,Target system repoid in custom repo file,Target system repo name in PES,Source system minor versions,Target system minor versions,architecture,type (rpm/srpm/debuginfo),source product type (ga/beta,htb),target product type (ga/beta/htb)
+
+source-repoid,target-custom-repoid,target-pes-repoid,all,all,x86_64,rpm,ga,ga
+```
+
+**Source system repoid** is the ID of a repository that is expected to be present on the system before the upgrade.
+
+**Target system repoid in custom repo file** is the ID of a repository listed in the associated package repository information (<vendor_name>.repo) file. It is supposed to be used during the upgrade process.
+
+**Target system repo name in PES** is the ID which is used to refer to the target system repository in the package migration event list (<vendor_name>.json).
+
+**Repository types**:
+- rpm: normal RPM packages
+- srpm: source packages
+- debuginfo: packages with debug information
+
+**Product types**:
+- GA: general availability repositories
+- Beta: beta-testing repositories
+- HTB: High Touch Beta repositories
+
+The repository mapping file also defines whether a vendor's packages will be included into the upgrade process at all. If at least one source repository listed in the file is present on the system, the vendor is considered active, and package repositories/PES events are enabled - otherwise, they will not affect the upgrade process.
+
+In the above example, vendor's data (including the .repo and .json files) will affect the upgrade process only if a repository with `source-repoid` ID is present on the system.
+
+
+### Package repository information
+
+This file defines the vendor's package repositories to be used during the upgrade.
+
+The file has the same format normal YUM/DNF package repository files do.
+
+> NOTE: The repositories listed in this file are only used *during* the upgrade. Package repositories on the post-upgrade system should be provided through updated packages or custom repository deployment.
+
+### Package signature list
+
+This file should contain the list of public signature headers that the packages are signed with, one entry per line.
+
+You can find signature headers for your packages by running the following command:
+
+`rpm -qa --queryformat "%{NAME} || %|DSAHEADER?{%{DSAHEADER:pgpsig}}:{%|RSAHEADER?{%{RSAHEADER:pgpsig}}:{(none)}|}|\n" <PACKAGE_NAME>`
+
+rpm will return an entry like the following:
+`package-name || DSA/SHA1, Mon Aug 23 08:17:13 2021, Key ID 8c55a6628608cb71`
+
+The value after "Key ID", in this case, `8c55a6628608cb71`, is what you should put into the signature list file.
+
+### Package migration event list
+
+The Leapp upgrade process uses information from the AlmaLinux PES (Package Evolution System) to keep track of how packages change between the OS versions. This data is located in `leapp-data/files/<target_system>/vendors.d/<vendor_name>.json` in the GitHub repository and in `/etc/leapp/files/vendors.d/<vendor_name>.json` on a system being upgraded.
+
+To add new rules to the list, add a new entry to the `packageinfo` array.
+**Important**: actions from PES json files will be in effect only for those packages that are signed **and** have their signatures in one of the active <vendor_name>.sigs files. Unsigned packages will be updated only if some signed package requires a new version, otherwise they will by left as they are.
+
+Required fields:
+
+- action: what action to perform on the listed package
+  - 0 - present
+  - 1 - removed
+  - 2 - deprecated
+  - 3 - replaced
+  - 4 - split
+  - 5 - merged
+  - 6 - moved to new repository
+  - 7 - renamed
+- arches: what system architectures the listed entry relates to
+- id: entry ID, must be unique
+- in_packageset: set of packages on the old system
+- out_packageset: set of packages to switch to, empty if removed or deprecated
+- initial_release: source OS release
+- release: target OS release
+
+`in_packageset` and `out_packageset` have the following format:
+
+```json
+      "in_packageset": {
+        "package": [
+          {
+            "module_stream": null,
+            "name": "PackageKit",
+            "repository": "base"
+          },
+          {
+            "module_stream": null,
+            "name": "PackageKit-yum",
+            "repository": "base"
+          }
+        ],
+        "set_id": 1592
+      },
+```
+
+For `in_packageset`, `repository` field defines the package repository the package was installed from on the source system.
+For `out_packageset`, `repository` field for packages should be the same as the "Target system repo name in PES" field in the associated vendor repository mapping file.
+Warning: leapp doesn't force packages from out_packageset to be installed from the specific repository; instead, it enables repo from out_packageset and then dnf installs the latest package version from all enabled repos.
+
+To take the above repository map example:
+
+```CSV
+Source system repoid,Target system repoid in custom repo file,Target system repo name in PES,Source system minor versions,Target system minor versions,architecture,type (rpm/srpm/debuginfo),source product type (ga/beta,htb),target product type (ga/beta/htb)
+
+source-repoid,target-custom-repoid,target-pes-repoid,all,all,x86_64,rpm,ga,ga
+```
+
+For this configuration, `in_packageset` entries would have `source-repoid` as the `repository` field, and `out_packageset` would have `target-pes-repoid` in theirs.
+
+Please refer to [PES contribution guide](https://wiki.almalinux.org/elevate/Contribution-guide.html) for additional information on entry fields.
+
+### Providing the data
+
+Once you've prepared the vendor data for migration, you can make a pull request to https://github.com/AlmaLinux/leapp-data/ to make it available publicly.
+Files should be placed in `files/<target-system>/vendors.d/`.
+
+Alternatively, you can deploy the vendor files on a system prior to starting the upgrade. In this case, place the files into the folder `/etc/leapp/files/vendors.d/`.

repos/system_upgrade/common/actors/checkenabledvendorrepos/actor.py

@@ -0,0 +1,55 @@
+from leapp.actors import Actor
+from leapp.libraries.stdlib import api
+from leapp.models import (
+    RepositoriesFacts,
+    VendorRepositoriesMapCollection,
+    ActiveVendorList,
+)
+from leapp.tags import FactsPhaseTag, IPUWorkflowTag
+
+
+class CheckEnabledVendorRepos(Actor):
+    """
+    Create a list of vendors whose repositories are present on the system.
+    Only those vendors' configurations (new repositories, PES actions, etc.)
+    will be included in the upgrade process.
+    """
+
+    name = "check_enabled_vendor_repos"
+    consumes = (RepositoriesFacts, VendorRepositoriesMapCollection)
+    produces = (ActiveVendorList)
+    tags = (IPUWorkflowTag, FactsPhaseTag.Before)
+
+    def process(self):
+        vendor_mapping_data = {}
+        active_vendors = []
+
+        # Make a dict for easy lookup of repoid -> vendor name.
+        for map_coll in api.consume(VendorRepositoriesMapCollection):
+            for map in map_coll.maps:
+                for repo in map.repositories:
+                    # Cut the .csv, keep only the vendor name.
+                    vendor_mapping_data[repo.from_repoid] = map.file[:-4]
+
+        # Is the repo listed in the vendor map as from_repoid present on the system?
+        for repos in api.consume(RepositoriesFacts):
+            for repo_file in repos.repositories:
+                for repo in repo_file.data:
+                    self.log.debug(
+                        "Looking for repository {} in vendor maps".format(repo.repoid)
+                    )
+                    if repo.repoid in vendor_mapping_data:
+                        # If the vendor's repository is present in the system, count the vendor as active.
+                        new_vendor = vendor_mapping_data[repo.repoid]
+                        self.log.debug(
+                            "Repository {} found, enabling vendor {}".format(
+                                repo.repoid, new_vendor
+                            )
+                        )
+                        active_vendors.append(new_vendor)
+
+        if active_vendors:
+            self.log.debug("Active vendor list: {}".format(active_vendors))
+            api.produce(ActiveVendorList(data=active_vendors))
+        else:
+            self.log.info("No active vendors found, vendor list not generated")

repos/system_upgrade/common/actors/peseventsscanner/actor.py

@@ -1,3 +1,6 @@
+import os
+import os.path
+
 from leapp.actors import Actor
 from leapp.libraries.actor.peseventsscanner import pes_events_scanner
 from leapp.models import (
@@ -7,10 +10,14 @@
     RepositoriesBlacklisted,
     RepositoriesSetupTasks,
     RpmTransactionTasks,
+    ActiveVendorList,
 )
 from leapp.reporting import Report
 from leapp.tags import FactsPhaseTag, IPUWorkflowTag
 
+LEAPP_FILES_DIR = "/etc/leapp/files"
+VENDORS_DIR = "/etc/leapp/files/vendors.d"
+
 
 class PesEventsScanner(Actor):
     """
@@ -20,10 +27,27 @@ class PesEventsScanner(Actor):
     message with relevant data will be produced to help DNF Upgrade transaction calculation.
     """
 
-    name = 'pes_events_scanner'
-    consumes = (InstalledRedHatSignedRPM, RepositoriesBlacklisted, RepositoriesMap, RpmTransactionTasks)
+    name = "pes_events_scanner"
+    consumes = (
+        InstalledRedHatSignedRPM,
+        RepositoriesBlacklisted,
+        RepositoriesMap,
+        RpmTransactionTasks,
+        ActiveVendorList,
+    )
     produces = (PESRpmTransactionTasks, RepositoriesSetupTasks, Report)
     tags = (IPUWorkflowTag, FactsPhaseTag)
 
     def process(self):
-        pes_events_scanner('/etc/leapp/files', 'pes-events.json')
+        pes_events_scanner(LEAPP_FILES_DIR, "pes-events.json")
+
+        active_vendors = []
+        for vendor_list in self.consume(ActiveVendorList):
+            active_vendors.extend(vendor_list.data)
+
+        if os.path.isdir(VENDORS_DIR):
+            vendor_pesfiles = list(filter(lambda vfile: ".json" in vfile, os.listdir(VENDORS_DIR)))
+
+            for pesfile in vendor_pesfiles:
+                if pesfile[:-5] in active_vendors:
+                    pes_events_scanner(VENDORS_DIR, pesfile)

repos/system_upgrade/common/actors/peseventsscanner/libraries/peseventsscanner.py

@@ -91,19 +91,16 @@ def _get_repositories_mapping():
     """
     repositories_mapping = {}
 
-    repositories_map_msgs = api.consume(RepositoriesMap)
-    repositories_map_msg = next(repositories_map_msgs, None)
-    if list(repositories_map_msgs):
-        api.current_logger().warning('Unexpectedly received more than one RepositoriesMap message.')
-    if not repositories_map_msg:
-        raise StopActorExecutionError(
-            'Cannot parse RepositoriesMap data properly',
-            details={'Problem': 'Did not receive a message with mapped repositories'}
-        )
+    for repositories_map_msg in api.consume(RepositoriesMap):
+        if not repositories_map_msg:
+            raise StopActorExecutionError(
+                'Cannot parse RepositoriesMap data from file {} properly'.format(repositories_map_msg.file),
+                details={'Problem': 'Did not receive a message with mapped repositories'}
+            )
 
-    for repository in repositories_map_msg.repositories:
-        if repository.arch == api.current_actor().configuration.architecture:
-            repositories_mapping[repository.to_pes_repo] = repository.to_repoid
+        for repository in repositories_map_msg.repositories:
+            if repository.arch == api.current_actor().configuration.architecture:
+                repositories_mapping[repository.to_pes_repo] = repository.to_repoid
 
     return repositories_mapping
 
@@ -177,7 +174,7 @@ def parse_pes_events(json_data):
     :return: List of Event tuples, where each event contains event type and input/output pkgs
     """
     data = json.loads(json_data)
-    if not isinstance(data, dict) or not data.get('packageinfo'):
+    if not isinstance(data, dict) or data.get('packageinfo') is None:
         raise ValueError('Found PES data with invalid structure')
 
     return [parse_entry(entry) for entry in data['packageinfo']]

repos/system_upgrade/common/actors/peseventsscanner/tests/files/sample04.json

@@ -0,0 +1,5 @@
+{
+    "legal_notice": "Copyright (c) 2022 CloudLinux Inc.",
+    "packageinfo": [],
+    "timestamp": "202207130941Z"
+}

repos/system_upgrade/common/actors/peseventsscanner/tests/unit_test_peseventsscanner.py

@@ -270,6 +270,10 @@ def test_get_events(monkeypatch):
     assert reporting.create_report.called == 1
     assert 'inhibitor' in reporting.create_report.report_fields['flags']
 
+    with open(os.path.join(CUR_DIR, 'files/sample04.json')) as f:
+        events = parse_pes_events(f.read())
+    assert len(events) == 0
+
 
 def test_pes_data_not_found(monkeypatch):
     def read_or_fetch_mocked(filename, directory="/etc/leapp/files", service=None, allow_empty=False):