Add support for using artifacts from another Job #185

fepitre · 2025-02-26T12:28:12Z

Our usecase is for cross-compiling Windows tools

codecov · 2025-02-26T13:17:20Z

Codecov Report

Attention: Patch coverage is 82.66667% with 13 lines in your changes missing coverage. Please review.

Project coverage is 74.76%. Comparing base (0dab1bc) to head (b8e359d).

Files with missing lines	Patch %	Lines
qubesbuilder/config.py	75.00%	6 Missing ⚠️
qubesbuilder/plugins/__init__.py	85.71%	6 Missing ⚠️
qubesbuilder/plugins/fetch/__init__.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #185      +/-   ##
==========================================
+ Coverage   74.69%   74.76%   +0.07%     
==========================================
  Files          51       51              
  Lines        5865     5913      +48     
==========================================
+ Hits         4381     4421      +40     
- Misses       1484     1492       +8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

marmarek · 2025-02-26T17:32:09Z

README.md

@@ -859,3 +859,34 @@ For the `fetch` stage, the Qubes executor with disposable template `qubes-builde
 For the `build` stage of `vm-fc42`, the Podman executor with container image `fedoraimg` will be used.
 For the `sign` stage, the Qubes executor with disposable template `signing-access-dvm` will be used for both `vm-fc42` and `vm-jammy`
 For the `prep` stage of `vm-jammy`, the Local executor with base directory `/some/path` will be used.
+
+### Cross-compile Build


Technically, cross-compile is something else, like building ARM binaries on X86. Better use different term, like cross-distribution dependencies

marmarek · 2025-02-26T17:32:59Z

README.md

+
+```yaml
+components:
+  - installer-qubes-os:


installer-qubes-os-windows-tools? (and below too)

marmarek · 2025-02-26T17:34:47Z

qubesbuilder/config.py

+            if (
+                isinstance(stage, dict)
+                and next(iter(stage)) == stage_name
+                and isinstance(stage[stage_name], dict)


At least some of this not matching should show a warning or even error. Otherwise it will be hard to find why it doesn't work if you make a typo (like extra - making it a list instead of dict).

marmarek · 2025-02-26T17:35:21Z

qubesbuilder/config.py

+                    if all(
+                        [
+                            need.get("component", None),
+                            need.get("distribution", None),
+                            need.get("stage", None),
+                            need.get("build", None),
+                        ]


and same here - if some part is missing, at least log a warning why it's ignored

marmarek · 2025-02-26T17:38:11Z

qubesbuilder/plugins/__init__.py

+                    copy_in.append(
+                        (artifact, self.executor.get_dependencies_dir())
+                    )


should they all be copied together, or maybe better in separate subdirs? what if file names conflict (for example you declare QWT use vm-win10 and vm-win11 if that would exist)?

Our usecase is for cross-compiling Windows tools

marmarek · 2025-03-11T13:12:14Z

From CI logs:

yaml.representer.RepresenterError: ('cannot represent an object', PosixPath('/home/gitlab-runner/gitlab/github-i88pso6b/artifacts/components/python-qasync/0.23.0-2/vm-bookworm/prep/python-qasync_0.23.0-2+deb12u1.dsc'))

  File "/home/gitlab-runner/builds/QubesOS/qubes-builderv2/qubesbuilder/plugins/build_deb/__init__.py", line 384, in run
    self.save_dist_artifacts_info(
  File "/home/gitlab-runner/builds/QubesOS/qubes-builderv2/qubesbuilder/plugins/__init__.py", line 577, in save_dist_artifacts_info
    return self.save_artifacts_info(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gitlab-runner/builds/QubesOS/qubes-builderv2/qubesbuilder/plugins/__init__.py", line 422, in save_artifacts_info
    raise PluginError(msg) from e
qubesbuilder.plugins.PluginError: python-qasync:debian-pkg_debian: Failed to write info for build stage.

Besides the error itself, I'd prefer to not use absolute paths in artifacts info. Specifically, relative to the relevant artifacts dir. So, in this example it would be just python-qasync_0.23.0-2+deb12u1.dsc. In case of rpm, if the rpm subdir is preserved, then for example:

files:
- rpm/qubes-core-dom0-4.3.17-1.126.fc41.noarch.rpm

fepitre · 2025-03-11T13:37:56Z

From CI logs:

yaml.representer.RepresenterError: ('cannot represent an object', PosixPath('/home/gitlab-runner/gitlab/github-i88pso6b/artifacts/components/python-qasync/0.23.0-2/vm-bookworm/prep/python-qasync_0.23.0-2+deb12u1.dsc'))
  File "/home/gitlab-runner/builds/QubesOS/qubes-builderv2/qubesbuilder/plugins/build_deb/__init__.py", line 384, in run
    self.save_dist_artifacts_info(
  File "/home/gitlab-runner/builds/QubesOS/qubes-builderv2/qubesbuilder/plugins/__init__.py", line 577, in save_dist_artifacts_info
    return self.save_artifacts_info(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gitlab-runner/builds/QubesOS/qubes-builderv2/qubesbuilder/plugins/__init__.py", line 422, in save_artifacts_info
    raise PluginError(msg) from e
qubesbuilder.plugins.PluginError: python-qasync:debian-pkg_debian: Failed to write info for build stage.
Besides the error itself, I'd prefer to not use absolute paths in artifacts info. Specifically, relative to the relevant artifacts dir. So, in this example it would be just python-qasync_0.23.0-2+deb12u1.dsc. In case of rpm, if the rpm subdir is preserved, then for example:
files:
- rpm/qubes-core-dom0-4.3.17-1.126.fc41.noarch.rpm

I don't see where is it supposed to write absolute path, indeed this is not intended.

It allows to know what has been created by jobs.

fepitre · 2025-03-11T17:04:09Z

Ok it should be good for review.

fepitre · 2025-03-11T17:04:49Z

Maybe I should add yml checks of files I forgot that but it's definitively being used by the dependencies_03 test.

fepitre force-pushed the needs branch 2 times, most recently from c8b3f84 to bbb0767 Compare February 26, 2025 16:07

marmarek requested changes Feb 26, 2025

View reviewed changes

marmarek mentioned this pull request Feb 26, 2025

Add QWT support #181

Merged

fepitre force-pushed the needs branch 5 times, most recently from 9593265 to 9c76892 Compare March 10, 2025 14:36

fepitre added 2 commits March 10, 2025 15:37

Support command for rpm build and provide verrel placeholder

75887ce

Add support for using artifacts from another Job

9740aba

Our usecase is for cross-compiling Windows tools

fepitre force-pushed the needs branch 3 times, most recently from 4552148 to 39e023f Compare March 10, 2025 14:49

fepitre force-pushed the needs branch from 39e023f to f1dedd0 Compare March 11, 2025 13:56

First pass on introducing "files" in artifacts file

b8e359d

It allows to know what has been created by jobs.

fepitre force-pushed the needs branch from f1dedd0 to b8e359d Compare March 11, 2025 14:36

fepitre marked this pull request as ready for review March 11, 2025 17:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for using artifacts from another Job #185

Add support for using artifacts from another Job #185

fepitre commented Feb 26, 2025

codecov bot commented Feb 26, 2025 •

edited

Loading

marmarek Feb 26, 2025

marmarek Feb 26, 2025

marmarek Feb 26, 2025

marmarek Feb 26, 2025

marmarek Feb 26, 2025

marmarek commented Mar 11, 2025

fepitre commented Mar 11, 2025

fepitre commented Mar 11, 2025

fepitre commented Mar 11, 2025

Add support for using artifacts from another Job #185

Are you sure you want to change the base?

Add support for using artifacts from another Job #185

Conversation

fepitre commented Feb 26, 2025

codecov bot commented Feb 26, 2025 • edited Loading

Codecov Report

marmarek Feb 26, 2025

Choose a reason for hiding this comment

marmarek Feb 26, 2025

Choose a reason for hiding this comment

marmarek Feb 26, 2025

Choose a reason for hiding this comment

marmarek Feb 26, 2025

Choose a reason for hiding this comment

marmarek Feb 26, 2025

Choose a reason for hiding this comment

marmarek commented Mar 11, 2025

fepitre commented Mar 11, 2025

fepitre commented Mar 11, 2025

fepitre commented Mar 11, 2025

codecov bot commented Feb 26, 2025 •

edited

Loading