Segment fault with Ubuntu 24.04 20250120.5.0 #11471

RadxaYuntian · 2025-01-26T10:46:40Z

Current workaround

GitHub has refused to acknowledge this problem. The current workaround can be found here.

Description

We have a scheduled job running every Sunday, which failed today, with no code change in last 2 weeks.

After checking the build log, it always failed at a dkms package installation. Once the workflow file is changed to print the dkms log, the error is always gcc segment fault.

Changing running environment to ubuntu22-04 fixed the segment fault. Action still failed but that's because the change we made to investigate this issue.

What may be unusual for us is that we are using binfmt to run aarch64 gcc in a devcontainer, because the final output is an aarch64 system image. So this is not some normal gcc failing.

Platforms affected

Azure DevOps
GitHub Actions - Standard Runners
GitHub Actions - Larger Runners

Runner images affected

Image version and build link

20250120.5.0

Is it regression?

20250105.1.0: https://github.com/RadxaOS-SDK/rsdk/actions/runs/12848906725

Expected behavior

DKMS install successfully without gcc segfault.

Actual behavior

gcc segfault:

   2025-01-26 07:47:36,252 bdebstrap ERROR: mmdebstrap failed with exit code 25. See above for details.
  
  /workspaces/rsdk
  
  DKMS make.log for radxa-overlays-0.1.20 for kernel 6.1.68-2-stable (aarch64)
  Sun Jan 26 07:47:16 UTC 2025
  make: Entering directory '/usr/src/linux-headers-6.1.68-2-stable'
  Segmentation fault (core dumped)
  warning: the compiler differs from the one used to build the kernel
    The kernel was built by: aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110
    You are using:           gcc (Debian 12.2.0-14) 12.2.0
    CC [M]  /var/lib/dkms/radxa-overlays/0.1.20/build/radxa-overlays.o
    DTC     /var/lib/dkms/radxa-overlays/0.1.20/build/arch/arm64/boot/dts/amlogic/overlays/meson-g12-disable-gpu.dtbo
    DTC     /var/lib/dkms/radxa-overlays/0.1.20/build/arch/arm64/boot/dts/amlogic/overlays/meson-g12-disable-hdmi.dtbo
    DTC     /var/lib/dkms/radxa-overlays/0.1.20/build/arch/arm64/boot/dts/rockchip/overlays/radxa-s0-ext-antenna.dtbo
  gcc: internal compiler error: Segmentation fault signal terminated program cc1
  Please submit a full bug report, with preprocessed source (by using -freport-bug).
  See <file:///usr/share/doc/gcc-12/README.Bugs> for instructions.
  make[2]: *** [scripts/Makefile.lib:409: /var/lib/dkms/radxa-overlays/0.1.20/build/arch/arm64/boot/dts/rockchip/overlays/radxa-s0-ext-antenna.dtbo] Error 4
  make[1]: *** [scripts/Makefile.build:500: /var/lib/dkms/radxa-overlays/0.1.20/build/arch/arm64/boot/dts/rockchip/overlays] Error 2
  make[1]: *** Waiting for unfinished jobs....

Repro steps

Clone https://github.com/RadxaOS-SDK/rsdk
Cherry pick RadxaYuntian/rsdk@090908a to view dkms log
Trigger workflow_dispatch for build.yaml

The text was updated successfully, but these errors were encountered:

RadxaYuntian · 2025-01-26T10:50:21Z

The gcc version Debian 12.2.0-14 was released on 2023/01/08, so the last successful run (2025/01/19) and today's failed run are both using the same version in the devcontainer.

deviantintegral · 2025-01-26T17:50:10Z

I can confirm this as well at https://github.com/pbkhrv/rtl_433-hass-addons/actions/runs/12972957498/job/36181006667. That job is compiling aarch64 in Docker under QEMU (I know, proper cross compiling would be better, but this is what the official Home Assistant builder action does so 🤷 ).

Is there a way to specify the runner image version to a previous 24.04 release to confirm the regression?

woblerr · 2025-01-26T19:36:16Z

The same problem for buildx for linux/arm64 via QEMU: https://github.com/woblerr/docker-pgbackrest/actions/runs/12965488407/job/36165276019#step:7:2658

Rollback to the ubuntu-22.04 runner solved the problem.

MyreMylar · 2025-01-26T20:09:50Z

Chiming in to say that we are seeing segfaults on our test runners for pygame-ce in the ppc64le architecture build since getting version 20250120.5.0. and, perhaps related, it also reporting that it can no longer detect the GNU compiler type for our S390x architecture build.

As @deviantintegral says it would be nice to have a way to roll back to a previous runner image to isolate the problem.

RaviAkshintala · 2025-01-27T05:19:46Z

Hi @RadxaYuntian Thank you for bringing this issue to our attention. We will look into this issue and will update you after investigating.

Due to an [issue](actions/runner-images#11471) with Ubuntu 24.04 20250120.5.0 runner image we have been seeing failures in our multi-arch images for the last few days which is blocking the release. I assume that the issue is something related to qemu, so downgrade to 22.04 until this issue is resolved. Signed-off-by: stevenhorsman <[email protected]>

BrianPugh · 2025-01-27T19:57:29Z

I'm also having very similar issues in tamp when using cibuildwheel to build python wheels for ppc64le and aarch64 targets.

actions/runner-images#11471

This issue is being currently discussed and fixed at actions/runner-images#11471 Signed-off-by: Rodrigo Tobar <[email protected]>

rtobar · 2025-01-28T04:41:14Z

Same issue here with gcc segfault, but in my case I saw it both with ubuntu-latest and ubuntu-20.04. Updating/downgrading to ubuntu-22.04 solved it as mentioned by other people.

Due to an [issue](actions/runner-images#11471) with Ubuntu 24.04 20250120.5.0 runner image we have been seeing failures in our multi-arch images for the last few days which is blocking the release. I assume that the issue is something related to qemu, so downgrade to 22.04 until this issue is resolved. Signed-off-by: stevenhorsman <[email protected]>

visstro · 2025-02-13T09:46:04Z

tonistiigi/binfmt:latest had large version bump just 2 days ago - tonistiigi/binfmt#165 (comment).
There is a new bug report tracking segfault issues: tonistiigi/binfmt#240

Looks like the solution for some people is to pin tonistiigi/binfmt:qemu-v7.0.0-28.

In my case tonistiigi/binfmt:qemu-v8.1.5 is still working fine (as I already noted in previous comment).

RadxaYuntian · 2025-02-17T04:04:57Z

qemu-v7.0.0-28: https://github.com/RadxaYuntian/rsdk/actions/runs/13304892587/job/37153541607#step:3:503
qemu-v8.1.5: https://github.com/RadxaYuntian/rsdk/actions/runs/13350294731/job/37285925408#step:3:503
Still getting segfaults with those.

actions/runner-images#11471 Signed-off-by: ZHANG Yuntian <[email protected]>

RadxaYuntian · 2025-02-17T09:02:02Z

Now even with 22.04 the segfault still happened with docker.io/tonistiigi/binfmt:latest.

22.04+qemu v7 seems to be fine.

- add arm64 builds (using ubuntu-22.04 runners for now, until actions/runner-images#11471 is fixed) - use new fancy cache feature of docker/build-push-action - run actions for all branches, but only push for main

ashwin153 · 2025-02-17T15:57:26Z

switching to ubuntu-22.04 + pinning to tonistiigi/binfmt:qemu-v7.0.0-28 worked for me.

    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-qemu-action@v3
        with:
          image: tonistiigi/binfmt:qemu-v7.0.0-28

actions/runner-images#11471 (comment)

…`jammy`), pin QEMU to avoid segfault 20524e4 fix: pin version of QEMU to avoid segfault (Kittywhiskers Van Gogh) c8ab705 revert: update containers and CI to use Ubuntu 24.04 LTS (`noble`) (Kittywhiskers Van Gogh) 2bcc90a revert: use default non-root user `ubuntu` introduced in Ubuntu 22.10 (Kittywhiskers Van Gogh) Pull request description: ## Additional Information * There is a regression in `noble` that has caused a build failure in trying to package our release container ([build](https://github.com/dashpay/dash/actions/runs/13376032021)) linked to [docker/setup-qemu-action#198](docker/setup-qemu-action#198). * In light of this, all non-CI containers and jobs have been moved back to Ubuntu 22.04 (`jammy`) though it seems that downgrading alone may be insufficient (see [actions/runner-images#11471](actions/runner-images#11471 (comment))). * To remedy this we are pinning the version of QEMU as suggested in [tonistiigi/binfmt#240](tonistiigi/binfmt#240 (comment)) ## Checklist: - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)** - [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)** - [x] I have made corresponding changes to the documentation **(note: N/A)** - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ ACKs for top commit: UdjinM6: utACK 20524e4 Tree-SHA512: 26bb2cd55a0267b56f86938d97ddfa32f0cdd8a2786c0366eedbcddf706e38b6af93cd29ab98ba420cbdbd112561ded61e2dba906c4b233ad737f24730f58ddc

…`jammy`), pin QEMU to avoid segfault 20524e48150c1552e60e833b4b42684b734e0040 fix: pin version of QEMU to avoid segfault (Kittywhiskers Van Gogh) c8ab7051afe03576f12789ed3fdbdaf63155ac59 revert: update containers and CI to use Ubuntu 24.04 LTS (`noble`) (Kittywhiskers Van Gogh) 2bcc90af46df03f4d0f1d0478187feb31b604c62 revert: use default non-root user `ubuntu` introduced in Ubuntu 22.10 (Kittywhiskers Van Gogh) Pull request description: ## Additional Information * There is a regression in `noble` that has caused a build failure in trying to package our release container ([build](https://github.com/dashpay/dash/actions/runs/13376032021)) linked to [docker/setup-qemu-action#198](docker/setup-qemu-action#198). * In light of this, all non-CI containers and jobs have been moved back to Ubuntu 22.04 (`jammy`) though it seems that downgrading alone may be insufficient (see [actions/runner-images#11471](actions/runner-images#11471 (comment))). * To remedy this we are pinning the version of QEMU as suggested in [tonistiigi/binfmt#240](tonistiigi/binfmt#240 (comment)) ## Checklist: - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)** - [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)** - [x] I have made corresponding changes to the documentation **(note: N/A)** - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ ACKs for top commit: UdjinM6: utACK 20524e48150c1552e60e833b4b42684b734e0040 Tree-SHA512: 26bb2cd55a0267b56f86938d97ddfa32f0cdd8a2786c0366eedbcddf706e38b6af93cd29ab98ba420cbdbd112561ded61e2dba906c4b233ad737f24730f58ddc

amotl · 2025-02-19T16:17:52Z

Dear @ashwin153,

thank you so much for your suggestion. We needed to apply that fix to fix the OCI build on ARM.

CI (ARM): Errors were encountered while processing: libc-bin (Segmentation fault) crate/mlflow-cratedb#208
OCI: Fix segfaults on ARM builds crate/mlflow-cratedb#210

With kind regards,
Andreas.

NB: Shall this ticket be re-opened? Do participants of this conversation have any other suggestions or news on this topic?

WyriHaximus · 2025-02-19T19:06:14Z

NB: Shall this ticket be re-opened? Do participants of this conversation have any other suggestions or news on this topic?

IMHO yes until it's all fully working again without having to pin versions, or bump to an older Ubuntu version etc.

Ref: actions/runner-images#11471

Due to actions/runner-images#11471 we see similar problems when using ubuntu24.04 to do cross build for arm64 images. A workaround is to downgrade to ubuntu22.04. Fixes confidential-containers#715 Signed-off-by: Xynnn007 <[email protected]>

Due to actions/runner-images#11471 we see similar problems when using ubuntu24.04 to do cross build for arm64 images. A workaround is to downgrade to ubuntu22.04 and use Qemu v7. Fixes confidential-containers#715 Signed-off-by: Xynnn007 <[email protected]>

Due to actions/runner-images#11471 we see similar problems when using ubuntu24.04 to do cross build for arm64 images. A workaround is to downgrade to ubuntu22.04 and use Qemu v8. Fixes confidential-containers#715 Signed-off-by: Xynnn007 <[email protected]>

amotl · 2025-02-22T00:33:22Z

Shall this ticket be re-opened?

IMHO yes until it's all fully working again without having to pin versions.

@kishorekumar-anchala: Based on the information above, can you please re-open this ticket, if you agree?

RadxaYuntian added bug report needs triage labels Jan 26, 2025

RaviAkshintala assigned kishorekumar-anchala Jan 27, 2025

RaviAkshintala added OS: Ubuntu OS: Ubuntu24 and removed needs triage labels Jan 27, 2025

blapie mentioned this issue Jan 27, 2025

Random failures with QEMU in Ubuntu 24.04 shibatch/sleef#614

Closed

dgraziotin mentioned this issue Jan 27, 2025

Build action fails under ubuntu-latest dgraziotin/docker-nginx-webdav-nononsense#161

Open

stevenhorsman mentioned this issue Jan 27, 2025

workflow: Downgrade github runner confidential-containers/cloud-api-adaptor#2263

Merged

muddyfish mentioned this issue Jan 27, 2025

Build on ubuntu22-04 awslabs/mountpoint-s3-csi-driver#361

Merged

hackeresq mentioned this issue Jan 27, 2025

workflow: Downgrade github runner to Ubunutu 22.04 investbrainapp/investbrain#55

Merged

busma13 mentioned this issue Jan 27, 2025

docker build is failing in CI terascope/base-docker-image#37

Closed

psyray added a commit to Security-Tools-Alliance/rengine-ng that referenced this issue Jan 27, 2025

Set ubuntu to ubuntu-22.04

1acc1c9

actions/runner-images#11471

nzin-alayacare mentioned this issue Jan 28, 2025

explicitely specify qemu targets AlayaCare/goliac-alayacare#140

Closed

rtobar added a commit to ICRAR/crc32c that referenced this issue Jan 28, 2025

Update to ubuntu-22.04 to avoid gcc sefault

fae6bdb

This issue is being currently discussed and fixed at actions/runner-images#11471 Signed-off-by: Rodrigo Tobar <[email protected]>

rtobar added a commit to ICRAR/ijson that referenced this issue Jan 28, 2025

Downgrade to ubuntu-22.04 to avoid gcc segfault

698b114

This issue is being currently discussed and fixed at actions/runner-images#11471 Signed-off-by: Rodrigo Tobar <[email protected]>

woblerr mentioned this issue Jan 28, 2025

Add pgBackRest v2.54.2. woblerr/docker-pgbackrest#56

Merged

charlesomer referenced this issue in mikebrady/shairport-sync Jan 28, 2025

Update docker-build-on-tag.yaml

bc824a2

alexanderkiel mentioned this issue Feb 14, 2025

Fix ARM64 Docker Build samply/blaze#2394

Closed

RadxaYuntian added a commit to RadxaOS-SDK/rsdk that referenced this issue Feb 17, 2025

fix: workaround segfaults

9c00d1b

actions/runner-images#11471 Signed-off-by: ZHANG Yuntian <[email protected]>

Piotr1215 mentioned this issue Feb 17, 2025

chore(ci): update ubuntu image runner loft-sh/guard#1

Merged

baszoetekouw mentioned this issue Feb 17, 2025

Fix Arm64 builds OpenConext/OpenConext-BaseContainers#32

Merged

flavorjones mentioned this issue Feb 17, 2025

Drop Struct::HTMLElementDescription class sparklemotion/nokogiri#3433

Merged

amotl mentioned this issue Feb 17, 2025

CI (ARM): Errors were encountered while processing: libc-bin (Segmentation fault) crate/mlflow-cratedb#208

Closed

ptr727 mentioned this issue Feb 17, 2025

GitHub Actions: qemu: uncaught target signal 11 (Segmentation fault) - core dumped ptr727/NxWitness#191

Open

hackeresq mentioned this issue Feb 18, 2025

Updated Dockerfile investbrainapp/investbrain#65

Open

soulgalore added a commit to sitespeedio/sitespeed.io that referenced this issue Feb 18, 2025

Try to fix build on Docker.

c3d53d6

actions/runner-images#11471 (comment)

avrodriguezgrad mentioned this issue Feb 18, 2025

Updates push.yml to fix runner and qemu segfault Gradiant/5g-images#336

Merged

kwvg mentioned this issue Feb 18, 2025

fix: revert deployment images back to Ubuntu 22.04 LTS (jammy), pin QEMU to avoid segfault dashpay/dash#6586

Merged

5 tasks

Brooooooklyn added a commit to napi-rs/napi-rs that referenced this issue Feb 20, 2025

ci: pin qemu and ubuntu version

f93ce16

Ref: actions/runner-images#11471

Brooooooklyn mentioned this issue Feb 20, 2025

ci: pin qemu and ubuntu version napi-rs/napi-rs#2474

Merged

Brooooooklyn added a commit to napi-rs/napi-rs that referenced this issue Feb 20, 2025

ci: pin qemu and ubuntu version (#2474)

4d77245

Ref: actions/runner-images#11471

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segment fault with Ubuntu 24.04 20250120.5.0 #11471

Segment fault with Ubuntu 24.04 20250120.5.0 #11471

RadxaYuntian commented Jan 26, 2025 •

edited

Loading

RadxaYuntian commented Jan 26, 2025 •

edited

Loading

deviantintegral commented Jan 26, 2025

woblerr commented Jan 26, 2025

MyreMylar commented Jan 26, 2025

RaviAkshintala commented Jan 27, 2025

BrianPugh commented Jan 27, 2025

rtobar commented Jan 28, 2025

visstro commented Feb 13, 2025

RadxaYuntian commented Feb 17, 2025

RadxaYuntian commented Feb 17, 2025

ashwin153 commented Feb 17, 2025 •

edited

Loading

amotl commented Feb 19, 2025

WyriHaximus commented Feb 19, 2025

amotl commented Feb 22, 2025 •

edited

Loading

Segment fault with Ubuntu 24.04 20250120.5.0 #11471

Segment fault with Ubuntu 24.04 20250120.5.0 #11471

Comments

RadxaYuntian commented Jan 26, 2025 • edited Loading

Current workaround

Description

Platforms affected

Runner images affected

Image version and build link

Is it regression?

Expected behavior

Actual behavior

Repro steps

RadxaYuntian commented Jan 26, 2025 • edited Loading

deviantintegral commented Jan 26, 2025

woblerr commented Jan 26, 2025

MyreMylar commented Jan 26, 2025

RaviAkshintala commented Jan 27, 2025

BrianPugh commented Jan 27, 2025

rtobar commented Jan 28, 2025

visstro commented Feb 13, 2025

RadxaYuntian commented Feb 17, 2025

RadxaYuntian commented Feb 17, 2025

ashwin153 commented Feb 17, 2025 • edited Loading

amotl commented Feb 19, 2025

WyriHaximus commented Feb 19, 2025

amotl commented Feb 22, 2025 • edited Loading

RadxaYuntian commented Jan 26, 2025 •

edited

Loading

RadxaYuntian commented Jan 26, 2025 •

edited

Loading

ashwin153 commented Feb 17, 2025 •

edited

Loading

amotl commented Feb 22, 2025 •

edited

Loading