Suggestion: add an aarch64 channel

[akirakyle]

I'm running nixos on a Pinebook Pro which uses an aarch64 board. I'm on the nixos-19.09 channel and I recently upgraded and much to my surprise nix started trying to build firefox which after an hour or so ran out of memory. I was under the impression that such a build would be cached from hydra given my experience with nix on x86_64 machines. However it seems that the nixos-19.09 channel doesn't actually critically depend on any aarch64 builds besides the minimal iso and sd card images.

This means that I have to find the last building firefox on hydra and pin nixpkgs at that version. The same can apply to other packages for example the ghc which I also have to pin at a different prior nixpkgs version.

It may smooth out the user experience on aarch64 to have a channel which critically depends on a similar set of aarch64 packages as x86_64 packages in the nixos-19.09 channel.

[kolbycrouch]

If you really must have firefox in the meantime, using zram and setting nix.buildCores to 1 or 2 should allow you to finish the build.

[akirakyle]

As I mentioned, I managed to get the latest firefox by pinning to the last building version on hydra, which is still the current firefox version. I'm not sure why the firefox build is failing now on hydra but as "just a user" on a release channel I shouldn't have to really care since the channel should wait to update until hydra successfully builds critical packages such as firefox. At least this is the way it works on x86_64 but on aarch64 there seems to be no equivalent of a stable nixos-19.09 channel that will wait to update until such critical packages are successfully built.

[akirakyle]

I've poked around hydra a bit more and I think I can better phrase what I'm asking. My understanding is that the nixos-19.09 channel points to the latest successful build of the nixos:release-19.09:tested job. The problem is that job's constituents are primarily x86_64 builds (except for an aarch64 minimal iso and sd image), thus when I am on that channel I may often upgrade to versions which don't successfully build on aarch64. There is an analogous job for the nixos:release-19.09-aarch64 jobset: nixos:release-19.09:tested where all the constituents are aarch64 builds. What I would like is a channel which tracks this job. However it seems there is no latest successful build for that job due to the nixos.tests.hibernate.aarch64-linux job always failing. Hopefully that can either be fixed or removed as a constituent job for that aarch64 tested channel.

More generally it might make sense to have aarch64 versions of all the existing channels so that aarch64 users can have a similar experience to x86_64 users when subscribing to release or unstable channels. Perhaps this may also alleviate issues such as #81081 #72880?

[doronbehar]

Well Ideally, the same way builds for x86_64 should be monitored for success or failure, aarch64 builds should be as well. There's this interesting project: https://github.com/makefu/hydra-check which you can use to get links to the builds of packages according to attribute path.

I'd advise to tackle down why Firefox for aarch (specifically) is not built on Hydra. What's interesting, is that I couldn't even find the build for it here and with hydra-check as well. Perhaps @eelco & @andir would have an opinion.

BTW did you try firefox-bin?

[akirakyle]

I think the problem is that since the only aarch64 builds whose test have to succeed before a channel update are the iso_minimal and sd_image which only depend on the bare minimum of tools you need for system. This leaves an aarch64 user always playing a game of roulette when doing a channel update.

Firefox is more of an example which prompted this issue than the issue itself (also mozilla doesn't build a linux aarch64 firefox so there is no aarch64 firefox-bin).

I see two solutions for aarch64 users to have an experience consistent what that on x86_64. Either a new set of channels for aarch64 users which mirror those for x86_64 or adding the aarch64 counterparts to the tests that must succeed before a channel moves forward. I think the latter probably makes life more difficult for x86_64 users as aarch64 builds often take longer and can hold up channel releases.

Thanks for pointing me to the hydra-check tool. That might make life much easier in the meantime where I have to pin packages manually to their last successful build on hydra.

[doronbehar]

I see. Your idea makes sense but it might need attention from bigger players then you and me. I think it'd be appropriate to raise the issue on discourse, I've encountered my self as well when I once tried to play with NixOS on a Raspberry... If you feel qualify, you may even reach https://github.com/NixOS/rfcs/ eventually.

[doronbehar]

BTW #82886 should fix our issue in finding Firefox build for aarch64 .

[samueldr]

Those PRs will fix that issue:

• Fix evaluation of release-20.03-aarch64 jobset #82886
• nixos/release-combined.nix: fix tested/supportedSystems (master version of #82886) #83249

---

The Mobile NixOS' system image depends on firefox, so it will end up being built at the very worst lagging a bit behind channel updates.

[akirakyle]

Great thanks for pointing me to those! I'm glad this issue is being discussed with the core maintainers and I look forward to the aarch64 build situation improving!

[akirakyle]

Sorry to reopen this issue but it seems that neither #82886 for 20.03 nor #83249 for master fundamentally changed anything. It looks like the tested jobset still doesn't get any aarch64 constituents beyond iso_minimal sd_image for the nixos:trunk-combined or nixos:release-20.03 jobsets which I understand to be what the nixos-unstable and nixos-release-20.03 channels track. It appears that those PRs only fixed the nixos:release-20.03-aarch64 jobset by removing x86_64 constituents. @vcunat brought up this issue in #83249

I believe we need some overall thought on the design we have around the jobsets and channels with regards to platforms. Currently we seem to be stuck half-way between separated and integrated aarch64.

For instance, if I were aarch64 user, I wouldn't really know what channel/commit to follow. The channel-advancing jobsets like trunk-combined only contain aarch64 tests but no packages, so channels might advance before most binaries are there. There are separate jobsets to get most aarch64 binaries (e.g. release-20.03-aarch64...), but I see no convenient way to "follow" these without a channel/branch.

In looking at nixos/release-combined.nix and the hydra configuration tab for the various nixos jobsets built by calling that nix expression I see that in fact the changes implemented by #82886 and #83249 would have an effect if supportedSystems was set on hydra to include aach64 for either trunk-combined or release-20.03 or both. This would make the normal channel update process for users on aarch64 consistent with that on x86-64 however it would potentially slow channel updates for x86-64 updates when aarch64 tested constituents fail (also then release-20.03 would strictly contain release-20.03-aarch64?).

As I've learned more about this, I think I've become convinced that it would be best to have a separate channel tracking release-20.03-aarch64 that then users could have their system track. Right now an aarch64 user that does a channel update could potentially encounter a case where something as basic as ssh breaks since the x86-64 tests might pass and the channel advances while the aarch64 tests fail. However an x86-64 user shouldn't have to wait for a critical security update on account of an aarch64 test failing.

Perhaps trunk-combined could set supportedSystems = [ "x86_64-linux" "aarch64-linux"] which might encourage faster fixing of issues on aarch64 as they occur and allow aarch64 users to follow the unstable channel more readily (this may also benefit the mobile-nixos project, see for example mobile-nixos/mobile-nixos#93).

[stale]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[lordcirth]

This is still an issue.

[doronbehar]

Maybe we should just add some aarch64 tests to those tested for nixos-unstable updates.

[vikanezrimaya]

In light of recent binary cache lags I'm strongly in favor of keeping a channel (and a related Nixpkgs branch like nixos-unstable branch on this repository - maybe named nixos-unstable-aarch64?) strictly for aarch64 - both so the builds won't block channel advances for x86_64, and so we could monitor more aarch64-related things - for example, rspamd was broken on aarch64-unstable for sometime - the issue was on tracker (filed by me) and fixed also by me, since apparently nobody else noticed it? (probably I'm the crazy one here - running a Raspberry Pi 4 as a production machine)

I started starring some aarch64 jobs on Hydra so I could monitor them, but a separate whole channel will be better IMO.

[andir]

Yeah that would be great. I am also starting to run more aarch64 hardware and having to build a lot of derivations even thought the channel advanced is a bit of a pity. I am also a bit concerned about the amount of hydra builders we have for aarch64. We should probably also seek some more funding/donations/… of proper aarch64 hardware?!

[vcunat]

Additional builders might not be too hard to negotiate. I think @grahamc might know (where to ask). It's true that in the past few months it's certainly been the most lagging platform, often taking days to catch up and parts getting cancelled anyway (on non-stable branches) because everyone else would have to wait.

[vcunat]

Even with the current Hydra state, I expect the stable branches should be usable – we do have jobsets (and the corresponding binaries in cache), but we don't have channels, which may make it harder to choose "the right" commit.

[plabadens]

We should probably also seek some more funding/donations/… of proper aarch64 hardware?!

Due to the time it takes to build a full system on lower end aarch64 hardware, I have spent some money on a private EC2 Graviton instance. I'd rather contribute that money on shared hardware instead, if it led to more up-to-date binary caches for everyone.

[andir]

I've opened a PR to add the channel for 20.09: NixOS/infra#142

For unstable we would need another hydra jobset.

[andir]

Next step would be to get a master jobset for aarch64.

@NixOS/infra Can any of you help here? What is the process to create one of those without making it a Tier1 platform?

[stale]

I marked this as stale due to inactivity. → More info

[Artturin]

we got it now https://status.nixos.org/

[doronbehar]

we got it now status.nixos.org

That's great news! Is there a chance for an aarch64-unstable channel as well?