Singularity+Docker(overlayfs)+Tar issue

[AaronTHolt]

I have a Dockerized Jenkins deployment (using overlay2) which is building Singularity images and pushing them to an sregistry we're hosting. I'm running into an issue where tar is failing when building a Singularity image tar: ./usr/share/man/man1: Directory renamed before its status could be extracted.... More specifically, Singularity fails to build after pulling from a local image. I've done some digging and found the issue is likely a docker+overlay2 issue. I'm curious if you've run into this before and have any workarounds. Thanks!

Version of Singularity:

2.4.2

Singularity fails to build mfix.img pulling local mfix_base.img

[move] holtat-mfix_base:latest.simg => /home/jenkins/workspace/test-mfix-final/mfix_base.img
+ sudo /usr/local/bin/singularity build mfix.img mfix.def
Using container recipe deffile: mfix.def
Sanitizing environment
[0mAdding base Singularity environment to container
Exporting contents of mfix_base.img to /tmp/.singularity-build.aSxAK6
tar: ./usr/share/man/man1: Directory renamed before its status could be extracted
tar: ./usr/share/man: Directory renamed before its status could be extracted
tar: ./usr/share/ghostscript/9.07/Resource/CMap: Directory renamed before its status could be extracted
tar: ./usr/share/ghostscript/9.07/Resource: Directory renamed before its status could be extracted
...

[dtrudg]

@AaronTHolt - please could you try with 2.5.1? The docker extraction has been completely rewritten and does not use gnu tar. This is known to address an issue with using it on top of an aufs system, and is likely to resolve for overlayfs too.

[vsoch]

@dctrud we have reproduced the error over here, using 2.5.1: #1570 Any updates to this issue? In the linked issue they suggest using a bsdtar but I'm hoping this extra dependency isn't required, since tar is a bread and butter sort of deal :)

[dtrudg]

Can you share a def file that triggers this, and a debug output? I'm thinking maybe I misread and inferred myself the build was using a docker base. Docker-extraction uses libarchive, which is the library the bsdtar is implemented with, hence my thought we'd be okay here. There are other paths that still use gnu tar though.

[dtrudg]

@vsoch - Ahh okay, I found the reproducer on #1606 which maybe you meant to reference instead of self-referencing this issue?

I don't think we'd want to change anything in Singularity here tbh. The underlying issue is a kernel bug in overlayfs that causes a problem when tar is used within a docker environment where docker is using the overlayfs storage driver. Apparently kernels >=4.13 fix this issue. Though bsdtar sounds like it is a workaround I don't think we'd want the extra dependency, and it'd definitely not come in a point release.

[vsoch]

Copy paste fail :_) See here! singularityhub/docker2singularity#30

[AaronTHolt]

I've finally gotten around to installing 2.5.1 and the issue persists. I agree that overlayfs and/or Docker needs to change and not Singularity. My current workaround is to build images in one go instead bootstrapping. Feel free to close this out whenever.

[vsoch]

@AaronTHolt and so awesome you have deployed an sregistry! Please ping me if you have any feature requests / other, I aim to add integrations / etc. that might be useful to you. Have you considered building Singularity images just using CI like Travis? I gave that a go this weekend and it was super easy! You would just add an sregistry push command at the end, and encrypt your environment variables in travis settings. https://vsoch.github.io/2018/singularity-ci/

[dtrudg]

I'm going to close this issue, per the comment above. Please feel free to reopen if there is anything further.