bin/make_tarball.sh: use bsdtar instead of tar

[pommi]

When using tar with overlayfs it could happen that you end up with for example
a /bin folder with 0700 rights, while it should be 0755. This is caused by:

tar: bin: Directory renamed before its status could be extracted

This is a known issue: docker/hub-feedback#727

In this case we have a "currupt" rootfs because the vcap user isn't able to do
anything anymore, because it's shell is /bin/bash (which is in /bin).

As a workaround we use bsdtar instead of tar. See:
coreos/bugs#1095 (comment)

Also generate a UTF-8 locale, so bsdtar can use it and you don't get:

bsdtar: Pathname can't be converted from UTF-8 to current locale.
bsdtar: <path/to/filename>: Can't translate pathname '<path/to/another_filename>' to UTF-8
bsdtar: <path/to/filename>: Can't translate linkname '<another_filename>' to UTF-8

[cfdreddbot]

Hey pommi!

Thanks for submitting this pull request! I'm here to inform the recipients of the pull request that you and the commit authors have already signed the CLA.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/144610517

The labels on this github issue will be updated when the story is started.

[sclevine]

Hi @pommi, thanks for the PR!

I'm not sure if we can take this as-is, because it modifies the host system and requires root privileges. I'll prioritize https://www.pivotaltracker.com/story/show/144610517 to investigate.

[pommi]

Hi @sclevine! Thanks for your quick follow-up!

This change only applies to the intermediate step (make cflinuxfs2.tar.gz) to create the tar.gz file using ./bin/make_tarball.sh. It spins up a docker instance and re-tar's the provided cflinuxfs2.tar file. ./bin/make_tarball.sh is only executed in this temporary docker instance.

[sesmith177]

Hello @pommi,

Our assumption here (based on reading the linked issues) is that the problem:

1. occurs occasionally
2. is that the .tar.gz file created by running make is bad

That is, the problem occurs while creating the .tar.gz file, not when extracting it later.

If this is the case, it may be better to add a check to bin/make_tarball.sh to confirm /bin was created with the proper permissions (or just fail if the tar command has any warnings)

We're not entirely comfortable switching to BSD tar if it's not absolutely necessary.

[pommi]

Hi @sesmith177,

Here is another issue that gives more in-depth information about the issue: moby/moby#19647. So it's overlayfs that is causing behaviour that tar isn't able to handle.

1. occurs occasionally

In our case it happens in 100% of our builds. I have to say that we add another 1GB of stuff to our rootfs, which cloud make things worse.

And it's not only /usr that is wrong, but sometimes a lot more:

tar: usr/lib/x86_64-linux-gnu/mit-krb5: Directory renamed before its status could be extracted
tar: usr/lib/x86_64-linux-gnu: Directory renamed before its status could be extracted
tar: usr/lib/ssl: Directory renamed before its status could be extracted
tar: usr/lib/python3.4/config-3.4m-x86_64-linux-gnu: Directory renamed before its status could be extracted
tar: usr/lib/python3.4: Directory renamed before its status could be extracted
tar: usr/lib/python2.7: Directory renamed before its status could be extracted
tar: usr/lib/openblas-base: Directory renamed before its status could be extracted
tar: usr/lib/gold-ld: Directory renamed before its status could be extracted
tar: usr/lib/gcc/x86_64-linux-gnu/4.8: Directory renamed before its status could be extracted
tar: usr/lib/gcc/x86_64-linux-gnu: Directory renamed before its status could be extracted
tar: usr/lib/gcc: Directory renamed before its status could be extracted
tar: usr/lib/compat-ld: Directory renamed before its status could be extracted
tar: usr/lib: Directory renamed before its status could be extracted
tar: usr/include/krb5: Directory renamed before its status could be extracted
tar: usr/include/kadm5: Directory renamed before its status could be extracted
tar: usr/include/gssrpc: Directory renamed before its status could be extracted
tar: usr/include/gssapi: Directory renamed before its status could be extracted
tar: usr/include: Directory renamed before its status could be extracted
tar: usr/bin: Directory renamed before its status could be extracted
tar: sbin: Directory renamed before its status could be extracted
tar: opt: Directory renamed before its status could be extracted
tar: lib64: Directory renamed before its status could be extracted
tar: lib/systemd/system/sysinit.target.wants: Directory renamed before its status could be extracted
tar: lib/systemd/system/sockets.target.wants: Directory renamed before its status could be extracted
tar: lib/systemd/system: Directory renamed before its status could be extracted
tar: lib/systemd: Directory renamed before its status could be extracted
tar: lib: Directory renamed before its status could be extracted
tar: etc/systemd/system/multi-user.target.wants: Directory renamed before its status could be extracted
tar: etc/systemd/system: Directory renamed before its status could be extracted
tar: etc/systemd: Directory renamed before its status could be extracted
tar: etc/ssl/certs: Directory renamed before its status could be extracted
tar: etc/ssl: Directory renamed before its status could be extracted
tar: etc/rcS.d: Directory renamed before its status could be extracted
tar: etc/rc6.d: Directory renamed before its status could be extracted
tar: etc/rc5.d: Directory renamed before its status could be extracted
tar: etc/rc4.d: Directory renamed before its status could be extracted
tar: etc/rc3.d: Directory renamed before its status could be extracted
tar: etc/rc2.d: Directory renamed before its status could be extracted
tar: etc/rc1.d: Directory renamed before its status could be extracted
tar: etc/rc0.d: Directory renamed before its status could be extracted
tar: etc/newt: Directory renamed before its status could be extracted
tar: etc/network: Directory renamed before its status could be extracted
tar: etc/init.d: Directory renamed before its status could be extracted
tar: etc/fonts/conf.d: Directory renamed before its status could be extracted
tar: etc/fonts: Directory renamed before its status could be extracted
tar: etc/apparmor.d/disable: Directory renamed before its status could be extracted
tar: etc/apparmor.d: Directory renamed before its status could be extracted
tar: etc/apparmor/init/network-interface-security: Directory renamed before its status could be extracted
tar: etc/apparmor/init: Directory renamed before its status could be extracted
tar: etc/apparmor: Directory renamed before its status could be extracted
tar: etc/alternatives: Directory renamed before its status could be extracted
tar: etc: Directory renamed before its status could be extracted
tar: dev: Directory renamed before its status could be extracted
tar: bin: Directory renamed before its status could be extracted
tar: Exiting with failure status due to previous errors

2. is that the .tar.gz file created by running make is bad

That is, the problem occurs while creating the .tar.gz file, not when extracting it later.

True

If this is the case, it may be better to add a check to bin/make_tarball.sh to confirm /bin was created with the proper permissions (or just fail if the tar command has any warnings)

That would mean that we will not be able to build a cflinuxfs2 anymore, because this will check will always fail.

We're not entirely comfortable switching to BSD tar if it's not absolutely necessary.

Can you share your specific concerns?

---

I just created 2 builds (1 using tar, 1 using bsdtar), did tar -ztvf and compared the results: http://paste.debian.net/plainh/c2599140 As you can see bin/ and dev/ have wrong permissions, the rest looks fine.

---

Now I'm thinking about this, why don't we just do:

gzip -f tmp/cflinuxfs2.tar

instead of:

docker run --rm -w /cflinuxfs2 -v `pwd`:/cflinuxfs2 $(docker_image) ./bin/make_tarball.sh cflinuxfs2

In 5586ddc it made sense, because there also were etc/hosts and etc/timezone that needed to be included, but those were removed in 7168dc0.

[sesmith177]

Hello @pommi,

Just zipping the final file looks interesting; could you try replacing the final line of the makefile with

docker run -w /cflinuxfs2 -v `pwd`:/cflinuxfs2 "ubuntu:trusty" bash -c "gzip -f tmp/cflinuxfs2.tar && mv tmp/cflinuxfs2.tar.gz ."