Crash: listxattr failed #1462

pepa65 · 2016-08-11T00:45:33Z

(Same issue as jborg/attic#120 it seems, very similar traceback.)

Borg (1.0.6-2~~ubuntu14.04.1~~ppa1) failed just now. From xfs to xfs (on same file system), on Ubuntu 14.04 trusty. Of my five directories (each a separate archive, same repo) only one worked. Yesterday all five worked.
Example failure:

borg create -v --show-rc --stats -C lz4 /data/store/borg::Public-20160810 /data/Public
Enter passphrase for key /data/store/borg:
Local Exception.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1609, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1546, in run
    return args.func(args)
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 81, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 247, in do_create
    create_inner(archive, cache)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 219, in create_inner
    read_special=args.read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 301, in _process     read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 275, in _process
     status = archive.process_file(path, st, cache, self.ignore_inode)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 683, in process_file
    item.update(self.stat_attrs(st, path))
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 573, in stat_attrs
    xattrs = xattr.get_all(path, follow_symlinks=False)
  File "/usr/lib/python3/dist-packages/borg/xattr.py", line 31, in get_all
    for name in listxattr(path, follow_symlinks=follow_symlinks))
  File "/usr/lib/python3/dist-packages/borg/xattr.py", line 115, in listxattr
    raise Exception('listxattr failed')
Exception: listxattr failed

Platform: Linux brain2 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64
Linux: Ubuntu 14.04 trusty
Borg: 1.0.6  Python: CPython 3.4.3
PID: 21726  CWD: /home/administrator
sys.argv: ['/usr/bin/borg', 'create', '-v', '--show-rc', '--stats', '-C', 'lz4', '/data/store/borg::Public-20160810', '/data/Public']
SSH_ORIGINAL_COMMAND: None

terminating with error status, rc 2

The text was updated successfully, but these errors were encountered:

ThomasWaldmann · 2016-08-11T00:47:36Z

Can you reproduce this with patched code if i give you some patch?

pepa65 · 2016-08-11T00:50:18Z

OK, should I build it then from github? (I was running from a repo package...)

ThomasWaldmann · 2016-08-11T00:51:54Z

diff --git a/borg/xattr.py b/borg/xattr.py
index e88d7ce..87531d4 100644
--- a/borg/xattr.py
+++ b/borg/xattr.py
@@ -112,7 +112,7 @@ if sys.platform.startswith('linux'):  # pragma: linux only
         namebuf = create_string_buffer(n)
         n2 = _check(func(path, namebuf, n), path)
         if n2 != n:
-            raise Exception('listxattr failed')
+            raise Exception('listxattr failed, n=%d, n2=%d' % (n, n2))
         return [os.fsdecode(name) for name in namebuf.raw.split(b'\0')[:-1] if not name.startswith(b'system.posix_acl_')]

     def getxattr(path, name, *, follow_symlinks=True):

Maybe make a copy of the original file, then apply patch.
You can also do that with a dist package's code, you just need to locate borg/xattr.py in the filesystem.

If you reproduce with the patch, the exception message will contain n and n2 values. Guess we can fix the issue if we know these. I suspect that they are both positive but not equal.

Also, I need the line from /proc/mounts of the filesystem where your backup source files are.

pepa65 · 2016-08-11T00:56:57Z

It says this:

File "/usr/lib/python3/dist-packages/borg/xattr.py", line 115
    raise Exception('listxattr failed', n=%d, n2=%d' % (n, n2))
                                          ^
SyntaxError: invalid syntax

ThomasWaldmann · 2016-08-11T00:58:58Z

You didn't apply the patch, you have one single-quote too much, after "failed".

ThomasWaldmann · 2016-08-11T01:04:17Z

Note: the code for freebsd and darwin is rather similar, so might need the same fix.

pepa65 · 2016-08-11T03:04:20Z

Sorry, I had to go somewhere. I'll have a look as soon as I get back.

-------- Original Message --------
From:TW [email protected]
Sent:Thu, 11 Aug 2016 08:28:38 +0700
To:borgbackup/borg [email protected]
Cc:PePa [email protected],Mention [email protected]
Subject:Re: [borgbackup/borg] Crash: listxattr failed (#1462)

@pepa65 still there?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/borgbackup/borg","title":"borgbackup/borg","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/borgbackup/borg"}},"updates":{"snippets":[{"icon":"PERSON","message":"@ThomasWaldmann in #1462: @pepa65 still there?"}],"action":{"name":"View Issue","url":"https://github.com/borgbackup/borg/issues/1462#issuecomment-239052371"}}}

pepa65 · 2016-08-11T04:00:55Z

Result:

borg create -v --show-rc --stats -C lz4 /data/store/borg::Public-20160810 /data/Public
Enter passphrase for key /data/store/borg: 
Local Exception.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1609, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1546, in run
    return args.func(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 81, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 247, in do_create
    create_inner(archive, cache)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 219, in create_inner
    read_special=args.read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 275, in _process
    status = archive.process_file(path, st, cache, self.ignore_inode)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 683, in process_file
    item.update(self.stat_attrs(st, path))
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 573, in stat_attrs
    xattrs = xattr.get_all(path, follow_symlinks=False)
  File "/usr/lib/python3/dist-packages/borg/xattr.py", line 31, in get_all
    for name in listxattr(path, follow_symlinks=follow_symlinks))
  File "/usr/lib/python3/dist-packages/borg/xattr.py", line 115, in listxattr
    raise Exception('listxattr failed, n=%d, n2=%d' % (n, n2))
Exception: listxattr failed, n=45, n2=24

Platform: Linux brain2 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64
Linux: Ubuntu 14.04 trusty
Borg: 1.0.6  Python: CPython 3.4.3
PID: 31818  CWD: /home/administrator
sys.argv: ['/usr/bin/borg', 'create', '-v', '--show-rc', '--stats', '-C', 'lz4', '/data/store/borg::Public-20160810', '/data/Public']
SSH_ORIGINAL_COMMAND: None

terminating with error status, rc 2

pepa65 · 2016-08-11T06:37:32Z

Also, I xfs_repaired the filesystem (no problems reported).
Then I initiated a new repo and tried the same 5 datasets, only www succeeds. It might be because of not having utf-8 filenames, or because simpler permissions?? (But they worked the first time...)

enkore · 2016-08-11T08:16:13Z

The code looks racy. Is this a consistent issue that always appears on the same file or a bit more random looking?

query object size

alloc buffer

query object

fail if size doesn't match

ThomasWaldmann · 2016-08-11T11:35:55Z

I've looked at some other xattr code, their strategy:

call xattr func with a small buffer (they used 128B, I'ld rather use 4kiB I guess)
if it does not succeed due to too small buffer, call it again, giving a buffer with the max allowed xattr size.

hmm, that's also problematic: https://bugs.python.org/issue13669

use similar approach as for lz4 buffer growing (thread-local module level buffer)?

enkore · 2016-08-11T12:14:50Z

use similar approach as for lz4 buffer growing (thread-local module level buffer)?

Yes. 4K is a sane starting buffer size and would often require no resizing at all, since often files have no xattrs at all, or only the security namespace or other small xattrs.

Also, the failure in step 4 in #1462 (comment) should absolutely not be fatal, since it's inherently racy (filesystem APIs that are racy... can't be! :)

…#1462 this also fixes the race condition seen in borgbackup#1462 because there is only 1 call now. either it succeeds, then we get the correct length as result and truncate the result value to that length. or it fails with ERANGE, then we grow the buffer to double size and repeat. or it fails with some other error, then we throw OSError.

pepa65 · 2016-08-11T14:55:37Z

Here is the /proc/mounts line for source and dest:
/dev/mapper/trusty-data /data xfs rw,relatime,attr2,inode64,noquota 0 0

…#1462 this also fixes the race condition seen in borgbackup#1462 because there is only 1 call now. either it succeeds, then we get the correct length as result and truncate the result value to that length. or it fails with ERANGE, then we grow the buffer to double size and repeat. or it fails with some other error, then we throw OSError.

enkore · 2016-08-11T16:44:46Z

There is at least one more race in the xattr code itself: #906

…#1462 this also fixes the race condition seen in borgbackup#1462 because there is only 1 call now. either it succeeds, then we get the correct length as result and truncate the result value to that length. or it fails with ERANGE, then we grow the buffer to double size and repeat. or it fails with some other error, then we throw OSError.

pepa65 · 2016-08-12T02:41:08Z

Could you give me some instructions how I could try this?

ThomasWaldmann · 2016-08-12T12:32:18Z

You would need to git pull or git clone from the branch the pull request is made from and install borg from source (see docs).

Or just wait until this got reviewed/merged, it will hopefully get into 1.0.7rc2 and can be tried there.

…#1462 this also fixes the race condition seen in borgbackup#1462 because there is only 1 call now. either it succeeds, then we get the correct length as result and truncate the result value to that length. or it fails with ERANGE, then we grow the buffer to double size and repeat. or it fails with some other error, then we throw OSError.

xattr: dynamically grow result buffer until it fits, fixes #1462

ThomasWaldmann · 2016-08-12T16:22:38Z

Fixed by #1464.

pepa65 · 2016-08-12T17:55:28Z

Sure, I'm eager to try! Which branch was the pull request made from??

ThomasWaldmann · 2016-08-12T18:03:51Z

As it is merged now, you can use 1.0-maint branch.

pepa65 · 2016-08-13T01:07:28Z

So far I did:

clone https://github.com/borgbackup/borg.git
cd borg
git checkout -t origin/1.0-maint
apt install python3-setuptools cython3 libssl-dev liblz4-dev 
python3 setup.py

Then it seems I am on a too old distribution (Ubuntu 14.04.5):

Installed /home/PeterPasschier1965/git/borg/setuptools_scm-1.11.1-py3.4.egg
your setuptools is too old (<12)
setuptools_scm functionality is degraded
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied

Don't want to mess too much with this box... If I store them on ext4 this bug won't be a problem, right??

ThomasWaldmann · 2016-08-13T03:37:57Z

I've just released 1.0.7rc2 (with a lot of xattr fixes/changes), so you could just take that.

pepa65 · 2016-08-13T06:53:36Z

I got the standalone binary borg-linux64, and did:

sudo cp borg-linux64 /usr/local/bin/borg
sudo chown root:root /usr/local/bin/borg
sudo chmod 755 /usr/local/bin/borg

borg -V outputs borg 1.0.7rc2
If I do the same command as at the start of this issue:
borg create -v --show-rc --stats -C lz4 /data/store/borg::Public-20160813 /data/Public
It comes back with: `terminating with success status, rc 0'!

Thank you very much Thomas and friends!

ThomasWaldmann · 2016-08-13T13:20:52Z

Thanks for testing.

ThomasWaldmann added this to the 1.0.7rc2 milestone Aug 11, 2016

ThomasWaldmann added the bug label Aug 11, 2016

ThomasWaldmann self-assigned this Aug 11, 2016

ThomasWaldmann mentioned this issue Aug 11, 2016

Dealing with attic issues #5

Closed

ThomasWaldmann mentioned this issue Aug 11, 2016

xattr: dynamically grow result buffer until it fits, fixes #1462 #1464

Merged

ThomasWaldmann added a commit that referenced this issue Aug 12, 2016

Merge pull request #1464 from ThomasWaldmann/fix-xattr-exception

4e5b80b

xattr: dynamically grow result buffer until it fits, fixes #1462

ThomasWaldmann closed this as completed Aug 12, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash: listxattr failed #1462

Crash: listxattr failed #1462

pepa65 commented Aug 11, 2016 •

edited by ThomasWaldmann

Loading

ThomasWaldmann commented Aug 11, 2016

pepa65 commented Aug 11, 2016

ThomasWaldmann commented Aug 11, 2016 •

edited

Loading

pepa65 commented Aug 11, 2016 •

edited

Loading

ThomasWaldmann commented Aug 11, 2016

ThomasWaldmann commented Aug 11, 2016

pepa65 commented Aug 11, 2016

pepa65 commented Aug 11, 2016

pepa65 commented Aug 11, 2016 •

edited

Loading

enkore commented Aug 11, 2016 •

edited

Loading

ThomasWaldmann commented Aug 11, 2016 •

edited

Loading

enkore commented Aug 11, 2016 •

edited

Loading

pepa65 commented Aug 11, 2016

enkore commented Aug 11, 2016 •

edited

Loading

pepa65 commented Aug 12, 2016

ThomasWaldmann commented Aug 12, 2016 •

edited

Loading

ThomasWaldmann commented Aug 12, 2016

pepa65 commented Aug 12, 2016

ThomasWaldmann commented Aug 12, 2016

pepa65 commented Aug 13, 2016

ThomasWaldmann commented Aug 13, 2016

pepa65 commented Aug 13, 2016

ThomasWaldmann commented Aug 13, 2016

Crash: listxattr failed #1462

Crash: listxattr failed #1462

Comments

pepa65 commented Aug 11, 2016 • edited by ThomasWaldmann Loading

ThomasWaldmann commented Aug 11, 2016

pepa65 commented Aug 11, 2016

ThomasWaldmann commented Aug 11, 2016 • edited Loading

pepa65 commented Aug 11, 2016 • edited Loading

ThomasWaldmann commented Aug 11, 2016

ThomasWaldmann commented Aug 11, 2016

pepa65 commented Aug 11, 2016

pepa65 commented Aug 11, 2016

pepa65 commented Aug 11, 2016 • edited Loading

enkore commented Aug 11, 2016 • edited Loading

ThomasWaldmann commented Aug 11, 2016 • edited Loading

enkore commented Aug 11, 2016 • edited Loading

pepa65 commented Aug 11, 2016

enkore commented Aug 11, 2016 • edited Loading

pepa65 commented Aug 12, 2016

ThomasWaldmann commented Aug 12, 2016 • edited Loading

ThomasWaldmann commented Aug 12, 2016

pepa65 commented Aug 12, 2016

ThomasWaldmann commented Aug 12, 2016

pepa65 commented Aug 13, 2016

ThomasWaldmann commented Aug 13, 2016

pepa65 commented Aug 13, 2016

ThomasWaldmann commented Aug 13, 2016

pepa65 commented Aug 11, 2016 •

edited by ThomasWaldmann

Loading

ThomasWaldmann commented Aug 11, 2016 •

edited

Loading

pepa65 commented Aug 11, 2016 •

edited

Loading

pepa65 commented Aug 11, 2016 •

edited

Loading

enkore commented Aug 11, 2016 •

edited

Loading

ThomasWaldmann commented Aug 11, 2016 •

edited

Loading

enkore commented Aug 11, 2016 •

edited

Loading

enkore commented Aug 11, 2016 •

edited

Loading

ThomasWaldmann commented Aug 12, 2016 •

edited

Loading