qdirstat-cache-writer creates broken cache file if some directory lacks Exec flag

[oliver]

If qdirstat-cache-writer encounters a directory which lacks the "executable" flag, it will create a broken cache file that cannot be completely read by qdirstat (only entries up to the broken line will be displayed in qdirstat).

The broken lines lack a size value, ie. there are two tab characters without a number inbetween.

Here's an example file: .qdirstat.cache.gz. It should list subdir1, subdir2 and subdir3, but since subdir2 is unreadable, subdir3 is not even listed anymore in qdirstat. Also, qdirstat has a total size of 2 MB for the entire tree, while du shows a size of 3 MB (and actually there are 4 MB of data, but the size of subdir2 cannot be detected obviously).

This is output of ls -lRa for that test directory:

br2wsl61:/tmp/test2(83)> ls -laR
.:
total 1620
drwxrwxr-x   5 og   og      4096 Feb  2 12:31 .
drwxrwxrwt. 74 root root  585728 Feb  2 12:31 ..
-rw-rw-r--   1 og   og       255 Feb  2 12:31 .qdirstat.cache.gz
drwxrwxr-x   2 og   og      4096 Feb  2 12:30 subdir1
drw-rw-r--   2 og   og      4096 Feb  2 12:28 subdir2
drwxrwxr-x   2 og   og      4096 Feb  2 12:30 subdir3
-rw-rw-r--   1 og   og   1048576 Feb  2 12:27 testfile0.bin

./subdir1:
total 1032
drwxrwxr-x 2 og og    4096 Feb  2 12:30 .
drwxrwxr-x 5 og og    4096 Feb  2 12:31 ..
-rw-rw-r-- 1 og og 1048576 Feb  2 12:30 testfile1.bin

./subdir2:
ls: cannot access ./subdir2/.: Permission denied
ls: cannot access ./subdir2/..: Permission denied
ls: cannot access ./subdir2/testfile2.bin: Permission denied
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
-????????? ? ? ? ?            ? testfile2.bin

./subdir3:
total 1032
drwxrwxr-x 2 og og    4096 Feb  2 12:30 .
drwxrwxr-x 5 og og    4096 Feb  2 12:31 ..
-rw-rw-r-- 1 og og 1048576 Feb  2 12:30 testfile3.bin

When I ran qdirstat-cache-writer on the test directory, these Perl warnings were printed:

Use of uninitialized value $_[0] in bitwise and (&) at /usr/lib64/perl5/Fcntl.pm line 221.
Use of uninitialized value $_[0] in bitwise and (&) at /usr/lib64/perl5/Fcntl.pm line 223.
Use of uninitialized value $_[0] in bitwise and (&) at /usr/lib64/perl5/Fcntl.pm line 225.
Use of uninitialized value $_[0] in bitwise and (&) at /usr/lib64/perl5/Fcntl.pm line 226.
Use of uninitialized value $_[0] in bitwise and (&) at /usr/lib64/perl5/Fcntl.pm line 227.
Use of uninitialized value $_[0] in bitwise and (&) at /usr/lib64/perl5/Fcntl.pm line 224.
Use of uninitialized value $size in concatenation (.) or string at censored/qdirstat/scripts/qdirstat-cache-writer line 375.
Use of uninitialized value $mtime in printf at censored/qdirstat/scripts/qdirstat-cache-writer line 376.
Use of uninitialized value $blocks in numeric gt (>) at censored/qdirstat/scripts/qdirstat-cache-writer line 378.
Use of uninitialized value $links in numeric gt (>) at censored/qdirstat/scripts/qdirstat-cache-writer line 379.

This happens with the latest qdirstat version (f33cba3), and it also happened with kdirstat-cache-writer.pl and k4dirstat 2.7.8-4-ge61717f.

[shundhammer]

Glad to see that finally after all these years (it's been in KDirStat for a long, long time) somebody really uses those features seriously. And even more glad to get such really high-quality bug reports with everything needed to track things down. Thank you!

I'll look into this.

[shundhammer]

This should be fixed with 0c2a3b1 .

Please check.

The handling is not as graceful as one could expect, though: The error is written to the cache file, but it's currently not visible in the QDirStat GUI that lstat() failed for the directory contents: It looks just like a normal directory with no content.

Handling this better would require extending the cache file format to properly store that information. I am somewhat reluctant to change that file format since I know that it seems to be in use for some home-cooked admin scripts or applications. Let me think how this could be handled.

[oliver]

The fix works fine. I tested newest qdirstat-cache-writer on the problematic directory here, and qdirstat correctly displayed the cache file.

Btw. thanks a lot for providing the cache writing script at all! I've been using kdirstat-cache-writer.pl in a cronjob for quite a while already, to index my data partition with its 3 million files; and kdirstat/qdirstat is simply the only tool that provides this feature and is sufficiently detailed. Filelight doesn't seem to have this feature at all, and Baobab got this cache-file feature only recently but it doesn't show single files at all. So kudos for writing kdirstat/qdirstat, it has been really helpful for me already!