Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git GC complains about path too long on windows for files that are not checked out #2222

Closed
YngveNPettersen opened this issue Jun 9, 2019 · 4 comments

Comments

@YngveNPettersen
Copy link

  • [ x] I was not able to find an open or closed issue matching what I'm seeing

Setup

  • Which version of Git for Windows are you using? Is it 32-bit or 64-bit?
git version 2.21.0.windows.1
cpu: x86_64
built from commit: 2481c4cbe949856f270a3ee80c802f5dd89381aa
sizeof-long: 4
sizeof-size_t: 8
  • Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?
Win 7 and Win10, both 64-bit
  • What options did you set as part of the installation? Or did you choose the
    defaults?
Editor Option: VIM
Custom Editor Path:
Path Option: BashOnly
Plink Path: C:\Program Files\PuTTY\plink.exe
SSH Option: Plink
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Disabled
Enable Symlinks: Disabled
  • Any other interesting things about your environment that might be related
    to the issue you're seeing?

Just some old commits with long path names (created by NPM). but have had issues with that with some files in chromium, too.

Details

  • Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash

git init longpath
cd longpath
# May need to be tweaked so that the path is 240 chars including work dir path
mkdir -p this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/path
touch this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/path/filename.txt
git add -A
git commit -m "Added long filename"
touch anormalfile.txt
git add anormalfile.txt
git commit -m "Added a normal file"
git rm this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/path/filename.txt
git commit -m "Removed long file"
cd ..
mkdir toolongdirname
cd toolongdirname
git clone ../longpath
cd longpath
git gc
  • What did you expect to occur after running these commands?

A quick and warning free gc operation

  • What actually happened instead?

The following was printed

warning: unable to access 'this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/.gitattributes': Filename too long
warning: unable to access 'this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/.gitattributes': Filename too long
warning: unable to access 'this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/.gitattributes': Filename too long
warning: unable to access 'this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/.gitattributes': Filename too long
Counting objects: 49, done.
Delta compression using up to 36 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (49/49), done.
Total 49 (delta 0), reused 45 (delta 0)
  • If the problem was occurring with a specific repository, can you provide the
    URL to that repository to help us with testing?

happens in any repository with this kind of long paths, if they get cloned into a directory that is too deep for the Windows path rule, even if the files are not present in the checkout

@dscho
Copy link
Member

dscho commented Oct 15, 2019

Does setting core.longpaths=true help?

@YngveNPettersen
Copy link
Author

I can't recall having seen it since I enabled that setting all over the place, since I was hitting this issue, and the real in-file-system issue, in a lot of my environments

However, my point is that this error should not be displayed for an operation that does not touch the real file system, this is displayed for a file that no longer exists in the checkout, it is only of the files that did exist in one of the commits, and have since been removed (but is still part of the history). GC should not restrict its internal maxium path length to an obsolete operating system specific max path length, even if that may be necessary when using the relevant, old APIs.

And just in case: If this is due to GC (for some reason) actually trying to access the (now-deleted) file in the real file system, then maybe there is a different bug in the GC code?

@dscho
Copy link
Member

dscho commented Oct 16, 2019

this is displayed for a file that no longer exists in the checkout

Ah, I missed this, and it is not actually a file that no longer exists, it is a file that never existed: .gitattributes.

Now, this seems to happen when read_attr() is called while processing tree entries, trying to see (in no_try_delta()) whether delta compression should be attempted by looking for the "delta" attribute. The full stack trace is:

#0  warn_on_inaccessible (
    path=0x357aa58 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/.gitattributes")
    at wrapper.c:425
#1  0x000000000064ab09 in warn_on_fopen_errors (
    path=0x357aa58 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/.gitattributes")
    at wrapper.c:431
#2  0x000000000064ab55 in fopen_or_warn (
    path=0x357aa58 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/.gitattributes",
    mode=0x6eacc8 <blank+294> "r") at wrapper.c:445
#3  0x00000000004e2b3a in read_attr_from_file (
    path=0x357aa58 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/.gitattributes", macro_ok=0)
    at attr.c:712
#4  0x00000000004e2dbb in read_attr (istate=0x7ae700 <the_index>,
    path=0x357aa58 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/.gitattributes", macro_ok=0)
    at attr.c:773
#5  0x00000000004e329f in prepare_attr_stack (istate=0x7ae700 <the_index>,
    path=0x357a028 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/path", dirlen=183,
    stack=0x35858b8) at attr.c:968
#6  0x00000000004e382a in collect_some_attrs (istate=0x7ae700 <the_index>,
    path=0x357a028 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/path", check=0x3585898)
    at attr.c:1111
#7  0x00000000004e38c5 in git_check_attr (istate=0x7ae700 <the_index>,
    path=0x357a028 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/path", check=0x3585898)
    at attr.c:1125
#8  0x0000000000472cab in no_try_delta (
    path=0x357a028 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/path")
    at builtin/pack-objects.c:982
#9  0x0000000000473239 in add_object_entry (oid=0x356b8f8, type=OBJ_TREE,
    name=0x357a028 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/path", exclude=0)
    at builtin/pack-objects.c:1192
#10 0x0000000000476b1d in show_object (obj=0x356b8f0,
    name=0x357a028 "this/is/a/ridicoulously/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/very/long/path", data=0x0)
    at builtin/pack-objects.c:2793
#11 0x00000000005623bb in process_tree (ctx=0x151de60, tree=0x356b8f0,
    base=0x151dde0, name=0x35affee "path") at list-objects.c:197
#12 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b8b8, base=0x151dde0) at list-objects.c:134
#13 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b8b8,
    base=0x151dde0, name=0x35afe5e "long") at list-objects.c:204
#14 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b880, base=0x151dde0) at list-objects.c:134
#15 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b880,
    base=0x151dde0, name=0x35afcce "very") at list-objects.c:204
#16 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b848, base=0x151dde0) at list-objects.c:134
#17 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b848,
    base=0x151dde0, name=0x35afc4e "very") at list-objects.c:204
#18 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b810, base=0x151dde0) at list-objects.c:134
#19 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b810,
    base=0x151dde0, name=0x35afabe "very") at list-objects.c:204
#20 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b7d8, base=0x151dde0) at list-objects.c:134
#21 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b7d8,
    base=0x151dde0, name=0x35af92e "very") at list-objects.c:204
#22 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b7a0, base=0x151dde0) at list-objects.c:134
#23 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b7a0,
    base=0x151dde0, name=0x35af79e "very") at list-objects.c:204
#24 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b768, base=0x151dde0) at list-objects.c:134
#25 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b768,
    base=0x151dde0, name=0x35af60e "very") at list-objects.c:204
#26 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b730, base=0x151dde0) at list-objects.c:134
#27 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b730,
    base=0x151dde0, name=0x35af47e "very") at list-objects.c:204
#28 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b6f8, base=0x151dde0) at list-objects.c:134
#29 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b6f8,
    base=0x151dde0, name=0x35af2ee "very") at list-objects.c:204
#30 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b6c0, base=0x151dde0) at list-objects.c:134
#31 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b6c0,
    base=0x151dde0, name=0x35af15e "very") at list-objects.c:204
#32 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b688, base=0x151dde0) at list-objects.c:134
#33 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b688,
    base=0x151dde0, name=0x35af04e "very") at list-objects.c:204
#34 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b650, base=0x151dde0) at list-objects.c:134
#35 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b650,
    base=0x151dde0, name=0x35aef3e "very") at list-objects.c:204
#36 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b618, base=0x151dde0) at list-objects.c:134
#37 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b618,
    base=0x151dde0, name=0x35aee2e "very") at list-objects.c:204
#38 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b5e0, base=0x151dde0) at list-objects.c:134
#39 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b5e0,
    base=0x151dde0, name=0x35aed1e "very") at list-objects.c:204
#40 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b5a8, base=0x151dde0) at list-objects.c:134
#41 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b5a8,
    base=0x151dde0, name=0x35aec0e "very") at list-objects.c:204
#42 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b570, base=0x151dde0) at list-objects.c:134
#43 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b570,
    base=0x151dde0, name=0x35aeafe "very") at list-objects.c:204
#44 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b538, base=0x151dde0) at list-objects.c:134
#45 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b538,
    base=0x151dde0, name=0x35ae9ee "very") at list-objects.c:204
#46 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b500, base=0x151dde0) at list-objects.c:134
#47 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b500,
    base=0x151dde0, name=0x35ae8de "very") at list-objects.c:204
#48 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b4c8, base=0x151dde0) at list-objects.c:134
#49 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b4c8,
    base=0x151dde0, name=0x356ad0e "very") at list-objects.c:204
#50 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b490, base=0x151dde0) at list-objects.c:134
#51 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b490,
    base=0x151dde0, name=0x356ad9e "very") at list-objects.c:204
#52 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b458, base=0x151dde0) at list-objects.c:134
#53 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b458,
    base=0x151dde0, name=0x35ae59e "very") at list-objects.c:204
#54 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b420, base=0x151dde0) at list-objects.c:134
#55 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b420,
    base=0x151dde0, name=0x35ae48e "very") at list-objects.c:204
#56 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b3e8, base=0x151dde0) at list-objects.c:134
#57 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b3e8,
    base=0x151dde0, name=0x35ae3be "very") at list-objects.c:204
#58 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b3b0, base=0x151dde0) at list-objects.c:134
#59 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b3b0,
    base=0x151dde0, name=0x35ae2ee "very") at list-objects.c:204
#60 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b378, base=0x151dde0) at list-objects.c:134
#61 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b378,
    base=0x151dde0, name=0x35ae21e "very") at list-objects.c:204
#62 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b340, base=0x151dde0) at list-objects.c:134
#63 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b340,
    base=0x151dde0, name=0x35ae19e "very") at list-objects.c:204
#64 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b308, base=0x151dde0) at list-objects.c:134
#65 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b308,
    base=0x151dde0, name=0x35ae0ce "very") at list-objects.c:204
#66 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b2d0, base=0x151dde0) at list-objects.c:134
#67 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b2d0,
    base=0x151dde0, name=0x35adf5e "very") at list-objects.c:204
#68 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b298, base=0x151dde0) at list-objects.c:134
#69 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b298,
    base=0x151dde0, name=0x35adf2e "very") at list-objects.c:204
#70 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b260, base=0x151dde0) at list-objects.c:134
#71 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b260,
    base=0x151dde0, name=0x35ade7e "very") at list-objects.c:204
#72 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b228, base=0x151dde0) at list-objects.c:134
#73 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b228,
    base=0x151dde0, name=0x35adc9e "very") at list-objects.c:204
#74 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b1f0, base=0x151dde0) at list-objects.c:134
#75 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b1f0,
    base=0x151dde0, name=0x35adb1e "very") at list-objects.c:204
#76 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b1b8, base=0x151dde0) at list-objects.c:134
#77 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b1b8,
    base=0x151dde0, name=0x35adbce "ridicoulously") at list-objects.c:204
#78 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b180, base=0x151dde0) at list-objects.c:134
#79 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b180,
    base=0x151dde0, name=0x35ada3e "a") at list-objects.c:204
#80 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b148, base=0x151dde0) at list-objects.c:134
#81 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b148,
    base=0x151dde0, name=0x35ad97e "is") at list-objects.c:204
#82 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b110, base=0x151dde0) at list-objects.c:134
#83 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b110,
    base=0x151dde0, name=0x35adab9 "this") at list-objects.c:204
#84 0x00000000005620c9 in process_tree_contents (ctx=0x151de60,
    tree=0x356b0a0, base=0x151dde0) at list-objects.c:134
#85 0x000000000056243e in process_tree (ctx=0x151de60, tree=0x356b0a0,
    base=0x151dde0, name=0x6fe2ff <__ac_HASH_UPPER+1167> "")
    at list-objects.c:204
#86 0x0000000000562bd5 in traverse_trees_and_blobs (ctx=0x151de60,
    base=0x151dde0) at list-objects.c:349
#87 0x0000000000562da3 in do_traverse (ctx=0x151de60) at list-objects.c:391
#88 0x0000000000562e8c in traverse_commit_list_filtered (
    filter_options=0x76d080 <filter_options>, revs=0x151e300,
    show_commit=0x476a47 <show_commit>, show_object=0x476ad2 <show_object>,
    show_data=0x0, omitted=0x0) at list-objects.c:429
#89 0x00000000004779f3 in get_object_list (ac=5, av=0x35515c8)
    at builtin/pack-objects.c:3153
#90 0x0000000000478fc9 in cmd_pack_objects (argc=0, argv=0x35308e0,
    prefix=0x0) at builtin/pack-objects.c:3501
#91 0x0000000000402f30 in run_builtin (p=0x6a0718 <commands+1752>, argc=10,
    argv=0x35308e0) at git.c:445
#92 0x00000000004032cb in handle_builtin (argc=10, argv=0x35308e0)
    at git.c:674
#93 0x000000000040351a in run_argv (argcp=0x151fd80, argv=0x151fd88)
    at git.c:741
#94 0x0000000000403959 in cmd_main (argc=10, argv=0x35308e0) at git.c:872
#95 0x00000000004c4551 in main (argc=11, argv=0x35308d8) at common-main.c:52

Now, a really big question is: why does it do this? Because gc tries to repack, that's why, and while repacking, it is rather important to know whether to try delta compression or not.

Does it make sense to read the attributes from the worktree? That is a splendid question for the Git mailing list. From past conversations, I would expect the answer to be: yes (because you cannot determine the gitattributes from looking at the tree objects, as there is no information about a parent tree in them, so we have to look at the Git index anyway, falling back to the worktree if no .gitattributes file was found in the index).

FWIW I modified the proof-of-concept to auto-tweak the path:

#!/bin/sh

set -ex

rm -rf longpath toolongdirname

git init longpath
cd longpath

very_long=this/is/a/ridicoulously/very/long/path
len="$(printf "%s" "$(pwd -W)/$very_long" | wc -c)"
# Tweak it so that the path is 240 chars including work dir path
while test $(($len+5)) -lt 240
do
	very_long=${very_long%/very/*}/very/very/${very_long##*/very/}
	len=$(($len+5))
done

mkdir -p $very_long
>$very_long/filename.txt
git add -A
git commit -m "Added long filename"
>anormalfile.txt
git add anormalfile.txt
git commit -m "Added a normal file"
git rm $very_long/filename.txt
git commit -m "Removed long file"
cd ..
mkdir toolongdirname
cd toolongdirname
git clone ../longpath
cd longpath
GIT_TRACE=1 git gc

I also added that GIT_TRACE=1 to the git gc call so that you can see that the warning is generated by a git pack-objects process that is spawned from git gc.

@dscho
Copy link
Member

dscho commented Oct 15, 2021

Closing this stale ticket.

@dscho dscho closed this as completed Oct 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants