fatal: Authentication failed after upgrade git client to 2.19.2

[AsherKa]

We use TFS 2018.2 hosted on-prem. Since upgrading git client from 2.19.1 to 2.19.2 (which comes with GCM v1.18.3) authentication stopped working.

I've enabled GCM_TRACE and this is what happenes:

18:24:33.410919 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.3) 'get'
18:24:33.528913 ...\Git\Where.cs:348    trace: [FindGitInstallations] found 1 Git installation(s).
18:24:33.543902 ...Configuration.cs:222 trace: [LoadGitConfiguration] git All config read, 82 entries.
18:24:33.675950 ...\Common.cs:85        trace: [CreateAuthentication] detecting authority type for 'https://tfs2018.mydomain/'.
18:24:33.695461 ...\Common.cs:224       trace: [CreateAuthentication] authority for 'https://tfs2018.mydomain/' is basic with NTLM=Auto.
18:24:33.695961 ...\Common.cs:765       trace: [QueryCredentials] querying 'Auto' for credentials.
18:24:33.906993 ...uthentication.cs:119 trace: [AcquireCredentials] 'https://tfs2018.mydomain/' supports NTLM, sending NTLM credentials instead
18:24:33.906993 ...\Common.cs:780       trace: [QueryCredentials] credentials found.
18:24:34.441006 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.3) 'erase'
18:24:34.557518 ...\Git\Where.cs:348    trace: [FindGitInstallations] found 1 Git installation(s).
18:24:34.570519 ...Configuration.cs:222 trace: [LoadGitConfiguration] git All config read, 82 entries.
18:24:34.650529 ...\Common.cs:85        trace: [CreateAuthentication] detecting authority type for 'https://tfs2018.mydomain/'.
18:24:34.682540 ...\Common.cs:224       trace: [CreateAuthentication] authority for 'https://tfs2018.mydomain/' is basic with NTLM=Auto.
18:24:34.683530 ...\Common.cs:252       trace: [DeleteCredentials] deleting basic credentials for 'https://tfs2018.mydomain/'.
18:24:34.688543 ...aseSecureStore.cs:59 trace: [Delete] credentials not found for 'git:https://tfs2018.mydomain'.
fatal: Authentication failed for 'https://tfs2018.mydomain/DefaultCollection/PROJ/_git/REPO/'

I'm aware of the option to set the GCM_AUTHORITY to BASIC but for now we chose to downgrade our clients to 2.19.1 instead.

I was wondering if the issue is known and are there any plans to push a fix in the near future?

[loldot]

I'm experiencing the same issue. The trace output is identical to @AsherKa

[jeschu1]

@AsherKa can you provide the output you have on the downgraded, successful, version?

It would also be helpful if you can install GCM 1.18.3 on the downgraded version and run the same test. That way the GCM version is the same on both runs and the only difference is the new version of git.

I'm not able to repro locally but we're looking into this.

[ljani]

@jeschu1: Here you go:

myrepo ❯❯❯ git version
git version 2.19.1.windows.1
myrepo ❯❯❯ git credential-manager version
09:32:43.203041 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.0) 'version'
Git Credential Manager for Windows version 1.18.0
myrepo ❯❯❯ git pull
09:32:50.657398 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.0) 'get'
09:32:50.705435 ...\Git\Where.cs:348    trace: [FindGitInstallations] found 1 Git installation(s).
09:32:50.711399 ...Configuration.cs:222 trace: [LoadGitConfiguration] git All config read, 68 entries.
09:32:50.750399 ...\Common.cs:85        trace: [CreateAuthentication] detecting authority type for 'https://tfs.example/'.
09:32:51.007938 ...\Common.cs:224       trace: [CreateAuthentication] authority for 'https://tfs.example/' is basic with NTLM=Auto.
09:32:51.009139 ...\Common.cs:765       trace: [QueryCredentials] querying 'Auto' for credentials.
09:32:51.148676 ...uthentication.cs:119 trace: [AcquireCredentials] 'https://tfs.example/' supports NTLM, sending NTLM credentials instead
09:32:51.149642 ...\Common.cs:780       trace: [QueryCredentials] credentials found.
09:32:51.457879 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.0) 'store'
09:32:51.507719 ...\Git\Where.cs:348    trace: [FindGitInstallations] found 1 Git installation(s).
09:32:51.512719 ...Configuration.cs:222 trace: [LoadGitConfiguration] git All config read, 68 entries.
09:32:51.539713 ...\Common.cs:85        trace: [CreateAuthentication] detecting authority type for 'https://tfs.example/'.
09:32:51.787562 ...\Common.cs:224       trace: [CreateAuthentication] authority for 'https://tfs.example/' is basic with NTLM=Auto.
09:32:51.788553 ...\Program.cs:513      trace: [Store] storing basic credentials for 'https://tfs.example/'.
Already up to date.

myrepo ❯❯❯ git version
git version 2.19.1.windows.1
myrepo ❯❯❯ git credential-manager version
09:34:26.199611 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.3) 'version'
Git Credential Manager for Windows version 1.18.3
myrepo ❯❯❯ git pull
09:34:29.592092 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.3) 'get'
09:34:29.648761 ...\Git\Where.cs:348    trace: [FindGitInstallations] found 1 Git installation(s).
09:34:29.653716 ...Configuration.cs:222 trace: [LoadGitConfiguration] git All config read, 68 entries.
09:34:29.694764 ...\Common.cs:85        trace: [CreateAuthentication] detecting authority type for 'https://tfs.example/'.
09:34:29.704763 ...\Common.cs:224       trace: [CreateAuthentication] authority for 'https://tfs.example/' is basic with NTLM=Auto.
09:34:29.705717 ...\Common.cs:765       trace: [QueryCredentials] querying 'Auto' for credentials.
09:34:29.977458 ...uthentication.cs:119 trace: [AcquireCredentials] 'https://tfs.example/' supports NTLM, sending NTLM credentials instead
09:34:29.978424 ...\Common.cs:780       trace: [QueryCredentials] credentials found.
09:34:30.244361 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.3) 'store'
09:34:30.297361 ...\Git\Where.cs:348    trace: [FindGitInstallations] found 1 Git installation(s).
09:34:30.302384 ...Configuration.cs:222 trace: [LoadGitConfiguration] git All config read, 68 entries.
09:34:30.330076 ...\Common.cs:85        trace: [CreateAuthentication] detecting authority type for 'https://tfs.example/'.
09:34:30.343082 ...\Common.cs:224       trace: [CreateAuthentication] authority for 'https://tfs.example/' is basic with NTLM=Auto.
09:34:30.344041 ...\Program.cs:513      trace: [Store] storing basic credentials for 'https://tfs.example/'.
Already up to date.

myrepo ❯❯❯ git version
git version 2.19.2.windows.1
myrepo ❯❯❯ git credential-manager version
09:35:57.332337 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.3) 'version'
Git Credential Manager for Windows version 1.18.3
myrepo ❯❯❯ git pull
09:35:59.910337 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.3) 'get'
09:35:59.964341 ...\Git\Where.cs:348    trace: [FindGitInstallations] found 1 Git installation(s).
09:35:59.970340 ...Configuration.cs:222 trace: [LoadGitConfiguration] git All config read, 68 entries.
09:36:00.023336 ...\Common.cs:85        trace: [CreateAuthentication] detecting authority type for 'https://tfs.example/'.
09:36:00.036338 ...\Common.cs:224       trace: [CreateAuthentication] authority for 'https://tfs.example/' is basic with NTLM=Auto.
09:36:00.037342 ...\Common.cs:765       trace: [QueryCredentials] querying 'Auto' for credentials.
09:36:00.308338 ...uthentication.cs:119 trace: [AcquireCredentials] 'https://tfs.example/' supports NTLM, sending NTLM credentials instead
09:36:00.309338 ...\Common.cs:780       trace: [QueryCredentials] credentials found.
09:36:00.497337 ...\Common.cs:744       trace: [Main] git-credential-manager (v1.18.3) 'erase'
09:36:00.551337 ...\Git\Where.cs:348    trace: [FindGitInstallations] found 1 Git installation(s).
09:36:00.556337 ...Configuration.cs:222 trace: [LoadGitConfiguration] git All config read, 68 entries.
09:36:00.594339 ...\Common.cs:85        trace: [CreateAuthentication] detecting authority type for 'https://tfs.example/'.
09:36:00.608357 ...\Common.cs:224       trace: [CreateAuthentication] authority for 'https://tfs.example/' is basic with NTLM=Auto.
09:36:00.608357 ...\Common.cs:252       trace: [DeleteCredentials] deleting basic credentials for 'https://tfs.example/'.
09:36:00.642856 ...aseSecureStore.cs:68 trace: [Delete] credentials for 'git:https://tfs.example' deleted from store.
fatal: Authentication failed for 'https://tfs.example/Collection/_git/Project/'

EDIT: I've messed around with GCM_AUTHORITY earlier this year, but I think I managed to clear all of the credentials. Though I'm not the only one with the problem in our company, so I guess the output is valid.

Also git for windows v2.19.2 has been demoted to a pre-release in GitHub although https://git-scm.com/ is still offering it.

[dscho]

I do not see any relevant code changes in Git v2.19.2 relative to v2.19.1. There is however one more thing that changed between Git for Windows v2.19.1 and v2.19.2: the cURL library was updated (which is the component responsible for talking via HTTPS).

To test whether this is the culprit, could I ask you to test with a portable Git v2.19.2, replacing mingw64\bin\libcurl-4.dll (and if it exists, mingw64\libexec\git-core\libcurl-4.dll) with the version from the portable Git v2.19.1?

If that does not "fix" things, could I ask you to perform the testing after setting GIT_TRACE_CURL=1 (which will be quite verbose)?

[ljani]

@dscho Spot on. Replacing mingw64\bin\libcurl-4.dll and mingw64\libexec\git-core\libcurl-4.dll with the old versions, authentication succeeds. What would be the next steps?

[dscho]

@ljani if possible, could you get a GIT_TRACE_CURL trace of both a failing and a succeeding run to us?

[ljani]

@dscho There seems to be lots of data, which I'm not sure I can censor properly, but here are the main diffs I spotted:

v2.19.2 seems to be using http/2:

== Info: ALPN, offering h2
...
== Info: Using HTTP2, server supports multi-use
== Info: Connection state changed (HTTP/2 confirmed)
== Info: Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

When v2.19.2 tries to authenticate, the connection fails with HTTP_1_1_REQUIRED whereas v2.19.1 continues with business as usual:

== Info: Issue another request to this URL: 'https://tfs.example/Collection/_git/Project/info/refs?service=git-upload-pack'
== Info: Couldn't find host tfs.example in the _netrc file; using defaults
== Info: Found bundle for host tfs.example: 0x3386fc0 [can multiplex]
== Info: Re-using existing connection! (#0) with host tfs.example
== Info: Connected to tfs.example (127.0.0.1) port 443 (#0)
== Info: Server auth using NTLM with user ''
== Info: Using Stream ID: 5 (easy handle 0x3380af0)
=> Send SSL data, 0000000005 bytes (0x00000005)
=> Send SSL data: .....
=> Send header, 0000000275 bytes (0x00000113)
=> Send header: GET /Collection/_git/Project/info/refs?service=git-upload-pack HTTP/2
=> Send header: Host: tfs.example
=> Send header: Authorization: NTLM <redacted>
=> Send header: User-Agent: git/2.19.2.windows.1
=> Send header: Accept: */*
=> Send header: Accept-Encoding: deflate, gzip
=> Send header: Pragma: no-cache
=> Send header:
<= Recv SSL data, 0000000005 bytes (0x00000005)
<= Recv SSL data: ....%
== Info: HTTP/2 stream 0 was not closed cleanly: HTTP_1_1_REQUIRED (err 13)
== Info: stopped the pause stream!
== Info: Connection #0 to host tfs.example left intact
fatal: Authentication failed for 'https://tfs.example/Collection/_git/Project/'

[dscho]

Okay, good, now we are getting somewhere. There are experimental patches in flight where you can force this via the config setting http.version, e.g.

git config http.https://tfs.example/Collection/_git/Project/.version HTTP/1.1

This feature is not supported in any official version, and not even in a pre-release yet, though.

[dscho]

This feature is not supported in any official version, and not even in a pre-release yet, though.

I backported it to Git for Windows' master, and it will be part of Git for Windows v2.20.0 (expected early this coming week) as well as a patched cURL so that you do not even have to specify http.version = HTTP/1.1 (unless you want to).

[dscho]

@AsherKa thank you for reporting, and @ljani thank you so much for assisting with your excellent analysis. I asked my teammates, and we all agree that you helped us prevent a major problem for lots of customers. And we all agree that we (and especially the help desk) owe you at least a 🍺 (if you drink alcohol, that is).

[ljani]

@dscho No problem! Thank you and your teammates for your hard work!

I'm happy to test out a snapshot build, if you can point me to one. I saw no artifacts in the Azure Pipelines builds, but it wouldn't be the first time I skipped over something.

[dscho]

@ljani there are no snapshot artifacts yet ;-) But in a few minutes, there will hopefully be a new and shiny Git for Windows v2.20.0 for you to test out.

[ljani]

@dscho I'm happy to report v2.20.0.windows.1 is working very well for me!

[AsherKa]

@AsherKa thank you for reporting, and @ljani thank you so much for assisting with your excellent analysis. I asked my teammates, and we all agree that you helped us prevent a major problem for lots of customers. And we all agree that we (and especially the help desk) owe you at least a 🍺 (if you drink alcohol, that is).

@dscho @ljani Thank you, guys! Cheers! 🍺