Investigate CI hangs in t3701-add-interactive.sh #4422
Comments
|
Hmm. While working on some massively-parallel build, I noticed that the output is pretty much garbled at times. I am very, very close to just re-disabling the apparently thoroughly-problematic pseudo console support again, by cherry-picking msys2/msys2-runtime@b74f834 |
|
The problem seems to be in t3701.58 "handle very large filtered diff". The |
|
Sadly, the same hang occurs with MSYS2's build of the MSYS2 runtime v3.4.6. |
|
I am in the middle of bisecting this (fun, fun...) and it seems that something between v3.3.6 and v3.4.0 introduced the bug. |
|
So this bisecting fun is a bit involved. Here are the reasons why:
I am finally at a point where I left
Based on past experience, I strongly suspect cygwin/cygwin@90788821b7 (with the same authorship as so many pseudo console commits that I battled in the past). |
Indeed it is. Reverting it on top of Git for Windows' current MSYS2 runtime "fixes" the issue. Now, what does this re-break if we revert that commit? Let's dig some. Apparently there was some conversation around merging a branch with the buggy commit as its tip commit: https://inbox.sourceware.org/cygwin-developers/[email protected]/. This was the culmination, apparently, of a thread that started on a different mailing list: https://inbox.sourceware.org/cygwin/CAEv6GOB8PXHgHoz7hdJy6Bia2GWEmUDnTd252gTGinz2vuv=hA@mail.gmail.com/. And that bug report was in response to none other than me myself identifying this as a bug in Cygwin: git-ecosystem/git-credential-manager#509 (comment). But that bug was in the scenario where a C# program (Git Credential Manager) talked to a Cygwin Plus, the suggested work-around to use So, under the assumption that we do not really need to bother with supporting GCM to talk to an MSYS And who knows? Maybe this could be the workaround we need to fix the mysterious hangs on Windows/ARM64? |
This reverts the problematic commit that causes a hang when `git add -p` has `cat.exe` configured as a diff filter, and that diff filter is fed input larger than the pipe's buffer size. This fixes git-for-windows/git#4422. Signed-off-by: Johannes Schindelin <[email protected]>
…rtup." When Git for Windows upgraded its MSYS2 runtime from v3.3.* to v3.4.* at long last, the t3701-add-interactive test of Git's test suite started to hang consistently, timing out. The culprit is a `git add -p` call that wants to read from a diff filter. This diff filter is `cat.exe`, i.e. nothing special, but that diff filter gets input that is larger than the pipe buffer, and therefore must not block. Yet that is exactly what it does. This was a regression that introduced by upgrading from MSYS2 runtime v3.3.*. After a tedious, multi-day bisection session, the commit introducing that regression was identified as 9078882 (Cygwin: pipe: Restore blocking mode for cygwin process at startup., 2021-11-17), which we hereby revert. So what bug does this reversion reintroduce? The commit seems to have been in response to https://inbox.sourceware.org/cygwin/CAEv6GOB8PXHgHoz7hdJy6Bia2GWEmUDnTd252gTGinz2vuv=hA@mail.gmail.com/ which reported a bug when a C# program writes 0-byte content to a Cygwin pipe. (Irony of ironies, this report originated from Git's realm, over at git-ecosystem/git-credential-manager#509.) The analysis revealed, back in December '21, that a `CYGWIN=pipe_byte` would work around the bug, too, but that did not fix the regression in Git's test suite. That leaves us with the somewhat unsatisfying conclusion that we _might_ reintroduce a regression when Git Credential Manager tries to talk to an _MSYS_ `git.exe`. But since we do not need that in Git for Windows, and since we need to have CI builds that do not time out after 6h causing failures, it is better to revert that commit, and in the hopefully unlikely event that this causes _other_ problems in Git for Windows' ecosystem, we will simply have to revisit this issue in more depth. This fixes git-for-windows/git#4422. Signed-off-by: Johannes Schindelin <[email protected]>
…rtup." When Git for Windows upgraded its MSYS2 runtime from v3.3.* to v3.4.* at long last, the t3701-add-interactive test of Git's test suite started to hang consistently, timing out. The culprit is a `git add -p` call that wants to read from a diff filter. This diff filter is `cat.exe`, i.e. nothing special, but that diff filter gets input that is larger than the pipe buffer, and therefore must not block. Yet that is exactly what it does. This was a regression that introduced by upgrading from MSYS2 runtime v3.3.*. After a tedious, multi-day bisection session, the commit introducing that regression was identified as 9078882 (Cygwin: pipe: Restore blocking mode for cygwin process at startup., 2021-11-17), which we hereby revert. So what bug does this reversion reintroduce? The commit seems to have been in response to https://inbox.sourceware.org/cygwin/CAEv6GOB8PXHgHoz7hdJy6Bia2GWEmUDnTd252gTGinz2vuv=hA@mail.gmail.com/ which reported a bug when a C# program writes 0-byte content to a Cygwin pipe. (Irony of ironies, this report originated from Git's realm, over at git-ecosystem/git-credential-manager#509.) The analysis revealed, back in December '21, that a `CYGWIN=pipe_byte` would work around the bug, too, but that did not fix the regression in Git's test suite. That leaves us with the somewhat unsatisfying conclusion that we _might_ reintroduce a regression when Git Credential Manager tries to talk to an _MSYS_ `git.exe`. But since we do not need that in Git for Windows, and since we need to have CI builds that do not time out after 6h causing failures, it is better to revert that commit, and in the hopefully unlikely event that this causes _other_ problems in Git for Windows' ecosystem, we will simply have to revisit this issue in more depth. This fixes git-for-windows/git#4422. Signed-off-by: Johannes Schindelin <[email protected]>
…rtup." When Git for Windows upgraded its MSYS2 runtime from v3.3.* to v3.4.* at long last, the t3701-add-interactive test of Git's test suite started to hang consistently, timing out. The culprit is a `git add -p` call that wants to read from a diff filter. This diff filter is `cat.exe`, i.e. nothing special, but that diff filter gets input that is larger than the pipe buffer, and therefore must not block. Yet that is exactly what it does. This was a regression that introduced by upgrading from MSYS2 runtime v3.3.*. After a tedious, multi-day bisection session, the commit introducing that regression was identified as 9078882 (Cygwin: pipe: Restore blocking mode for cygwin process at startup., 2021-11-17), which we hereby revert. So what bug does this reversion reintroduce? The commit seems to have been in response to https://inbox.sourceware.org/cygwin/CAEv6GOB8PXHgHoz7hdJy6Bia2GWEmUDnTd252gTGinz2vuv=hA@mail.gmail.com/ which reported a bug when a C# program writes 0-byte content to a Cygwin pipe. (Irony of ironies, this report originated from Git's realm, over at git-ecosystem/git-credential-manager#509.) The analysis revealed, back in December '21, that a `CYGWIN=pipe_byte` would work around the bug, too, but that did not fix the regression in Git's test suite. That leaves us with the somewhat unsatisfying conclusion that we _might_ reintroduce a regression when Git Credential Manager tries to talk to an _MSYS_ `git.exe`. But since we do not need that in Git for Windows, and since we need to have CI builds that do not time out after 6h causing failures, it is better to revert that commit, and in the hopefully unlikely event that this causes _other_ problems in Git for Windows' ecosystem, we will simply have to revisit this issue in more depth. This fixes git-for-windows/git#4422. Signed-off-by: Johannes Schindelin <[email protected]>
…rtup." When Git for Windows upgraded its MSYS2 runtime from v3.3.* to v3.4.* at long last, the t3701-add-interactive test of Git's test suite started to hang consistently, timing out. The culprit is a `git add -p` call that wants to read from a diff filter. This diff filter is `cat.exe`, i.e. nothing special, but that diff filter gets input that is larger than the pipe buffer, and therefore must not block. Yet that is exactly what it does. This was a regression that introduced by upgrading from MSYS2 runtime v3.3.*. After a tedious, multi-day bisection session, the commit introducing that regression was identified as 9078882 (Cygwin: pipe: Restore blocking mode for cygwin process at startup., 2021-11-17), which we hereby revert. So what bug does this reversion reintroduce? The commit seems to have been in response to https://inbox.sourceware.org/cygwin/CAEv6GOB8PXHgHoz7hdJy6Bia2GWEmUDnTd252gTGinz2vuv=hA@mail.gmail.com/ which reported a bug when a C# program writes 0-byte content to a Cygwin pipe. (Irony of ironies, this report originated from Git's realm, over at git-ecosystem/git-credential-manager#509.) The analysis revealed, back in December '21, that a `CYGWIN=pipe_byte` would work around the bug, too, but that did not fix the regression in Git's test suite. That leaves us with the somewhat unsatisfying conclusion that we _might_ reintroduce a regression when Git Credential Manager tries to talk to an _MSYS_ `git.exe`. But since we do not need that in Git for Windows, and since we need to have CI builds that do not time out after 6h causing failures, it is better to revert that commit, and in the hopefully unlikely event that this causes _other_ problems in Git for Windows' ecosystem, we will simply have to revisit this issue in more depth. This fixes git-for-windows/git#4422. Signed-off-by: Johannes Schindelin <[email protected]>
…rtup." When Git for Windows upgraded its MSYS2 runtime from v3.3.* to v3.4.* at long last, the t3701-add-interactive test of Git's test suite started to hang consistently, timing out. The culprit is a `git add -p` call that wants to read from a diff filter. This diff filter is `cat.exe`, i.e. nothing special, but that diff filter gets input that is larger than the pipe buffer, and therefore must not block. Yet that is exactly what it does. This was a regression that introduced by upgrading from MSYS2 runtime v3.3.*. After a tedious, multi-day bisection session, the commit introducing that regression was identified as 9078882 (Cygwin: pipe: Restore blocking mode for cygwin process at startup., 2021-11-17), which we hereby revert. So what bug does this reversion reintroduce? The commit seems to have been in response to https://inbox.sourceware.org/cygwin/CAEv6GOB8PXHgHoz7hdJy6Bia2GWEmUDnTd252gTGinz2vuv=hA@mail.gmail.com/ which reported a bug when a C# program writes 0-byte content to a Cygwin pipe. (Irony of ironies, this report originated from Git's realm, over at git-ecosystem/git-credential-manager#509.) The analysis revealed, back in December '21, that a `CYGWIN=pipe_byte` would work around the bug, too, but that did not fix the regression in Git's test suite. That leaves us with the somewhat unsatisfying conclusion that we _might_ reintroduce a regression when Git Credential Manager tries to talk to an _MSYS_ `git.exe`. But since we do not need that in Git for Windows, and since we need to have CI builds that do not time out after 6h causing failures, it is better to revert that commit, and in the hopefully unlikely event that this causes _other_ problems in Git for Windows' ecosystem, we will simply have to revisit this issue in more depth. This fixes git-for-windows/git#4422. Signed-off-by: Johannes Schindelin <[email protected]>
…rtup." When Git for Windows upgraded its MSYS2 runtime from v3.3.* to v3.4.* at long last, the t3701-add-interactive test of Git's test suite started to hang consistently, timing out. The culprit is a `git add -p` call that wants to read from a diff filter. This diff filter is `cat.exe`, i.e. nothing special, but that diff filter gets input that is larger than the pipe buffer, and therefore must not block. Yet that is exactly what it does. This was a regression that introduced by upgrading from MSYS2 runtime v3.3.*. After a tedious, multi-day bisection session, the commit introducing that regression was identified as 9078882 (Cygwin: pipe: Restore blocking mode for cygwin process at startup., 2021-11-17), which we hereby revert. So what bug does this reversion reintroduce? The commit seems to have been in response to https://inbox.sourceware.org/cygwin/CAEv6GOB8PXHgHoz7hdJy6Bia2GWEmUDnTd252gTGinz2vuv=hA@mail.gmail.com/ which reported a bug when a C# program writes 0-byte content to a Cygwin pipe. (Irony of ironies, this report originated from Git's realm, over at git-ecosystem/git-credential-manager#509.) The analysis revealed, back in December '21, that a `CYGWIN=pipe_byte` would work around the bug, too, but that did not fix the regression in Git's test suite. That leaves us with the somewhat unsatisfying conclusion that we _might_ reintroduce a regression when Git Credential Manager tries to talk to an _MSYS_ `git.exe`. But since we do not need that in Git for Windows, and since we need to have CI builds that do not time out after 6h causing failures, it is better to revert that commit, and in the hopefully unlikely event that this causes _other_ problems in Git for Windows' ecosystem, we will simply have to revisit this issue in more depth. This fixes git-for-windows/git#4422. Signed-off-by: Johannes Schindelin <[email protected]>
…rtup." When Git for Windows upgraded its MSYS2 runtime from v3.3.* to v3.4.* at long last, the t3701-add-interactive test of Git's test suite started to hang consistently, timing out. The culprit is a `git add -p` call that wants to read from a diff filter. This diff filter is `cat.exe`, i.e. nothing special, but that diff filter gets input that is larger than the pipe buffer, and therefore must not block. Yet that is exactly what it does. This was a regression that introduced by upgrading from MSYS2 runtime v3.3.*. After a tedious, multi-day bisection session, the commit introducing that regression was identified as 9078882 (Cygwin: pipe: Restore blocking mode for cygwin process at startup., 2021-11-17), which we hereby revert. So what bug does this reversion reintroduce? The commit seems to have been in response to https://inbox.sourceware.org/cygwin/CAEv6GOB8PXHgHoz7hdJy6Bia2GWEmUDnTd252gTGinz2vuv=hA@mail.gmail.com/ which reported a bug when a C# program writes 0-byte content to a Cygwin pipe. (Irony of ironies, this report originated from Git's realm, over at git-ecosystem/git-credential-manager#509.) The analysis revealed, back in December '21, that a `CYGWIN=pipe_byte` would work around the bug, too, but that did not fix the regression in Git's test suite. That leaves us with the somewhat unsatisfying conclusion that we _might_ reintroduce a regression when Git Credential Manager tries to talk to an _MSYS_ `git.exe`. But since we do not need that in Git for Windows, and since we need to have CI builds that do not time out after 6h causing failures, it is better to revert that commit, and in the hopefully unlikely event that this causes _other_ problems in Git for Windows' ecosystem, we will simply have to revisit this issue in more depth. This fixes git-for-windows/git#4422. Signed-off-by: Johannes Schindelin <[email protected]>
…rtup." When Git for Windows upgraded its MSYS2 runtime from v3.3.* to v3.4.* at long last, the t3701-add-interactive test of Git's test suite started to hang consistently, timing out. The culprit is a `git add -p` call that wants to read from a diff filter. This diff filter is `cat.exe`, i.e. nothing special, but that diff filter gets input that is larger than the pipe buffer, and therefore must not block. Yet that is exactly what it does. This was a regression that introduced by upgrading from MSYS2 runtime v3.3.*. After a tedious, multi-day bisection session, the commit introducing that regression was identified as 9078882 (Cygwin: pipe: Restore blocking mode for cygwin process at startup., 2021-11-17), which we hereby revert. So what bug does this reversion reintroduce? The commit seems to have been in response to https://inbox.sourceware.org/cygwin/CAEv6GOB8PXHgHoz7hdJy6Bia2GWEmUDnTd252gTGinz2vuv=hA@mail.gmail.com/ which reported a bug when a C# program writes 0-byte content to a Cygwin pipe. (Irony of ironies, this report originated from Git's realm, over at git-ecosystem/git-credential-manager#509.) The analysis revealed, back in December '21, that a `CYGWIN=pipe_byte` would work around the bug, too, but that did not fix the regression in Git's test suite. That leaves us with the somewhat unsatisfying conclusion that we _might_ reintroduce a regression when Git Credential Manager tries to talk to an _MSYS_ `git.exe`. But since we do not need that in Git for Windows, and since we need to have CI builds that do not time out after 6h causing failures, it is better to revert that commit, and in the hopefully unlikely event that this causes _other_ problems in Git for Windows' ecosystem, we will simply have to revisit this issue in more depth. This fixes git-for-windows/git#4422. Signed-off-by: Johannes Schindelin <[email protected]>
In https://github.com/git-for-windows/git-sdk-64/actions/runs/4965633499/jobs/8886995022, the
test-minimal-sdk (5)job timed out after six hours. The only test that is not reported as having finished is:t3701-add-interactive.sh.This most likely has something to do with the 64-bit Git for Windows SDK having been updated from MSYS2 runtime v3.3.6 to v3.4.6. It is quite possible that the "hang" is merely an interactive
addwaiting for user input, caused by the MSYS2 runtime now defaulting toenable_pcon.The text was updated successfully, but these errors were encountered: