<mutex>: recursive_mutex::lock misbehaves in debug builds when the ownership level is extremely large

[frederick-vs-ja]

Describe the bug
recursive_mutex::lock behaves incorrectly when the ownership level is extremely large (e.g., when larger than 0x100000000).

In debug visualizer, ownership_levels can become a negative value, and then a small positive value (i.e. can wrap).

recursive_mutex::try_lock probably has a similar issue.

Command-line test case

C:\repro>type repro.cpp
#include <cassert>
#include <iostream>
#include <mutex>

int main() {
    std::recursive_mutex m;
    unsigned long long count = 0ull;
    try {
        while (count != 0x1'0000'0010ull) {
            m.lock();
            ++count;
            if (count % 0x2000'0000ull == 0x10ull) {
                std::cout << std::hex << count << '\n';
            }
        }
    } catch (std::system_error& e) {
        std::cout << e.what() << '\n';
        // assert(e.code().value() != static_cast<int>(std::errc::device_or_resource_busy)); // LWG-2309
    } catch (...) {
        assert(false);
    }
    while (count > 0) {
        if (count % 0x2000'0000ull == 0x10ull) {
            std::cout << std::hex << count << '\n';
        }
        --count;
        m.unlock();
    }
}

C:\reprocl.exe /std:c++latest /MTd /EHsc /W4 /WX .\repro.cpp
用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.34.31937 版
版权所有(C) Microsoft Corporation。保留所有权利。

/std:c++latest 作为最新的 C++
working 草稿中的语言功能预览提供。我们希望你提供有关 bug 和改进建议的反馈。
但是，请注意，这些功能按原样提供，没有支持，并且会随着工作草稿的变化
而更改或移除。有关详细信息，请参阅
https://go.microsoft.com/fwlink/?linkid=2045807。

repro.cpp
Microsoft (R) Incremental Linker Version 14.34.31937.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:repro.exe
repro.obj

C:\repro>repro.exe
10
20000010
40000010
60000010
80000010
a0000010
c0000010
e0000010
100000010
100000010
D:\a\_work\1\s\src\vctools\crt\github\stl\src\mutex.cpp(163): unlock of unowned mutex

And an assertion failure dialog is shown.

Expected behavior
An exception of types std::system_error is thrown ([thread.mutex.recursive]/3), and e.what() is printed. And then count is decreased to zero, with some values printed.

STL version
Microsoft Visual Studio Community 2022 (64 位) - Current 版本 17.4.4

Additional context
The root cause is probably that when _Mtx_recursive is set in type (which is the case for recursive_mutex), the lock operation never reports failure, and count is still increased even if it's equal to INT_MAX (technically UB, wraps currently).

STL/stl/src/mutex.cpp

Lines 97 to 103 in 9ae1b3f

	int res = WAIT_TIMEOUT;
	if (target == nullptr) { // no target --> plain wait (i.e. infinite timeout)
	if (mtx->thread_id != static_cast<long>(GetCurrentThreadId())) {
	mtx->_get_cs()->lock();
	}
	res = WAIT_OBJECT_0;

STL/stl/src/mutex.cpp

Lines 133 to 142 in 9ae1b3f

	if (res == WAIT_OBJECT_0 || res == WAIT_ABANDONED) {
	if (1 < ++mtx->count) { // check count
	if ((mtx->type & _Mtx_recursive) != _Mtx_recursive) { // not recursive, fixup count
	--mtx->count;
	res = WAIT_TIMEOUT;
	}
	} else {
	mtx->thread_id = static_cast<long>(GetCurrentThreadId());
	}
	}