Reverse PInvokes do not call TraceCall on non-x86 platforms

[davmason]

Description

Filed after discussion in #46238 (comment)

There is no call to Debugger::TraceCall for reverse pinvokes on non-x86 platforms. This means when doing mixed mode debugging on x64, if a breakpoint is set in a reverse pinvoked function as the first call on a new thread the breakpoint will not be hit.

Example program:

using System;
using System.Runtime.InteropServices;
using System.Threading;

namespace ReversePInvokeBP
{
    class Program
    {
        [DllImport("PInvokeDLL.dll")]
        private static extern unsafe void PassReversePInvokeAddr(delegate* unmanaged<void> pCalInfoEnumProcExEx);

        [UnmanagedCallersOnly]
        private static void CallMe()
        {
            Console.WriteLine("Here!");
        }

        static unsafe void Main(string[] args)
        {
            PassReversePInvokeAddr(&CallMe);

            Thread.Sleep(Timeout.Infinite);
        }
    }
}

#include "pch.h"
#include <thread>

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

typedef void (*myFunc)(void);

VOID ThreadFunc(void* addr)
{
    myFunc target = (myFunc)addr;
    printf("Calling managed function...\n");
    target();
}

VOID APIENTRY PassReversePInvokeAddr(void *addr)
{
    std::thread thread(ThreadFunc, addr);
    thread.join();
}

On x86 a breakpoint in CallMe will be hit in mixed mode debugging, on x64 it will not. Managed debugging hits the breakpoint for both x86 and x64. Similarly, if the reverse pinvoke is done on the current managed thread it will work for both x86 and x64 even with mixed mode debugging.

[ghost]

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Filed after discussion in #46238 (comment)

There is no call to Debugger::TraceCall for reverse pinvokes on non-x86 platforms. This means when doing mixed mode debugging on x64, if a breakpoint is set in a reverse pinvoked function as the first call on a new thread the breakpoint will not be hit.

Example program:

using System;
using System.Runtime.InteropServices;
using System.Threading;

namespace ReversePInvokeBP
{
    class Program
    {
        [DllImport("PInvokeDLL.dll")]
        private static extern unsafe void PassReversePInvokeAddr(delegate* unmanaged<void> pCalInfoEnumProcExEx);

        [UnmanagedCallersOnly]
        private static void CallMe()
        {
            Console.WriteLine("Here!");
        }

        static unsafe void Main(string[] args)
        {
            PassReversePInvokeAddr(&CallMe);

            Thread.Sleep(Timeout.Infinite);
        }
    }
}

#include "pch.h"
#include <thread>

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

typedef void (*myFunc)(void);

VOID ThreadFunc(void* addr)
{
    myFunc target = (myFunc)addr;
    printf("Calling managed function...\n");
    target();
}

VOID APIENTRY PassReversePInvokeAddr(void *addr)
{
    std::thread thread(ThreadFunc, addr);
    thread.join();
}

On x86 a breakpoint in CallMe will be hit in mixed mode debugging, on x64 it will not. Managed debugging hits the breakpoint for both x86 and x64. Similarly, if the reverse pinvoke is done on the current managed thread it will work for both x86 and x64 even with mixed mode debugging.

Author:	davmason
Assignees:	-
Labels:	area-Diagnostics-coreclr
Milestone:	-

[AaronViviano]

I believe I am also seeing this exact issue in my new .Net 5 x64 project. I have a thread in my C++ code that sends data back to my .Net UI. I can't debug the reverse Pinvoked .Net method.

[jkoritzinsky]

Yes, this should be fixed in .NET 6

[jkoritzinsky]

Fixed by #46238

[AaronViviano]

@jkoritzinsky Great to hear. Thank you.