[release/10.0] Fix InlinedCallFrameMarker bit-0 check on 64-bit platforms in DAC stack walk

[jkotas]

Fixes Issue #131606

main PR N/A (targeted servicing fix; the full fix for main is in #131642 / #131654, which are too invasive for a servicing branch)

Description

On 64-bit platforms, an unmanaged calli target can be encoded into InlinedCallFrame::m_Datum as (target << 1) | 1. This left-shift moves bit 0 of the target address into bit 1 of m_Datum. Two sites in src/coreclr/debug/daccess/dacdbiimplstackwalk.cpp (GetCountOfInternalFrames and EnumerateInternalFrames) test bit 1 of m_Datum against InlinedCallFrameMarker::ExceptionHandlingHelper without first checking bit 0 to confirm the field actually holds a MethodDesc* (with the marker bit set) rather than a shifted-and-tagged raw calli target. As a result, when the linker happens to place the native calli target at an odd address, the shifted-in low bit is misread as the EH-helper marker, and the frame is incorrectly skipped during stack walking.

This PR adds a TARGET_64BIT-guarded check for bit 0 before interpreting bit 1 as the InlinedCallFrameMarker::ExceptionHandlingHelper marker at both affected sites, matching the precedent already established in InlinedCallFrame::GetFunction_Impl (src/coreclr/vm/frames.h), which masks out the marker bits before treating m_Datum as a pointer.

Customer Impact

Visual Studio's mixed-mode debugger fails to show native C++ call stack frames when a C++/CLI layer calls native code through an unmanaged calli whose target address happens to be odd (a common occurrence with delay-loaded imports and incrementally-linked thunks). This makes debugging native code called from managed C++/CLI unreliable and appears intermittent/random to customers, since it depends on a single bit of a linker-chosen address.

Regression

Yes. This is a regression from .NET 8, introduced by the exception handling rewrite.

Testing

Verified the corrected bit-check logic in isolation (bit 0 check first, then bit 1), confirming that on 64-bit an odd-tagged calli target (bit0=1, bit1=1) is no longer misidentified as the EH-helper marker, while behavior for actual EH-helper-marked frames (bit0=0, bit1=1) and for 32-bit platforms (unaffected, no shift-tagging) is unchanged.

Risk

Low. The change is narrowly scoped to two conditional checks in the DAC stack-walk code, guarded by TARGET_64BIT, and does not alter behavior for 32-bit platforms or for frames that are genuinely marked as EH helpers.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @tommcdon, @dotnet/dotnet-diag
See info in area-owners.md if you want to be subscribed.