Skip to content

feat(aarch64): add GDB debugging for Linux MSHV - #2052

Open
cshung wants to merge 3 commits into
hyperlight-dev:mainfrom
cshung:cshung/mshv-aarch64-gdb
Open

cshung wants to merge 3 commits into
hyperlight-dev:mainfrom
cshung:cshung/mshv-aarch64-gdb

Conversation

@cshung

@cshung cshung commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

ARM64 Linux MSHV GDB

MSHV guests support register and memory access, two hardware breakpoints,
BRK software breakpoints, and hardware single-step.
Guest exception vectors forward debug stops through HVC.
Debugger register access uses the saved exception context and active stack pointer.
Replacement-VM reset preserves debugger configuration.

The hypercall parser validates message type and payload length.
Native ARM64 MSHV GDB tests run in debug CI.
The default-driver build retains non-debug KVM use and rejects KVM debugging.
KVM debugger support is independent in #2016.

Commit layers

main
`-- D: MSHV backend, #1626
    `-- E: cherry-pick of shared GDB support A, #2050
        `-- F: MSHV-specific GDB support, this PR

Each layer is one signed commit. A and E have identical zero-context patch IDs.
WHP and KVM debugger support have independent branches.

MSHV GDB-only comparison.

The graph and update rules are in
docs/gdb-commit-layers.md.
The full PR diff includes unmerged prerequisites #1626 and #2050.

Validation

Native Azure ARM64 Linux/MSHV validation before this rebase:

  • Formatting, standard Clippy, MSHV-only GDB Clippy and host build pass.
  • The full just test recipe passes with 1,076 successful test executions.
  • All 26 debugger, backend and recovery tests pass with default and MSHV-only
    GDB features.
  • All seven real GDB client tests pass, including bootstrap stack preservation,
    software and hardware breakpoints, single-step, code writes and snapshots.
  • Debug and release Rust and C guest builds pass.
  • Native regression failures demonstrate rejected-resume handling, unbacked MMIO,
    listener setup, bootstrap stack corruption, stale exception-context selection
    and the extra instruction after an MMIO step.
  • MSHV backend code and stress limits are unchanged.
  • Windows formatting, Clippy, build and the full test recipe pass after the
    rebase. Linux and ARM64 cross-Clippy pass on all six layers.
  • The shared ARM64 test constructor passes the configuration required by
    upstream Sandbox::recreate.
  • Shared support passes Windows x64 Clippy, WHP ARM64 cross-Clippy, Linux
    just clippyw, and Linux x64 and ARM64 KVM Clippy.

Native ARM64 runtime tests were not repeated for this rebase.

The dedicated MSHV validation VM was last verified deallocated.

November 16 rebase validation

Base: 5fe5c8f8. Each layer is one signed commit. A and E are patch-equivalent.
The shared ARM64 test helpers use upstream's guest startup and memory-layout APIs.

  • All six layers pass formatting and Linux ARM64 cross-Clippy.
  • KVM-only and MSHV-only ARM64 GDB configurations pass cross-Clippy.
  • WHP passes Windows ARM64 cross-Clippy and Linux just clippyw.
  • The final MSHV GDB tip passes Windows x64 formatting, Clippy, build, the full
    just test recipe and three focused debugger tests. Rust and C guests are
    rebuilt in debug and release for the guest startup ABI.

Native ARM64 runtime results above predate this rebase. They were not repeated
against the guest startup ABI in this base.

@cshung cshung added the C-enhancement For PRs adding features, improving functionality, docs, tests, etc. label Nov 11, 2026
Copilot AI balanced review requested due to automatic review settings November 11, 2026 20:15
@cshung cshung added the S-needs-review PR is ready for (re-)review label Nov 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical debugging defects can strand the GDB listener, expose or modify the wrong ARM64 register context, and corrupt live bootstrap stack data.

6 open issues
What changed in this PR

This PR adds ARM64 Linux/MSHV guest debugging to Hyperlight. Its full scope includes the MSHV backend and shared debugger prerequisites from #1626 and #2050.

Changes:

  • Implements MSHV register access, breakpoints, single-step, exception forwarding, and debugger state preservation across VM reset.
  • Shares debugger handling across architectures, including page-sized memory access and snapshot recovery.
  • Adds native ARM64 MSHV CI, bounded test concurrency, debugger tests, and documentation.
File Description
typos.toml Allows an upstream MSHV identifier spelling.
src/​hyperlight_host/​tests/​sandbox_host_tests.rs Bounds ARM64 MSHV test concurrency.
src/​hyperlight_host/​tests/​integration_test.rs Reduces concurrent MSHV partitions in stress tests.
src/​hyperlight_host/​src/​sandbox/​uninitialized_evolve.rs Initializes guest debug vectors during setup.
src/​hyperlight_host/​src/​sandbox/​snapshot/​memory/​backing.rs Exposes backing ranges for debugger cache flushing.
src/​hyperlight_host/​src/​sandbox/​snapshot/​file/​config.rs Accepts Arm CPU implementers in Linux tests.
src/​hyperlight_host/​src/​sandbox/​initialized.rs Guards snapshots against breakpoints and tests recovery.
src/​hyperlight_host/​src/​mem/​mgr.rs Exposes host memory ranges to ARM64 debugging.
src/​hyperlight_host/​src/​mem/​memory_region.rs Decodes ARM64 MSHV memory access flags.
src/​hyperlight_host/​src/​mem/​exe.rs Carries debug-vector metadata.
src/​hyperlight_host/​src/​mem/​elf.rs Locates and relocates guest debug vectors.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​whp/​x86_64.rs Adapts the debugger translation interface.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​mshv/​x86_64.rs Adapts the debugger translation interface.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​mshv/​aarch64.rs Implements the ARM64 MSHV backend and debugger.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​mod.rs Adds ARM64 debug exit and setup error support.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​kvm/​x86_64.rs Adapts the debugger translation interface.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​kvm/​aarch64.rs Explicitly rejects unsupported KVM debugging.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​x86_64.rs Uses shared debugger handling and breakpoint storage.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​mod.rs Integrates ARM64 debug stops and cancellation handling.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​debug.rs Provides shared requests, memory access, and breakpoint handling.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​aarch64.rs Connects MSHV, debug vectors, and initial stops.
src/​hyperlight_host/​src/​hypervisor/​gdb/​target.rs Supports ARM64 register packets and target operations.
src/​hyperlight_host/​src/​hypervisor/​gdb/​mod.rs Adds page-table translation and instruction-cache flushing.
src/​hyperlight_host/​src/​hypervisor/​gdb/​event_loop.rs Handles ARM64 initial-stop notifications.
src/​hyperlight_host/​src/​hypervisor/​gdb/​arch/​x86_64.rs Isolates x86 breakpoint and stop classification.
src/​hyperlight_host/​src/​hypervisor/​gdb/​arch/​aarch64.rs Defines ARM64 BRK encoding and alignment checks.
src/​hyperlight_host/​src/​hypervisor/​gdb/​arch.rs Selects architecture-specific debugger helpers.
src/​hyperlight_host/​examples/​guest-debugging/​main.rs Adds ARM64 real-client debugger tests.
src/​hyperlight_host/​build.rs Enables debug-build GDB support for ARM64 Linux/MSHV.
src/​hyperlight_guest_bin/​src/​arch/​aarch64/​exception/​entry.rs Forwards synchronous exceptions through HVC.
src/​hyperlight_common/​src/​arch/​aarch64/​exn.rs Defines the shared debug HVC immediate.
Justfile Supports nextest selection and broader debugger test filters.
docs/​how-to-debug-a-hyperlight-guest.md Documents MSHV debugging and snapshot constraints.
docs/​gdb-commit-layers.md Documents shared and backend-specific commit dependencies.
CHANGELOG.md Records backend, debugging, and recovery support.
.github/​workflows/​ValidatePullRequest.yml Adds ARM64 MSHV debug and release jobs.
.github/​workflows/​dep_build_test.yml Configures native MSHV runners and debugger validation.
.config/​nextest.toml Limits concurrent partition-heavy tests.

Review effort: Balanced

@cshung
cshung force-pushed the cshung/mshv-aarch64-gdb branch 2 times, most recently from d38fe01 to 46beba7 Compare November 12, 2026 00:12
@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/mshv-aarch64-gdb branch from 46beba7 to 19159c6 Compare November 12, 2026 18:15
@hyperlight-gh-bot

This comment has been minimized.

@yoshuawuyts yoshuawuyts added S-nominated Item nominated for the next triage meeting O-aarch64 Specific to 64-bit Arm processors O-linux Linux host support O-mshv Linux MSHV backend for Microsoft Hyper-V labels Nov 16, 2026
cshung and others added 3 commits November 16, 2026 09:56
MSHV provides partition creation, memory mappings, MMIO dispatch, batched register access and replacement-VM reset on ARM64. Native CI uses tagged hierarchical-virtualization runners and bounded scheduling for the partition limit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>
Architecture adapters, software breakpoints, page-sized memory access and recovery handling are shared by backend layers. Guest debug vectors and common tests support ARM64. Backend layers control platform enablement. The commit-layer document defines dependencies and shared-update validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>
MSHV forwards guest debug exceptions through intercepted HVCs. Register access uses the saved exception context and active stack pointer. Two hardware breakpoints and single-step survive replacement-VM reset. Native ARM64 GDB tests run in debug CI. Non-debug KVM use is available with default features.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>
@cshung
cshung force-pushed the cshung/mshv-aarch64-gdb branch from 19159c6 to e1fef27 Compare November 16, 2026 18:16
@hyperlight-gh-bot

Copy link
Copy Markdown

Benchmark Results

Measured commit: e1fef2752ffa
Baseline commit: 5fe5c8f8c44f

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 760.06 ns (➖ 1.00x faster)
vec_bytes 633.59 ns (➖ 1.07x slower)
372.35 µs (➖ 1.00x faster)

payload_allocation

slot_pool_segmented
262144 522.56 ns (➖ 1.01x faster)
65536 142.02 ns (➖ 1.00x faster)

sandboxes

create_initialized_and_drop
medium 13.26 ms (➖ 1.01x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
7.69 ns (➖ 1.00x slower) 7.69 ns (➖ 1.00x faster) 7.72 ns (➖ 1.00x slower)

snapshot_files

load_snapshot_unverified
small 99.01 µs (➖ 1.01x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.34 µs (➖ 1.04x slower) 7.08 µs (➖ 1.00x slower)
65536 1.93 µs (➖ 1.09x faster) 2.04 µs (➖ 1.07x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 26.95 µs (➖ 1.02x faster)
65536 6.13 µs (➖ 1.08x faster) 6.21 µs (➖ 1.04x faster)
8192 1.02 µs (➖ 1.01x faster) 1.09 µs (➖ 1.08x slower)
kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 666.02 ns (➖ 1.01x faster)
vec_bytes 528.29 ns (➖ 1.01x slower)
660.00 µs (➖ 1.02x faster)

payload_allocation

slot_pool_segmented
262144 494.80 ns (➖ 1.09x faster)
65536 172.40 ns (➖ 1.01x slower)

sandboxes

create_initialized_and_drop
medium 14.49 ms (➖ 1.01x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
6.79 ns (➖ 1.00x slower) 6.82 ns (➖ 1.01x slower) 6.79 ns (➖ 1.01x faster)

snapshot_files

load_snapshot_unverified
small 48.72 µs (➖ 1.01x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.49 µs (➖ 1.00x faster) 7.47 µs (➖ 1.00x faster)
65536 2.11 µs (➖ 1.01x slower) 2.10 µs (➖ 1.01x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 28.69 µs (➖ 1.01x faster)
65536 7.04 µs (➖ 1.00x faster) 6.96 µs (➖ 1.01x faster)
8192 693.99 ns (➖ 1.00x faster) 708.94 ns (➖ 1.03x faster)
mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 885.34 ns (➖ 1.03x faster)
vec_bytes 782.15 ns (➖ 1.12x slower)
319.67 µs (➖ 1.01x slower)

payload_allocation

slot_pool_segmented
262144 752.53 ns (➖ 1.12x slower)
65536 186.38 ns (➖ 1.02x slower)

sandboxes

create_initialized_and_drop
medium 64.52 ms (➖ 1.03x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
9.87 ns (➖ 1.01x faster) 9.87 ns (➖ 1.00x faster) 10.15 ns (➖ 1.02x slower)

snapshot_files

load_snapshot_unverified
small 113.73 µs (➖ 1.06x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 9.53 µs (➖ 1.01x faster) 9.40 µs (➖ 1.01x slower)
65536 2.48 µs (➖ 1.01x slower) 2.62 µs (➖ 1.02x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 34.81 µs (➖ 1.02x faster)
65536 7.96 µs (➖ 1.02x faster) 8.12 µs (➖ 1.00x faster)
8192 1.42 µs (➖ 1.13x slower) 1.29 µs (➖ 1.00x slower)
mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 915.22 ns (➖ 1.02x faster)
vec_bytes 656.46 ns (➖ 1.01x faster)
777.98 µs (➖ 1.00x faster)

payload_allocation

slot_pool_segmented
262144 659.45 ns (➖ 1.00x slower)
65536 272.70 ns (➖ 1.03x slower)

sandboxes

create_initialized_and_drop
medium 81.40 ms (➖ 1.10x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
8.69 ns (➖ 1.01x faster) 8.69 ns (➖ 1.00x faster) 8.72 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 54.58 µs (➖ 1.01x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.90 µs (➖ 1.01x faster) 7.93 µs (➖ 1.00x faster)
65536 2.25 µs (➖ 1.01x faster) 2.29 µs (➖ 1.01x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 37.60 µs (➖ 1.06x faster)
65536 7.45 µs (➖ 1.01x faster) 7.51 µs (➖ 1.01x faster)
8192 892.87 ns (➖ 1.03x faster) 898.83 ns (➖ 1.08x faster)
hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.20 µs (➖ 1.01x slower)
vec_bytes 818.61 ns (➖ 1.07x slower)
2.00 ms (➖ 1.00x slower)

payload_allocation

slot_pool_segmented
262144 787.76 ns (➖ 1.00x faster)
65536 231.62 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 12.56 ms (➖ 1.02x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
10.02 ns (➖ 1.05x faster) 10.60 ns (➖ 1.06x faster) 9.91 ns (➖ 1.35x faster)

snapshot_files

load_snapshot_unverified
small 794.11 µs (➖ 1.03x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 9.21 µs (➖ 1.00x slower) 9.17 µs (➖ 1.01x slower)
65536 2.34 µs (➖ 1.01x faster) 2.31 µs (➖ 1.03x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 39.29 µs (➖ 1.03x slower)
65536 8.65 µs (➖ 1.10x faster) 8.85 µs (➖ 1.03x slower)
8192 1.29 µs (➖ 1.04x faster) 1.28 µs (➖ 1.03x faster)
hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.21 µs (➖ 1.01x slower)
vec_bytes 793.78 ns (➖ 1.01x slower)
3.11 ms (➖ 1.00x slower)

payload_allocation

slot_pool_segmented
262144 761.03 ns (➖ 1.02x slower)
65536 210.90 ns (➖ 1.00x faster)

sandboxes

create_initialized_and_drop
medium 13.50 ms (➖ 1.12x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
10.05 ns (➖ 1.03x slower) 10.11 ns (➖ 1.07x faster) 10.15 ns (➖ 1.06x slower)

snapshot_files

load_snapshot_unverified
small 689.22 µs (➖ 1.01x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.76 µs (➖ 1.01x slower) 7.48 µs (➖ 1.03x faster)
65536 2.99 µs (➖ 1.29x slower) 2.27 µs (➖ 1.02x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 41.31 µs (➖ 1.00x faster)
65536 8.94 µs (➖ 1.06x slower) 8.47 µs (➖ 1.02x faster)
8192 1.06 µs (➖ 1.13x faster) 1.12 µs (➖ 1.04x slower)

Reported by cargo ci bench-report --candidate run:43927113336 --config-file bench_report.toml.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C-enhancement For PRs adding features, improving functionality, docs, tests, etc. O-aarch64 Specific to 64-bit Arm processors O-linux Linux host support O-mshv Linux MSHV backend for Microsoft Hyper-V S-needs-review PR is ready for (re-)review S-nominated Item nominated for the next triage meeting

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants