Fix v3 native CI shutdown and FreeBSD evidence transport

.github/workflows/v3-package-validation.yml

@@ -155,6 +155,7 @@ jobs:
       - name: freebsd_instance
         uses: vmactions/freebsd-vm@c9f815bc7aa0d34c9fdd0619b034a32d6ca7b57e
         with:
+          copyback: 'false'
           release: '14.3'
           mem: 8192
           cpu: 2

.github/workflows/v3-release-acceptance.yml

@@ -289,6 +289,7 @@ jobs:
         cp -R "$RUNNER_TEMP/selected" build/ci-selected-input
     - uses: vmactions/freebsd-vm@c9f815bc7aa0d34c9fdd0619b034a32d6ca7b57e
       with:
+        copyback: 'false'
         release: '14.3'
         mem: 8192
         cpu: 2

examples/host_poll.hpp

@@ -41,6 +41,7 @@
 #include <cstdint>
 #include <optional>
 #include <stdexcept>
+#include <string>
 #include <vector>
 
 #include <httpserver/server/readiness.hpp>
@@ -99,7 +100,10 @@ inline std::vector<srv::readiness_event> wait(
     for (const auto& interest : interests) fds.push_back(registration(interest));
 #if defined(_WIN32)
     const auto ready = ::WSAPoll(fds.data(), static_cast<ULONG>(fds.size()), timeout(deadline));
-    if (ready < 0 && ::WSAGetLastError() != WSAEINTR) throw std::runtime_error("WSAPoll failed");
+    if (ready < 0) {
+        const int error = ::WSAGetLastError();
+        if (error != WSAEINTR) throw std::runtime_error("WSAPoll failed (WSA error " + std::to_string(error) + ")");
+    }
 #else
     const auto ready = ::poll(fds.data(), static_cast<nfds_t>(fds.size()), timeout(deadline));
     if (ready < 0 && errno != EINTR) throw std::runtime_error("poll failed");

specs/tasks/FEATURE-ci-v3-runtime-green.md

@@ -0,0 +1,64 @@
+# FEATURE-ci-v3-runtime-green
+
+Status: Stages 1 and 3 implemented with focused proof; complete CI and remaining supported runtime repairs are pending. Groundwork validation is explicitly prohibited until FULL CI IS GREEN.
+Base: `v3` at `8f3083977f49706e66cefbb3ee8b6f6685469719`.
+Working branch: `task/FEATURE-ci-v3-runtime-green` (retain this one worktree through CI).
+Plan: primary checkout `.groundwork-plans/FEATURE-ci-v3-runtime-green-plan.md`.
+Journal: primary checkout `.debug/v3-ci-green-VSvXxx/journal.md`.
+Stage evidence: `.debug/v3-ci-green-VSvXxx/runtime-green-stage1/`.
+
+## Stage 1: default Windows timer shutdown
+
+- [x] Compare cached shutdown detection with event substitution and unconditional singleton leakage.
+- [x] Preserve actual native62 RED and reliable LLDB unwind; do not relaunch original captured suites.
+- [x] Add and observe a failing bounded Windows DLL exit regression, with ordinary explicit timer retirement coverage.
+- [x] Cache the documented Windows shutdown query before constructing the default timer thread; abandon that service only during actual process termination. Preserve ordinary cleanup, POSIX behavior and public headers/API/ABI.
+- [x] Register `task_timer_shutdown` in Automake; verify DLL exit, explicit retirement, ordinary static-executable teardown, existing Linux resume/lifetime tests, changed-file lint and diff.
+- [ ] Full exact-source CI on the working branch, including actual MSYS2 package/archive lanes.
+- [ ] Groundwork validation, only after full CI is green; caller owns publication and eventual merge/cleanup.
+
+The actual original CI/local native62 program completes all 734 checks but hangs during DLL process detach. The accepted 24-frame LLDB unwind reaches `timer_queue` condition-variable destruction from `ExitProcess`/`LdrShutdownProcess`. Evidence is retained at `.debug/ci-windows-dump-entry-dbl8orzx/offline-analysis/mac-approved-result.json` and `mac-lldb-session.stdout.txt`; the earlier DbgHelp `.bss` chain is rejected. The original 180-second gate remains FAILED/NOT_RECHECKED.
+
+Microsoft documents that process exit kills the other threads before DLL detach, so even lock acquisition, joining and synchronization-object destruction can deadlock. The private Windows owner resolves `RtlDllShutdownInProgress` before creating its queue/thread, and releases its ownership only when that cached function reports process termination. There is no module lookup, allocation, locking, joining or synchronization destruction on that path. The operating system reclaims this single process-lifetime service. Ordinary destruction still deletes the queue; the POSIX factory is byte-identical. If the documented Windows 10+ query is unavailable, the ordinary cleanup fallback remains; legacy-platform shutdown is not claimed fixed.
+
+Requirements preserved: PRD-V3N-REQ-024/025 and architecture v3 section 3.1 (timeout/cancellation, exactly-once delivery and executor affinity). Public representation, scheduling, cancellation, deadlines, explicit timer ownership and callback delivery are unchanged. No forced application exit, DLL pinning or public shutdown interface is introduced.
+
+## Focused proof and context limits
+
+A C-only consumer of the retained actual CI DLL reaches its default timer factory and prints `DEFAULT_TIMER_READY`, then fails to exit within 9 seconds; its retained handle is retired, with total 9.449 seconds. No C++ standard-library object crosses that ABI boundary. This is a new minimal regression, not another original native62 launch.
+
+The new registered C++ regression runs a matching small three-owner DLL (`resume_signal.cpp`, `task.cpp`, `executor.cpp`) with explicit retirement and default-exit children. The final test against original timer source is RED: exit 1, child wait timeout 258 and destruction-during-shutdown marker, total 9.372 seconds. Against fixed source it is GREEN: exit 0, total 0.491 seconds, no shutdown destruction marker. Each child has a 9-second wait and at most 1 second for owned retirement. An inherited event observes pending-capture destruction during actual process termination, so the assertion also protects runtimes whose condition variable does not itself hang. Static-executable normal CRT destruction is permitted by this test and passes separately. Ordinary explicit retirement joins the worker and releases a capture with an hour-long pending deadline.
+
+Linux focused tests pass: `task_timer_shutdown`; `task_resume_signal` 11 tests/134 checks; `task_resume_signal_lifetime` 3 tests/1816 checks. These use current source compiled with GNU13/C++20/pthreads and only the three owning concurrency translation units. Changed-file cpplint and `git diff --check` pass. `autoreconf -fi` registers the new program and inherited library linkage; existing AC_OUTPUT and .PHONY warnings are recorded without unrelated repair. No full baseline or broad suite was run.
+
+Native Windows context is Windows 11 build26200, cross LLVM-MinGW23.1.4/MinGW15 with its UCRT startup/headers, MSYS2 GCC15.2 standard-library headers/import libraries in an owned cache, and the retained actual CI MSVCRT libstdc++/libgcc/winpthread DLLs. Actual CI uses Windows Server2022 and LLVM23.1.0 with MSYS2 mingw64 throughout. This focused mixed-CRT proof is qualified; it does not replace full actual CI. No provider, SDK or compiler was built or globally installed. A first downloaded UCRT GCC package was unused after original imports identified MSVCRT.
+
+Preserved setup failures: sandboxed WSL native launch was unavailable before normal escalation; initial future-based fixture had unresolved libstdc++ TLS symbols, then a corrected TLS build exposed missing runtime once-call exports (native entry-point status C0000139). That setup failure is not RED. Replacing only test readiness synchronization with an already-supported Win32 event corrected the fixture boundary; the corrected native failure above is the behavioral RED. No production workaround was based on the setup error. Raw commands, logs, receipts, hashes and bounded ownership cleanup are retained in stage evidence.
+
+Darwin PSK failures, Windows host WSAPoll consumer failure and incidental Linux/macOS/BSD assessments retain their previous UNKNOWN/closed-budget states. No unrelated source repair, original diagnostic retry, push, CI dispatch, review agent, Groundwork validation, merge or cleanup occurred in stage 1.
+
+## Related WSAPoll error visibility
+
+- [x] Immediately save `WSAGetLastError()` when the existing `WSAPoll` call fails and include its numeric value in the existing exception message.
+- [x] Preserve interrupted-poll handling and all polling, timing and shutdown semantics.
+
+This caller-authorized message change follows proven timer stage commit `70d0fba7`. It supplies useful error evidence in the next already-required CI run; it does not diagnose or repair the prior consumer exit 6, whose cause remains UNKNOWN and captures remain closed. There are no per-poll traces, socket queries, clocks, locks or error suppression changes.
+
+A bounded Windows test substitutes only the external WSAPoll boundary to return WSAEINVAL through the actual Winsock last-error API. Before the change it returns 2 because the exception omits numeric 10022 (0.310s); after it exits 0 (0.303s), also checking that WSAEINTR still produces no exception. This controlled message regression is not a reproduction of the original consumer failure. Actual Windows compilation, GNU syntax checking of `examples/external_event_loop.cpp`, changed-header cpplint and diff checks pass. Commands, sources and receipts are under stage evidence `host-poll-message/`.
+
+Additional ordinary cleanup proof: a static-linked Windows default service releases its pending callback during normal executable CRT teardown before a later registered atexit observer checks the capture; exit 0 in 0.391s (`ordinary-default-native-test.json`). The default service is therefore not unconditionally immortal.
+
+## Stage 3: FreeBSD evidence transport setup
+
+- [x] Preserve actual current8f native package PASS, redundant automatic-copyback FAIL, and explicit fresh SCP recovery/upload PASS as separate receipts.
+- [x] Set `copyback: 'false'` on the pinned FreeBSD VM action in package validation and release acceptance.
+- [x] Pass the existing 13 focused workflow tests, actionlint and exact source-preservation proof.
+- [ ] Full working-branch CI with the corrected VM setup (not dispatched here).
+
+Current8f package job `121358499066` passed installed native TLS=no at 03:49UTC, TLS=yes and identical installed headers at 04:03UTC. Its later automatic whole-workspace rsync copyback failed with protocol error 12 after receiving 4,049,536,821 bytes. Explicit fresh SCP recovery then succeeded and uploaded artifact `12626459963`, 776,035,888 bytes, producer-reported digest `597ded253263153a8ae4bb617df42defaaceb55c930978e013e4063253532894`. This is retained raw-log/metadata evidence; the artifact was not downloaded for stage3. It does not prove the separate cookie archive translation-unit gate, which remains pending.
+
+The exact pinned action implementation SHA256 is `7f9e5e36ceae15937ec0aabbf7e4ce4ab2855b36aa3dfa0ce8e79e7c0fb9569e`; its post-run step 7 runs only when the lowercased copyback input is not `'false'`. Initial synchronization and guest execution are distinct earlier operations. The input is declared in the [pinned action metadata](https://raw.githubusercontent.com/vmactions/freebsd-vm/c9f815bc7aa0d34c9fdd0619b034a32d6ca7b57e/action.yml). Disabling only this redundant post-run copyback leaves the explicit required SCP transport responsible for retrieving evidence. The underlying rsync connection-loss mechanism is not claimed diagnosed.
+
+Whole-file byte comparison proves that removing the single added scalar from each workflow restores the original file exactly. Guest prepare/run/trap/native commands, initial sync settings, pinned actions/toolchains, source checks, every gate, SCP error propagation, artifact paths and existing fallback handling are therefore unchanged. An existing fallback is preserved if present; automatic copyback is no longer a promised source for it. Failed SCP and missing evidence remain failures.
+
+Focused checks: `python3 scripts/test-v3-ci-workflows.py FreeBSDNodePrerequisites -v` (9 tests), `python3 scripts/test-v3-package-workflow.py -v` (4 tests), cached actionlint on both workflows, and `git diff --check` all pass. No new mirror test, VM, timeout increase, retry, transport framework, baseline suite, CI dispatch, push, merge or Groundwork validation was performed. Stage3 commands, outputs, source hashes and preservation proof are retained at `.debug/v3-ci-green-VSvXxx/runtime-green-stage3/` in the primary checkout. PSK source remains untouched while its fresh native authorization is pending.

src/concurrency/resume_signal.cpp

@@ -4,6 +4,13 @@
 #include <utility>
 #include <vector>
 
+#ifdef _WIN32
+#ifndef NOMINMAX
+#define NOMINMAX
+#endif
+#include <windows.h>
+#endif
+
 namespace httpserver::detail {
 
 auto resume_node::claim() noexcept -> bool {
@@ -121,8 +128,27 @@ auto timer_queue::run() -> void {
 }
 
 timer_queue& default_timer_queue() {
+#ifdef _WIN32
+    struct service {
+        using shutdown_query = BOOLEAN (NTAPI*)();
+        // Resolve before starting the timer thread. Late loader lookups can
+        // themselves acquire locks orphaned by Windows process termination.
+        shutdown_query terminating = reinterpret_cast<shutdown_query>(GetProcAddress(GetModuleHandleW(L"ntdll.dll"), "RtlDllShutdownInProgress"));
+        std::unique_ptr<timer_queue> queue = std::make_unique<timer_queue>();
+
+        ~service() {
+            // ExitProcess has already killed other threads before DLL detach.
+            // Leave this process-lifetime queue to the OS; even destroying its
+            // mutex/condition_variable can deadlock on a killed waiter.
+            if (terminating && terminating()) static_cast<void>(queue.release());
+        }
+    };
+    static service owner;
+    return *owner.queue;
+#else
     static timer_queue queue;
     return queue;
+#endif
 }
 
 }  // namespace httpserver::detail

test/Makefile.am

@@ -584,6 +584,11 @@ EXTRA_DIST += integ/native_websocket_e2e.sh
 
 task_resume_signal_lifetime_SOURCES = unit/task_resume_signal_lifetime_test.cpp
 task_resume_signal_lifetime_LDADD = $(top_builddir)/src/libhttpserver.la
+
+# Windows children bound DLL process-detach retirement; POSIX checks ordinary
+# explicit timer ownership without changing its process lifetime semantics.
+check_PROGRAMS += task_timer_shutdown
+task_timer_shutdown_SOURCES = unit/task_timer_shutdown_test.cpp
 websocket_drain_SOURCES = unit/websocket_drain_test.cpp
 websocket_drain_LDADD = $(top_builddir)/src/libhttpserver_v3core.la
 websocket_drain_race_SOURCES = unit/websocket_drain_race_test.cpp

test/unit/task_timer_shutdown_test.cpp

@@ -0,0 +1,120 @@
+/* Copyright (C) 2026 Sebastiano Merlino; SPDX-License-Identifier: LGPL-2.1-or-later */
+#ifdef _WIN32
+#ifndef NOMINMAX
+#define NOMINMAX
+#endif
+#include <windows.h>
+#endif
+
+#include <chrono>
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#ifndef _WIN32
+#include <future>
+#endif
+#include <memory>
+#include <string>
+#include <utility>
+#include <httpserver/concurrency/resume_signal.hpp>
+
+namespace {
+using httpserver::detail::timer_queue;
+
+bool timer_runs(timer_queue& queue) {
+#ifdef _WIN32
+    std::shared_ptr<void> ready(CreateEventW(nullptr, TRUE, FALSE, nullptr), [](void* event) { CloseHandle(event); });
+    if (!ready) return false;
+    queue.schedule(timer_queue::clock::now(), [ready] { SetEvent(ready.get()); });
+    return WaitForSingleObject(ready.get(), 1000) == WAIT_OBJECT_0;
+#else
+    auto ready = std::make_shared<std::promise<void>>();
+    auto done = ready->get_future();
+    queue.schedule(timer_queue::clock::now(), [ready] { ready->set_value(); });
+    return done.wait_for(std::chrono::seconds(1)) == std::future_status::ready;
+#endif
+}
+
+int retire_explicit_timer() {
+    std::weak_ptr<int> pending_capture;
+    {
+        timer_queue queue;
+        if (!timer_runs(queue)) return 1;
+        auto capture = std::make_shared<int>(42);
+        pending_capture = capture;
+        queue.schedule(timer_queue::clock::now() + std::chrono::hours(1), [capture] {});
+        capture.reset();
+        if (pending_capture.expired()) return 2;
+    }
+    // Retirement joins the worker and releases pending callbacks promptly,
+    // even though their deadlines are an hour away.
+    return pending_capture.expired() ? 0 : 3;
+}
+
+#ifdef _WIN32
+int run_child(const wchar_t* mode) {
+    wchar_t image[32768];
+    const DWORD length = GetModuleFileNameW(nullptr, image, 32768);
+    if (length == 0 || length == 32768) return 4;
+    SECURITY_ATTRIBUTES attributes{sizeof(attributes), nullptr, TRUE};
+    HANDLE destroyed = CreateEventW(&attributes, TRUE, FALSE, nullptr);
+    if (destroyed == nullptr) return 5;
+    std::wstring command = L"\"" + std::wstring(image, length) + L"\" " + mode + L" " + std::to_wstring(reinterpret_cast<std::uintptr_t>(destroyed));
+    STARTUPINFOW startup{};
+    startup.cb = sizeof(startup);
+    PROCESS_INFORMATION child{};
+    if (!CreateProcessW(image, command.data(), nullptr, nullptr, TRUE, 0, nullptr, nullptr, &startup, &child)) {
+        CloseHandle(destroyed);
+        return 5;
+    }
+    CloseHandle(child.hThread);
+    const DWORD wait = WaitForSingleObject(child.hProcess, 9000);
+    DWORD code = 6;
+    if (wait == WAIT_OBJECT_0) {
+        if (!GetExitCodeProcess(child.hProcess, &code)) code = 7;
+    } else {
+        // Only this retained child handle is retired; no name-based cleanup.
+        TerminateProcess(child.hProcess, 8);
+        WaitForSingleObject(child.hProcess, 1000);
+    }
+    CloseHandle(child.hProcess);
+    // Captures may retire normally in a static executable's CRT teardown,
+    // but must survive DLL process detach after other threads are killed.
+    if (WaitForSingleObject(destroyed, 0) != WAIT_TIMEOUT) code = 10;
+    CloseHandle(destroyed);
+    if (code != 0) std::fprintf(stderr, "timer child failed: wait=%lu exit=%lu\n", wait, code);
+    return code == 0 ? 0 : 1;
+}
+#endif
+}  // namespace
+
+int main(int argc, char** argv) {
+#ifdef _WIN32
+    if (argc == 3 && std::strcmp(argv[1], "--default-exit") == 0) {
+        using shutdown_query = BOOLEAN (NTAPI*)();
+        const auto terminating = reinterpret_cast<shutdown_query>(GetProcAddress(GetModuleHandleW(L"ntdll.dll"), "RtlDllShutdownInProgress"));
+        if (!terminating) return 11;  // documented Windows 10+ contract
+        auto& queue = httpserver::detail::default_timer_queue();
+        if (!timer_runs(queue)) return 9;
+        HANDLE destroyed = reinterpret_cast<HANDLE>(static_cast<std::uintptr_t>(std::strtoull(argv[2], nullptr, 10)));
+        auto marker = std::shared_ptr<void>(destroyed, [terminating](void* event) {
+            if (terminating()) SetEvent(event);
+        });
+        queue.schedule(timer_queue::clock::now() + std::chrono::hours(1), [marker = std::move(marker)] {});
+        // Leave the DLL timer worker waiting when ExitProcess retires other
+        // threads, before invoking DLL static destructors under loader lock.
+        Sleep(100);
+        std::puts("default timer active; returning from main");
+        std::fflush(stdout);
+        return 0;
+    }
+    if (argc == 3 && std::strcmp(argv[1], "--retire") == 0) return retire_explicit_timer();
+    if (run_child(L"--retire") != 0) return 1;
+    return run_child(L"--default-exit");
+#else
+    static_cast<void>(argc);
+    static_cast<void>(argv);
+    return retire_explicit_timer();
+#endif
+}