Repository navigation
feat(taskblock): compose Linux native I/O producers with shared socket hooks (PROF-14931) #666
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
kaahos
wants to merge
70
commits into
paul.fournillon/taskblock-jvm-producers
from
paul.fournillon/taskblock-native-io
Draft
Changes from all commits
Commits
Show all changes
70 commits
Select commit
Hold shift + click to select a range
bd9563f
feat(taskblock): capture stacks synchronously at block exit
kaahos 6089b93
fix: restrict block suppression to all-thread scope
kaahos a5ae1b5
fix: preserve JVMTI frames in overlapping buffers
kaahos ee6d17c
fix: address sphinx review
kaahos 2441b5f
fix(taskblock): preserve unowned wall fallback outside owned blocks
kaahos c6c7a7b
fix: fix emission problems
kaahos 3951400
test(taskblock): migrate core context setup
kaahos 1fd6329
fix(taskblock): report park entry ownership
kaahos 79eadad
fix(taskblock): migrate TaskBlock tests to the jafar-backed JFR API
kaahos 5046d8a
fix: address review comments
kaahos dde0bc1
fix: address sphinx comments
kaahos 5bcc2b7
fix: address review comments
kaahos 51e879e
fix: address review comments
kaahos fdae4ff
fix: trim unused BlockRunSnapshot fields
kaahos c632b1f
fix: fix bug
kaahos 554f450
feat: squash commit for PR 664 rebase
kaahos f4dc61b
fix
kaahos cc4d7b8
feat(taskblock): squashed commit: instrument blocking native I/O for …
kaahos f57e4ae
feat: squashed commit for native_socket_netty rebase onto taskblock-n…
kaahos 19cda16
fix: capture Gradle results locally and reap owned processes
kaahos 82635c8
fix: preserve native I/O transparency across ownership and fd reuse
kaahos 5f38745
docs: add canonical agent instruction redirect
kaahos 629f48b
fix: preserve task-block boundaries and restore sanitizer setup
kaahos 69427a2
test: publish sanitizer XML and verify forwarding behavior
kaahos cd0c904
test: verify transaction forwarding without GOT identity
kaahos e88728b
fix: isolate native I/O patch failures by installation generation
kaahos 0646821
fix: reserve native I/O owner activation before discovery
kaahos fb34de3
Merge TaskBlock library stack into native I/O profiling
kaahos ae8c8ed
Merge #665 unowned-tail test setup fix
kaahos b1b8e6f
Resolve native test fixtures independently of working directory
kaahos 1d3208b
Protect native patch-count diagnostics with the patcher lock
kaahos 21d462e
Skip high-FD classifier reuse test when the hard limit is too low
kaahos 48d6ad3
test: skip high-FD I/O qualification on constrained runners
kaahos cfe37a2
test: isolate blocking-hook and wall-precheck measurements
kaahos c552340
test: account for OpenJ9 allocation reporting batches
kaahos 413218b
fix: preserve full-width TaskBlock in-flight test counts
kaahos 6cefa68
Fix native I/O cancellation cleanup and owner classification
kaahos a0a4207
Fix JVM support gtest inline definition include
kaahos 1d97b88
Fix native socket runner fixtures for poller topology and carrier reu…
kaahos 4c141ad
Merge cumulative TaskBlock custody into native I/O producers
kaahos 6a131f2
Merge crash bootstrap and recorder telemetry into native I/O TaskBlocks
kaahos c5fadd5
Merge upstream #663 test follow-ups into #666
kaahos 0264999
Merge upstream exit-order repair into #666
kaahos 312aeaf
Scope native socket lifecycle checks to completion boundaries
kaahos ac9bdfa
Merge wall dump epoch guard into #666
kaahos 5388563
Mark unprovisionable musl/OpenJ9 combinations not applicable
kaahos 7c88682
test(ci): make the pipeline exclusion literal explicit
kaahos ac6af81
Merge #665 repairs into #666 preserving native I/O unload ownership
kaahos c34e62a
fix(natsock): compose JDK 8 restartablePoll adapter with the poll hook
kaahos ab4beaf
perf(natsock): use cached fd verdict on sampler hook admission
kaahos c9a7fe2
docs(natsock): state retained fd classifier memory
kaahos 002263f
refactor(natsock): move Linux-only native I/O sources to _linux files
kaahos f0fe242
chore(stack): merge #665 fixes into #666
kaahos b53f8c1
chore(stack): merge #665 fixes into #666
kaahos fc0518a
chore(stack): merge #665 fixes into #666
kaahos a88a09d
chore(stack): merge #665 fixes into #666
kaahos 4a0f02c
chore(stack): merge #665 fixes into #666
kaahos 7d0d3ad
fix(codecache): return the JUMP_SLOT as an import's first slot
kaahos 9458398
test(nativeio): run the low-FD errno pass under a 65536 FD limit
kaahos 62eaa55
chore(stack): merge #665 fixes into #666
kaahos 3ff304d
ci(sanitizer): defer the gtest JUnit report to a separate change
kaahos 9383cbf
chore(stack): merge #665 fixes into #666
kaahos 1a7137e
test(nativeio): look for IBM Java 8 bridge hooks in libjava
kaahos 7e0d181
chore(stack): merge #665 fixes into #666
kaahos 9ab25d8
test(wall): wait on counters, not dumps, before release retained-dump…
kaahos 38f29dd
chore(stack): merge #665 fixes into #666
kaahos d980c30
chore(stack): merge #665 fixes into #666
kaahos 9de7589
test(wall): hold origin A quietly before the unfiltered release trans…
kaahos b1126a6
test(wall): gate release unowned-tail transitions on the worker's own…
kaahos 23d43ed
chore(stack): merge #665 fixes into #666
kaahos File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| #!/usr/bin/env python3 | ||
| # Copyright 2026, Datadog, Inc. | ||
| """Capture Gradle output and status locally, including on failure or interruption.""" | ||
|
|
||
| import ctypes | ||
| import datetime | ||
| import json | ||
| import os | ||
| from pathlib import Path | ||
| import signal | ||
| import subprocess | ||
| import sys | ||
| import time | ||
|
|
||
|
|
||
| def adopt_gradle_descendants(args): | ||
| """Own detached single-use daemons on Linux, not just the launcher session.""" | ||
| if sys.platform != "linux" or "--no-daemon" not in args: | ||
| return False | ||
| libc = ctypes.CDLL(None, use_errno=True) | ||
| # PR_SET_CHILD_SUBREAPER: orphaned descendants reparent here rather than PID 1. | ||
| if libc.prctl(36, 1, 0, 0, 0) != 0: | ||
| raise OSError(ctypes.get_errno(), "cannot own Gradle descendant processes") | ||
| return True | ||
|
|
||
|
|
||
| def reap_adopted_children(): | ||
| """Terminate and reap only children adopted by this wrapper's subreaper.""" | ||
| children_file = Path("/proc/self/task") / str(os.getpid()) / "children" | ||
| started = time.monotonic() | ||
| signalled = set() | ||
| while True: | ||
| while True: | ||
| try: | ||
| pid, _ = os.waitpid(-1, os.WNOHANG) | ||
| except ChildProcessError: | ||
| return | ||
| if pid == 0: | ||
| break | ||
| children = [int(pid) for pid in children_file.read_text().split()] | ||
| for pid in children: | ||
| sig = signal.SIGKILL if time.monotonic() - started >= 10 else signal.SIGTERM | ||
| if (pid, sig) not in signalled: | ||
| try: | ||
| os.kill(pid, sig) | ||
| except ProcessLookupError: | ||
| # The adopted child can exit between the snapshot and kill(). | ||
| pass | ||
| signalled.add((pid, sig)) | ||
| time.sleep(0.02) | ||
|
|
||
|
|
||
| def main(args): | ||
| """Forward arguments unchanged and publish evidence, not inferred test passes.""" | ||
| args = args or ["build"] | ||
| for directory in ("build/logs", "build/reports/claude", ".claude/out"): | ||
| Path(directory).mkdir(parents=True, exist_ok=True) | ||
| stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%d-%H%M%S-%f") | ||
| label = args[0].replace("/", "_").replace(":", "_") | ||
| log = Path("build/logs") / (stamp + "-" + label + ".log") | ||
| command = ["./gradlew"] + args | ||
| if not any(arg in ("-i", "--info", "-d", "--debug") for arg in args): | ||
| command.append("-i") | ||
| if not any(arg == "--console" or arg.startswith("--console=") for arg in args): | ||
| command.append("--console=plain") | ||
| interrupted = [] | ||
| process = None | ||
| subreaper = False | ||
|
|
||
| def interrupt(signum, frame): | ||
| interrupted.append(signum) | ||
| if process is not None: | ||
| try: | ||
| os.killpg(process.pid, signal.SIGTERM) | ||
| except ProcessLookupError: | ||
|
github-code-quality[bot] marked this conversation as resolved.
Fixed
|
||
| # The launcher group may have exited before this signal handler ran. | ||
| pass | ||
|
|
||
| previous = {sig: signal.signal(sig, interrupt) for sig in (signal.SIGINT, signal.SIGTERM)} | ||
| print("Logging full Gradle output to: " + str(log), flush=True) | ||
| try: | ||
| with log.open("wb") as output: | ||
| try: | ||
| subreaper = adopt_gradle_descendants(args) | ||
| process = subprocess.Popen(command, stdout=output, stderr=subprocess.STDOUT, | ||
| start_new_session=True) | ||
| termination_started = None | ||
| while True: | ||
| try: | ||
| status = process.wait(timeout=0.2) | ||
| break | ||
| except subprocess.TimeoutExpired: | ||
| if interrupted: | ||
| if termination_started is None: | ||
| termination_started = time.monotonic() | ||
| elif time.monotonic() - termination_started >= 10: | ||
| os.killpg(process.pid, signal.SIGKILL) | ||
| if status < 0: | ||
| status = 128 - status | ||
| except OSError as error: | ||
| output.write((str(error) + "\n").encode()) | ||
| status = 127 | ||
| finally: | ||
| # Gradle is run without a persistent daemon by qualification callers. | ||
| # Reap any still-owned descendants after their launcher exits. | ||
| if process is not None: | ||
| try: | ||
| os.killpg(process.pid, signal.SIGKILL) | ||
| except ProcessLookupError: | ||
|
github-code-quality[bot] marked this conversation as resolved.
Fixed
|
||
| # The owned process group may already have exited. | ||
| pass | ||
| if subreaper: | ||
| reap_adopted_children() | ||
| if interrupted: | ||
| status = 128 + interrupted[0] | ||
| lines = log.read_text(errors="replace").splitlines() | ||
| build = next((line for line in reversed(lines) if line.startswith("BUILD ")), "No BUILD status") | ||
| report = {"command": command, "log": str(log), "exitCode": status, | ||
| "buildStatus": build, "testAcceptance": "not evaluated; inspect fresh XML populations"} | ||
| root = Path("build/reports/claude/gradle-summary") | ||
| root.with_suffix(".json").write_text(json.dumps(report, indent=2) + "\n") | ||
| root.with_suffix(".md").write_text( | ||
| "# Gradle summary\n\n" + build + "\n\nExit status: " + str(status) | ||
| + "\n\nFull log: `" + str(log) + "`\n\nTest acceptance requires fresh XML counts and skips.\n") | ||
| print(build + " (exit " + str(status) + ")", flush=True) | ||
| print(str(root.with_suffix(".md"))) | ||
| print(str(root.with_suffix(".json"))) | ||
| return status | ||
| finally: | ||
| for sig, handler in previous.items(): | ||
| signal.signal(sig, handler) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| sys.exit(main(sys.argv[1:])) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| # Copyright 2026, Datadog, Inc. | ||
| """Hermetic tests of local capture/status/owned-process wrapper behavior.""" | ||
|
|
||
| import json | ||
| import os | ||
| from pathlib import Path | ||
| import signal | ||
| import subprocess | ||
| import sys | ||
| import tempfile | ||
| import time | ||
| import unittest | ||
|
|
||
| WRAPPER = Path(__file__).with_name("build-and-summarize").resolve() | ||
|
|
||
|
|
||
| class GradleSummaryTest(unittest.TestCase): | ||
| def setUp(self): | ||
| self.temp = tempfile.TemporaryDirectory() | ||
| self.addCleanup(self.temp.cleanup) | ||
| self.root = Path(self.temp.name) | ||
|
|
||
| def launcher(self, body): | ||
| path = self.root / "gradlew" | ||
| path.write_text("#!/usr/bin/env python3\n" + body) | ||
| path.chmod(0o755) | ||
|
|
||
| def report(self): | ||
| return json.loads((self.root / "build/reports/claude/gradle-summary.json").read_text()) | ||
|
|
||
| def run_wrapper(self, *args): | ||
| return subprocess.run([str(WRAPPER), *args], cwd=self.root, | ||
| stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=20) | ||
|
|
||
| def test_default_success_and_complete_stream_capture(self): | ||
| self.launcher("import sys\nprint('BUILD SUCCESSFUL')\nprint('stderr marker', file=sys.stderr)\n") | ||
| result = self.run_wrapper() | ||
| self.assertEqual(0, result.returncode, result.stderr) | ||
| report = self.report() | ||
| self.assertEqual(["./gradlew", "build", "-i", "--console=plain"], report["command"]) | ||
| self.assertIn("stderr marker", (self.root / report["log"]).read_text()) | ||
| self.assertIsNone(report["tests"]["total"]) | ||
| self.assertTrue(any("XML/JFR" in action for action in report["actions"])) | ||
|
|
||
| def test_failure_status_and_arguments_are_preserved(self): | ||
| self.launcher("import sys\nprint('BUILD FAILED')\nsys.exit(23)\n") | ||
| result = self.run_wrapper(":module:test", "--console=plain", "-d", "-Ptests=a b") | ||
| self.assertEqual(23, result.returncode, result.stderr) | ||
| self.assertEqual(["./gradlew", ":module:test", "--console=plain", "-d", "-Ptests=a b", | ||
| "-i", "--console=plain"], | ||
| self.report()["command"]) | ||
| self.assertEqual(23, self.report()["exitCode"]) | ||
|
|
||
| def test_missing_launcher_still_produces_failure_evidence(self): | ||
| self.assertEqual(127, self.run_wrapper().returncode) | ||
| self.assertEqual(127, self.report()["exitCode"]) | ||
|
|
||
| @unittest.skipUnless(sys.platform == "linux", "Linux subreaper ownership requires Linux") | ||
| def test_no_daemon_reaps_detached_descendant(self): | ||
| self.launcher("import os, time\nchild = os.fork()\nif child == 0:\n" | ||
| " os.setsid()\n open('detached.pid', 'w').write(str(os.getpid()))\n" | ||
| " time.sleep(60)\nelse:\n" | ||
| " while not os.path.exists('detached.pid'): time.sleep(0.01)\n" | ||
| " print('BUILD SUCCESSFUL')\n") | ||
| result = self.run_wrapper("--no-daemon") | ||
| self.assertEqual(0, result.returncode, result.stderr) | ||
| child = int((self.root / "detached.pid").read_text()) | ||
| with self.assertRaises(ProcessLookupError): | ||
| os.kill(child, 0) | ||
|
|
||
| def test_sigterm_reaps_owned_group_and_reports_signal(self): | ||
| self.launcher("import os, time\nopen('launcher.pid', 'w').write(str(os.getpid()))\n" | ||
| "time.sleep(60)\n") | ||
| process = subprocess.Popen([str(WRAPPER), "--no-daemon"], cwd=self.root, | ||
| stdout=subprocess.PIPE, stderr=subprocess.PIPE) | ||
| self.addCleanup(lambda: process.poll() is None and process.kill()) | ||
| for _ in range(200): | ||
| if (self.root / "launcher.pid").exists(): | ||
| break | ||
| time.sleep(0.01) | ||
| self.assertTrue((self.root / "launcher.pid").exists()) | ||
| child = int((self.root / "launcher.pid").read_text()) | ||
| process.send_signal(signal.SIGTERM) | ||
| process.communicate(timeout=15) | ||
| self.assertEqual(143, process.returncode) | ||
| self.assertEqual(143, self.report()["exitCode"]) | ||
| with self.assertRaises(ProcessLookupError): | ||
| os.kill(child, 0) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| unittest.main() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.