fix(ocsf): emit schema-conformant OCSF 1.8 events

[zanetworker]

Summary

Native OCSF 1.8.0 events failed the official OCSF validator: HTTP Activity emitted an attribute the class doesn't define, Process Activity events had no actor, connection_info lacked direction_id, and SSH Activity events named no endpoint. The vendored-schema tests missed these because they checked only top-level, non-profile attributes.

This makes every builder's native output pass https://schema.ocsf.io/api/1.8.0/validate (apart from top-level container, open on #4283) and adds tests that keep it that way, including an e2e test that validates the supervisor's real JSONL.

The schema downgrade is out of scope. #4284 tracks picking the one version OpenShell emits (1.1 or 1.8) and removing the downgrade; until then the existing downgrade is unchanged, so output with ocsf_schema_version set is still not conformant. This PR is needed either way:

• If 1.8 stays, this is the conformance work, and bug(ocsf): schema downgrade labels events 1.1 or 1.3 that do not conform to those versions #4284 only removes the downgrade and deprecates the setting.
• If OpenShell moves to 1.1, the builder fixes carry over (the new endpoint fields exist in every version), and the validator and e2e show exactly which emitters need work once the 1.1 schemas are vendored and OCSF_VERSION changes.

Related Issue

Closes #4283

Changes

• HTTP Activity no longer emits is_src_dst_assignment_known, which no OCSF version defines on that class.
• ProcessActivityBuilder requires an acting process before build() (type-state, like the HTTP and Network builders). Sandbox process events name the openshell-sandbox launcher with its real pid.
• connection_info carries the required direction_id and direction, defaulting to Unknown.
• SshActivityBuilder requires a source or destination endpoint before build(). All eight SSH events had none; they now name the supervisor's SSH server as dst_endpoint (svc_name: "ssh" and the listen socket path in name), using a new Endpoint::svc_name field.
• Endpoint::from_domain and the policy DNS endpoint also set hostname.
• validate_required_fields recurses into nested objects, rejects undefined attributes, enforces profile attributes only when declared, and loads object schemas for the event's metadata.version. New test every_builder_emits_schema_conformant_events covers one event per builder, including Authentication.
• New e2e test ocsf_jsonl_conformance (Podman CI set) enables JSONL, generates denied traffic, copies the supervisor's JSONL with podman cp, and validates every record. It covers production call sites rather than builders, and found the SSH endpoint gap. The e2e crate takes openshell-ocsf (with test-support) as a dev-dependency so both suites share one validator.
• With test-support, vendored schemas are compiled into the binary (validation/embedded.rs, drift-checked by embedded_schemas_match_vendored_files), so archived e2e runs on CI machines need no source files. The fallback EventContext carries a sandbox identity, so producer tests emit container and device fields as production does. Top-level container is the single named exception in the validator, linked to bug(ocsf): native OCSF 1.8 events fail the official schema validator #4283.
• schemas/ocsf/README.md documents the embedded table, regeneration, and validation with the official OCSF server.

User-visible: HTTP events drop is_src_dst_assignment_known; SSH events gain dst_endpoint; connection_info gains direction_id; endpoints built from a domain gain hostname.

Deferred: top-level container is undefined in every OCSF version but identifies the affected sandbox on gateway events (#4283). The SIGTERM-failure Process Activity event still uses the Process::new("unknown", 0) fallback.

Testing

On main at b58e24c9d, after #4677 (device.os.type_id), which this branch builds on.

• Real run on Podman (OPENSHELL_E2E_PODMAN_TEST=ocsf_jsonl_conformance mise run e2e:podman, gateway and supervisor images built from this branch): supervisor_jsonl_conforms_to_the_native_schema passes, validating every supervisor JSONL record against 1.8.0, alongside the rest of the Podman smoke set (5 suites, 0 failures). Before the SSH fix it failed with "SSH Activity" 1.8.0 "SSH connection accepted on supervisor Unix socket": Missing at_least_one field from [dst_endpoint, src_endpoint].
• cargo test for openshell-ocsf (with test-support), openshell-server, openshell-supervisor, openshell-supervisor-network, openshell-supervisor-process, openshell-sandbox, openshell-otel
• cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, and no clippy findings in the new e2e file
• Unit tests added/updated

ssh::exec_input_tests::eof_preserves_queued_input_until_the_child_reads hits its 5 s deadline on this machine on both main and this branch.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[johntmyers]

/ok to test 78621e8

[github-actions]

Label test:e2e applied for 78621e8. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

PR Review Status

The independent review found no blocking defects in the OCSF conformance and schema downgrade changes. @zanetworker, I checked your note about top-level container: it is present in the base, remains an acknowledged design question, and the downgrade preserves its data under unmapped.

Current-head testing has started with test:e2e and test:windows. Trivy and Helm are green; Branch Checks, E2E, and both Windows lanes are pending. The E2E Label Help bot requests Re-run all jobs, but the identified run still has an active first attempt.

Action required: rerun Branch E2E Checks after the active attempt finishes. Gator has operator authorization and the narrow sandbox permission to perform that rerun in a subsequent cycle.

Blocking findings: None.

Carried findings: None.

Gator metadata

• Validation: Focused OCSF correctness fixes for linked issues #4283 and #4284; operator requested review and relevant tests.
• Docs: Fern documentation updates describe OS identity, mixed-version JSONL, downgrade preservation, gateway configuration, and the kept-native metric.
• Checks: DCO, OpenShell / Helm Lint, and OpenShell / Trivy Changes pass; Branch Checks and both Windows MSVC PR lint and test lanes are pending on the reviewed head.
• E2E: test:e2e applied; current-head mirror verified; Branch E2E Checks run 37657018590 attempt 1 is active. Bot-required rerun remains outstanding. Relevant E2E coverage includes the new Podman JSONL conformance test.
• Head SHA: 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963
• Base SHA: 3fc93e28273fe304db634e6d5e9a52d57200389e
• Merge base SHA: 834b79a8c2351370e7c6c363423f5be18d440256
• Patch ID: 33dfd9e90bc20c4d4b8493efc716d9a8d3c32edb
• Gator payload: 11
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:blocked
• Blocked reason: test_dispatch_required

[johntmyers]

gator-agent

Author Follow-Up Nudge

This PR has been in gator:in-review for more than 48 business hours and required CI still needs attention.

@zanetworker, please make the vendored OCSF schemas available to the relocated Podman E2E test and push an update. The rootless Podman run fails in supervisor_jsonl_conforms_to_the_version_each_record_claims because the loader uses the build machine's CARGO_MANIFEST_DIR and cannot find schemas/ocsf/v1.1.0/classes/base_event.json on the remote test machine. Verify that the archived test can load its schemas and validate the JSONL records there.

@NVIDIA/openshell-maintainers, both Windows lanes also need a gate resolution: Clippy reports unnecessary_wraps and needless_pass_by_ref_mut in crates/openshell-cli/src/ssh.rs at lines 369, 376, and 423. That file is unchanged by this PR.

Gator metadata

• Head SHA: 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963
• Gator payload: 11
• Next state: gator:in-review
• Nudge scope: Existing CI follow-up; no new code-review disposition.

[zanetworker]

Fixed in aae174742: the vendored schemas are now compiled into the test binary (crates/openshell-ocsf/src/validation/embedded.rs, generated with a drift test), so archived binaries need no files from the build machine. Verified by running the ocsf_jsonl_conformance nextest archive from an unrelated directory with --workspace-remap and the repo's schemas/ hidden, against a Podman gateway: 1 passed. The Windows Clippy findings are in crates/openshell-cli/src/ssh.rs, which this PR doesn't change.

@johntmyers could you take a look, and /ok to test aae17474233af3ddd272f43557a01a7cf0c7f677 when you get a chance?

[johntmyers]

/ok to test aae1747

[johntmyers]

gator-agent

PR Review Status

Thanks @zanetworker (Adel Zaalouk), I checked your update about relocating the archived test: the loader now reads compiled-in schemas, and all 112 vendored schemas are included. That addresses the build-machine path dependency. Your note about the earlier Windows Clippy failures also holds: ssh.rs is unchanged by this PR.

The independent follow-up review found one new Windows regression in the drift test. It generates platform-native paths while the committed table uses forward slashes. The inline finding covers both the failing comparison and regeneration.

Action required: normalize the generated schema paths to forward slashes and verify the focused drift test on Windows.

Blocking findings:

• GATOR-aae17474-01: Windows path separators break the new embedded-schema drift check and regeneration.

Carried findings: None. The prior schema-relocation obligation is addressed by the loader change; current-head E2E will verify it in CI.

I posted /ok to test for the full current head. Branch Checks, Helm, E2E, and Windows workflows are queued or running. The existing test:e2e and test:windows labels remain appropriate. Gator stays in review while the author fixes the path normalization.

Gator metadata

• Validation: Focused OCSF correctness changes for linked issues #4283 and #4284; continuing the previously validated PR.
• Docs: Existing Fern updates cover the user-visible changes; schema README documents embedding and regeneration.
• Checks: DCO passes; current-head Branch Checks and Helm workflows are active, Windows is queued. Trivy is not yet confirmed green.
• E2E: Current-head Branch E2E Checks run 38081852521 is active. The older bot-required rerun instruction referred to the previous head; no current-head rerun instruction was present when checked.
• Head SHA: aae17474233af3ddd272f43557a01a7cf0c7f677
• Base SHA: eeba0e7954c0fb4d8e9e2e29d1bfa68e290eb8b3
• Merge base SHA: 834b79a8c2351370e7c6c363423f5be18d440256
• Patch ID: 892df22a6665f52060e783e6810464d88dae8f16
• Gator payload: 11
• Review mode: follow_up
• Previous reviewed SHA: 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[johntmyers]

/ok to test 4f8ea84

[johntmyers]

gator-agent

PR Review Status

Thanks @zanetworker (Adel Zaalouk), I checked your reply about Windows paths. The comparison and regeneration now use the same helper to join native path components with forward slashes, and the new regression test checks that key. The independent follow-up found no new blockers, and I resolved GATOR-aae17474-01.

I posted /ok to test for the current head. Branch Checks, Helm, E2E, and both Windows lanes are running or queued. The earlier E2E rerun instruction applies to an older head; there is no current-head rerun request. Native Windows verification remains with the Windows lanes.

Action required: @NVIDIA/openshell-maintainers, approve the current-head Trivy Changes workflow, which is awaiting workflow approval. Gator will continue inspecting the active tests; the earlier unrelated Windows CLI Clippy failures still need a maintainer resolution if they recur.

Blocking findings: None.

Carried findings: None. The Windows schema-path finding is fixed and its thread is resolved.

Gator metadata

• Validation: Continuing the validated OCSF correctness fixes for linked issues #4283 and #4284.
• Docs: Existing Fern updates cover user-visible changes; this delta changes test path normalization only.
• Checks: DCO passes. Branch Checks run 38129872736, Helm run 38129872726, and Windows run 38129872735 are active. Trivy Changes run 38129078889 concluded action_required and has no passing current-head result.
• E2E: test:e2e remains applied; current-head Branch E2E Checks run 38129873026 is in progress. test:windows remains applied and both lanes are active.
• Head SHA: 4f8ea841df8a061170bd0623124c8e109e9174c4
• Base SHA: eeba0e7954c0fb4d8e9e2e29d1bfa68e290eb8b3
• Merge base SHA: 834b79a8c2351370e7c6c363423f5be18d440256
• Patch ID: 3646f9d2c586ab7fe3c85fd231645d9dc6f787f2
• Gator payload: 11
• Review mode: follow_up
• Previous reviewed SHA: aae17474233af3ddd272f43557a01a7cf0c7f677
• Review budget exhausted: no
• Maintainer decision required: no — no review convergence decision is needed.
• Next state: gator:blocked
• Blocked reason: workflow_approval_required

[feloy]

The fix #4194 should help make the test:windows pass

[rhuss]

cc-review summary

Reviewed at 4f8ea841d with a 6-dimension agent pass plus CodeRabbit. The PR is functionally solid: it builds, both drift tests pass, the downgrade logic is correct (stripped attributes are preserved under unmapped.downgraded_attributes, versions are labeled only when they validate, else kept native), security is clean, and all four goals are delivered with genuine deferrals. No blocking defect.

Three non-blocking recommendations below, worth considering before or shortly after merge. Minor and Low findings (SIGTERM process pid, 1.1 unmapped test coverage, ConnectionDirectionId dead variants, kept-native observability split detail, setter duplication) are omitted to keep this focused; happy to share them if useful.

Nice touches: the downgrade never fabricates data and asserts byte-identity on kept-native; per-version required-field differences are grounded in the vendored schemas; type-state builders make missing required fields a compile error; both generated files are drift-guarded.

Source: cc-review (correctness, architecture, security, production, test-quality, goal-alignment agents + CodeRabbit)

[rhuss]

Important (observability): kept-native events are reported two ways. The gateway path emits openshell_ocsf_log_kept_native_total{class_uid,target}, while the sandbox path feeds the process-global KEPT_NATIVE mutex that the supervisor drains into a warn!. An operator alerting on the metric misses sandbox kept-native events, and kept_native_tally() reflects only one caller.

Consider a single sink: both paths emitting the metric, or both routing through record_kept_native with the metric/warn emitted from one drain point.

[zanetworker]

Addressed in 632853564: every output path now records kept-native events through the shared record_kept_native tally, so kept_native_tally() covers the gateway JSONL, supervisor JSONL, and OTLP export paths. The gateway also emits openshell_ocsf_log_kept_native_total (JSONL) and openshell_ocsf_otlp_kept_native_total (OTLP); the supervisor reports the tally as a warning from its settings poll loop, since it has no metrics exporter and logging from inside the tracing layer is not reliable. A supervisor metric would need a supervisor metrics pipeline, which is out of scope here.

[krishicks]

I do not think we should merge this. We should pick a specific schema version that is widely supported and stick to it rather than attempt lossy downgrades from a schema version no SIEM may support (e.g. 1.8)

[rhuss]

@krishicks on which schema version would you settle ? This directly impacts #2640 where to add any open traces to the event. 1.8 would be ideal as there is a typed section in the schema, otherwise its just an untyped key.

[zanetworker]

I've narrowed this PR to native OCSF 1.8 conformance. The downgrade changes are out, so it no longer touches the 1.1/1.3 downgrade and closes only #4283.

1.8 is what OpenShell emits by default today, and this PR makes that output correct:

• Native events from every builder, including Authentication, pass the official 1.8.0 validator (apart from top-level container, still open on bug(ocsf): native OCSF 1.8 events fail the official schema validator #4283).
• The schema validator recurses into nested objects, rejects undefined attributes, and honours declared profiles.
• Process Activity requires an actor and SSH Activity requires an endpoint at compile time.
• A Podman e2e test validates the supervisor's real JSONL against 1.8.0.

It's rebased on #4677 and uses that PR's device.os change.

#4284 stays open to pick the one version OpenShell emits (1.1 or 1.8) and to remove the downgrade. Until then the existing downgrade is unchanged. Nothing here is lost either way: if we stay on 1.8, #4284 only removes the downgrade and deprecates the setting; if we move to 1.1, that work builds on top of this one, since the builder fixes carry over and the validator and e2e show exactly which emitters need changes.

To inform that decision, we still need discovery on how the main SIEMs handle 1.8 records (Security Lake, Google SecOps, Splunk, Elastic): accepted, rejected, or stored but not queryable. That happens separately, with any doc or code changes in follow-up PRs.

@krishicks could you take another look?

[zanetworker]

SIEM spike results for the version decision are on #4284: #4284 (comment) (details in https://gist.github.com/zanetworker/dd3bf999d1f18e6557fa5b95bf87712d). In short, none of the ten sinks tested rejects native 1.8.0, which is the output this PR makes conformant.