[zanetworker]

Summary

Native OCSF 1.8.0 events failed the official OCSF validator: HTTP Activity emitted an attribute the class doesn't define, Process Activity events had no actor, connection_info lacked direction_id, and SSH Activity events named no endpoint. The vendored-schema tests missed these because they checked only top-level, non-profile attributes.

This makes every builder's native output pass https://schema.ocsf.io/api/1.8.0/v2/validate (apart from top-level container, open on #4283) and adds structural checks that keep it that way, including an e2e test that validates the supervisor's real JSONL. The checks cover required and defined attributes, profiles, and at_least_one constraints, but not value types or enum ranges; the official validator stays the reference for those (see schemas/ocsf/README.md).

The schema downgrade is out of scope. #4284 tracks picking the one version OpenShell emits (1.1 or 1.8) and removing the downgrade; until then the existing downgrade is unchanged, so output with ocsf_schema_version set is still not conformant. This PR is needed either way:

• If 1.8 stays, this is the conformance work, and bug(ocsf): schema downgrade labels events 1.1 or 1.3 that do not conform to those versions #4284 only removes the downgrade and deprecates the setting.
• If OpenShell moves to 1.1, the validator and e2e show exactly which emitters need work once the 1.1 schemas are vendored and OCSF_VERSION changes.

Related Issue

Refs #4283

With #4677 (device.os.type_id), this meets #4283's acceptance items except the first: events still fail the official validator on top-level container, which this PR leaves unchanged, so the issue stays open for that.

Changes

• HTTP Activity no longer emits is_src_dst_assignment_known, which no OCSF version defines on that class.
• ProcessActivityBuilder requires an acting process before build() (type-state, like the HTTP and Network builders). Sandbox process events name the openshell-sandbox launcher with its real pid.
• connection_info carries the required direction_id and direction, defaulting to Unknown.
• SshActivityBuilder requires a source or destination endpoint before build(). All eight SSH events had none; they now name the supervisor's SSH server as dst_endpoint (svc_name: "ssh" and the listen socket path in name), using a new Endpoint::svc_name field.
• validate_required_fields and validate_known_fields share one recursive walker. Both reject undefined attributes at every level and, when validating an event, attributes from profiles the event doesn't declare. validate_required_fields also checks required attributes and at_least_one constraints, and fails any event not labelled 1.8.0. New test every_builder_emits_schema_conformant_events covers one event per builder, including Authentication and a fully populated Detection Finding with an ATT&CK mapping (tactic and technique are vendored for it).
• New e2e test ocsf_jsonl_conformance (Podman CI set) enables JSONL, generates denied TCP traffic and a plain HTTP request through the sandbox proxy, waits until Network, HTTP and SSH Activity records are all present, copies the supervisor's JSONL with podman cp, and validates every record. It covers those production call sites rather than builders, and found the SSH endpoint gap. When records can't be copied, the failure names the podman command and its error. The e2e crate takes openshell-ocsf (with test-support) as a dev-dependency so both suites share one validator.
• With test-support, vendored schemas are compiled into the binary (validation/embedded.rs, drift-checked by embedded_schemas_match_vendored_files), so archived e2e runs on CI machines need no source files. The fallback EventContext carries a sandbox identity, so producer tests exercise the top-level container exception as a supervisor with a sandbox ID emits it. Top-level container is the single named exception in the validator, linked to bug(ocsf): native OCSF 1.8 events fail the official schema validator #4283.
• schemas/ocsf/README.md documents the embedded table, regeneration, and validation with the official OCSF server through its v2 endpoint (/api/v2/validate locally, /api/1.8.0/v2/validate hosted), which reports constraint failures such as at_least_one that the older /api/validate misses. The unused v1.7.0 schemas are removed, so only 1.8.0 is vendored and embedded.
• embedded_schemas_match_vendored_files normalizes line endings, so CRLF checkouts on Windows pass, and skips when the source checkout isn't present, as in a relocated nextest archive.
• Docs: ocsf-json-export.mdx describes the per-class field changes (SSH dst_endpoint, Process actor.process, connection_info.direction_id, HTTP Activity without is_src_dst_assignment_known) and notes that top-level container is the one attribute OCSF 1.8.0 doesn't define.

User-visible: HTTP events drop is_src_dst_assignment_known; SSH events gain dst_endpoint; connection_info gains direction_id; Process Activity events gain actor.process (the openshell-sandbox launcher).

Deferred: top-level container is undefined in every OCSF version but identifies the affected sandbox on gateway events (#4283). The SIGTERM-failure Process Activity event still uses the Process::new("unknown", 0) fallback.

Testing

Rebased on main at 4b7486e73, which includes #4677 (device.os.type_id) and the gateway authentication events from #4728, #4761 and #4762.

• Real run on Podman (OPENSHELL_E2E_PODMAN_TEST=ocsf_jsonl_conformance mise run e2e:podman, gateway and supervisor images built from this branch): supervisor_jsonl_conforms_to_the_native_schema passes. It runs an L7 policy against a local test server, requires Network, HTTP and SSH Activity records, and validates every supervisor JSONL record against 1.8.0. An earlier version of the test, with raw sockets only, produced no HTTP Activity at all; requiring the classes caught that. Before the SSH fix it failed with "SSH Activity" 1.8.0 "SSH connection accepted on supervisor Unix socket": Missing at_least_one field from [dst_endpoint, src_endpoint].
• cargo test for openshell-ocsf (with test-support), openshell-server, openshell-supervisor, openshell-supervisor-network, openshell-supervisor-process, openshell-sandbox, openshell-otel: all pass (openshell-ocsf 202 unit tests, the other six crates 4299). The embedded-schema drift test also passes against a CRLF copy of embedded.rs.
• cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, and no clippy findings in the new e2e file
• Unit tests added/updated

ssh::exec_input_tests::eof_preserves_queued_input_until_the_child_reads has a 5 s deadline and hit it on this machine on both main and this branch in one run; it passed in the final run.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[johntmyers]

/ok to test 78621e8

[github-actions]

Label test:e2e applied for 78621e8. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

PR Review Status

The independent review found no blocking defects in the OCSF conformance and schema downgrade changes. @zanetworker, I checked your note about top-level container: it is present in the base, remains an acknowledged design question, and the downgrade preserves its data under unmapped.

Current-head testing has started with test:e2e and test:windows. Trivy and Helm are green; Branch Checks, E2E, and both Windows lanes are pending. The E2E Label Help bot requests Re-run all jobs, but the identified run still has an active first attempt.

Action required: rerun Branch E2E Checks after the active attempt finishes. Gator has operator authorization and the narrow sandbox permission to perform that rerun in a subsequent cycle.

Blocking findings: None.

Carried findings: None.

Gator metadata

• Validation: Focused OCSF correctness fixes for linked issues #4283 and #4284; operator requested review and relevant tests.
• Docs: Fern documentation updates describe OS identity, mixed-version JSONL, downgrade preservation, gateway configuration, and the kept-native metric.
• Checks: DCO, OpenShell / Helm Lint, and OpenShell / Trivy Changes pass; Branch Checks and both Windows MSVC PR lint and test lanes are pending on the reviewed head.
• E2E: test:e2e applied; current-head mirror verified; Branch E2E Checks run 37657018590 attempt 1 is active. Bot-required rerun remains outstanding. Relevant E2E coverage includes the new Podman JSONL conformance test.
• Head SHA: 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963
• Base SHA: 3fc93e28273fe304db634e6d5e9a52d57200389e
• Merge base SHA: 834b79a8c2351370e7c6c363423f5be18d440256
• Patch ID: 33dfd9e90bc20c4d4b8493efc716d9a8d3c32edb
• Gator payload: 11
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:blocked
• Blocked reason: test_dispatch_required

[johntmyers]

gator-agent

Author Follow-Up Nudge

This PR has been in gator:in-review for more than 48 business hours and required CI still needs attention.

@zanetworker, please make the vendored OCSF schemas available to the relocated Podman E2E test and push an update. The rootless Podman run fails in supervisor_jsonl_conforms_to_the_version_each_record_claims because the loader uses the build machine's CARGO_MANIFEST_DIR and cannot find schemas/ocsf/v1.1.0/classes/base_event.json on the remote test machine. Verify that the archived test can load its schemas and validate the JSONL records there.

@NVIDIA/openshell-maintainers, both Windows lanes also need a gate resolution: Clippy reports unnecessary_wraps and needless_pass_by_ref_mut in crates/openshell-cli/src/ssh.rs at lines 369, 376, and 423. That file is unchanged by this PR.

Gator metadata

• Head SHA: 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963
• Gator payload: 11
• Next state: gator:in-review
• Nudge scope: Existing CI follow-up; no new code-review disposition.

[zanetworker]

Fixed in aae174742: the vendored schemas are now compiled into the test binary (crates/openshell-ocsf/src/validation/embedded.rs, generated with a drift test), so archived binaries need no files from the build machine. Verified by running the ocsf_jsonl_conformance nextest archive from an unrelated directory with --workspace-remap and the repo's schemas/ hidden, against a Podman gateway: 1 passed. The Windows Clippy findings are in crates/openshell-cli/src/ssh.rs, which this PR doesn't change.

@johntmyers could you take a look, and /ok to test aae17474233af3ddd272f43557a01a7cf0c7f677 when you get a chance?

[johntmyers]

/ok to test aae1747

[johntmyers]

gator-agent

PR Review Status

Thanks @zanetworker (Adel Zaalouk), I checked your reply about Windows paths. The comparison and regeneration now use the same helper to join native path components with forward slashes, and the new regression test checks that key. The independent follow-up found no new blockers, and I resolved GATOR-aae17474-01.

I posted /ok to test for the current head. Branch Checks, Helm, E2E, and both Windows lanes are running or queued. The earlier E2E rerun instruction applies to an older head; there is no current-head rerun request. Native Windows verification remains with the Windows lanes.

Action required: @NVIDIA/openshell-maintainers, approve the current-head Trivy Changes workflow, which is awaiting workflow approval. Gator will continue inspecting the active tests; the earlier unrelated Windows CLI Clippy failures still need a maintainer resolution if they recur.

Blocking findings: None.

Carried findings: None. The Windows schema-path finding is fixed and its thread is resolved.

Gator metadata

• Validation: Continuing the validated OCSF correctness fixes for linked issues #4283 and #4284.
• Docs: Existing Fern updates cover user-visible changes; this delta changes test path normalization only.
• Checks: DCO passes. Branch Checks run 38129872736, Helm run 38129872726, and Windows run 38129872735 are active. Trivy Changes run 38129078889 concluded action_required and has no passing current-head result.
• E2E: test:e2e remains applied; current-head Branch E2E Checks run 38129873026 is in progress. test:windows remains applied and both lanes are active.
• Head SHA: 4f8ea841df8a061170bd0623124c8e109e9174c4
• Base SHA: eeba0e7954c0fb4d8e9e2e29d1bfa68e290eb8b3
• Merge base SHA: 834b79a8c2351370e7c6c363423f5be18d440256
• Patch ID: 3646f9d2c586ab7fe3c85fd231645d9dc6f787f2
• Gator payload: 11
• Review mode: follow_up
• Previous reviewed SHA: aae17474233af3ddd272f43557a01a7cf0c7f677
• Review budget exhausted: no
• Maintainer decision required: no — no review convergence decision is needed.
• Next state: gator:blocked
• Blocked reason: workflow_approval_required

[feloy]

The fix #4194 should help make the test:windows pass

[rhuss]

cc-review summary

Reviewed at 4f8ea841d with a 6-dimension agent pass plus CodeRabbit. The PR is functionally solid: it builds, both drift tests pass, the downgrade logic is correct (stripped attributes are preserved under unmapped.downgraded_attributes, versions are labeled only when they validate, else kept native), security is clean, and all four goals are delivered with genuine deferrals. No blocking defect.

Three non-blocking recommendations below, worth considering before or shortly after merge. Minor and Low findings (SIGTERM process pid, 1.1 unmapped test coverage, ConnectionDirectionId dead variants, kept-native observability split detail, setter duplication) are omitted to keep this focused; happy to share them if useful.

Nice touches: the downgrade never fabricates data and asserts byte-identity on kept-native; per-version required-field differences are grounded in the vendored schemas; type-state builders make missing required fields a compile error; both generated files are drift-guarded.

Source: cc-review (correctness, architecture, security, production, test-quality, goal-alignment agents + CodeRabbit)

[rhuss]

Important (observability): kept-native events are reported two ways. The gateway path emits openshell_ocsf_log_kept_native_total{class_uid,target}, while the sandbox path feeds the process-global KEPT_NATIVE mutex that the supervisor drains into a warn!. An operator alerting on the metric misses sandbox kept-native events, and kept_native_tally() reflects only one caller.

Consider a single sink: both paths emitting the metric, or both routing through record_kept_native with the metric/warn emitted from one drain point.

[zanetworker]

Addressed in 632853564: every output path now records kept-native events through the shared record_kept_native tally, so kept_native_tally() covers the gateway JSONL, supervisor JSONL, and OTLP export paths. The gateway also emits openshell_ocsf_log_kept_native_total (JSONL) and openshell_ocsf_otlp_kept_native_total (OTLP); the supervisor reports the tally as a warning from its settings poll loop, since it has no metrics exporter and logging from inside the tracing layer is not reliable. A supervisor metric would need a supervisor metrics pipeline, which is out of scope here.

[krishicks]

I do not think we should merge this. We should pick a specific schema version that is widely supported and stick to it rather than attempt lossy downgrades from a schema version no SIEM may support (e.g. 1.8)

[rhuss]

@krishicks on which schema version would you settle ? This directly impacts #2640 where to add any open traces to the event. 1.8 would be ideal as there is a typed section in the schema, otherwise its just an untyped key.

[zanetworker]

I've narrowed this PR to native OCSF 1.8 conformance. The downgrade changes are out, so it no longer touches the 1.1/1.3 downgrade and closes only #4283.

1.8 is what OpenShell emits by default today, and this PR makes that output correct:

• Native events from every builder, including Authentication, pass the official 1.8.0 validator (apart from top-level container, still open on bug(ocsf): native OCSF 1.8 events fail the official schema validator #4283).
• The schema validator recurses into nested objects, rejects undefined attributes, and honours declared profiles.
• Process Activity requires an actor and SSH Activity requires an endpoint at compile time.
• A Podman e2e test validates the supervisor's real JSONL against 1.8.0.

It's rebased on #4677 and uses that PR's device.os change.

#4284 stays open to pick the one version OpenShell emits (1.1 or 1.8) and to remove the downgrade. Until then the existing downgrade is unchanged. Nothing here is lost either way: if we stay on 1.8, #4284 only removes the downgrade and deprecates the setting; if we move to 1.1, that work builds on top of this one, since the builder fixes carry over and the validator and e2e show exactly which emitters need changes.

To inform that decision, we still need discovery on how the main SIEMs handle 1.8 records (Security Lake, Google SecOps, Splunk, Elastic): accepted, rejected, or stored but not queryable. That happens separately, with any doc or code changes in follow-up PRs.

@krishicks could you take another look?

[zanetworker]

SIEM spike results for the version decision are on #4284: #4284 (comment) (details in https://gist.github.com/zanetworker/dd3bf999d1f18e6557fa5b95bf87712d). In short, none of the ten sinks tested rejects native 1.8.0, which is the output this PR makes conformant.

[krishicks]

Thanks for doing the testing. This makes me feel much better about choosing 1.8, and makes more sense why SIEMs might say they support specific versions only (because that support is regarding specific OCSF classes), and why they might not necessarily upgrade promptly. I think we should do two things: Fix 1.8 (this PR), and then remove everything not related to 1.8 (downgrade, 1.7 schemas). With that in mind, I see a bunch of 1.1/1.3 or otherwise non-1.8 stuff in this PR that could/should probably be removed:

• Endpoint::hostname (crates/openshell-ocsf/src/objects/endpoint.rs:19, set at :44): from_domain now fills hostname as well as domain. In 1.8, domain alone is valid, so in native output hostname is a duplicate. Please drop the field and its Nones in the other constructors.
• hostname assertion and its 1.1/1.3 comment (endpoint.rs:140-141): "OCSF 1.1 and 1.3 accept only hostname as the endpoint's name." This tests behavior that only matters for versions we don't support.
• Policy DNS endpoint hostname (crates/openshell-supervisor-network/src/policy_dns/mod.rs:485) and the updated expectations at :990 and :1370: same issue, please revert.
• load_class_schema_for_version (crates/openshell-ocsf/src/validation/schema.rs:34, re-exported from validation/mod.rs): the only caller is the e2e test, which passes OCSF_VERSION, so load_class_schema is enough.
• Version selection in validate_required_fields (schema.rs:73-75): it loads object schemas for the event's metadata.version. With one supported version it should validate against OCSF_VERSION, and a record with any other version should be a hard failure rather than a schema lookup. The version parameter threaded through validate_attributes and the in OCSF {version} parts of the messages can go too.
• v1.7.0 schemas in embedded.rs (crates/openshell-ocsf/src/validation/embedded.rs, 25 v1.7.0/... entries): the generator embeds every vendored version, so 1.7.0 schemas are compiled into test-support binaries even though nothing loads them. Please limit the table to v1.8.0, or delete schemas/ocsf/v1.7.0/ entirely. (This could be done in a follow-up PR; this is leftover from the 1.8 upgrade.)
• README note (crates/openshell-ocsf/schemas/ocsf/README.md:10): "v1.7.0 is the previous native version; no test loads it." This should go along with the 1.7.0 schemas.
• "Any OCSF version" comment (crates/openshell-ocsf/src/builders/http.rs, in the HTTP builder test): "HTTP Activity does not define this attribute in any OCSF version." This assertion tests a field that was deleted, and every_builder_emits_schema_conformant_events already covers undefined attributes, so the test line can go too.

The 🤖 also found this, which would be nice to resolve in this PR:

The PR now has two overlapping validators with different rules:

• validate_known_fields (existing): top-level only, and rejects profile attributes whose profile isn't declared.
• validate_required_fields (expanded): recursive, but only checks that an attribute exists in the schema. An attribute from an undeclared profile passes.

Probably known and required fields should both be checked recursively.

[zanetworker]

Thanks @krishicks, all addressed in the updated commit:

• Endpoint::hostname removed, along with its constructor Nones, the 1.1/1.3 assertion, and the policy DNS endpoint change and its test expectations.
• load_class_schema_for_version removed; the e2e uses load_class_schema.
• Validation is against OCSF_VERSION only; an event labelled with any other version fails outright. The version parameter and the "in OCSF {version}" message parts are gone.
• v1.7.0 schemas deleted, with their embedded-table entries and the README note.
• The "any OCSF version" HTTP builder test line removed.
• The two validators share one walker. Both reject undefined attributes at every level and, when validating an event, attributes from profiles the event doesn't declare; validate_required_fields also checks required attributes and at_least_one. Objects are validated as objects, without event profile rules.

A review pass also turned up a few things, fixed here too:

• The embedded-schema drift test compared bytes exactly and would fail on CRLF Windows checkouts; it now normalizes line endings.
• tactic and technique are vendored, so a Detection Finding with an ATT&CK mapping validates. The per-builder test uses a fully populated finding.
• The e2e only ever produced Network Activity. It now runs an L7 policy against a local test server, so HTTP Activity is produced and validated too, and it requires Network, HTTP and SSH records. Copy failures report the podman error.

The PR now says "Refs #4283": #4283 stays open for top-level container, the one remaining validator exception. The tests check structure (required and defined attributes, profiles, at_least_one), not value types or enum ranges.

[houman44]

I captured supervisor JSONL from a released build (0.1.5, Homebrew, Docker Desktop on macOS arm64) and ran it through the official validator, as a before-picture for this PR.

Capture: ocsf_json_enabled=true on one sandbox, a denied TCP connect to example.com:443, and three sandbox exec calls. 12 records, taken from the supervisor's stderr (OCSF-JSON lines): 6 Base Event, 3 SSH Activity, 2 Network Activity, 1 Device Config State Change. All are labelled metadata.version 1.8.0.

Results against https://schema.ocsf.io/api/1.8.0/v2/validate:

Class	Records	Errors
Base Event	6	attribute_unknown container
SSH Activity	3	attribute_unknown container; constraint_failed at_least_one [dst_endpoint, src_endpoint]
Network Activity	2	attribute_unknown container
Device Config State Change	1	attribute_unknown container

That matches the description: top-level container is the #4283 exception, and the SSH endpoint gap is the one the new e2e found. No HTTP or Process Activity records came through in this run, so I can't confirm those fixes from released output.

Possible gap in the README's validation recipe. schemas/ocsf/README.md says to POST each record to /api/validate and that "an empty JSON object means the record is valid". On the hosted server, the v1 endpoint (/api/1.8.0/validate) returns only {"container": {"error": "Undefined attribute name", …}} for the SSH Activity records. It does not report the at_least_one failure that the v2 endpoint catches. Someone following the recipe would see SSH Activity pass apart from container. Could the recipe use /api/v2/validate? It returns errors, error_count and constraint failures. Also, an empty v1 object reads the same as a check that never ran.

I tested the hosted schema.ocsf.io only, not a local ocsf-server build, so local behaviour may differ.

Happy to rerun this on a build from this branch if that would help.

[zanetworker]

Thanks @houman44, this is a useful before-picture, and it matches what the PR targets: container (#4283, unchanged here) and the SSH Activity endpoint gap.

You're right about the recipe. The v1 endpoint misses at_least_one: for an SSH record without endpoints, /api/1.8.0/validate reports only container, while /api/1.8.0/v2/validate also reports the constraint failure. I've switched the README to the versioned v2 endpoint and to checking error_count (unversioned /api/v2/validate validates against 1.9.0, so the version is pinned).

A rerun on this branch would help, especially for HTTP and Process Activity, which your capture didn't produce. The e2e gets HTTP Activity by running an L7 policy against a local server and sending a request the policy denies; plain TCP to a denied host only produces Network Activity.

[krishicks]

/ok to test 8dbc022

[krishicks]

A few remaining items found by the bot on re-review:

1. Duplicate tests that use the deleted field (crates/openshell-ocsf/src/validation/schema.rs:279 and :473): rejects_attributes_the_class_does_not_define and known_fields_rejects_attributes_outside_the_schema both inject is_src_dst_assignment_known into a base event and expect a rejection. Now that both validators share one walker, these run the same code. Please keep one, and use a neutral name such as "not_an_ocsf_attribute" so the test doesn't refer to a field we removed.
2. Docs describe a field we don't emit (docs/observability/ocsf-json-export.mdx:155): "HTTP Activity records omit is_src_dst_assignment_known…" reads like a changelog entry; it tells readers about something that isn't in the output. Please drop this bullet. The SSH dst_endpoint, Process actor and connection_info direction bullets are useful and should stay.
3. Test of str::replace (schema.rs:411): embedded_table_comparison_ignores_crlf_checkouts only tests with_lf_line_endings, which is a single replace("\r\n", "\n"). Please keep the normalization in the drift test but remove this test.
4. Drift check can skip without anyone noticing (schema.rs:426): when the source checkout is missing, embedded_schemas_match_vendored_files prints a message and passes. That makes sense for relocated nextest archives. But if the CI job that runs this test only uses archived binaries, the drift check never actually runs. Can you confirm it runs from a full checkout in at least one CI job?
5. Test comment doesn't match what it shows (schema.rs:342): in object_schemas_are_validated_as_objects_not_events, the comment says "The top-level container exemption applies to events only". The device half passes because device defines container, not because of the exemption. Only the user half (from :346) shows what the comment describes. Please either move the comment to the user half or drop the device half.