Skip to content

graph: bulk edge ingest has no batch form — every loader pays one statement per edge; add GRAPH INSERT/DELETE EDGES #369

Closed

Description

@EnRaiha

Version / build tested against

origin/main @ 1ff3551

Deployment mode

Origin, single node (local)

Engine(s) involved

graph, pgwire

Summary

There is no batch form for edge writes at the SQL front door. Inside the
server, batch already exists end to end: the physical plan carries GraphOp::EdgePutBatch
and EdgeDeleteBatch, staging has stage_edge_put_batch, the WAL encode/decode pair
exists, and the data plane has execute_edge_put_batch and execute_edge_delete_batch.
The only missing link is the statement: parse_insert_edge builds a single-edge
GraphStmt::GraphInsertEdge, and the DDL dispatch handles one edge.

A bulk loader therefore has to send one GRAPH INSERT EDGE per edge, and a delete plus
insert pair per edge when it needs idempotent replay. A 66k edge import that should take
minutes takes hours, because every edge costs a full statement round trip and its own
apply burst. Because the lower layers already batch, I expect the change to be smaller
than it looks: statement variant, parser, and the DDL handler that reaches the existing
batch ops.

I say this as the author of a separate ETL tool, but the gap is generic: any graph
import, any migration, and any backup restore hits it. The deciding question I applied
when writing this: would a generic bulk client need it? Yes.

Steps to reproduce

-- seed two nodes, then time the per-edge form
GRAPH INSERT EDGE IN 'kg_code_edges' FROM 'a' TO 'b' TYPE 'CONTAINS';
GRAPH INSERT EDGE IN 'kg_code_edges' FROM 'a' TO 'c' TYPE 'CONTAINS';

Observed: one apply per statement, and at scale the write settles into single-digit
edges per second under pressure. A 66k edge drain, measured on this machine, ran for
hours with rate pacing to stay inside cluster pressure.

Expected behavior

One statement carries N edges. Proposal:

GRAPH DELETE EDGES IN 'kg_code_edges' VALUES
  ('a','b','CONTAINS'), ('a','c','CONTAINS');
GRAPH INSERT EDGES IN 'kg_code_edges' VALUES
  ('a','b','CONTAINS'), ('a','c','CONTAINS');

Semantics I would expect, and would document in the PR:

  • One statement is one transaction, one apply burst, one response.
  • RLS is evaluated per edge, exactly as the single-edge form does.
  • A hard cap per statement, 1000 edges or 1 MiB of SQL text, whichever comes first,
    with a clear error when exceeded.
  • Idempotency stays with the caller: delete batch then insert batch, which is what a
    loader already does today.

Also please expose it natively

The batch should be a native protocol op too (GraphOp::EdgeInsertBatch and
EdgeDeleteBatch or equivalent), not only a pgwire statement, so native clients get the
same capability without going through SQL text.

What actually happened? (check all that are true)

  • Acknowledged/committed data was lost, corrupted, or silently wrong
  • The server crashed, hung, or failed to start
  • A security or isolation boundary was crossed
  • Core functionality is broken with no acceptable workaround
  • A workaround exists (send one statement per edge and pace the rate)

Proposed severity

SEV-3, feature gap with a workaround. Throughput only, no data risk.

Reproducibility

Always, every attempt.

Last known-good version / commit (if a regression)

Not a regression. The batch form never existed as far as I can find.

Environment & logs

Linux x86_64, single-node origin, trust mode, psql 16 and a Rust client over pgwire.
Bench numbers available on request; I will attach a before and after table in the PR if
someone picks this up, or in the issue if that helps triage.

Before submitting

  • I searched existing issues and this is not a duplicate.
  • I reproduced this on current main (not a stale branch).
  • This is not a security vulnerability.

Consolidated from #373: test(graph): batch write paths need RLS and GRANT probes

#373 cannot land before this issue (no batch statement exists yet), and its RLS and GRANT probes are this feature's acceptance tests. They are now part of this issue's acceptance criteria. Closing #373 as a duplicate of this issue.

Original body of #373

Version / build tested against

origin/main @ 1ff3551 (probe exists for the single-edge path only)

Deployment mode

Origin, single node (local)

Engine(s) involved

graph, security

Summary

Single-edge writes have RLS probes (graph_timeseries_rls_probe.rs and the
authorization cases). If batch write paths land (see the batch edge issue), they need the
same coverage before anyone trusts them with a least-privilege loader. A batch path that
evaluates policy once for the statement instead of once per edge would be an isolation
bug, and no test would catch it today.

Expected behavior

  • A wire test where a least-privilege user writes a batch containing one allowed edge
    and one denied edge, and asserts the denied edge does not land while the allowed one does.
    The single-edge path evaluates policy in resolve_edge_write_rls
    (nodedb/src/control/server/shared/ddl/neutral/graph_ops/edge_rls.rs:32) and carries the
    verdict per write in EdgePutParams.rls_write_check
    (nodedb/src/data/executor/handlers/graph_edge_write.rs:45). A batch must carry one
    verdict per edge the same way, and plan_permission.rs:171 already lists the batch
    variants in its permission map.
  • The same probe for UPSERT batches on node collections, since bulk loaders use both.

Actual behavior

Not applicable yet; batch paths do not exist. This issue is the coverage requirement that
should travel with them.

What actually happened? (check all that are true)

  • Acknowledged/committed data was lost, corrupted, or silently wrong
  • The server crashed, hung, or failed to start
  • A security or isolation boundary was crossed (hypothetically, untested)
  • Core functionality is broken with no acceptable workaround
  • A workaround exists (stay on single-edge writes, which are covered)

Proposed severity

SEV-2 if a batch path lands without this coverage. SEV-4 as a coverage request today.

Reproducibility

Not applicable.

Before submitting

  • I searched existing issues and this is not a duplicate.
  • This is not a security vulnerability report. It asks for test coverage, not a fix
    for an active hole.

Activity

  1. self-assigned this
    on Sep 27, 2026
  2. farhan-syah commented on Nov 3, 2026

    @farhan-syah
    Member

    Fixed in #443. GRAPH INSERT EDGE and GRAPH DELETE EDGE take several edges in one statement:

    GRAPH INSERT EDGE IN 'edges'
      FROM 'a' TO 'b' TYPE 'CONTAINS',
      FROM 'a' TO 'c' TYPE 'CONTAINS';
    • One statement is one write unit and one response. Edges on several vShards commit as one Calvin transaction.
    • RLS decides each edge. One refused edge refuses the statement, and no edge lands.
    • Caps: 1000 edges (54000) and 1 MiB of statement text.
    • Native and remote clients send a list as one batch request.

    The syntax extends the existing statement instead of adding EDGES ... VALUES.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions