Skip to content

JVM GraphBatch: a list, dict or date-time property sends create_vertices and new_edges to per-element Python #243

Closed

Description

@tae898

On the JVM wheel, one list, tuple, dict, date-time, date, Decimal or bytes property in a row moves the whole GraphBatch.create_vertices call, and the whole GraphBatch.new_edges call with properties, off the JSON bulk path onto per-element Python.

_create_vertices_json_bulk in src/arcadedb/graph_batch.py returns None as soon as json_bulk_scalar_ok refuses one value, and the chunk goes through _to_java_property_matrix (one JPype conversion per element). new_edges does the same and then buffers every edge with new_edge (one crossing and one RID build per edge). #82 added tagged values to the JSON text for insert_many (json_bulk_dumps_tagged, decoded by DocumentBatcher.untag), so the text can already carry a date-time, date, Decimal and bytes that sit directly in a row, and any nested list or map of JSON values. The graph batches kept the untagged check.

Evidence so far is UNTIMED orientation only: a list property cost 91.4 us per row against 9.3 us for the same rows without it (one run, no timed lock). Scalar-only properties are engine bound (profile of ingest_graph on the JVM wheel: engine 80.7% of the calling thread, Java glue 1.7%), so only these shapes are affected. #496 (#33) covered declared ARRAY_OF_FLOATS vectors only.

Proposed change, in a draft PR: carry those shapes through json_bulk_dumps_tagged, decode them in VertexBatcher and EdgeBatcher with DocumentBatcher.untag (and toMap(), so a nested list or map is stored as a plain List or Map as insert_many does), and keep everything the text cannot carry on the old path: a NaN, an integer beyond 64 bits or a date-time inside a list or map, a subnormal, a non-str dict key, a set, a numpy or pandas value. The stored values must equal what the per-element path stores, types included; the PR's tests compare both.

A timed A/B of the PR (wheel against wheel, since the bridge jar changes) follows once CI has built the wheels.

Metadata

Metadata

Assignees

No one assigned

    Labels

    jvmThe arcadedb-jvm package (JPype build)performanceSpeed or memory of the bindings, with a measurement

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions