Repository navigation
JVM GraphBatch: list, dict and date-time properties stay on the JSON bulk path - #244
Conversation
|
Test runs for this PR, read on 2026-11-04:
Every test that fails on main is one of the new shape tests (the shape gate, the tagged writer and the bridge decode do not exist there). The tests-only branch is deleted after this read. The timed A/B (one-library overlay on main's CI JVM wheel, rule written before the run) is queued on the laptop's timed lane. |
|
Timed A/B on head
Clean pairs per figure: 10, 11, 12; every A/A median between 0.968 and 1.043. By the rule written before the run (revised after the untimed probe and before any timed run, as the study README records) this is a SHIP: every gating interval is below 0.90, the list-of-floats guard shows no harm, and the scalar control is flat. The merge waits for the end of the current batch of timed studies that pin main (merging now would move their pins); the PR leaves draft at that point, with its docs. |
…hrough the JSON bulk path (#243) create_vertices and new_edges refused a whole call when one property was a list, dict, date-time, date, Decimal or bytes, and went per element through JPype. They now use json_bulk_dumps_tagged (insert_many's format from #82) and VertexBatcher/EdgeBatcher decode with DocumentBatcher.untag after toMap(). A chunk (vertices) or call (edges) holding a value the text cannot carry still takes the old path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…243) Step-0 probe (untimed): a list of 128 doubles cost 117 us per row through the text against 31 us on the property matrix (one double[]), and a scalar row 6.18 against 5.22 us. A list or tuple of only int/float now keeps main's path, and an input whose values are all scalars runs main's untagged json.dumps and gate (no shape test, no tagged writer); Java reads scalars as before (DocumentBatcher.value adds two type tests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the NaN (CI run 41731925082) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fallback rule, the laptop A/B figures (#243) Docs and docstring only. Figures copied by script from the study ab.txt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4d6c9c0 to
7627337
Compare
Closes #243
On the JVM wheel one list, tuple, dict, date-time, date, Decimal or bytes property sent the whole
GraphBatch.create_verticescall, and the wholeGraphBatch.new_edgescall with properties, down per-element Python (_to_java_property_matrix, ornew_edgeper edge). This carries those shapes throughjson_bulk_dumps_tagged, the tagged-value JSON #82 added forinsert_many.What changes:
json_bulk_shape_ok(type_conversion.py) is the cheap per-value gate for whatjson_bulk_scalar_okrefuses: the exact typeslist,tuple,dict,datetime,date,Decimal,bytes,bytearray. A non-empty list or tuple of onlyintandfloatis refused and keeps main's path: the untimed step-0 probe measured 117 us per row through the text against 31 us on the property matrix for a list of 128 doubles (onedouble[]). Subclasses (pandasTimestamp), numpy values, sets and pandas NA stay refused as before.json.dumps, and in Java the same per-key loop (DocumentBatcher.valueadds two type tests per value). The tagged writer and the shape gate are only reached when a row holds a shape._create_vertices_json_bulkandnew_edgeswrite the text withjson_bulk_dumps_tagged. A chunk of vertices that holds a value the text cannot carry (a NaN, an integer beyond 64 bits or a date-time inside a list or map, a subnormal, a non-str dict key) goes through the property-matrix path on its own. For edges every chunk is written before the first one crosses, so the same values start the per-edge path over with nothing buffered. A declaredARRAY_OF_FLOATSproperty whose list the float matrix cannot carry still sends its chunk to the matrix path, as in #496.VertexBatcherandEdgeBatcherdecode throughDocumentBatcher.value: a nested object or array becomes a plainMaporList(asDocumentBatcher.filldoes) and a tag is decoded byDocumentBatcher.untag(now package-private).Tests (
tests/test_graph_batch_tagged_shapes.py, JVM only, listed inNATIVE_NOT_APPLICABLE): every shape, declared and schemaless, is stored bycreate_verticesandnew_edgesas the property-matrix path andnew_edgeper edge store it (Python type andreprof every property compared); spies on_to_java_property_matrixandnew_edgeprove which path ran; a list of only numbers keeps the matrix path (alone and beside a dict); an input of scalars never reaches the shape gate or the tagged writer; values the text cannot carry keep the old path and store the same; chunk-level fallback with_BULK_CHUNKlowered; declared vector next to list, dict and date-time properties; the gates.tests/test_graph_batch.pyused adatetimeto reach the property-matrix path in two tests, and now uses a NaN.Known difference inside a nested container: a small integer inside a list or map arrives from the JSON text as a Java
Integerwhere the per-element path builds aLong. It reads back as the same Pythonint(the tests compare it); scalar integers already behaved this way on the bulk path.Overlap with #221 (
perf/jvm-create-vertices-jarray): #221 changes onlyis_vector_candidateintype_conversion.py; this PR adds functions elsewhere in that file and editsgraph_batch.pyand the bridge classes, so the two merge without a conflict. A JPypefloat[]ordouble[]on an undeclared property is not a shape and keeps the old path here; on a declaredARRAY_OF_FLOATSproperty #221 sends it to the float matrix.Status: the pre-registered timed wheel-against-wheel A/B on head
690d014shipped by its rule (results). Since then the head moved only by docs, a docstring, and a rebase onmainafter #236; the measured code is unchanged. The untimed step-0 probe figures that stood here are superseded by the timed ones.Docs updated in 7627337745 (rebased onto main after #236; the
:description:ofhow-to/bulk-loads.adocnow covers both topics and both sections are kept): a section inhow-to/bulk-loads.adoc(shapes on the bulk path, numeric lists keep the older path, the fallback rule, the laptop A/B figures with 95% intervals) and thenew_edgesdocstring. Edited 2026-11-04 to add this line.🤖 Generated with Claude Code
Edited 2026-11-05: the status line now gives the timed result; the PR is ready for review.