Skip to content

JVM readers: a record or row to dict, the property names and result-set iteration in one crossing each - #233

Open
tae898 wants to merge 2 commits into
mainfrom
perf/jvm-reader-crossings
Open

tae898 wants to merge 2 commits into
mainfrom
perf/jvm-reader-crossings

Conversation

@tae898

@tae898 tae898 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #227

The JVM build read a record, a row, a list of property names and each row of a result set through many JPype calls: Document.to_dict() took 18 reads of Java objects for 3 properties, Result.to_dict() 14, the property names 12, and for row in rs two calls per row (counts of an untimed census, see the issue; no timing yet). A new bridge class, ReaderCalls (src/java/com/arcadedb/python/ReaderCalls.java, compiled with the other bridge classes into the same jar), cuts each to one call:

  • Result.to_dict() (results.py) and Document.to_dict() (graph.py): ReaderCalls.rowJson / documentJson write the row or record as the typed JSON of TypedRows (the transport ResultSet.to_list() already uses). Python makes one call, reads one array element and one string, and parses with the C json module; the side array of the engine's own objects is read only when the JSON carries the tag character. Values have the types convert_java_to_python gives them, and the key order is the engine's.
  • get_property_names() and property_names of both classes (_property_names_tuple): rowNames / documentNames return the names as one JSON text instead of one JPype read per name.
  • ResultSet.__next__ (results.py): nextOrNull does hasNext() and next() in one call. A row is still read when it is asked for, an error in the statement surfaces from the call as before, the end is the same None as hasNext() false, and the closed and exhausted states are unchanged.

Without the class (a jar that lacks it) every caller takes the old path. The native build is not touched. TypedRows.appendRow and TypedRows.quote lose private so the new class can use them.

One difference on purpose: Document.to_dict() now stores the names of the read it just made, so a property set after an earlier to_dict() or get_property_names() is no longer missing from the next to_dict() (the cached tuple of names was never refreshed on a mutable record). A record whose wrapped Java object is not a com.arcadedb.database.Document (a binary record) now raises a TypeError from the bridge call where it raised AttributeError for the missing getPropertyNames.

Tests (tests/test_reader_crossings.py, JVM only, listed in NATIVE_NOT_APPLICABLE): a counting proxy on the bridge classes and the Java objects, which also counts the reads of the arrays and the Java set they return, pins the exact calls (to_dict is the call and one array read; the old path is at least 7 for a record and 9 for a row; iteration is one call per row against two), and every value type of the wide table, the values SQL cannot produce, tag-named properties and unusual property names are compared with an independent per-property read of the Java object (same Python types, same key order). Closed-database errors, laziness of iteration and the close semantics are pinned too.

Overlap with #214 (issue #213): #214 edits RowAccess.java, DbCalls.java, results.py (ResultSet.one, ResultSet.count), graph.py (Document.get_raw) and core.py. This PR adds its Java in a new file, edits ResultSet.__next__, Result._property_names_tuple, Result.to_dict, Document._property_names_tuple and Document.to_dict, and one line of tests/conftest.py (next to the line #214 adds). The hunks do not touch, so the two merge in either order; the only shared line region is the NATIVE_NOT_APPLICABLE table in tests/conftest.py, which may need a trivial rebase.

Status: the pre-registered timed A/B of the main CI JVM wheel against this PR's CI JVM wheel shipped by its rule. Figures and scope are in this comment and in the docs passage this PR now adds.

Docs: the Results section of how-to/databases-and-queries.adoc has the four timed figures (Document.to_dict at 3 and 12 properties, iterating a result into dicts at 100 and 10,000 rows) with their 95% intervals and the control, added in 7cacef9.

🤖 Generated with Claude Code

Edited 2026-11-05: the status line now gives the timed result; the PR is ready for review.

@tae898

tae898 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

CI on head 3403dd2: workflow_dispatch run 41716185993 of test-jvm.yml, 18 of 18 jobs succeeded, including all 10 test legs (the new tests/test_reader_crossings.py ran on each). No timed run exists yet; the PR stays draft. The timed A/B is prepared with its rule and the GRAY rerun clause written before any run.

@tae898

tae898 commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Timed A/B of this PR on the JVM wheel (laptop timed lane, tools/ab.py --noise, ABBA, round 0 dropped, an A/A control). Arm A = main's CI JVM wheel 1b315c1318 (run 41716288781), arm B = this PR's CI JVM wheel at 3403dd2764 (run 41716185993). B/A median with its 95% interval (below 1 = this PR is faster):

workload B/A median [95% CI] A/A median [95% CI]
record to dict, 1,000 records of 3 properties 0.307 [0.303, 0.311] 0.997 [0.964, 1.002]
record to dict, 1,000 records of 12 properties 0.133 [0.132, 0.135] 1.000 [0.991, 1.005]
iterate a result into dicts, 100 rows 0.657 [0.649, 0.664] 0.995 [0.976, 1.014]
iterate a result into dicts, 10,000 rows 0.474 [0.454, 0.480] 1.001 [0.992, 1.005]

Clean pairs per figure: 12. By the rule written before the run (every gating upper end below 0.90, every control median inside [0.95, 1.05], a clean A/A) this is a SHIP. The control times a pure-Python json.dumps of rows fetched before the timer (no query, no JPype call, no code this PR changes); its B/A medians are 0.996, 0.987, 0.993, 1.000 on the four workloads.

This is the third run. The first two were invalid by the same rule and report nothing: their control was to_json_list over the same rows, whose timed region includes the query, and its A/A did not settle. The new control was declared in the study before this run started. to_json_list is still measured and reported (it never decides); on the 3-property records its B/A is 1.046 [1.012, 1.067].

The PR leaves draft at the end of the current batch of timed studies that pin main, with its docs, and merges on green. I can re-run the study on the branch after any change.

tae898 and others added 2 commits November 5, 2026 07:57
…et iteration in one crossing each (#227)

Document.to_dict took 18 reads of Java objects for 3 properties, Result.to_dict 14, a
property-names call 12, and each row of a result set two calls (hasNext, next). The new
bridge class ReaderCalls writes a record or a row as the typed JSON of TypedRows (one call,
one string read, the C json decoder), returns the names as one JSON text, and hands over
the next row in one call. Values keep the types convert_java_to_python gives them; the side
array is read only when the JSON refers to it; a result set is still read one row per call.
Without the class the old paths run. Document.to_dict now also refreshes the cached names,
so a property set after an earlier read is no longer missing from get_property_names().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lt iteration)

The study that adopts this PR timed record to dict at 3 and 12 properties and iteration into dicts at 100 and 10,000 rows on the arcadedb-jvm package. The Results section gets those four gating figures with their 95% intervals, the unit (time per record or row), the control that the change does not touch, and the scope (JVM build, these workloads, one laptop). The numbers are generated from the study's ab-score.md by script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 force-pushed the perf/jvm-reader-crossings branch from 7cacef9 to 64efea3 Compare November 4, 2026 22:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JVM wheel: record and row readers cross into the JVM many times (to_dict, property names, result-set iteration)

1 participant