Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 48 additions & 1 deletion docs/modules/ROOT/pages/how-to/async-executor.adoc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= The async executor
:description: What db.async_executor() returns, how to submit SQL and transaction blocks without waiting, how to wait for them, how errors and callbacks reach you, what close() does to every caller, and what is thread-safe, on both builds.
:description: What db.async_executor() returns, how to submit SQL and transaction blocks without waiting, how to wait for them, how errors and callbacks reach you, what close() does to every caller, and what is thread-safe, on both builds, and how the native build hands records over in batches.

// Sources: src/arcadedb/async_executor.py (the JVM build), src/arcadedb/_native/extras.py (NativeAsyncExecutor, the native build), core.py (async_executor), and ArcadeDB 26.11.1 DatabaseAsyncExecutorImpl.java. Pinned by tests/test_async_executor.py, tests/test_async_transaction.py and tests/test_async_executor_behavior.py.

Expand Down Expand Up @@ -125,6 +125,53 @@ no error raised 51

Give every call whose result you need an `error_callback`. `executor.on_error(callback)` sets one handler for the whole executor, and `executor.on_ok(callback)` one that runs after each batch commit; see their docstrings for what each receives. A per-record callback of `create_record` runs when the record is applied to the batch, before the commit, and `on_ok` runs after it (`tests/test_async_executor.py` pins this order).

[[native-create-record]]
== Native: `create_record` hands records over in batches

On the `arcadedb` (native) package, from 26.11.1.post4, the async executor collects the records that `create_record` receives for one type and kind, and hands up to 1,000 of them to the library at once as one columnar frame. Each record keeps its own callback and its own error callback. Before, every record was one call with one JSON text. A thread of the executor hands a batch over when its first record has waited 0.05 seconds, so a record is not held back when no other call reaches the executor. The first 128 records of an executor go one by one, as before, and so does a hand-over of fewer than 64 collected records, because a frame has a fixed cost that a handful of records does not repay. The `arcadedb-jvm` package does not change.

An executor that receives only about 100 records pays for the per-record counter that lets the first records run one callback each: an async `create_record` of 100 records reads 1.060 [1.055, 1.067] on the laptop and 1.050 [1.045, 1.052] on the mini, that is about 6 percent more time for executors of 100 records. Executors of 1,000 records or more are faster. A linger thread shared by the process, tracked in ArcadeData/arcadedb-python#406, would remove this cost.

The table is one study of the `arcadedb` (native) package on a laptop and on a second Linux machine, the mini: the time for each operation of the post4 wheel against the 26.11.1.post3 wheel. The ratio is the median of the pair ratios (below 1 means less time) with its 95% interval. Setup: Python 3.12, one fresh process for each run, two cores for the process, the two builds alternated in both orders (ABBA), the first round dropped, and an A/A control (the post3 wheel twice) whose median ratio was between 0.994 and 1.002 on the laptop and between 0.989 and 1.002 on the mini. The post4 wheels were built by CI from commit e83947e, the commit of the tag, and the post3 wheels were the ones that CI built from the post3 commit. The files on PyPI were checked for the same engine as 26.11.1 (see xref:which-package.adoc#engine-of-post4[The engine of 26.11.1.post4]) and were not timed again.

[cols="5,3,3,2"]
|===
| What was timed | Ratio on the laptop, post4 over post3 (95% interval) | Ratio on the mini (95% interval) | Clean pairs (laptop, mini)

| async `create_record` of 100 records of one type, then `wait_completion()`
| 1.060 [1.055, 1.067]
| 1.050 [1.045, 1.052]
| 20, 16

| async `create_record` of 1,000 records of one type, then `wait_completion()`
| 0.342 [0.339, 0.345]
| 0.324 [0.322, 0.326]
| 20, 16

| async `create_record` of 100,000 records of one type, then `wait_completion()`
| 0.166 [0.164, 0.167]
| 0.153 [0.150, 0.155]
| 20, 16

| async `create_record` with a callback and an error callback, 1,000 records of one type, then `wait_completion()`
| 0.673 [0.668, 0.680]
| 0.561 [0.556, 0.568]
| 20, 16

| async `create_record` with a callback and an error callback, 100,000 records of one type, then `wait_completion()`
| 0.703 [0.700, 0.706]
| 0.591 [0.583, 0.599]
| 20, 16

| async `create_record` of 100,000 wide records of one type, then `wait_completion()`
| 0.210 [0.209, 0.212]
| 0.208 [0.206, 0.211]
| 20, 16

|===

The records that the batching holds before it hands them over use memory: up to 1,000 record copies at a time. Across the 172 figures of the whole post4 study the median of the peak resident size, post4 over post3, was 1.004, and the async figures that batch records used more (for example, a control figure of 1,000 records read 234 to 326 MiB on the mini).

== Blocks in a transaction

`executor.transaction(block)` runs `block` on a worker inside a transaction: the engine begins it, commits it, and rolls it back when the block raises. Inside the block, use `db` as you do in your own thread. `retries`, `ok_callback` and `error_callback` are optional, and `slot=n` sends the block to the worker with that number. Blocks sent to the same slot run in the order you submitted them, and the tests pin no other order, so do not depend on the order of `command` calls.
Expand Down
47 changes: 45 additions & 2 deletions docs/modules/ROOT/pages/how-to/bulk-loads.adoc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Bulk loads and commit size
:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, how insert_many treats Python's cyclic garbage collector on arcadedb-jvm, which property shapes a graph batch loads in bulk on the JVM build, how to pass time-series columns to append_samples (the address path is native only), a known limitation: a graph batch does not reuse the space of deleted edges, and how columnar loads, parallel loads and vertex loads through insert_dataframe and insert_arrow behave, including what a failed parallel load leaves behind.
:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, how insert_many treats Python's cyclic garbage collector on arcadedb-jvm, which property shapes a graph batch loads in bulk on the JVM build, how to pass time-series columns to append_samples (the address path is native only), a known limitation: a graph batch does not reuse the space of deleted edges, and how columnar loads, parallel loads and vertex loads through insert_dataframe and insert_arrow behave, including what a failed parallel load leaves behind, and how `GraphBatch.create_vertices` sends columns on the native build.

`insert_many`, `graph_batch` and `GraphBatch.create_vertices` load data in transactions. The commit size is the number of rows in one transaction. It decides how much work the engine holds in memory before it commits, and how many rows stay in the database when a load fails. Data that comes as a file, a DataFrame or an Arrow table has its own pages: xref:how-to/import-export.adoc[Importing and exporting data] and xref:how-to/dataframes-and-columns.adoc[DataFrames, Arrow and columns].

Expand Down Expand Up @@ -112,7 +112,7 @@ What a failure leaves behind:
* On the native package, an interrupt (`KeyboardInterrupt`) during a parallel call waits for the writers before it propagates, so no writer still runs when the exception reaches you.
* In a transaction you opened, `insert_arrow` and `insert_dataframe` with `parallel=True` do not join it. The writers commit on their own, and a rollback of your transaction does not remove those rows.

For one record at a time, `db.async_executor().create_record(document, callback=..., error_callback=...)` queues the record. The writers call `error_callback` with the exception when they reject it (a duplicate key under a `UNIQUE` index, or a batch abandoned at a failed commit). Without an `error_callback`, the failure reaches only the executor-wide `on_error` handler, if you set one. Call `wait_completion()` before you read the records back.
For one record at a time, `db.async_executor().create_record(document, callback=..., error_callback=...)` queues the record (on the native package it hands the records over in batches, see xref:how-to/async-executor.adoc#native-create-record[Native: create_record hands records over in batches]). The writers call `error_callback` with the exception when they reject it (a duplicate key under a `UNIQUE` index, or a batch abandoned at a failed commit). Without an `error_callback`, the failure reaches only the executor-wide `on_error` handler, if you set one. Call `wait_completion()` before you read the records back.

[[vertex-loads]]
== Vertex types in `insert_dataframe` and `insert_arrow`
Expand Down Expand Up @@ -183,6 +183,49 @@ The table is one laptop study of the `arcadedb-jvm` package: per-row time of thi

|===

== Native: `GraphBatch.create_vertices` sends columns

On the `arcadedb` (native) package, from 26.11.1.post4, a chunk of rows that `GraphBatch.create_vertices` creates crosses into the library as typed columns by address, in the layout of `insert_columns`, and the library writes the new record ids into one int64 buffer. Before, every chunk was one JSON text and the ids came back as a JSON list of text.

The route is taken when the rows of the chunk have the same keys in the same order and plain values. A chunk that the columns cannot carry (other keys or key order, nested lists or maps, a mixed-type column, huge integers) keeps the JSON route of 26.11.1.post3, so nothing is lost. A chunk of fewer than 64 rows keeps the JSON route as well, because the columns have a fixed cost that a handful of rows does not repay. The environment variable `ARCADEDB_NATIVE_GB_COLP_MIN_ROWS` sets that limit, and 0 turns it off. `ARCADEDB_NATIVE_GB_COLP=0` switches the whole route off. The engine method that creates the vertices, the transaction, the retries and the commit are the same as before. Float32 array vectors are stored after the other properties of the vertex, as they were.

Reads after a load by this route are not all faster. In the L2 graph study on the mini, with the same wheel and the route switched off for comparison, a point read of one vertex took 1.230 [1.196, 1.248] of the post3 time with the route on (24.056 to 29.494 microseconds per read) and 0.983 [0.976, 0.991] with it off. A read of the neighbours of one vertex took 0.774 [0.769, 0.791] with the route on and 0.995 [0.985, 0.998] with it off, and the load itself 0.830 [0.829, 0.834] with the route on. The cause of the read difference is not known yet. Measure the read path of your own workload after a column load, and set `ARCADEDB_NATIVE_GB_COLP=0` to keep the JSON route if point reads matter more than the load.

The table is one study of the `arcadedb` (native) package on a laptop and on a second Linux machine, the mini: the time for each operation of the post4 wheel against the 26.11.1.post3 wheel. The ratio is the median of the pair ratios (below 1 means less time) with its 95% interval. A ratio is quoted only when its interval is narrow enough to read. Setup: Python 3.12, one fresh process for each run, two cores for the process, the two builds alternated in both orders (ABBA), the first round dropped, and an A/A control (the post3 wheel twice) whose median ratio was between 0.997 and 1.006 on the laptop and between 0.981 and 1.000 on the mini. The post4 wheels were built by CI from commit e83947e, the commit of the tag, and the post3 wheels were the ones that CI built from the post3 commit. The files on PyPI were checked for the same engine as 26.11.1 (see xref:which-package.adoc#engine-of-post4[The engine of 26.11.1.post4]) and were not timed again.

[cols="5,3,3,2"]
|===
| What was timed | Ratio on the laptop, post4 over post3 (95% interval) | Ratio on the mini (95% interval) | Clean pairs (laptop, mini)

| `GraphBatch.create_vertices` of rows with the vector as a Python list, 100,000 rows per call
| 0.134 [0.133, 0.135]
| 0.118 [0.118, 0.118]
| 16, 17

| `GraphBatch.create_vertices` of rows with an int and a str property, 100,000 rows per call
| 0.778 [0.760, 0.786]
| 0.735 [0.734, 0.736]
| 16, 17

| `GraphBatch.create_vertices` of rows with an int and a str property, 100 rows per call
| 0.944 [0.911, 0.960]
| 0.894 [0.890, 0.901]
| 16, 17

| `GraphBatch.create_vertices` of rows with an int and a str property, 1 row per call
| no claim (interval too wide)
| 0.885 [0.872, 0.896]
| 16, 17

| `GraphBatch.create_vertices` of rows with a float32 vector as a numpy array, 100,000 rows per call
| 0.990 [0.983, 0.995]
| 0.953 [0.952, 0.964]
| 16, 17

|===

Memory did not improve. Across the 172 figures of the whole post4 study, the median of the peak resident size, post4 over post3, was 1.004, so there is no systematic change, and this page makes no memory claim for this route.

== Time-series samples with `append_samples`

`db.async_executor().append_samples(type_name, timestamps, *columns)` loads a columnar batch into a TIMESERIES type: the timestamps, then one column for each tag and field in the order of the type's declaration. Call `wait_completion()` before you read the samples back. How to declare the type is in the main ArcadeDB docs: see {arcadedb-docs-url}/concepts/timeseries[Time series]. The call exists on both packages. The rest of this section describes the fast path of the `arcadedb` (native) package. On `arcadedb-jvm`, a NumPy array for the timestamps or for a numeric column crosses as one buffer copy, and any other sequence converts value by value.
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/ROOT/pages/how-to/concurrency.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ This page covers what is specific to the Python bindings: which calls work from

== How to read the numbers

Every number on this page, except the ratios in <<reads-fewer-calls,Reads with fewer calls into the JVM>>, comes from probe scripts that were run on 2026-10-24 on one Linux x86_64 machine. The setup of each probe is next to its numbers. The builds in the table below are older than the current release, 26.11.1.post3, and the probes were not run again on it, so read each number as a measurement of the build that produced it. The scripts and their raw logs are not part of this repository yet.
Every number on this page, except the ratios in <<reads-fewer-calls,Reads with fewer calls into the JVM>>, comes from probe scripts that were run on 2026-10-24 on one Linux x86_64 machine. The setup of each probe is next to its numbers. The builds in the table below are older than the current release, 26.11.1.post4, and the probes were not run again on it, so read each number as a measurement of the build that produced it. The scripts and their raw logs are not part of this repository yet.

[cols="1,3,2"]
|===
Expand Down
49 changes: 49 additions & 0 deletions docs/modules/ROOT/pages/how-to/databases-and-queries.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -381,6 +381,55 @@ print(db.lookup_by_rid(rid).get("name")) # the same record, from

Compare record ids as `str(rid)`. A row also has an identity of its own: `result.get_rid()` returns it for a record row, an `expand()` row and a projection of `@rid` that does not alias it, and `None` for any other projection.

==== JVM: reading a record id is one call

On the `arcadedb-jvm` package, from 26.11.1.post4, `Result.get_rid()`, `Result.get_vertex()`, `Result.get_edge()` and `Result.get_element()` each make one call into the JVM, through one method of a small Java helper in the bridge jar. Before, each of these made three or four JPype calls. When the bridge class is not available, the methods keep the plain path. `Document.get_rid()` of a record object is not changed: it keeps the plain path of 26.11.1.post3. The `arcadedb` (native) package does not change.

The table is one study of the `arcadedb-jvm` package on a laptop and on a second Linux machine, the mini: the time for each operation of the post4 wheel against the 26.11.1.post3 wheel. The ratio is the median of the pair ratios (below 1 means less time) with its 95% interval. Setup: Python 3.12, one fresh process for each run, two cores for the process, the two builds alternated in both orders (ABBA), the first round dropped, and an A/A control (the post3 wheel twice) whose median ratio was between 0.988 and 1.011 on the laptop and between 0.994 and 1.010 on the mini. The post4 wheels were built by CI from commit e83947e, the commit of the tag, and the post3 wheels were the ones that CI built from the post3 commit. The files on PyPI were checked for the same engine as 26.11.1 (see xref:which-package.adoc#engine-of-post4[The engine of 26.11.1.post4]) and were not timed again.

[cols="5,3,3,2"]
|===
| What was timed | Ratio on the laptop, post4 over post3 (95% interval) | Ratio on the mini (95% interval) | Clean pairs (laptop, mini)

| `Result.get_rid()` over 1 row of a document type
| 0.735 [0.699, 0.754]
| 0.680 [0.666, 0.688]
| 15, 13

| `Result.get_rid()` over 100,000 rows of a document type
| 0.618 [0.605, 0.630]
| 0.596 [0.592, 0.605]
| 15, 13

| `Result.get_rid()` over 100,000 rows of a vertex type
| 0.604 [0.592, 0.622]
| 0.599 [0.589, 0.605]
| 15, 13

| `Result.get_rid()` over 100,000 rows of an edge type
| 0.616 [0.612, 0.626]
| 0.595 [0.591, 0.600]
| 15, 13

| `Result.get_vertex()` over 100,000 rows
| 0.743 [0.728, 0.767]
| 0.738 [0.724, 0.756]
| 15, 13

| `Result.get_edge()` over 100,000 rows
| 0.742 [0.727, 0.752]
| 0.739 [0.722, 0.763]
| 15, 13

| `Result.get_element()` over 100,000 rows
| 0.762 [0.748, 0.764]
| 0.755 [0.740, 0.777]
| 15, 13

|===

Memory did not improve: across the 172 figures of the whole post4 study the median of the peak resident size, post4 over post3, was 1.004, and this page makes no memory claim for these calls.

=== Pagination

Page a large type by `@rid` instead of `SKIP`: each page is `WHERE @rid > ? LIMIT ?`, and the cursor is the `@rid` of the last row of the previous page. Start with `"#-1:-1"`. The script above runs the loop with a page size of one so that two short pages print.
Expand Down
12 changes: 12 additions & 0 deletions docs/modules/ROOT/pages/which-package.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,18 @@ The API is the same. The runtime is not. These are the differences you can plan

Both packages run the same ArcadeDB engine, with the same SQL and openCypher and the same database format.

[[engine-of-post4]]
=== The engine of 26.11.1.post4

The engine of 26.11.1.post4 is the engine of 26.11.1 on both packages. Only the bindings changed. The check compared the Linux x86_64 wheels of 26.11.1.post4, as published on PyPI, with the published 26.11.1 wheels (read on 2026-11-16).

* `{jvm-pkg-name}`: all 70 engine jars of the wheel are the same files as in the 26.11.1 wheel, and `arcadedb-engine-26.11.1.jar` carries the same build number in both (`a3a5d124572608412aa187f9c4b290e071699473`). Only the jar of the bindings, `arcadedb-python-bridge.jar`, differs.
* `{native-pkg-name}`: the engine inside the library has the same build number and the same engine version, 26.11.1. The library file is built again for every wheel, so the file differs, and so does the source commit of the bindings (`e83947e646614bd43f149a8ea0054c8f682dcf11` for post4, the commit of the tag).

Two builds of the same source can differ in speed on one operation, so a figure that moves between two wheels of the same source is not by itself a change of the code (ArcadeData/arcadedb-python#404).

The single-record `UPDATE` through `command()` is slower in a paper cell on the published 26.11.1.post3 native wheel than on 26.11.1 (ArcadeData/arcadedb-python#400). The cause is the build of that wheel, not the bindings source: the wheel that CI built from the same commit runs the statement as fast as 26.11.1 (the table and the proposed process change are in #404). #400 stays open until a post4 native wheel has been measured in that cell on the mini.

[[switching]]
== Switch from one to the other

Expand Down
Loading