[tae898]

find_nearest keeps its default: every hit carries its indexed vector, and the default search is the code of #203 (query as one float32 buffer), unchanged. This PR adds a keyword-only include_vector=False for callers that need only the ids, the distances or other properties of the hits.

On the native build, find_nearest(q, k, include_vector=False) runs SELECT *, !<vector property>, !record FROM (SELECT expand(vectorNeighbors(...))) and builds each hit without the vector (768 or 1536 floats per hit as JSON text, most of it read by nobody). A hit loads its record once, by RID, the first time the vector is read (get, has_property) or the whole record is needed (to_dict, get_property_names, modify, save). Values, property order and types are those of lookup_by_rid, and a hit deleted since the search raises ArcadeDBError on that first read. A caller that opts in and then reads the vector of every hit pays one extra lookup per hit, so the keyword is for searches that do not read it. The JVM build accepts the keyword and ignores it: it loads whole records either way, so the call costs the same (stated in the docstring and in docs/modules/ROOT/pages/how-to/vectors.adoc, which gets a section "Hits without the vector" with the figures below).

Why an opt-in and not the default. The first version of this PR made every hit lazy; its timed A/B showed that is a cost for callers that read the vector (256 dimensions: 1.336 [1.333, 1.346] for id and vector read, above the 1.10 cap for a default). So the default stays as it is. The timed A/B of this design (tools/ab.py --noise, ABBA, round 0 dropped, an A/A control, answers identical in every run; native CI wheel of main 7e64c5e, engine inside it 26.10.2; arm A is #203's code, arm B is A plus this PR's files at its pre-rebase head c91a828, whose change the rebase left as it was (range-diff differs in context lines only); find_nearest k = 10 on 100,000 vectors per dimension count; 12 clean pairs per figure) is below. It was run by the rule written before it (the opt-in faster with the upper interval end below 1.00, the default call non-inferior with no lower end above 1.03), and the rule gives SHIP as an explicit opt-in. B/A of the p50 per call, 95% interval, below 1 is faster:

call	p50 B/A at 1536-d	p50 B/A at 256-d
include_vector=False, hits not read	0.637 [0.633, 0.641]	0.8 [0.793, 0.818]
include_vector=False, hit ids read	0.637 [0.633, 0.638]	0.797 [0.793, 0.803]
include_vector=False, then every vector read (one lookup per hit)	1.0 [0.999, 1.004]	1.348 [1.34, 1.352]
default call, hits not read	1.0 [0.996, 1.002]	1.005 [0.996, 1.011]
default call, hit ids read	1.001 [0.997, 1.005]	1.002 [0.994, 1.005]
default call, every vector read	0.997 [0.995, 1.005]	0.997 [0.993, 1.007]

The A/A control is clean and the statement control (sql_id) is flat. The keyword is for callers that do not read the vector: at 256 dimensions the lookup per hit costs more than shipping the vector, so opting in and then reading every vector is slower than the default (1.348), which the docs say. The engine inside the wheel is 26.10.2, which still reads each record for the search row; whether the figures carry over to the 26.11.1 engine is not measured here.

This PR was stacked on #203; #203 is merged, and the branch is rebased onto main (5d6e47c) with the diff reduced to this change (src/arcadedb/_native/records.py, src/arcadedb/_native/schema_objects.py, src/arcadedb/vector.py) plus docs and tests. The conflicts were in save and modify of records.py, next to main's lost-update check (base=, #194) and point-operation glue (own=, #230); both are kept: a lazy hit loads its record before modify() takes the values save() compares, so the check sees the whole record.

Tests. tests/test_vector_hit_records.py runs on both backends: the default call and the opt-in call equal lookup_by_rid in type, property names and order, every value and the vector, for a four-bucket type with a subtype and in every first-access order; a hit can be modified and saved; ef_search and allowed_rids work with the keyword; the keyword is keyword-only and changes no answer; an opt-in hit of a type with properties called record and distance (the names of the search row's own columns) equals the lookup. tests/test_native_vector_hit_lazy.py is native only: the default search ships the vector, runs a statement that drops nothing and looks up nothing; the opt-in search ships no vector and looks up nothing, then one lookup per record however the vector is reached, the loaded vector equals the default's, and a closed database does a lookup for none. Distances in test_the_keyword_changes_no_answer compare as main's _assert_same does: exactly on the native build, to a few float32 ulps on the JVM build. Not run locally after the rebase (no workload on this machine); CI runs them. compile, pyflakes and the repo's pre-commit hooks (black, isort) are clean on the changed files.

Known and left alone: on the default path (this PR does not touch it), a type with a property called record or distance still gets a wrong or failing hit on native find_nearest, because those names are the search row's own columns. The opt-in path handles them. A fix of the default path would be a separate change.

Refs #19

Edited 2026-11-04: reworked from "lazy by default" to an explicit opt-in (include_vector=False), stacked on #203; the first body described the lazy-by-default design and its side fix for properties named record or distance on the default path.

Edited 2026-11-05: rebased onto main (#203 merged), docs section carries the measured figures of the opt-in study, which replace the earlier "study not run yet" paragraph.

🤖 Generated with Claude Code

[tae898]

Timed A/B (native wheel; arm A = main's CI native wheel, arm B = the same tree with only this PR's files, A/A control; laptop timed lane, tools/ab.py --noise, ABBA, round 0 dropped). find_nearest p50 latency, B/A median with 95% CI (below 1 = this PR is faster):

caller	1536-d	256-d
hits not read	0.66 [0.657, 0.663]	0.805 [0.803, 0.815]
each hit's id read	0.663 [0.659, 0.665]	0.808 [0.807, 0.812]
each hit's id and vector read	1.011 [1.008, 1.016]	1.336 [1.333, 1.346]
statement path (control, not changed by this PR)	1.001 [0.994, 1.006]	1.007 [0.994, 1.012]

Clean pairs per figure: 12; answers equal on both arms in every round. By the rule written before the run this is a DROP as the default: a caller who reads the vector of each hit pays one extra lookup per hit, 1.336 [1.333, 1.346] at 256 dimensions, above the rule's 1.10 regression cap, while callers who do not read the vector gain clearly.

So the cut is real but only for callers who do not need the vectors, which makes it an API question rather than a default. The options as I see them, with no choice made here: (1) an explicit opt-in on find_nearest that returns hits without their vector (other vector stores expose the same choice as an include-vector flag); (2) leave find_nearest as it is and close this PR. This PR stays draft until that is decided; an opt-in version would get its own timed check of both the opt-in and the default path.

[tae898]

Change of design, 2026-11-04. The first version of this PR made every find_nearest hit lazy. Its timed A/B (main's native wheel 7e64c5e, engine 26.10.2, 12 clean pairs, ab.py 95% CI) showed that it helps callers that do not read the vector (hits not read p50 B/A 0.66 [0.657, 0.663] at 1536-d and 0.805 [0.803, 0.815] at 256-d) and hurts callers that do (id and vector read 1.011 [1.008, 1.016] and 1.336 [1.333, 1.346], above the 1.10 cap for a default). So the default is back to what it is today: hits carry their vector, and the default search is #203's code. The lazy hits are now an explicit opt-in, find_nearest(..., include_vector=False), for callers who need only ids or distances; the JVM build accepts the keyword and ignores it.

The PR is rebased onto #203 (head 02103b5) and its base is now perf/native-find-nearest-ndarray; new head c91a828. The body is rewritten to match. A new pre-registered timed study will measure both paths (the default must stay non-inferior to #203, the opt-in must win when hits are not read or only the id is read); until it runs there is no speed claim and the PR stays a draft.

[tae898]

Timed A/B of the explicit opt-in on the native build (laptop timed lane, tools/ab.py --noise, ABBA, round 0 dropped, an A/A control; identical answers checked in every run). Arm A = main's CI wheel 7e64c5e5de with #203's head 02103b58fa, arm B = A plus this PR's head c91a8287d6. p50 per call, B/A median with its 95% interval (below 1 = this PR is faster):

call	role	1536-d	256-d
include_vector=False, hits used without the vector	gating	0.637 [0.633, 0.641]	0.8 [0.793, 0.818]
include_vector=False, hit ids read	gating	0.637 [0.633, 0.638]	0.797 [0.793, 0.803]
default call, hits used without the vector	non-inferiority	1.0 [0.996, 1.002]	1.005 [0.996, 1.011]
default call, hit ids read	non-inferiority	1.001 [0.997, 1.005]	1.002 [0.994, 1.005]
default call, every vector read	non-inferiority	0.997 [0.995, 1.005]	0.997 [0.993, 1.007]
include_vector=False, then every vector read (one lookup per hit)	reported	1.0 [0.999, 1.004]	1.348 [1.34, 1.352]

Clean pairs per figure: 12. By the rule written before the run this is a SHIP as an explicit opt-in: the opt-in calls are faster on both dimensions, the default call (which keeps today's behaviour, every hit carrying its vector) shows no lower interval end above 1.03, and the A/A control is clean. Opting in and then reading every vector costs one lookup per hit: 1.0 [0.999, 1.004] at 1536 dimensions and 1.348 [1.34, 1.352] at 256, so the keyword is for callers that do not read the vector; its docs will say so, with these figures, before it merges.

This PR is stacked on #203: it merges after #203, in the batch at the end of the current timed studies that pin main, with its docs (the keyword and these figures).