You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Windows Python 3.14 parity: test_vector_read_your_writes missed 1 of 200 committed vectors once on main (intermittent) #409
The scheduled run of test-native.yml on main of 2026-11-16 (run 43799876573, commit 4c95ed1) failed one job, "Parity against the installed wheel (windows-x64, Python 3.14)", with one failed test out of 3,447: tests/test_vector_read_your_writes.py::test_through_the_facade_a_task_finds_the_vector_it_committed_in_the_call_before.
The assertion: 1 of 200 committed vectors were not found by the next search: [(396, [162, 169, 427, 165, 454])]. It is not a crash (the existing --retry-crash of the Windows legs did not apply: crash 0, flaky_crash 0) and not a timeout. The same job passed on a re-run of the failed job (attempt 2), and the two previous scheduled runs (2026-11-15 and 2026-11-14) were green, so it is intermittent.
Why it is worth an issue and not only a re-run: the test checks the property that a vector committed in one call is found by the next search, the one an agent loop depends on (#306). One miss in 200 can be an approximate-search recall miss for a vector queried by itself, or a real visibility race between the commit and the index on this platform. The artifact native-parity-windows-x64-py3.14 of the run holds the per-file log and XML.
Proposal: keep rerunning a failed Windows leg once, as we do, and treat a second failure as real; do not auto-retry assertion failures (only crashes are retried today). For this test: run it in a loop on a Windows runner with the number of misses logged, then decide between a bug in the commit-to-search path and a test that should query with a slack (the vector must be among the top k, not first). If it fails again in the next two weeks it is quarantined with a marker and a tracking line here.
The policy for rerunning Windows legs is also not written in CONTRIBUTING.md or the CI docs; I will add it in the docs pull request of 26.11.1.post4.
A second hit, and on Linux this time: PR #410 (the post4 release line into main), run 43863995512, job "Test arcadedb (linux/amd64, Python 3.12)", 3,596 passed and 1 failed: the same test, 1 of 200 committed vectors were not found by the next search: [(447, [271, 253, 169, 401, 386])]. So it is not Windows-specific and not a Python 3.14 matter. Hit 1 was the scheduled run on main at 4c95ed1 (before post4), so the cause is older than the post4 changes; the failing job is re-run to see whether it repeats on the same bytes.
Two hits in two full runs is too few to give a rate, so my next step is a Java repro on the engine API (a committed vector missing from the very next search), in a loop on JDK 21 and 25, before anyone treats it as a CI flake.
(Edited: the first version of this comment gave a rate of "two hits in about 7,000 test runs of this file"; that number counted the tests of two full suites, not runs of this test, so I removed it.)
Correction to my comment above: I removed a rate that I could not support (it counted the tests of two full suites, not runs of this test). The failing job on #410 is re-run; the loop repro comes next.
The failure reproduces locally, on Linux, and not only through the facade. The loop (600 planned iterations, one run of tests/test_vector_read_your_writes.py each, 4 tests per iteration) runs on the mini (i9-12900HK) in a systemd scope at nice 19 (I meant it to run on the 2 efficiency cores 12 and 13, but a user scope with AllowedCPUs does not pin a process on this host, so it ran on all 20 cores), with the native wheel that CI built from main (the artifact native-wheel-linux-x86_64 of run 44011962939, 26.12.1.dev0, commit 259e33639b), without conftest.py. The mini was loaded at the same time (load 10 to 24 from other jobs), which is closer to a shared CI runner than to a quiet machine.
Result so far: 194 iterations, 193 pass and 1 failure, at iteration 82 (2026-11-17 06:07:07Z): test_a_task_finds_the_vector_it_committed_in_the_call_before (the plain database path, not the AsyncDatabase facade of the CI failure) with 1 of 200 committed vectors were not found by the next search: [(349, [144, 164, 326, 253, 95])]. The same assertion as the CI hits (the corpus is searched with ef_search of 400 and every vector is the nearest neighbour of itself at distance 0, so a miss is a write that the next search did not see, not approximate recall).
So the race is not Windows-specific, not Python 3.14 specific and not a facade matter: it is in the commit-to-search path of the native build, at a rate of about one iteration in 190 under load. The failing output is saved with the loop's log (~/stage409/keep/ on the mini).
Next: let the loop finish its 600 iterations for a rate, run the JVM build and the published 26.11.1.post4 native wheel (without the two facade tests, which need the module that only main has) for comparison, and then reduce it to a Java loop on the engine API, to see whether the engine or the binding is the place.
(Edited 2026-11-17: the first version said the loop ran on 2 efficiency cores. It did not; it ran unpinned at nice 19 on a loaded machine. The failure counts are as read.)
More data, 2026-11-17 06:39Z, from the loop of the comment above. At 06:31Z it stood at 194 iterations with 1 failure (that count was right at that time); at 06:39Z it stands at 218 iterations with 9, that is 8 failures in the last 24 iterations, after I started four more loops on the same mini (the load average went from about 10 to 40). I re-read every saved output and counted a run as a failure only when pytest itself printed failed (a JVM run that prints 2 passed and then exits nonzero is the separate JVM exit crash, and is not counted here). All runs: nice 19 and NOT pinned to any cores (my pin did not work, see the correction below), the mini loaded by other jobs:
build
iterations
clean
real test failures
what failed
native from main (CI wheel, 26.12.1.dev0)
218
209
9
1 of 200 committed vectors were not found (8), 2 of 200 ... (1)
native, the published 26.11.1.post4 (the two tests without the facade, which post4 does not have)
30
24
6
1 of 200 ... (6)
JVM from main
13
0
4
1 of 200 ... (3) and RecordNotFoundException: Record #1:446 not found raised from the vector search (1); the other 9 runs passed and then exited nonzero
JVM, the published 26.11.1.post4 (two tests, no facade)
9
0
1
1 of 200 ... (1); the other 8 passed and exited nonzero
So the published release is affected, on the native and on the JVM build, not only main. The rate depends on the load of the machine: 1 failure in the first 194 iterations of the native loop, 8 in the next 24 while the load was four times higher. That points at a timing race in the engine's commit-to-search path and not at the Python binding. The failing vector is not always the same (349, 492, 447, 456 and others), and (447, [271, 253, 169, 401, 386]) is exactly the signature of the CI failure on #410: the JVM run reproduced it.
Next: a Java program on the engine API that replays the test (4 writers, 50 inserts each, a conflict retry, the search right after each commit) in a loop, against the released 26.11.1 jars and the main jars, so that it can be filed against the engine.
(Edited 2026-11-17: the first version of this comment said that my earlier count of 1 failure in 194 was wrong and called the mini quiet during the first 100 iterations. Neither was right: the count was correct at 06:31Z and the load during those iterations was not measured.)
(Second edit, 2026-11-17: the table's runs were unpinned, not on 2 efficiency cores each, as the first version of this line said; a user-level systemd scope with AllowedCPUs does not restrict cores on this host.)
Correction to my comment of 06:39Z: it said that my count of 1 failure in 194 was wrong. It was right at the time (06:31Z); the failures came later, when the load of the mini rose. I edited the comment in place.
Correction: my two comments above said the loop ran on 2 efficiency cores. It was not pinned: a user-level systemd scope with AllowedCPUs does not restrict cores on this host (no cpuset controller, Cpus_allowed_list stays 0-19), so it ran at nice 19 on all cores of a loaded machine. I edited both comments in place. The failure counts are as read.
The race is in the engine, and it needs more than one writer. I wrote a Java repro against the engine API (no Python, no bindings, Repro.java, about 470 lines): writer threads each commit an insert (its own transaction, retried on NeedRetryException, as run_in_transaction(retries=60) does) and then, in the same thread, search k=5, efSearch=400 for the vector they just committed; a miss is that vector absent from the hits. A scenario is a fresh database with 300 base vectors of dimension 8 and an index whose graph is built first, then 4 writers with 50 inserts each (or 1 writer with 200). The JVM sees 2 cpus (affinity read back: 12 and 13, or 14 and 15, nice 19, on the mini). Every number below is read from the TOTALS block of its run.
engine
JDK
writers
scenarios
committed inserts
scenarios with a miss
inserts missed
search exceptions
found by the immediate re-searches
still missing on them
missing after all writers ended
26.11.1 (a3a5d1245)
25
4
200
40,000
47 (24%)
152 (0.38%)
6
124
34
1
26.11.1 (a3a5d1245)
21
4
100
20,000
16 (16%)
18 (0.09%)
0
15
3
0
main 515aff93c6
25
4
200
40,000
63 (32%)
154 (0.39%)
5
126
33
2
main 515aff93c6
21
4
100
20,000
26 (26%)
28 (0.14%)
0
22
6
0
26.11.1 (a3a5d1245)
25
1
100
20,000
0 (0%)
0 (0.00%)
0
0
0
0
main 515aff93c6
25
1
100
20,000
0 (0%)
0 (0.00%)
0
0
0
0
main 515aff93c6, rebuild threshold raised to 1,000,000 mutations
25
4
100
20,000
16 (16%)
21 (0.10%)
1
17
5
0
The two re-search columns count misses and search exceptions together (RecordNotFoundException on loading a RID the index returned: 6 and 5 in the 200-scenario runs on JDK 25, none on JDK 21).
What it shows:
One writer never misses (100 scenarios, 20,000 inserts, on both engines). With four writers 0.1 to 0.4 percent of the committed inserts are missed, on JDK 21 and on JDK 25, on the released 26.11.1 and on main. So it is not the Python bindings and not one JDK.
Most misses are found by the three immediate re-searches straight after; a few are still missing on them, and of those some appear within a second and a few are not found within 3 seconds (JDK 21, main: 2 of 28). A vector that was still missing after every writer had ended was seen once and twice in the two 200-scenario runs on JDK 25 and never on JDK 21. That last one could mean a lost index entry; I have not looked into it yet.
The graph rebuild contributes but is not all of it: raising the rebuild threshold to 1,000,000 mutations lowers the rate (63 of 200 scenarios on main became 16 of 100) but not to zero.
The engine logs one WARNING per recovery in these runs: LSMVectorIndex Recovered 1 vectors from in-memory index (page-parsed: N, in-memory: N+1). 1,212 and 1,222 lines in the two 200-scenario runs on JDK 25, none in the one-writer runs. I do not know yet whether it is a cause or a symptom.
What I did not control: the runs shared the mini with other jobs at nice 19 and the load was not the same in each, so compare the rows by the yes or no of a miss, not as rates between JDKs (the rate rises with load, see the comment of 06:39Z).
Next: upstream has an open PR on the same subject, ArcadeData/arcadedb#12087 ("a commit replay holds its vectors back from searches until its commit concludes"). I am building its head and running the same repro on both JDKs, and then I will file the engine issue with Repro.java and these numbers.
Result of the check I announced in my last comment: the engine fix is in an open upstream pull request. I built the head of ArcadeData/arcadedb#12087 (ba7eabaa00, five commits over its merge-base 2eb615cd26: the three of #11969 and the two of that PR) and ran the same Java repro on JDK 21 and JDK 25, 100 scenarios each. With the PR head none of 40,000 committed inserts was missed (0 of 100 scenarios on each JDK). Its own merge-base, built the same way and run on the same cores, missed in 18 of 100 scenarios on JDK 21 (22 inserts) and 14 of 100 on JDK 25 (15 inserts), and main 515aff93c6 in 26 of 100 on JDK 21 and 63 of 200 on JDK 25. The table, the repro and the commands are in my comment on the PR: ArcadeData/arcadedb#12087 (comment)
So I will not file a separate engine issue; the PR already addresses it (a commit replay holds its vectors back from searches until its commit concludes), and it conflicts with main at the moment. I will run the repro again on its rebased head and on the first main build that contains it.
What this means for the bindings: until an engine release contains that change, tests/test_vector_read_your_writes.py can fail on any CI run when the load is high, on the released 26.11.1 engine that both published wheels carry, and so can an application that searches for a vector right after committing it from several threads. I have not changed any test or wheel for it; whether to mark the test as a known flake until then is open.
Activity
tae898 commentedon Nov 16, 2026
A second hit, and on Linux this time: PR #410 (the post4 release line into main), run 43863995512, job "Test arcadedb (linux/amd64, Python 3.12)", 3,596 passed and 1 failed: the same test,
1 of 200 committed vectors were not found by the next search: [(447, [271, 253, 169, 401, 386])]. So it is not Windows-specific and not a Python 3.14 matter. Hit 1 was the scheduled run onmainat 4c95ed1 (before post4), so the cause is older than the post4 changes; the failing job is re-run to see whether it repeats on the same bytes.Two hits in two full runs is too few to give a rate, so my next step is a Java repro on the engine API (a committed vector missing from the very next search), in a loop on JDK 21 and 25, before anyone treats it as a CI flake.
(Edited: the first version of this comment gave a rate of "two hits in about 7,000 test runs of this file"; that number counted the tests of two full suites, not runs of this test, so I removed it.)
tae898 commentedon Nov 16, 2026
Correction to my comment above: I removed a rate that I could not support (it counted the tests of two full suites, not runs of this test). The failing job on #410 is re-run; the loop repro comes next.
tae898 commentedon Nov 17, 2026
The failure reproduces locally, on Linux, and not only through the facade. The loop (600 planned iterations, one run of
tests/test_vector_read_your_writes.pyeach, 4 tests per iteration) runs on the mini (i9-12900HK) in a systemd scope at nice 19 (I meant it to run on the 2 efficiency cores 12 and 13, but a user scope withAllowedCPUsdoes not pin a process on this host, so it ran on all 20 cores), with the native wheel that CI built frommain(the artifactnative-wheel-linux-x86_64of run 44011962939,26.12.1.dev0, commit259e33639b), withoutconftest.py. The mini was loaded at the same time (load 10 to 24 from other jobs), which is closer to a shared CI runner than to a quiet machine.Result so far: 194 iterations, 193 pass and 1 failure, at iteration 82 (2026-11-17 06:07:07Z):
test_a_task_finds_the_vector_it_committed_in_the_call_before(the plain database path, not theAsyncDatabasefacade of the CI failure) with1 of 200 committed vectors were not found by the next search: [(349, [144, 164, 326, 253, 95])]. The same assertion as the CI hits (the corpus is searched withef_searchof 400 and every vector is the nearest neighbour of itself at distance 0, so a miss is a write that the next search did not see, not approximate recall).So the race is not Windows-specific, not Python 3.14 specific and not a facade matter: it is in the commit-to-search path of the native build, at a rate of about one iteration in 190 under load. The failing output is saved with the loop's log (
~/stage409/keep/on the mini).Next: let the loop finish its 600 iterations for a rate, run the JVM build and the published
26.11.1.post4native wheel (without the two facade tests, which need the module that onlymainhas) for comparison, and then reduce it to a Java loop on the engine API, to see whether the engine or the binding is the place.(Edited 2026-11-17: the first version said the loop ran on 2 efficiency cores. It did not; it ran unpinned at nice 19 on a loaded machine. The failure counts are as read.)
tae898 commentedon Nov 17, 2026
More data, 2026-11-17 06:39Z, from the loop of the comment above. At 06:31Z it stood at 194 iterations with 1 failure (that count was right at that time); at 06:39Z it stands at 218 iterations with 9, that is 8 failures in the last 24 iterations, after I started four more loops on the same mini (the load average went from about 10 to 40). I re-read every saved output and counted a run as a failure only when pytest itself printed
failed(a JVM run that prints2 passedand then exits nonzero is the separate JVM exit crash, and is not counted here). All runs: nice 19 and NOT pinned to any cores (my pin did not work, see the correction below), the mini loaded by other jobs:main(CI wheel,26.12.1.dev0)1 of 200 committed vectors were not found(8),2 of 200 ...(1)26.11.1.post4(the two tests without the facade, which post4 does not have)1 of 200 ...(6)main1 of 200 ...(3) andRecordNotFoundException: Record #1:446 not foundraised from the vector search (1); the other 9 runs passed and then exited nonzero26.11.1.post4(two tests, no facade)1 of 200 ...(1); the other 8 passed and exited nonzeroSo the published release is affected, on the native and on the JVM build, not only
main. The rate depends on the load of the machine: 1 failure in the first 194 iterations of the native loop, 8 in the next 24 while the load was four times higher. That points at a timing race in the engine's commit-to-search path and not at the Python binding. The failing vector is not always the same (349, 492, 447, 456 and others), and(447, [271, 253, 169, 401, 386])is exactly the signature of the CI failure on #410: the JVM run reproduced it.Next: a Java program on the engine API that replays the test (4 writers, 50 inserts each, a conflict retry, the search right after each commit) in a loop, against the released 26.11.1 jars and the
mainjars, so that it can be filed against the engine.(Edited 2026-11-17: the first version of this comment said that my earlier count of 1 failure in 194 was wrong and called the mini quiet during the first 100 iterations. Neither was right: the count was correct at 06:31Z and the load during those iterations was not measured.)
(Second edit, 2026-11-17: the table's runs were unpinned, not on 2 efficiency cores each, as the first version of this line said; a user-level systemd scope with
AllowedCPUsdoes not restrict cores on this host.)tae898 commentedon Nov 17, 2026
Correction to my comment of 06:39Z: it said that my count of 1 failure in 194 was wrong. It was right at the time (06:31Z); the failures came later, when the load of the mini rose. I edited the comment in place.
tae898 commentedon Nov 17, 2026
Correction: my two comments above said the loop ran on 2 efficiency cores. It was not pinned: a user-level systemd scope with
AllowedCPUsdoes not restrict cores on this host (no cpuset controller,Cpus_allowed_liststays 0-19), so it ran at nice 19 on all cores of a loaded machine. I edited both comments in place. The failure counts are as read.tae898 commentedon Nov 17, 2026
The race is in the engine, and it needs more than one writer. I wrote a Java repro against the engine API (no Python, no bindings,
Repro.java, about 470 lines): writer threads each commit an insert (its own transaction, retried onNeedRetryException, asrun_in_transaction(retries=60)does) and then, in the same thread, searchk=5,efSearch=400for the vector they just committed; a miss is that vector absent from the hits. A scenario is a fresh database with 300 base vectors of dimension 8 and an index whose graph is built first, then 4 writers with 50 inserts each (or 1 writer with 200). The JVM sees 2 cpus (affinity read back: 12 and 13, or 14 and 15, nice 19, on the mini). Every number below is read from the TOTALS block of its run.a3a5d1245)a3a5d1245)515aff93c6515aff93c6a3a5d1245)515aff93c6515aff93c6, rebuild threshold raised to 1,000,000 mutationsThe two re-search columns count misses and search exceptions together (
RecordNotFoundExceptionon loading a RID the index returned: 6 and 5 in the 200-scenario runs on JDK 25, none on JDK 21).What it shows:
LSMVectorIndex Recovered 1 vectors from in-memory index (page-parsed: N, in-memory: N+1). 1,212 and 1,222 lines in the two 200-scenario runs on JDK 25, none in the one-writer runs. I do not know yet whether it is a cause or a symptom.What I did not control: the runs shared the mini with other jobs at nice 19 and the load was not the same in each, so compare the rows by the yes or no of a miss, not as rates between JDKs (the rate rises with load, see the comment of 06:39Z).
Next: upstream has an open PR on the same subject, ArcadeData/arcadedb#12087 ("a commit replay holds its vectors back from searches until its commit concludes"). I am building its head and running the same repro on both JDKs, and then I will file the engine issue with
Repro.javaand these numbers.tae898 commentedon Nov 17, 2026
Result of the check I announced in my last comment: the engine fix is in an open upstream pull request. I built the head of ArcadeData/arcadedb#12087 (
ba7eabaa00, five commits over its merge-base2eb615cd26: the three of #11969 and the two of that PR) and ran the same Java repro on JDK 21 and JDK 25, 100 scenarios each. With the PR head none of 40,000 committed inserts was missed (0 of 100 scenarios on each JDK). Its own merge-base, built the same way and run on the same cores, missed in 18 of 100 scenarios on JDK 21 (22 inserts) and 14 of 100 on JDK 25 (15 inserts), and main515aff93c6in 26 of 100 on JDK 21 and 63 of 200 on JDK 25. The table, the repro and the commands are in my comment on the PR: ArcadeData/arcadedb#12087 (comment)So I will not file a separate engine issue; the PR already addresses it (a commit replay holds its vectors back from searches until its commit concludes), and it conflicts with main at the moment. I will run the repro again on its rebased head and on the first main build that contains it.
What this means for the bindings: until an engine release contains that change,
tests/test_vector_read_your_writes.pycan fail on any CI run when the load is high, on the released 26.11.1 engine that both published wheels carry, and so can an application that searches for a vector right after committing it from several threads. I have not changed any test or wheel for it; whether to mark the test as a known flake until then is open.