Repository navigation
Commit 99d034a
authored
perf(knowledge): reconcile a completed listing in one pass over its windows (#9449)
The reconciliation count read every window of a connector, then the ACL,
soft-delete and hard-delete walks each read the same windows again. The tally
now counts every decision input and keeps the rows each removal acts on in the
same window statement, so removals page through those rows instead of walking
the connector again. A condition matching more than 10,000 documents keeps only
its count and falls back to the existing walk, so memory and each statement's
result stay bounded.1 parent 6a269f9 commit 99d034a
9 files changed
Lines changed: 603 additions & 191 deletions
File tree
- apps/sim/lib/knowledge
- __integration__
- connectors
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
966 | 966 | | |
967 | 967 | | |
968 | 968 | | |
969 | | - | |
970 | | - | |
| 969 | + | |
| 970 | + | |
971 | 971 | | |
972 | 972 | | |
973 | 973 | | |
| |||
1002 | 1002 | | |
1003 | 1003 | | |
1004 | 1004 | | |
1005 | | - | |
1006 | | - | |
| 1005 | + | |
| 1006 | + | |
1007 | 1007 | | |
1008 | | - | |
| 1008 | + | |
1009 | 1009 | | |
1010 | 1010 | | |
1011 | 1011 | | |
| |||
1042 | 1042 | | |
1043 | 1043 | | |
1044 | 1044 | | |
1045 | | - | |
| 1045 | + | |
1046 | 1046 | | |
1047 | | - | |
1048 | | - | |
| 1047 | + | |
| 1048 | + | |
1049 | 1049 | | |
1050 | 1050 | | |
1051 | 1051 | | |
| |||
1069 | 1069 | | |
1070 | 1070 | | |
1071 | 1071 | | |
1072 | | - | |
| 1072 | + | |
1073 | 1073 | | |
1074 | 1074 | | |
1075 | 1075 | | |
| |||
1082 | 1082 | | |
1083 | 1083 | | |
1084 | 1084 | | |
1085 | | - | |
1086 | | - | |
1087 | | - | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
1088 | 1091 | | |
1089 | 1092 | | |
1090 | 1093 | | |
1091 | 1094 | | |
1092 | | - | |
| 1095 | + | |
1093 | 1096 | | |
1094 | 1097 | | |
1095 | 1098 | | |
| |||
1109 | 1112 | | |
1110 | 1113 | | |
1111 | 1114 | | |
1112 | | - | |
| 1115 | + | |
1113 | 1116 | | |
1114 | 1117 | | |
1115 | 1118 | | |
| |||
1119 | 1122 | | |
1120 | 1123 | | |
1121 | 1124 | | |
1122 | | - | |
1123 | | - | |
| 1125 | + | |
| 1126 | + | |
1124 | 1127 | | |
1125 | | - | |
| 1128 | + | |
| 1129 | + | |
1126 | 1130 | | |
1127 | 1131 | | |
1128 | 1132 | | |
1129 | | - | |
1130 | | - | |
| 1133 | + | |
1131 | 1134 | | |
1132 | 1135 | | |
1133 | 1136 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
131 | 131 | | |
132 | 132 | | |
133 | 133 | | |
134 | | - | |
135 | | - | |
136 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
137 | 137 | | |
138 | 138 | | |
139 | 139 | | |
| |||
278 | 278 | | |
279 | 279 | | |
280 | 280 | | |
281 | | - | |
282 | | - | |
283 | | - | |
| 281 | + | |
284 | 282 | | |
285 | 283 | | |
286 | 284 | | |
| |||
367 | 365 | | |
368 | 366 | | |
369 | 367 | | |
370 | | - | |
| 368 | + | |
371 | 369 | | |
372 | 370 | | |
373 | 371 | | |
| |||
855 | 853 | | |
856 | 854 | | |
857 | 855 | | |
858 | | - | |
| 856 | + | |
859 | 857 | | |
860 | 858 | | |
861 | 859 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
| 5 | + | |
6 | 6 | | |
7 | | - | |
8 | | - | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
9 | 11 | | |
10 | | - | |
11 | | - | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
12 | 17 | | |
13 | 18 | | |
14 | | - | |
15 | | - | |
16 | | - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
17 | 22 | | |
| 23 | + | |
18 | 24 | | |
19 | 25 | | |
20 | | - | |
21 | | - | |
22 | | - | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
23 | 31 | | |
24 | 32 | | |
25 | 33 | | |
| |||
43 | 51 | | |
44 | 52 | | |
45 | 53 | | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | 54 | | |
53 | | - | |
54 | | - | |
55 | | - | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
56 | 59 | | |
57 | | - | |
58 | | - | |
59 | | - | |
| 60 | + | |
| 61 | + | |
60 | 62 | | |
61 | | - | |
62 | | - | |
63 | | - | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
64 | 66 | | |
65 | 67 | | |
66 | 68 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | | - | |
10 | | - | |
| 10 | + | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | | - | |
14 | | - | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
15 | 38 | | |
16 | 39 | | |
17 | 40 | | |
18 | | - | |
| 41 | + | |
19 | 42 | | |
20 | 43 | | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
25 | 48 | | |
26 | 49 | | |
27 | 50 | | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
| 51 | + | |
36 | 52 | | |
37 | 53 | | |
38 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
39 | 61 | | |
40 | 62 | | |
41 | 63 | | |
42 | | - | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
43 | 92 | | |
44 | 93 | | |
45 | | - | |
46 | | - | |
| 94 | + | |
| 95 | + | |
47 | 96 | | |
48 | 97 | | |
49 | 98 | | |
50 | 99 | | |
51 | 100 | | |
52 | 101 | | |
53 | | - | |
| 102 | + | |
54 | 103 | | |
55 | 104 | | |
56 | 105 | | |
0 commit comments