Repository navigation
perf: CSV import ~1.5x faster (pipelined, byte-level parse, UTF-8 bytes into the writer) - #590
Merged
Merged
Conversation
dfa1
force-pushed
the
perf/csv-import-compact
branch
2 times, most recently
from
October 22, 2026 20:06
d727bfa to
12dc3ef
Compare
dfa1
changed the title
dfa1
force-pushed
the
perf/csv-import-compact
branch
from
October 22, 2026 20:53
877e5a3 to
e6541b5
Compare
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DictEncodingEncoder picked a string column's min/max with String::compareTo, UTF-16 order, where every other string bound in the writer uses UTF-8 byte order (Utf8Order), as Rust compares strings. A dictionary mixing a supplementary character (a surrogate pair in UTF-16) with U+E000..U+FFFF reported the bounds the wrong way round. The default vortex.zoned layout computes its own bounds, so this reached files only through the encoder stats (the legacy vortex.stats layout). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
writeChunk takes a Utf8 column as byte[][] as well as String[]: a caller that already holds UTF-8 (a CSV parser) skips decoding to String only for every encoder to encode it back. Rows are checked to be well-formed UTF-8 (Utf8Validation, RFC 3629, as Rust validates a Utf8 array). The encoders told Utf8 from Binary by carrier (String[] vs byte[][]); they now ask the dtype, through VarBinEncodingEncoder.textMinMax for the bounds, and Dict and Constant key and compare the bytes directly. The global dictionary keys Utf8 by String either way, so a column mixing both carriers codes each value once. Utf8BytesInputTest writes 5 column shapes x 3 option sets both ways and asserts byte-identical files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Profiled a 2.7 GB / 45M-row CSV import (YourKit, JFR with DebugNonSafepoints) and fixed what showed up. Output is byte-identical in default and compact modes. CSV import: - Parse the next chunk while a background thread converts and writes the previous one: parsing is the import's one sequential step. - CsvByteReader parses bytes into a column-wise ColumnChunk instead of FastCSV's chars and a String per field, and Utf8 columns reach the writer as UTF-8 byte[][]. Same format as FastCSV's defaults: CsvByteReaderTest reads 200 random files (quotes, "", every line ending, empty lines, multi-byte and malformed UTF-8, a BOM, 1-7-byte reads) against FastCSV. The delimiter must now be ASCII. - Numbers parse in place; a plain decimal whose digits fit 2^53 is digits / 10^k, bit-identical to Double.parseDouble (Clinger's fast path, 20 000 random decimals checked). A string column is checked for non-ASCII once per chunk, eight bytes at a time. Writer (CPU the profile pinned on it): - Zstd orders string zone stats by the bytes it already encoded (6.9 s -> 1.2 s of a compact import) instead of comparing Strings again. - Zone min/max kept as primitives: an ascending id boxed a Long per row. - Repartitioner sizes strings with getBytes(UTF_8).length, twice as fast as the hand-written count and exact by definition. - OnPair's first two trie levels are direct-indexed and deeper edges sit in a one-array table: it wins high-cardinality strings by default and its per-byte hashing was 38% of a default import. 2.7 GB CSV, interleaved runs: default 13.13 s -> 7.69 s, compact (loaded machine) 15.65 s -> 10.71 s. Tried and dropped: pooled zstd contexts, regrouping fields at parse time, and parallel block parsing (correct, but slower on a CPU-bound 10-core laptop). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
force-pushed
the
perf/csv-import-compact
branch
from
October 22, 2026 20:58
e6541b5 to
dbae512
Compare
dfa1
added a commit
that referenced
this pull request
Oct 22, 2026
Cut the Unreleased entries into 0.18.1 (2026-10-22) and list the dict bounds fix from #590 under Fixed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Profiled a 2.7 GB / 45M-row
import --compact(YourKit + JFR withDebugNonSafepoints) and fixed what showed up. Separate commits, rebase-merge.Result: original → this PR, 4 interleaved runs on a loaded machine: 15.65 s → 10.71 s (−32%), 583G → 520G instructions. Quiet machine: ~10.6 s → ~6.5 s. Output is byte-identical to
mainthroughout (sha of the 614 MB file unchanged).CsvByteReaderparses bytes into a column-wiseColumnChunk(nochardecode, noStringper field); numbers parse in place, strings go to the writer as UTF-8. Same format as FastCSV's defaults —CsvByteReaderTestchecks 200 random files (quotes,"", every line ending, empty lines, multi-byte and malformed UTF-8, BOM, 1–7-byte reads) against FastCSV itself. Only difference: the delimiter must be ASCII.writeChunkaccepts aUtf8column as UTF-8byte[][](validated,Utf8Validation, RFC 3629). Encoders tell Utf8 from Binary by dtype instead of carrier.Utf8BytesInputTest: 5 column shapes × 3 option sets written both ways → byte-identical files.String::compareTo(UTF-16 order) instead of UTF-8 byte order; reached files only via the legacyvortex.statslayout. No vortex-jni cross-check added for this one (needs a legacy-edition filtered read); unit test covers the order.Repartitionersizes strings withgetBytes(UTF_8).length.Tried and dropped: pooling zstd contexts, regrouping fields at parse time, and parallel block parsing — correct (speculative split + verification, byte-identical) but slower on a 4P+6E laptop: the machine is CPU-bound, so parse threads take CPU from the encoders (kept on a local
spike/parallel-csv-parsebranch for many-core machines)../mvnw verify -pl core,writer,csv,cli,integration -amgreen (3025 integration tests).🤖 Generated with Claude Code