Repository navigation
perf: preserve and process Parquet dictionaries efficiently - #26250
Closed
Rich-T-kid wants to merge 6 commits into
Closed
Rich-T-kid wants to merge 6 commits into
Rich-T-kid wants to merge 6 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #26250 +/- ##
========================================
Coverage 82.92% 82.93%
========================================
Files 1154 1154
Lines 457872 458338 +466
Branches 457872 458338 +466
========================================
+ Hits 379712 380134 +422
- Misses 55136 55153 +17
- Partials 23024 23051 +27 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
|
will keep this PR in draft mode and wait for the next arrow-rs bump as apache/arrow-rs#11407 should close the gap for the sort operator, and #26186 & #25874 will remove the un-needed cast in the project operator. Good results overall, once the parquet scan speed up is in I expect dictionarys to be faster than utf8View for low to medium queries |
This was referenced Oct 13, 2026
Rich-T-kid
force-pushed
the
dict-encoding-bench
branch
from
October 17, 2026 01:26
2b497c8 to
fdf73af
Compare
Contributor
Author
|
run benchmark parquet_rle_dictionary_scan baseline: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"CPU Details (lscpu)Details
Resource Usageparquet_rle_dictionary_scan — base (merge-base)
parquet_rle_dictionary_scan — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark parquet_rle_dictionary_scan baseline: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"CPU Details (lscpu)Details
Resource Usageparquet_rle_dictionary_scan — base (merge-base)
parquet_rle_dictionary_scan — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to fbf7737 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "fbf77371a0512765ebf81fdc60e4a6cdceb8101f"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark parquet_rle_dictionary_scan baseline: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"Results will be posted here when complete File an issue against this benchmark runner |
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned env: baseline: changed: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
Hi @Rich-T-kid, thanks for the request (#26250 (comment)). Usage: Any benchmark name is accepted: Per-side configuration ( env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2GPod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal): resources:
cpu: "16"
memory: "128Gi"
arch: arm64 # arm64 or amd64Independent benchmark shards (optional; 1–8, default 1): run benchmark arrow_writer
shards: 4Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion. File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 72e7d68 (72e7d68) to 72e7d68 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 72e7d68 (72e7d68) to 6096dab diff Run configurationrun benchmark parquet_rle_dictionary_scan
baseline:
ref: "6096dab24154d1a954e86004d5ac5a9a067f26e1"
changed:
ref: "72e7d6871e886864d7abb533185f586ccecb938c"CPU Details (lscpu)Details
Resource Usageparquet_rle_dictionary_scan — base (merge-base)
parquet_rle_dictionary_scan — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 38b3d06 (38b3d06) to 0179f67 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "0179f67a5ec2392018f0deebaca89b74cf24ce07"
changed:
ref: "38b3d06b895a45ffebf48c01ddc043de46afeb5c"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 38b3d06 (38b3d06) to 38b3d06 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "38b3d06b895a45ffebf48c01ddc043de46afeb5c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "38b3d06b895a45ffebf48c01ddc043de46afeb5c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
dict-encoding-bench
branch
from
October 20, 2026 05:51
38b3d06 to
cb09c73
Compare
Contributor
Author
|
run benchmarks tpch clickbench_partitioned env: baseline: changed: |
Contributor
Author
|
run benchmarks tpch clickbench_partitioned env: baseline: changed: |
Contributor
Author
|
run benchmark dictionary_group_values baseline: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing cb09c73 (cb09c73) to cb09c73 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "cb09c73e3"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "cb09c73e3"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing cb09c73 (cb09c73) to 6213ad2 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "6213ad2e3"
changed:
ref: "cb09c73e3"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 38b3d06 (38b3d06) to 38b3d06 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "38b3d06b895a45ffebf48c01ddc043de46afeb5c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "38b3d06b895a45ffebf48c01ddc043de46afeb5c"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing cb09c73 (cb09c73) to cb09c73 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "cb09c73e3"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "cb09c73e3"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 38b3d06 (38b3d06) to 0179f67 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "0179f67a5ec2392018f0deebaca89b74cf24ce07"
changed:
ref: "38b3d06b895a45ffebf48c01ddc043de46afeb5c"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing cb09c73 (cb09c73) to 6213ad2 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "6213ad2e3"
changed:
ref: "cb09c73e3"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 38b3d06 (38b3d06) to 0179f67 diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "0179f67a5ec2392018f0deebaca89b74cf24ce07"
changed:
ref: "38b3d06b895a45ffebf48c01ddc043de46afeb5c"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing cb09c73 (cb09c73) to 6213ad2 diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "6213ad2e3"
changed:
ref: "cb09c73e3"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing cb09c73 (cb09c73) to cb09c73 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "cb09c73e3"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "cb09c73e3"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing cb09c73 (cb09c73) to 6213ad2 diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "6213ad2e3"
changed:
ref: "cb09c73e3"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark with_hashes baseline: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 1014e2e (1014e2e) to 6213ad2 diff Run configurationrun benchmark with_hashes
baseline:
ref: "6213ad2e3"
changed:
ref: "1014e2ed8"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing cb09c73 (cb09c73) to cb09c73 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "cb09c73e3"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "cb09c73e3"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing cb09c73 (cb09c73) to 6213ad2 diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_EXECUTION_PARQUET_SCHEMA_FORCE_VIEW_TYPES: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "6213ad2e3"
changed:
ref: "cb09c73e3"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing cb09c73 (cb09c73) to 6213ad2 diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "6213ad2e3"
changed:
ref: "cb09c73e3"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 1014e2e (1014e2e) to 6213ad2 diff Run configurationrun benchmark with_hashes
baseline:
ref: "6213ad2e3"
changed:
ref: "1014e2ed8"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
None. This PR fixes dictionary expansion in Parquet
binary_as_stringscansand the downstream sparse-dictionary costs that became visible after preserving
the Parquet dictionaries.
Why
Whether dictionary output beats
Utf8Viewdepends on more than Parquet decodespeed. The dictionary must remain compact throughout the physical plan.
ClickBench's string columns are unannotated Parquet
BYTE_ARRAYs.binary_as_stringchanges their public table type toDictionary<Int32, Utf8>, but the per-file opener still sees physicalBinary. Schema coercion rejected that binary/string family mismatch, so thereader expanded each Parquet dictionary to plain binary and
DecoderProjectionhashed every row to rebuild it. This was the first sourceof the roughly 3x low-cardinality Q10/Q11 regression.
After fixing the reader path, high-cardinality Q12/Q33 were still slow. The
remaining problem was that small logical batches retained very large dictionary
value arrays:
(276-545 MiB),
coalescer repeatedly merged it, and
group table and a second dictionary-local table.
The key invariant is therefore not just "decode to dictionary". Dictionary
values must also be compact relative to each batch at operator boundaries.
What changed
Parquet reader path
Dictionary<K, Binary>(orLargeBinary) from the Parquet readerwhen the logical output is
Dictionary<K, Utf8>(orLargeUtf8). The publicscan schema remains the requested string dictionary.
DecoderProjectionand reuse them across output batches.binary_as_string + RLE_DICTIONARY + per-file schema coercion, including multiple output batches and dictionary reuse.parquet_rle_dictionary_scan, a 1,000,000-row Criterion sweep over NDVs1 through 500,000 for annotated UTF-8 and unannotated binary sources.
Downstream dictionary operators
the array slice; dense dictionaries retain the hash-once/scatter path.
small batch does not retain the complete set of groups.
guarantee instead of hashing and deduplicating the same strings again.
coalescing; dense dictionaries remain a zero-copy no-op.
multi-column hashes, compact aggregate output, alternate dictionary key
mappings, and repartition compaction.
Physical-plan construction remains outside timed Parquet benchmarks. Every
timed drain receives a fresh physical plan because Parquet scan plans contain
one-shot stream state.
The immutable benchmark commits
acc69cd00and1121f2b3btemporarily pinnedArrow to apache/arrow-rs#11462,
which is the intended post-merge configuration. The current head restores the
released Arrow 60 dependency so unrelated post-release Arrow changes do not
leave DataFusion CI red.
Results
Parquet scan regression
The Neoverse-V2 Criterion run
compared the immutable pre/fixed reader commits:
Utf8ViewUtf8ViewUtf8Viewby about 8%The annotated UTF-8 control was unchanged, isolating the delta to the
unannotated-binary coercion path.
End-to-end ClickBench
The reader-only benchmark-bot comparison
made Q10 3.08x faster and Q11 2.80x faster, bringing those low-cardinality
queries to parity with
Utf8View. It did not improve high-cardinality Q33,which confirmed that its remaining cost was downstream of the scan.
The final benchmark-bot comparison
holds RLE-dictionary output enabled on both sides and compares the reader-only
commit with the complete operator fixes:
URL, count distinctUserIDSearchPhraseURLUserID/SearchEngineIDUserID, filtered URLURLEleven queries improved, none regressed, and 32 were unchanged by the operator
fixes. In particular, Q10/Q11 were already fixed by the reader-path change and
remain unchanged here.
Q33 summed operator metrics show where the time went. These are diagnostic
parallel-worker sums, so they are not additive wall-clock time:
This confirms that repeated sparse-dictionary merging—not Parquet decoding—was
the dominant Q33 regression after the reader fix.
The separate final-head
Utf8Viewversus dictionary runshows the remaining end-to-end gap:
Utf8ViewThe PR removes the pathological expansion and sparse-backing costs; it does
not make dictionary output universally faster than
Utf8View. The remaininggap is ordinary dictionary decode, hashing, key indirection, value merging, and
group-table work. It is no longer a second expansion/rebuild cycle.
Focused operator benchmarks
Sparse dictionary hashing now scales with logical rows rather than retained
dictionary values. The table is a local controlled A/B because the sparse
cases do not exist on the baseline commit:
The hosted sparse-hash run
also showed the pre-existing dense dictionary cases unchanged. Its new
8,192-row sparse cases took 39.1 us with 8,192 values and 79.3-91.3 us with
65,536-1,048,576 values, confirming that runtime is no longer proportional to
the entire backing dictionary.
The hosted 65,536-cardinality dictionary group benchmark
also improved:
take_nTPC-H control
TPC-H strings are annotated UTF-8 and generally use compact dictionaries. In
the hosted reader-only versus complete-fix control,
total SF1 time moved from 798.11 to 782.69 ms: 2 queries improved, 0 regressed,
and 20 were unchanged. This is expected because the new sparse paths are gated
and dense dictionaries retain the existing behavior.
The final
Utf8Viewversus dictionary TPC-H runwas 715.04 versus 756.93 ms in total. Most queries were unchanged; Q10 was
1.42x and Q12 was 1.17x slower with dictionaries, while Q19 was 1.05x faster.
Test coverage
datafusion-common::hash_utilstests pass.force_hash_collisionsenabled.cargo clippy -p datafusion-common --lib -- -D warningscargo clippy -p datafusion-common --bench with_hashes -- -D warningscargo clippy -p datafusion-physical-plan --lib -- -D warnings16 fixture-dependent cases require the
parquet-testingsubmodule, which isnot present in this checkout.
Are there any user-facing changes?
No API or output-schema change. RLE-dictionary Parquet scans avoid unnecessary
decode/re-encode work, and downstream operators avoid work proportional to
unreferenced dictionary values.