Skip to content

fix(spark)!: reject file format subclasses - #1459

Open
alexandrefimov wants to merge 2 commits into
substrait-io:mainfrom
alexandrefimov:spark-reject-format-subclasses-1388
Open

alexandrefimov wants to merge 2 commits into
substrait-io:mainfrom
alexandrefimov:spark-reject-format-subclasses-1388

Conversation

@alexandrefimov

Copy link
Copy Markdown
Contributor

Reject file-format subclasses instead of exporting them as plain CSV, Parquet or ORC. Path-based Delta reads can otherwise lose deletion vectors or column mapping and return wrong rows.

Closes #1388

BREAKING CHANGE: Spark export now rejects file-format subclasses, including DeltaParquetFileFormat. Use a format-aware exporter for those sources.

File format subclasses can depend on metadata that plain file reads and
writes do not carry. Reject them instead of exporting a different operation.

Closes substrait-io#1388

BREAKING CHANGE: Spark export rejects file-format subclasses, including
path-based Delta reads. Use a format-aware exporter for these sources.
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • Bug Fixes
    • CSV, Parquet, and ORC file operations now accept only the corresponding standard formats; custom subclasses are reported as unsupported.
    • Error messages for unsupported file formats now identify the format’s class, making it easier to understand why a read or write failed.
    • Reads from empty sources using supported formats continue to return no rows, while non-empty reads preserve their rows.

Spark file-format conversion now rejects subclasses of CSV, Parquet, and ORC, and reports the unsupported format’s class name. A new test suite checks these rejections and round trips for supported formats.

Changes: Spark file formats

Layer / File(s) Summary
Format conversion and validation
spark/src/main/scala/io/substrait/spark/logical/ToSubstraitRel.scala, spark/src/test/scala/io/substrait/spark/FileFormatSuite.scala
Local file scans convert the format before listing files. Conversion accepts only exact CSV, Parquet, and ORC classes. Tests check subclass rejection for partitioned and unpartitioned reads, empty sources, and writes, plus round trips for supported formats.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: andrew-coleman, bvolpato

Merge Risk: Minimal · up to 837a4

Spark export now rejects file-format subclasses instead of silently exporting them as plain CSV, Parquet or ORC, which prevents incorrect rows from Delta-backed sources. This is an intentional breaking change, and no merge-blocking risk was found.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title follows Conventional Commits with a breaking-change marker and names the main change: Spark export now rejects file-format subclasses instead of exporting them as plain CSV, Parquet, or ORC.…
Description check Passed The description gives the rationale (subclasses such as DeltaParquetFileFormat can lose deletion vectors or column mapping on path-based Delta reads), links issue #1388, and includes a BREAKING CHANGE…
Linked Issues check Passed Issue #1388 requires that Spark export not map Delta-backed file formats to plain Parquet. It asks for exact class matching for CSV, Parquet, and ORC, and rejection of other subclasses so conversion f…
Out of Scope Changes check Passed The changes are limited to the format check in ToSubstraitRel.scala, which includes a single conversion of the Hadoop file format in buildLocalFileScan and the exception message change, and to the new…
Full details: Title check

Explanation

The title follows Conventional Commits with a breaking-change marker and names the main change: Spark export now rejects file-format subclasses instead of exporting them as plain CSV, Parquet, or ORC. The changeset matches this: convertFileFormat accepts only exact base classes, and FileFormatSuite tests that subclasses are rejected.

Full details: Description check

Explanation

The description gives the rationale (subclasses such as DeltaParquetFileFormat can lose deletion vectors or column mapping on path-based Delta reads), links issue #1388, and includes a BREAKING CHANGE footer with migration guidance. It does not describe the tests added in FileFormatSuite or the empty-source behavior, and it does not follow the template's comment structure beyond the footer. These gaps are minor, so the description is mostly complete.

Full details: Linked Issues check

Explanation

Issue #1388 requires that Spark export not map Delta-backed file formats to plain Parquet. It asks for exact class matching for CSV, Parquet, and ORC, and rejection of other subclasses so conversion fails. The reviewed change in spark/src/main/scala/io/substrait/spark/logical/ToSubstraitRel.scala makes convertFileFormat accept CSV, Parquet, and ORC only when the format is exactly the base class. Subclasses, including DeltaParquetFileFormat, now reach the unsupported-format exception, which reports the class name. The new spark/src/test/scala/io/substrait/spark/FileFormatSuite.scala covers this with automated tests. The suite checks rejection of Parquet, ORC, and CSV subclasses for partitioned and unpartitioned reads, empty sources, and file writes. It also checks that plain reads still round-trip. Limitation: the convertFileFormat hunk was past the read output limit, so the source logic was confirmed only from the supplied change summary and the test expectations, not from the code text itself.

Full details: Out of Scope Changes check

Explanation

The changes are limited to the format check in ToSubstraitRel.scala, which includes a single conversion of the Hadoop file format in buildLocalFileScan and the exception message change, and to the new FileFormatSuite tests. Both directly implement the rejection behavior requested by #1388 and the breaking change described in the PR. No unrelated changes were identified in the supplied evidence. Limitation: the full ToSubstraitRel.scala diff could not be read, so the scope check relies on the change summary for the exact hunks.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 6af0912 and 837a41c.

Files selected for processing: 2

  • spark/src/main/scala/io/substrait/spark/logical/ToSubstraitRel.scala
  • spark/src/test/scala/io/substrait/spark/FileFormatSuite.scala

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spark: a path-based Delta read is exported as plain Parquet, so deleted rows reappear and column-mapped columns read as null

1 participant