[hzmsh]

Rationale for this change

Filters on float columns can miss rows, and delete() can delete the wrong rows:

import pyarrow as pa

from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import FloatType, IntegerType, NestedField

catalog = InMemoryCatalog("default")
catalog.create_namespace("default")
schema = Schema(NestedField(1, "id", IntegerType()), NestedField(2, "value", FloatType()))
tbl = catalog.create_table("default.t", schema=schema)
tbl.append(pa.table({"id": pa.array([1, 2, 3], pa.int32()), "value": pa.array([0.1, 0.4, 0.7], pa.float32())}))

print(tbl.scan(row_filter="value = 0.1").to_arrow()["id"].to_pylist())  # main: [], expected: [1]
tbl.delete("value != 0.1")
print(tbl.scan().to_arrow()["id"].to_pylist())  # main: [], expected: [1]

Why. A float column stores 32-bit values, and 0.1 as a 32-bit float is 0.10000000149011612. PyIceberg keeps the filter value as the 64-bit 0.1. To decide which files to read, it compares the filter value with each file's min and max (or with the partition value). These two numbers are not equal, so the file with 0.1 is skipped. Reading rows inside a file is fine, because the row filter turns the value into a 32-bit Arrow scalar (_convert_scalar). So the bug shows up when the value is a file's min or max, or a partition value.

Deletes use the same min/max check. For delete("value != 0.1"), PyIceberg decides that no row in the file is 0.1 and drops the whole file.

Data [0.1, 0.4, 0.7] in one file:

	main	this PR
value = 0.1	[]	[1]
value <= 0.1	[]	[1]
value >= 0.7	[]	[3]
delete("value != 0.1"), then scan	[]	[1]
delete("value = 0.1"), then scan	[1, 2, 3]	[2, 3]

Fix. Convert the value to 32 bits when the float literal is created, in FloatLiteral.__init__. The spec defines float as a 32-bit IEEE 754 value (spec.md#L281). Java does the same: binding a double literal to a float column creates new FloatLiteral(value().floatValue()) (Literals.java#L380). This also removes _value32, which is now the same as .value.

A float initial-default read from table metadata is converted too, like Java (SingleValueParser.java#L79). Otherwise a filter on a column that is missing from older files would stop matching its default.

Reproduced on 0.12.0 and main.

Are these changes tested?

Yes.

• tests/catalog/test_catalog_behaviors.py: new scan and delete tests on a float column, with and without an identity partition, for each catalog fixture. All 18 fail on main and pass with this change.
• tests/expressions/test_literals.py::test_double_to_float_within_bound expected the old 64-bit value. It now expects the 32-bit value of 34.56 (34.560001373291016).
• tests/expressions/test_visitors.py: new test for a missing float column with initial_default=0.1. It fails if only the filter value is converted.
• tests/test_conversions.py::test_json_float_is_read_as_float32: a float default read from JSON (22.27) now comes back as its float32 value, while a double stays 22.27. It fails on main.
• Unit tests on Linux (Python 3.11, with pyiceberg-core), after rebasing on current main (4ba3ba4): 4351 passed, 4331 on main. The difference is the 20 tests above that fail on main. 12 other failures are the same on both, from optional packages I don't have (Kerberos, Azure). My setup has no Cython Avro decoder, so tests/avro was not collected, and I ignored the pure-Python fallback warning on both sides. I couldn't run the Docker integration tests locally, so I rely on CI for them.
• tests/integration/test_rest_schema.py::test_initial_default_all_columns: the float case set a default of 22.27, which is not exact in float32 and now reads back as 22.270000457763672 (CI caught this). It now uses 22.25, which is exact. The double case still uses 22.27.
• prek run --files on the changed files passes.

Are there any user-facing changes?

Yes. Filters and deletes on float columns now return and remove the right rows. double columns are not affected.

Small visible changes: FloatLiteral(0.1).value (and its repr) is now 0.10000000149011612, and float default values are rounded to float32 too, like Java: a default of 0.1 set with add_column or set_default_value, passed to NestedField, or read from table metadata becomes 0.10000000149011612. On main, DoubleLiteral(0.1) == FloatLiteral(0.1) was True and the other way round was False; now both are False, like Java.

---

Written with AI assistance (Claude Code, Claude Opus 5.5). The Java references were read from apache/iceberg main, and the tests above were run on this branch.

[hzmsh]

The integration failure was caused by this PR: float defaults are now stored as float32 values, so the float case in test_initial_default_all_columns (default 22.27) read back as 22.270000457763672. Fixed in 39a7ff3 by using 22.25, which is exact in float32. I also added a unit test for reading a float default from JSON and updated the description.

The lint-and-unit-test (3.10) failure is unrelated: google.api_core now raises a FutureWarning on Python 3.10, so tests/catalog/test_bigquery_metastore.py fails to import (the 3.11-3.14 jobs were cancelled after it failed). Other open PRs show the same failure, and #4100 fixes it.

[hzmsh]

Hi @Fokko, a gentle ping on this one. Filtering a float column by a value like 0.1 can miss rows today; this PR rounds the literal to float32, as the spec and Java do. It's rebased on the latest main, and all checks pass except cibw-dev-env-smoke-test, which was cancelled while building the wheel (log). Could you re-run it and take a look when you have a moment? Thanks!