[SPARK-60322][SQL] Fix Parquet and ORC filter pushdown dropping rows for comparisons with a zero

[SEPURI-SAI-KRISHNA]

What changes were proposed in this pull request?

ParquetFilters and OrcFilters push a comparison of a FLOAT or DOUBLE column with 0.0 or -0.0 down in a form that keeps exactly the values that the Spark predicate keeps:

Spark predicate, z = 0.0 or -0.0	Parquet	ORC
f = z, f <=> z	or(eq(f, 0.0), eq(f, -0.0))	or(equals(f, 0.0), equals(f, -0.0)), nullSafeEquals for <=>
f IN (z, ...)	both zeros in the set	both zeros in the list
f != z	and(notEq(f, 0.0), notEq(f, -0.0))	(pushed as not(equals), covered above)
f < z, f >= z	bound -0.0	lessThan with bound -0.0 (>= is not(lessThan))
f <= z, f > z	bound 0.0	lessThanEquals with bound 0.0 (> is not(lessThanEquals))

Other values are pushed down as before.

Why are the changes needed?

Spark treats 0.0 and -0.0 as equal (SELECT 0.0D = -0.0D is true), but both formats tell them apart, and the pushed predicate compared with the literal unchanged. A row group, stripe or file that holds only the other zero was skipped, and the Filter above the scan never saw its rows. With the default configuration, on 3.5.3, 4.0.3 and 4.2.0 (official binaries) and on master:

CREATE TABLE p USING parquet AS SELECT id, -0.0D * id AS f FROM range(0, 1000, 1, 1);
SELECT count(*) FROM p WHERE f = 0;      -- 0, expected 1000

CREATE TABLE o USING orc AS SELECT id, -0.0D * id AS f FROM range(1, 1000, 1, 1);
SELECT count(*) FROM o WHERE f = 0;      -- 0, expected 999
SELECT count(*) FROM o WHERE f >= 0;     -- 0, expected 999

With spark.sql.parquet.filterPushdown=false / spark.sql.orc.filterPushdown=false they return 1000, 999 and 999.

Where the formats tell the zeros apart:

• The Parquet dictionary filter and bloom filter, and the ORC bloom filter: a dictionary or bloom filter of -0.0 does not contain 0.0. This drops the Parquet rows above.
• ORC statistics compare with Double.compareTo, so a min and max of -0.0 rule out f = 0.0 and f >= 0.0.
• Parquet statistics and the column index: Parquet 1.17 and earlier write -0.0 as the min and 0.0 as the max of a chunk of zeros, so these did not skip rows. Parquet 1.18 (SPARK-58869, master only) writes FLOAT and DOUBLE columns with the IEEE 754 total order and exact bounds, so on master they skip rows too.

A range takes the zero on its side as the bound, rather than any zero that keeps enough values, because a pushed Not negates the pushed predicate: with a bound that keeps more values, NOT (f < 0.0) would become not(lt(f, 0.0)), which skips a file of -0.0 values. ORC builds > and >= that way, as not(lessThanEquals) and not(lessThan).

Does this PR introduce any user-facing change?

Yes. Queries that compare a FLOAT or DOUBLE column of a Parquet or ORC table with 0.0 or -0.0 now return the rows that hold the other zero, where they could return fewer rows before.

How was this patch tested?

New tests in ParquetFilterSuite and OrcFilterSuite, which run with DSv1 and DSv2. They write one file of 0.0, one of -0.0 and one of 1.0, for FLOAT and DOUBLE, and compare 30 predicates with each zero (=, <=>, IN below and above the IN threshold, !=, <, <=, >, >=, and NOT of them with BooleanSimplification excluded) with the same filter on the data in memory. The Parquet test writes the files once dictionary encoded and once with a bloom filter, checks that in the footer, and reads them with the record-level filter on. Both tests fail without this change. Each part of the change was also reverted on its own (equality, IN, not-equal and each of the four range bounds, for each format), and each of the 14 reverts fails the tests.

ParquetV1FilterSuite, ParquetV2FilterSuite, OrcFilterSuite, OrcV1FilterSuite and the other suites in the parquet and orc packages pass.

A separate matrix compared these predicates, and BETWEEN and OR built from them, with pushdown on and off over files of 0.0, -0.0, other values and NULL: Parquet and ORC, FLOAT and DOUBLE, DSv1 and DSv2, vectorized and not, dictionaries and bloom filters on and off, only the statistics, the column index or the dictionary filter on, the record-level filter and the storage filter on and off. 3,558 of 7,904 combinations were wrong before this change and none after.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)