Table.delete() with an OR filter deletes the wrong rows when a column is NULL #4184

Apache Iceberg version

0.12.0 (latest release)

Please describe the bug 🐞

Currently, delete(f) keeps a rewritten file's rows where NOT f OR c IS NULL holds, with one c IS NULL per column c that f reads but never tests for null, so a NULL keeps rows that match and deletes rows that don't. scan(row_filter=f) returns the right rows.

import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog

cat = SqlCatalog("c", uri="sqlite:///:memory:", warehouse=f"file://{tempfile.mkdtemp()}")
cat.create_namespace("ns")
df = pa.table({"x": [1, 2, 3], "p": [1, 2, 2], "v": [None, None, "c"]})
t = cat.create_table("ns.t", schema=df.schema)
t.append(df)
f = "(p == 1 AND v IS NULL) OR (p == 2 AND v == 'c')"

print(sorted(t.scan(row_filter=f).to_arrow()["x"].to_pylist()))
# [1, 3]

t.delete(f)
print(sorted(t.scan().to_arrow()["x"].to_pylist()))
# [], should be [2]
# x=2 has p=2 and a NULL v, so it matches neither branch

p == 1 OR v == 'c' keeps x=1 instead, and overwrite(df, overwrite_filter=f) does the same as delete(f).

tested on pyiceberg 0.12.0 and main

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time