[Rahul-cs50]

Closes #4184

Rationale for this change

When delete() (and overwrite() with an overwrite_filter) has to rewrite a data file, it keeps the rows that _expression_to_complementary_pyarrow selects. That filter was NOT f, OR'd with c IS NULL for every column c that f reads without a null check. Under pyarrow's three-valued logic that goes wrong both ways when a column is null:

• NOT f is null where f is null, so the row is dropped even though f did not match it: x=2 in the issue (p == 2, v null) is deleted by (p == 1 AND v IS NULL) OR (p == 2 AND v == 'c').
• c IS NULL keeps a row whenever c is null, even where f is still true: p == 1 OR v == 'c' keeps x=1 (p == 1, v null).

A row should be deleted exactly where f is true, which are the rows scan(row_filter=f) returns. The complement is now NOT coalesce(f, false). The NaN handling (OR is_nan(c) for columns the filter does not NaN-check) is left as it was.

Are these changes tested?

Yes. test_delete_and_overwrite_filter_with_null in tests/catalog/test_catalog_behaviors.py runs the issue's two filters plus v != 'c' and NOT (v == 'c') through delete() and overwrite() on every local catalog, and checks that the rows left are the ones scan(row_filter=...) does not return. It fails on main and passes with this change. test_expression_to_complementary_pyarrow is updated for the new expression. Unit tests in tests/io, tests/table, tests/catalog and tests/expressions: 2658 passed (the only failures are the four Kerberos tests in test_hive.py, which need the hive-kerberos extra I did not install). ruff check and ruff format --check are clean.

Are there any user-facing changes?

delete() and overwrite(..., overwrite_filter=...) no longer delete rows the filter does not match, or keep rows it does, when a column the filter reads is null.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.