Skip to content

Accept partial multi-character separators in unquoted values - #388

Open
bardrop wants to merge 1 commit into
ruby:mainfrom
bardrop-oss:fix-unquoted-row-separator-chars
Open

bardrop wants to merge 1 commit into
ruby:mainfrom
bardrop-oss:fix-unquoted-row-separator-chars

Conversation

@bardrop

@bardrop bardrop commented Nov 18, 2026

Copy link
Copy Markdown

GitHub: fix GH-107

The robust parser stops an unquoted value at every character of a multi-character row separator. When that character isn't a complete separator, parsing fails with "TODO: Meaningful message":

CSV.parse(%Q{a|b,x|\n}, row_sep: "|\n")
# => [["a|b", "x"]]   (loose parser)
CSV.parse(%Q{a|b,"say ""hi"""|\n}, row_sep: "|\n")
# => CSV::MalformedCSVError: TODO: Meaningful message in line 1.

So the same a|b field parses or fails depending on whether another field in the row has a "" escape or a quoted new line (that is what moves the row to parse_quotable_robust). The same happens with the first character of a multi-character col_sep at the start of a value (CSV.parse(%Q{"""":::a\n}, col_sep: "::") fails, x:::a gives [["x", ":a"]]), and with any non-first character of a row separator ("""",c>d<> with row_sep: "<>").

The fix generalizes the existing multi-character col_sep loop in parse_unquoted_column_value: a separator character that isn't a complete column or row separator is kept in the value, as the loose parser does. CR and LF are still never allowed in an unquoted value, so row_sep: "\r\n" behaves as before. The separator check uses one keep_start/keep_back level per separator, like the old loop; nesting parse_column_end/parse_row_end inside another keep lost data at chunk edges with CSV_PARSER_SCANNER_TEST.

Checks:

  • 3 new tests fail on main and pass here.
  • ruby run-test.rb: 538 tests pass, also with CSV_PARSER_SCANNER_TEST=yes.
  • Differential fuzz of the loose path against the robust path (same rows with a """" field in front, 14 column/row separator pairs, ~38k cases): main has 1,090 differences in 9,472 cases, this branch has 0. The robust results are the same with scanner chunk sizes 1, 2, 3, 5, 7 and 64.
  • Default ,/\n and \r\n parsing don't run the new loop. The robust path with col_sep: "::" is about 6% slower in a quick benchmark (50k rows), because it now also checks for the row separator.

This was written with an AI agent (Claude); I ran and checked the tests and fuzzing above.

GitHub: fix rubyGH-107

The robust parser stopped an unquoted value at every character of a
multi-character row separator, and at the first character of a
multi-character column separator only when it was not at the start of
the value. When such a character wasn't a complete separator, parsing
failed with "TODO: Meaningful message":

    CSV.parse(%Q{a|b,"say ""hi"""|\n}, row_sep: "|\n")
    # => CSV::MalformedCSVError

The loose parser splits by the whole separators, so the same row
without a "" escape or a quoted new line was parsed as
[["a|b", "say \"hi\""]].

Treat any separator character that isn't a complete column or row
separator as a part of the unquoted value, as the loose parser does.
CR and LF are still not allowed in an unquoted value.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CSV parser fails with mixed quoting and non-standard row_sep

1 participant

⌂ Proxy