Repository navigation
Preserve row delimiters sharing a column separator prefix - #375
OskarEichler wants to merge 1 commit into
Conversation
| is_column_end = @column_ends.all? do |column_end| | ||
| @scanner.scan(column_end) | ||
| end | ||
| is_separator_end = parse_column_end || parse_row_end |
There was a problem hiding this comment.
Calling parse_column_end / parse_row_end here nests a second keep_start/keep_back (the multi-character branch of each) inside the outer keep_start. InputsScanner loses data when a nested keep crosses a chunk edge, so values get cut on IO input.
Repro: same line, read from a pipe so the 8 KiB chunks are used:
require "csv"
n = 8190
line = %Q{""""::#{"a" * n}:b::c\n}
rd, wr = IO.pipe; wr.write(line); wr.close
p CSV.new(rd, col_sep: "::").read.map { |r| r.map { |f| f && f.size } }
# main: [[1, 8192, 1]] ("aaa...:b", "c")
# this PR: [[1, 8190]] (":b" and the "c" column are gone)With n from 8170 to 8200, 28 of 31 lengths are wrong on this branch and 0 on main. It also shows with the CI scanner mode, e.g. CSV_PARSER_SCANNER_TEST=yes CSV_PARSER_SCANNER_TEST_CHUNK_SIZE=1: CSV.parse(%Q{"""\"::a:b::c\n}, col_sep: "::") gives [["\"", "a"]] instead of [["\"", "a:b", "c"]].
Suggested fix: keep one keep level per check, like the old loop, e.g.
def at_separator?(separator, separator_characters)
@scanner.keep_start
if separator_characters
found = separator_characters.all? {|c| @scanner.scan(c)}
else
found = @scanner.scan(separator)
end
@scanner.keep_back
found
endand break if at_separator?(@column_end, @column_ends) or at_separator?(@row_end, @row_ends) instead of the nested calls. A with_chunk_size("1") test would catch it.
Summary
When scanning an unquoted field with a multi-character column separator, check for a complete row delimiter before consuming an unmatched column-prefix character. Reuse the existing chunk-aware separator scanners.
Reproduction
CSV.parse("x||y||z|~|", col_sep: "||", row_sep: "|~|", liberal_parsing: true)raises MalformedCSVError: the final row delimiter's first|is consumed as field data.Verification
54 focused cases pass in both parser modes across three encodings, three diverging delimiter-prefix pairs and varying row counts. The release fails the direct reproduction. Together with the separate writer quote fix, all 1,080 corpus checks pass in each parser mode; neither patch alone addresses the other cause.
This isolated patch passes the unchanged upstream suite in both normal and
CSV_PARSER_SCANNER_TEST=yesmodes: 527 tests / 4,050 assertions in each mode, zero failures/errors. Commands use Ruby 4.0.6 through rbenv (bundle exec ruby run-test.rb). Syntax andgit diff --checkpass. The combined release-based consumer branch also passes gem build andrake warning:error rdoc.No tests were added or changed, per the requesting project's explicit no-new-tests policy. Focused checks and measurements live outside this repository. The upstream no-RuboCop policy is respected. Verification was local on macOS/Ruby 4.0.6; the supported OS/Ruby matrix was not run locally. Current main and installed 3.3.6 share base commit
0873ab362d996f12796c3c3e8998b5be657a9b12; no unrelated changes are included.Breaking changes and limitations
None intended. This fixes diverging separators that share an initial prefix. Identical delimiters or a full delimiter being a prefix of the other are not claimed unambiguous or newly supported.