Skip to content

Preserve row delimiters sharing a column separator prefix - #375

Open
OskarEichler wants to merge 1 commit into
ruby:mainfrom
OskarEichler:codex/csv-shared-separator-prefix
Open

OskarEichler wants to merge 1 commit into
ruby:mainfrom
OskarEichler:codex/csv-shared-separator-prefix

Conversation

@OskarEichler

Copy link
Copy Markdown
Contributor

Summary

When scanning an unquoted field with a multi-character column separator, check for a complete row delimiter before consuming an unmatched column-prefix character. Reuse the existing chunk-aware separator scanners.

Reproduction

CSV.parse("x||y||z|~|", col_sep: "||", row_sep: "|~|", liberal_parsing: true) raises MalformedCSVError: the final row delimiter's first | is consumed as field data.

Verification

54 focused cases pass in both parser modes across three encodings, three diverging delimiter-prefix pairs and varying row counts. The release fails the direct reproduction. Together with the separate writer quote fix, all 1,080 corpus checks pass in each parser mode; neither patch alone addresses the other cause.

This isolated patch passes the unchanged upstream suite in both normal and CSV_PARSER_SCANNER_TEST=yes modes: 527 tests / 4,050 assertions in each mode, zero failures/errors. Commands use Ruby 4.0.6 through rbenv (bundle exec ruby run-test.rb). Syntax and git diff --check pass. The combined release-based consumer branch also passes gem build and rake warning:error rdoc.

No tests were added or changed, per the requesting project's explicit no-new-tests policy. Focused checks and measurements live outside this repository. The upstream no-RuboCop policy is respected. Verification was local on macOS/Ruby 4.0.6; the supported OS/Ruby matrix was not run locally. Current main and installed 3.3.6 share base commit 0873ab362d996f12796c3c3e8998b5be657a9b12; no unrelated changes are included.

Breaking changes and limitations

None intended. This fixes diverging separators that share an initial prefix. Identical delimiters or a full delimiter being a prefix of the other are not claimed unambiguous or newly supported.

Comment thread lib/csv/parser.rb
is_column_end = @column_ends.all? do |column_end|
@scanner.scan(column_end)
end
is_separator_end = parse_column_end || parse_row_end

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling parse_column_end / parse_row_end here nests a second keep_start/keep_back (the multi-character branch of each) inside the outer keep_start. InputsScanner loses data when a nested keep crosses a chunk edge, so values get cut on IO input.

Repro: same line, read from a pipe so the 8 KiB chunks are used:

require "csv"
n = 8190
line = %Q{""""::#{"a" * n}:b::c\n}
rd, wr = IO.pipe; wr.write(line); wr.close
p CSV.new(rd, col_sep: "::").read.map { |r| r.map { |f| f && f.size } }
# main:     [[1, 8192, 1]]   ("aaa...:b", "c")
# this PR:  [[1, 8190]]      (":b" and the "c" column are gone)

With n from 8170 to 8200, 28 of 31 lengths are wrong on this branch and 0 on main. It also shows with the CI scanner mode, e.g. CSV_PARSER_SCANNER_TEST=yes CSV_PARSER_SCANNER_TEST_CHUNK_SIZE=1: CSV.parse(%Q{"""\"::a:b::c\n}, col_sep: "::") gives [["\"", "a"]] instead of [["\"", "a:b", "c"]].

Suggested fix: keep one keep level per check, like the old loop, e.g.

def at_separator?(separator, separator_characters)
  @scanner.keep_start
  if separator_characters
    found = separator_characters.all? {|c| @scanner.scan(c)}
  else
    found = @scanner.scan(separator)
  end
  @scanner.keep_back
  found
end

and break if at_separator?(@column_end, @column_ends) or at_separator?(@row_end, @row_ends) instead of the nested calls. A with_chunk_size("1") test would catch it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ Proxy