Repository navigation
Conversation
GitHub: fix rubyGH-107 The robust parser stopped an unquoted value at every character of a multi-character row separator, and at the first character of a multi-character column separator only when it was not at the start of the value. When such a character wasn't a complete separator, parsing failed with "TODO: Meaningful message": CSV.parse(%Q{a|b,"say ""hi"""|\n}, row_sep: "|\n") # => CSV::MalformedCSVError The loose parser splits by the whole separators, so the same row without a "" escape or a quoted new line was parsed as [["a|b", "say \"hi\""]]. Treat any separator character that isn't a complete column or row separator as a part of the unquoted value, as the loose parser does. CR and LF are still not allowed in an unquoted value.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub: fix GH-107
The robust parser stops an unquoted value at every character of a multi-character row separator. When that character isn't a complete separator, parsing fails with "TODO: Meaningful message":
So the same
a|bfield parses or fails depending on whether another field in the row has a""escape or a quoted new line (that is what moves the row toparse_quotable_robust). The same happens with the first character of a multi-charactercol_sepat the start of a value (CSV.parse(%Q{"""":::a\n}, col_sep: "::")fails,x:::agives[["x", ":a"]]), and with any non-first character of a row separator ("""",c>d<>withrow_sep: "<>").The fix generalizes the existing multi-character
col_seploop inparse_unquoted_column_value: a separator character that isn't a complete column or row separator is kept in the value, as the loose parser does. CR and LF are still never allowed in an unquoted value, sorow_sep: "\r\n"behaves as before. The separator check uses onekeep_start/keep_backlevel per separator, like the old loop; nestingparse_column_end/parse_row_endinside another keep lost data at chunk edges withCSV_PARSER_SCANNER_TEST.Checks:
ruby run-test.rb: 538 tests pass, also withCSV_PARSER_SCANNER_TEST=yes.""""field in front, 14 column/row separator pairs, ~38k cases): main has 1,090 differences in 9,472 cases, this branch has 0. The robust results are the same with scanner chunk sizes 1, 2, 3, 5, 7 and 64.,/\nand\r\nparsing don't run the new loop. The robust path withcol_sep: "::"is about 6% slower in a quick benchmark (50k rows), because it now also checks for the row separator.This was written with an AI agent (Claude); I ran and checked the tests and fuzzing above.