Skip to content

Fix paragraph mode gets with extra empty lines or a limit - #244

Open
bardrop wants to merge 1 commit into
ruby:masterfrom
bardrop-oss:fix-gets-paragraph-newlines-limit
Open

bardrop wants to merge 1 commit into
ruby:masterfrom
bardrop-oss:fix-gets-paragraph-newlines-limit

Conversation

@bardrop

@bardrop bardrop commented Nov 18, 2026

Copy link
Copy Markdown

Paragraph mode (gets(""), readlines(""), each_line("")) does not match IO and String in a few ways:

require "stringio"

StringIO.new("a\n\n\n\nb").gets("")
# => "a\n\n\n\n"   (IO and String: "a\n\n")

StringIO.new("\n\nabc").gets("", 1)
# => nil         (IO: "a") -- nil before the end of the stream

io = StringIO.new("\n\n\n")
io.gets("")      # => nil
io.pos           # => 0     (IO: 3)

For the first one, core Ruby already tests this for String and IO:

  • test/ruby/test_string.rb: "hello\n\n\nworld".each_line('') gives "hello\n\n"
  • IO returns "a\n\n" and skips the rest of the empty lines (swallow in io.c)

The second one comes from the limit: it was applied from pos, before the leading newlines were skipped. When the limit window held only newlines, gets returned nil as if the stream had ended, so a while (line = io.gets("", n)) loop stops early.

Change

In the paragraph branch of strio_getline (C and Java):

  • skip the leading newlines first, then apply the limit
  • return the paragraph with only one empty line, then move pos past all the following empty lines (also after a line cut by the limit, like IO)
  • when only newlines are left, move pos to the end before returning nil

The limit code moved to a small helper so both places use the same logic. Other separators are not changed.

On JRuby, paragraph_end used 0 to mean "not found", but 0 is a valid index, so StringIO.new("\r\n\r\nabc").each_line("", chomp: true) gave ["\r\n\r\n", "abc"] while the C extension (and String) give ["", "abc"]. It now uses -1.

test_each expected the old output ("foo\nbar\n\n\n"), so I updated it to what IO and String return.

Testing

  • New tests in test_stringio.rb (extra empty lines, only newlines, limit after leading newlines, CRLF paragraph at the start with chomp).
  • CRuby 4.0.9: 4 tests fail before, rake test passes after (122 tests).
  • JRuby 10.1.3.0: 5 tests fail before, rake test passes after (119 tests, 2 omissions).
  • Differential check against File#gets on random paragraph-mode op sequences (gets("") with and without limit/chomp, read, getc, readlines("")): 7648 of 30k sequences differ with stringio 3.2.0, 18 after. The 18 left all start with pos in the middle of a multibyte character (an older, separate case).
  • Same 40k random sequences with other separators (nil, "\n", "ab", "\r\n", ...) give identical results before and after.
  • JRuby and CRuby give the same results on 20k random paragraph-mode sequences.

`gets("")`, `readlines("")` and `each_line("")` differed from IO and
String:

* The returned paragraph kept all the following empty lines, such as
  `"a\n\n\n\n"` for `"a\n\n\n\nb"`.  IO and String return `"a\n\n"`
  and skip the rest.
* The limit counted the leading newlines that are skipped, so
  `StringIO.new("\n\nabc").gets("", 1)` returned `nil` before the end
  of the stream.  IO returns `"a"`.
* After a line cut by the limit, the following empty lines were not
  skipped as IO does.
* When only newlines were left, `gets("")` returned `nil` without
  moving `pos` to the end.

On JRuby, a paragraph at the start of the string, such as
`"\r\n\r\nabc"` with `chomp: true`, was not chomped because 0 was used
as "not found" for the paragraph end.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ Proxy