Speed up finding line endings in TextIOWrapper #161199

Feature or enhancement

With newline='', the mode the csv documentation recommends, TextIOWrapper.readline() and iteration search for \r or \n one character at a time (in the universal newlines branch of _PyIO_find_line_ending()). With newline=None and newline='\n', they use memchr(). As a result, the same file reads much more slowly with newline=''. With a PGO+LTO build of main on Linux:

$ ./python -m timeit "for line in open('data.csv', newline=''): pass"
10 loops, best of 5: 21.9 msec per loop
$ ./python -m timeit "for line in open('data.csv', newline='\n'): pass"
20 loops, best of 5: 12.3 msec per loop

(data.csv is 24 MB: 300,000 rows of ASCII CSV.)

The newline decoder has two similar loops:

• With newline='', it scans each chunk for \r and \n to keep the newlines attribute up to date, and stops only after seeing all three kinds of line ending. For a file with only \r\n endings, which is what csv.writer writes, it scans every chunk.
• With the default newline=None, it translates \r\n and \r to \n one character at a time in every chunk that contains \r.

For 1-byte strings (no character above U+00FF), all three loops can skip ahead one size_t word at a time: a cheap test skips words with no byte below 0x0E, an exact test finds \r and \n in the rest, and ctz() gives the position, as in find_first_nonascii() in unicodeobject.c. With this change, the first command above takes 13.8 msec per loop.

In interleaved runs on 24 MB files with PGO builds (gcc and MSVC):

• Iterating over lines with newline='' is 1.35x to 1.6x faster on CSV files like this one, and up to 2x faster on tab-separated files.
• Iterating over a file with \r\n endings with the default newline=None is 1.3x faster.
• csv.reader is 5% to 11% faster.

Lines of 5 to 15 characters also get faster. 2- and 4-byte strings and the other newline modes run the same code as before.

I have a PR ready.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-161199: Speed up finding line endings in TextIOWrapper #161200