Skip to content

gh-159635: Use quote frequency to break csv.Sniffer ties - #159636

Open
MrGreyfun wants to merge 1 commit into
python:mainfrom
MrGreyfun:fix-csv-sniffer-quotechar-tie
Open

MrGreyfun wants to merge 1 commit into
python:mainfrom
MrGreyfun:fix-csv-sniffer-quotechar-tie

Conversation

@MrGreyfun

Copy link
Copy Markdown

csv.Sniffer.sniff() can select the wrong quote character because the first of two equally scoring candidates wins, regardless of how often each quote character starts a field.

To fix this bug, the sniffer now compares how many fields begin with each quote character when candidates have the same score, delimiter, and escape character. It counts only records included in each candidate's trial parse and skips quoted field contents; the count is computed only when needed to resolve a tie. If the counts are equal, the existing candidate order is preserved.

The regression tests cover both quote-character directions, mixed quote characters, and quoted-looking field data. All 157 test_csv tests pass.

Fixes gh-159635

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

csv.Sniffer ignores the dominant quotechar when candidates tie

1 participant