Skip to content

Preserve carriage returns in values when reading and rewriting .env files - #724

Open
Sreekant13 wants to merge 2 commits into
theskumar:mainfrom
Sreekant13:fix-carriage-return-roundtrip
Open

Sreekant13 wants to merge 2 commits into
theskumar:mainfrom
Sreekant13:fix-carriage-return-roundtrip

Conversation

@Sreekant13

Copy link
Copy Markdown

Problem

set_key() quotes a value, in its own words, "for the value to survive a write/read round-trip", yet a value containing a carriage return does not survive it:

import tempfile, os
from dotenv import set_key, dotenv_values

p = os.path.join(tempfile.mkdtemp(), ".env")
open(p, "w").close()
set_key(p, "KEY", "a\rb")                  # default quote_mode="always"
print(dotenv_values(p))                    # {'KEY': 'a\nb'}  <- expected 'a\rb'

The parser itself is not at fault. Feeding the exact same bytes through a stream returns the value intact:

import io
from dotenv import dotenv_values
dotenv_values(stream=io.StringIO("KEY='a\rb'\n"))   # {'KEY': 'a\rb'}

The difference is that file reads (DotEnv._get_stream and rewrite) open with universal-newline translation (open(..., newline=None)), which rewrites \r and \r\n to \n before the parser runs. So dotenv_values(path) and dotenv_values(stream=...) disagree for identical bytes, and a carriage return in a value is silently lost. This has stayed hidden because the parser tests all feed StringIO, which does no translation.

Fix

Open the read streams and the rewrite temp file with newline="". The parser already recognises \r\n, \r and \n on its own (see _newline, _end_of_line, _unquoted_value), so disabling the file-layer translation makes path reads agree with stream reads and keeps values byte-exact. Round-trips now hold for \r, \n, \r\n and mixed line endings.

A regression case is added to the existing test_set_key_round_trips (it fails before this change for the \r values and passes after).

Note on line endings

Because the temp file is now written with newline="", lines that set_key/unset_key write go out with \n. Untouched lines are echoed back with their original bytes. On a file that currently uses CRLF, an edited line is therefore written with LF while untouched lines keep CRLF. Fidelity of the stored value is the gain; I am happy to instead match the surrounding file's newline style if you prefer that behaviour.

Reads opened files with universal-newline translation, so a carriage
return inside a value was rewritten to "\n" before the parser saw it.
set_key quotes a value "for the value to survive a write/read round-trip"
(its own words), yet a value such as "a\rb" was read back as "a\nb". The
parser already recognises "\r\n", "\r" and "\n" itself, so the translation
was both redundant and lossy, and reading a path disagreed with reading
the same bytes from a stream.

Open the read streams and the rewrite temp file with newline="" so no
newline translation happens at the file layer. Values now round-trip
byte-for-byte for "\r", "\n" and "\r\n".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant