Repository navigation
fix: DEncode depth check false positives on string payloads - #810
Merged
Merged
Conversation
_checkDEncodeDepth counted every d/l/t/e byte in the stream as structure, including characters inside DEncode string payloads (s<len>:payload), whose content is arbitrary bytes. Legacy stored states whose payload contained enough container-like characters falsely triggered 'DEncode data exceeds maximum nesting depth (50)' and 'DEncode data has unbalanced nesting', turning valid data into HTTP 500 errors in web_listAppState. Walk the DEncode stream token by token instead, skipping string payloads and scalar values, so that only real containers (d/l/t) contribute to the nesting depth. Malformed data is now rejected earlier with clearer messages, and genuinely deep nesting is still rejected, keeping the DoS guard intact. Also make web_listAppState skip vars that cannot be decoded instead of failing the whole listing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_checkDEncodeDepthinUPHandler.pycounted everyd/l/t/ebyte in the DEncode stream as structure. But DEncode strings are encoded ass<len>:payload, and the payload is arbitrary bytes, so container-looking characters inside string payloads were counted as nesting.Legacy pre-JSON stored states whose payload contains enough
d/l/t/echaracters (e.g.{"data": "d"*50}: real nesting depth 2, old counter reports 56) falsely raisedDEncode data exceeds maximum nesting depth (50)orDEncode data has unbalanced nesting, turning valid data into HTTP 500 inweb_listAppState.The fix walks the DEncode stream token by token, skipping string payloads and scalar values (including the float
e+/-exponent case, mirroringDEncode.decodeFloat), so only real containers (d/l/t) contribute to depth. Malformed data is rejected earlier with clearer messages. The DoS guard is intact: genuinely deep nesting is still rejected at depth > 50, and the 5 MB size limit and post-decode type validation are unchanged.Additionally,
web_listAppStatenow logs and skips vars that cannot be decoded instead of failing the whole listing.Evidence
Against the real
DIRAC.Core.Utilities.DEncode:DEncode.encode({"data": "d"*50})→ old checker reports maxDepth 56 →ValueError: DEncode data exceeds maximum nesting depth (50)→ HTTP 500 onlistAppState(seen in production:WebApp/UPHandler ERROR: Exception serving request DEncode data exceeds maximum nesting depth (50)).b"d"*200 + b"e"*200still rejected with the depth error; unbalanced and malformed streams rejected.Merge Danger
Door: two-way
Blast Radius: UPHandler only
Affects only decoding of legacy (pre-JSON) application/desktop states in the user profile. Newly saved states are pure JSON and never enter the DEncode path. The security hardening (size limit, depth cap, type validation) is preserved.