Skip to content

fix: DEncode depth check false positives on string payloads - #810

Merged
fstagni merged 1 commit into
DIRACGrid:integrationfrom
fstagni:fix_dencode_depth
Oct 28, 2026
Merged

fstagni merged 1 commit into
DIRACGrid:integrationfrom
fstagni:fix_dencode_depth

Conversation

@fstagni

@fstagni fstagni commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Summary

_checkDEncodeDepth in UPHandler.py counted every d/l/t/e byte in the DEncode stream as structure. But DEncode strings are encoded as s<len>:payload, and the payload is arbitrary bytes, so container-looking characters inside string payloads were counted as nesting.

 _safeDecode(data)
   json.loads(raw)                      # fails for legacy DEncode data
-  _checkDEncodeDepth(raw)              # counts d/l/t/e bytes in string payloads too
+  _checkDEncodeDepth(raw)              # token-aware: skips string payloads & scalars
   DEncode.decode(raw)[0]

Legacy pre-JSON stored states whose payload contains enough d/l/t/e characters (e.g. {"data": "d"*50}: real nesting depth 2, old counter reports 56) falsely raised DEncode data exceeds maximum nesting depth (50) or DEncode data has unbalanced nesting, turning valid data into HTTP 500 in web_listAppState.

The fix walks the DEncode stream token by token, skipping string payloads and scalar values (including the float e+/- exponent case, mirroring DEncode.decodeFloat), so only real containers (d/l/t) contribute to depth. Malformed data is rejected earlier with clearer messages. The DoS guard is intact: genuinely deep nesting is still rejected at depth > 50, and the 5 MB size limit and post-decode type validation are unchanged.

Additionally, web_listAppState now logs and skips vars that cannot be decoded instead of failing the whole listing.

Evidence

Against the real DIRAC.Core.Utilities.DEncode:

  • Before: DEncode.encode({"data": "d"*50}) → old checker reports maxDepth 56 → ValueError: DEncode data exceeds maximum nesting depth (50) → HTTP 500 on listAppState (seen in production: WebApp/UPHandler ERROR: Exception serving request DEncode data exceeds maximum nesting depth (50)).
  • After: same input passes; round-trip of all DEncode types (ints, longs, floats, bools, strings, lists, tuples, dicts, datetimes) passes; b"d"*200 + b"e"*200 still rejected with the depth error; unbalanced and malformed streams rejected.

Merge Danger

Door: two-way

Blast Radius: UPHandler only

Affects only decoding of legacy (pre-JSON) application/desktop states in the user profile. Newly saved states are pure JSON and never enter the DEncode path. The security hardening (size limit, depth cap, type validation) is preserved.

_checkDEncodeDepth counted every d/l/t/e byte in the stream as
structure, including characters inside DEncode string payloads
(s<len>:payload), whose content is arbitrary bytes. Legacy stored
states whose payload contained enough container-like characters
falsely triggered 'DEncode data exceeds maximum nesting depth (50)'
and 'DEncode data has unbalanced nesting', turning valid data into
HTTP 500 errors in web_listAppState.

Walk the DEncode stream token by token instead, skipping string
payloads and scalar values, so that only real containers (d/l/t)
contribute to the nesting depth. Malformed data is now rejected
earlier with clearer messages, and genuinely deep nesting is still
rejected, keeping the DoS guard intact.

Also make web_listAppState skip vars that cannot be decoded instead
of failing the whole listing.
@fstagni
fstagni merged commit b81459b into DIRACGrid:integration Oct 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant