Skip to content

perf: speed up _unescape_special_chars for large documents #748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nicholasserra merged 6 commits into from
Nov 9, 2026
Merged

perf: speed up _unescape_special_chars for large documents #748

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fb5899e
perf: speed up _unescape_special_chars for large documents
SwastikTripathi Oct 25, 2026
4bf3ddb
Address review: nonlocal flag, define _sub once, ASCII comments
SwastikTripathi Oct 27, 2026
ed6330a
Merge remote-tracking branch 'origin/master' into md2-748-merge
SwastikTripathi Oct 27, 2026
58cd6cb
Merge remote-tracking branch 'origin/master' into perf/unescape-speci…
SwastikTripathi Nov 7, 2026
be9bb4e
Simplify the unescape loop to while replaced
SwastikTripathi Nov 7, 2026
a13e621
Merge branch 'master' into perf/unescape-special-chars
nicholasserra Nov 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
## python-markdown2 2.5.6 (not yet released)

- [pull #762] Speed up documents with many links by skipping the backslash-escape scan for text without a backslash
- [pull #748] Speed up `_unescape_special_chars` by skipping text with no hashes and dropping the full-text comparison per pass (#635)
- [pull #758] Fix `footnotes` extra adding an empty footnotes block when no footnote is referenced.
- [pull #746] Preserve query strings and fragments in `markdown-file-links` conversions, including reference-link targets.
- [pull #739] Fix `header-ids` extra putting sanitized HTML into header ids in safe mode, which allowed attribute injection (#649)
Expand Down
36 changes: 25 additions & 11 deletions lib/markdown2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2425,19 +2425,33 @@ def _encode_email_address(self, addr: str) -> str:

def _unescape_special_chars(self, text: str) -> str:
# Swap back in all the special characters we've hidden.
hashmap = tuple(self._escape_table.items()) + tuple(self._code_table.items())
if 'md5-' not in text:
return text
# Build the hash -> original mapping directly, without intermediate tuples.
# html_blocks table is in format {hash: item} compared to usual {item: hash}
hashmap += tuple(tuple(reversed(i)) for i in self.html_blocks.items())
replacements = {}
for ch, hash in hashmap:
replacements: dict[str, str] = {}
for ch, hash in self._escape_table.items():

@Crozzers Crozzers Nov 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be perhaps faster and simpler if instead of building a replacements dict, you just did this in the _sub function:

val = (
    self._escape_table.get(match[0])
    or self._code_table.get(match[0])
    or self.html_blocks.get(match[0])
)

Maybe more memory efficient?

@SwastikTripathi SwastikTripathi Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That works for html_blocks, which is keyed by hash, but _escape_table and _code_table go the other way ({original: hash}, e.g. self._code_table[text] = hashed), so .get(match[0]) on them would always miss. To skip the dict build we would need reverse tables kept in sync at every place those two are written (the escape table is also written per URL in _protect_url). The early md5- check already skips the build for most calls, so I kept the dict here. Happy to add the reverse tables if you prefer that.

@Crozzers Crozzers Nov 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, forgot about that. In that case building the map is probably the best way

replacements.setdefault(hash, ch)
while True:
orig_text = text
# Scan once per nesting level instead of once for every stored hash.
text = re.sub(r'md5-[0-9a-f]{32}',
lambda match: replacements.get(match[0], match[0]), text)
if text == orig_text:
break
for ch, hash in self._code_table.items():
replacements.setdefault(hash, ch)
for hash, item in self.html_blocks.items():
replacements.setdefault(hash, item)

def _sub(match: re.Match) -> str:
nonlocal replaced
val = replacements.get(match[0])
if val is None:
return match[0]
replaced = True
return val

# Scan once per nesting level instead of once for every stored hash.
# Stop when a pass makes no replacement, rather than comparing the
# whole text against the previous pass.
replaced = True
while replaced:
replaced = False
text = re.sub(r'md5-[0-9a-f]{32}', _sub, text)
return text

def _outdent(self, text: str) -> str:
Expand Down
Loading