Repository navigation
Conversation
The PrefixTree allocated one Node, and one empty children Hash, for every character of every indexed name. On a large index (~53k names) this accounted for about 1.4M objects and ~129MB, roughly half of the memory retained by the indexer. Store runs of non-branching characters as a single node and lazily allocate the children hash. The public API and result ordering are unchanged. In the same benchmark, the tree drops to ~13MB and total indexer RSS growth from ~295MB to ~167MB. Add tests for edge splitting and for searching and deleting keys that end in the middle of an edge.
Fryguy
commented
Nov 16, 2026
| @@ -36,10 +36,7 @@ module RubyIndexer | |||
| class PrefixTree | |||
Author
There was a problem hiding this comment.
Note I did not rename this to RadixTree, and kept it as PrefixTree. I can rename it if you think that's more appropriate, but I didn't want to introduce that much churn.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On large projects the Ruby LSP uses a lot of memory. Indexing a ManageIQ checkout (43,691 files, 330,729 distinct names, including its bundled gems) grows the process by ~1.87 GB.
Profiling the index shows
RubyIndexer::PrefixTreeis the largest single cost. It allocates oneNodefor every character of every indexed name, plus an empty childrenHashon every node. For ManageIQ that is ~3.46M nodes and ~3.46M hashes, about 810 MB, or over 40% of everything the indexer retains.Implementation
PrefixTreeis now a radix tree:Hashunless they get a child.The public API (
insert,search,delete) and the order of search results are unchanged, so no caller needed to change.Things to watch for in review:
find_nodecan end in the middle of an edge. In that case it returns the node at the end of the edge, because every key below it still matches the prefix.deleteremoves the node returned byfind_node, so a partial key still deletes everything under it (same as before).Results indexing ManageIQ with its own bundle, Ruby 3.4.10,
mainvs. this branch, same machine:PrefixTree::NodeobjectsThese numbers cover the index only (measured with
Index#index_all), not open documents. Remaining index memory is mostlyString(~165 MB),Location(~129 MB) andArray(~90 MB), which could be follow-ups.Automated Tests
I added tests to
prefix_tree_test.rbfor edge splitting, keys that end in the middle of an edge, searching prefixes that end mid-edge, and deleting after a split. They also pass against the previous implementation, so they check behavior rather than the new structure.Manual Tests
Save this as
measure_index.rb:From the root of this repository, run it on
mainand then on this branch:bundle exec ruby -Ilib /path/to/measure_index.rb. On this repository (1,870 files) I get 119 MB / 1.01M live slots onmainand 74 MB / 538k live slots on this branch. Larger projects show a bigger difference.To check completion in VS Code, open this repository's folder on this branch. The extension runs the local
exe/ruby-lspwhen the workspace folder is namedruby-lsp.Create a scratch Ruby file in this repo with
# typed: ignoreas the first line (the Ruby LSP leaves constant completion to Sorbet in other files). Once indexing finishes, typeRubyIndexer::Preand expectRubyIndexer::PrefixTree. Then typerequire "ruby_indexer/and expect the files underlib/ruby_indexer/lib/ruby_indexer/. These use@entries_treeand@require_paths_tree.In another open file, define a new constant, then confirm it shows up in completion from the scratch file. This exercises
PrefixTree#deleteand#insert.Possible follow-ups
Remaining index memory on ManageIQ after this change (1.1 GB growth) is mostly entry data, not the tree. None of these are part of this PR, and the savings are not measured.
Locationobjects (~129 MB, 1.7M objects): everyMethodentry holds two (locationandname_location). They could be packed into a singleIntegeror computed lazily from the file when requested.Stringobjects (~165 MB, 2.5M objects): 330k distinct names are spread across a much larger number of entries. Deduplicating names when entries are created (-name) would avoid keeping separate copies.Signatureand parameter objects (~30 MB): a third of the methods in a smaller sample (25.6k of 77.5k) have no parameters and could share one frozen emptySignature.RBS::Environmentinindex_ruby_core: it is built only for its side effects and then discarded, which costs about 20 MB of RSS on its own. Dropping it also dropsStringIO, becauseEnvironment.from_loaderadds thestringiolibrary, so it would needloader.add(library: "stringio")instead.Hashes remain. Nodes with a small fan-out could use a small array of pairs.