Skip to content

Commit 1926dff

Browse files
authored
Improve performance of loading dump files in cppcheckdata.py (#8989)
Loading a dump file in `cppcheckdata.py` is now about 2.5x faster. Addons that iterate over the configurations more than once no longer parse the dump file on each pass. The parsed objects are unchanged. - Objects are built directly from the XML parser callbacks, without creating `ElementTree` elements first. - The cyclic garbage collector is disabled while parsing, because it took about half of the loading time. It is enabled again before each configuration is returned to the caller. - The `CppcheckData` constructor skips the `<dump>` elements and parses only the rest of the file. Before, it parsed the whole file to reach the raw tokens and suppressions, which are written after the configurations. - `data.configurations` loads the configurations once and keeps them. Later accesses and `iterconfigurations()` reuse them, while `iterconfigurations()` on its own still parses one configuration at a time. `misc.py` now uses `data.configurations` for its three passes, and `misra.py --verify` parses the file once instead of twice. Every parsed object was compared between the old and new code on dumps of the current layout, the old layout (raw tokens before the configurations) and clang-import dumps. New tests are in `test/addon`. ### Timings Fastest of several runs on Python 3.12, with peak memory after the slash. Load the dump file and iterate over the configurations once: | Dump | Size | Configurations | Before | After | Speedup | |---|---|---|---|---|---| | misra-test.c | 26 MB | 6 | 2.06s / 239 MB | 0.91s / 181 MB | 2.3x | | token.cpp | 15 MB | 1 | 1.20s / 161 MB | 0.53s / 157 MB | 2.3x | | valueflow.cpp | 18 MB | 1 | 1.93s / 275 MB | 0.78s / 243 MB | 2.5x | | symboldatabase.cpp | 33 MB | 1 | 3.44s / 363 MB | 1.31s / 357 MB | 2.6x | `misc.py` (three passes over the configurations): | Dump | Before | After | Speedup | |---|---|---|---| | misra-test.c | 5.08s / 239 MB | 1.28s / 253 MB | 4.0x | | token.cpp | 3.30s / 169 MB | 0.73s / 158 MB | 4.5x | | valueflow.cpp | 4.97s / 326 MB | 1.13s / 243 MB | 4.4x | | symboldatabase.cpp | 9.03s / 422 MB | 1.92s / 358 MB | 4.7x | `misc.py` uses more memory on misra-test.c because it now keeps all 6 configurations in memory at once.
1 parent 8140605 commit 1926dff

3 files changed

Lines changed: 249 additions & 173 deletions

File tree

0 commit comments

Comments
 (0)