Repository navigation
Commit 1926dff
authored
Improve performance of loading dump files in cppcheckdata.py (#8989)
Loading a dump file in `cppcheckdata.py` is now about 2.5x faster.
Addons that iterate over the configurations more than once no longer
parse the dump file on each pass. The parsed objects are unchanged.
- Objects are built directly from the XML parser callbacks, without
creating `ElementTree` elements first.
- The cyclic garbage collector is disabled while parsing, because it
took about half of the loading time. It is enabled again before each
configuration is returned to the caller.
- The `CppcheckData` constructor skips the `<dump>` elements and parses
only the rest of the file. Before, it parsed the whole file to reach the
raw tokens and suppressions, which are written after the configurations.
- `data.configurations` loads the configurations once and keeps them.
Later accesses and `iterconfigurations()` reuse them, while
`iterconfigurations()` on its own still parses one configuration at a
time. `misc.py` now uses `data.configurations` for its three passes, and
`misra.py --verify` parses the file once instead of twice.
Every parsed object was compared between the old and new code on dumps
of the current layout, the old layout (raw tokens before the
configurations) and clang-import dumps. New tests are in `test/addon`.
### Timings
Fastest of several runs on Python 3.12, with peak memory after the
slash.
Load the dump file and iterate over the configurations once:
| Dump | Size | Configurations | Before | After | Speedup |
|---|---|---|---|---|---|
| misra-test.c | 26 MB | 6 | 2.06s / 239 MB | 0.91s / 181 MB | 2.3x |
| token.cpp | 15 MB | 1 | 1.20s / 161 MB | 0.53s / 157 MB | 2.3x |
| valueflow.cpp | 18 MB | 1 | 1.93s / 275 MB | 0.78s / 243 MB | 2.5x |
| symboldatabase.cpp | 33 MB | 1 | 3.44s / 363 MB | 1.31s / 357 MB |
2.6x |
`misc.py` (three passes over the configurations):
| Dump | Before | After | Speedup |
|---|---|---|---|
| misra-test.c | 5.08s / 239 MB | 1.28s / 253 MB | 4.0x |
| token.cpp | 3.30s / 169 MB | 0.73s / 158 MB | 4.5x |
| valueflow.cpp | 4.97s / 326 MB | 1.13s / 243 MB | 4.4x |
| symboldatabase.cpp | 9.03s / 422 MB | 1.92s / 358 MB | 4.7x |
`misc.py` uses more memory on misra-test.c because it now keeps all 6
configurations in memory at once.1 parent 8140605 commit 1926dff
3 files changed
Lines changed: 249 additions & 173 deletions
0 commit comments