Skip to content

gh-161751: Fix race in mimalloc abandoned segment visit - #162003

Merged
corona10 merged 1 commit into
python:mainfrom
corona10:gh-161751-mimalloc-abandoned-visit-race
Nov 19, 2026
Merged

corona10 merged 1 commit into
python:mainfrom
corona10:gh-161751-mimalloc-abandoned-visit-race

Conversation

@corona10

@corona10 corona10 commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

@corona10

Copy link
Copy Markdown
Member Author

@DinoV @nascheme @daanx (For mimalloc upstream report)

Since mimalloc 2.5.2, abandoned segments of all sub-processes are tracked in one shared arena bitmap, so a reclaim in one sub-process can temporarily clear the bit of a segment that belongs to another sub-process. A full visit read that bitmap before taking abandoned_visit_lock, so it could see a temporarily cleared bit and skip that segment. As a downstream change, a full visit now takes the lock before reading the bitmap.

@corona10

corona10 commented Nov 14, 2026 •

Copy link
Copy Markdown
Member Author

I also reproduced the reported issue and confirmed that this fix resolves it.

@corona10

corona10 commented Nov 15, 2026 •

Copy link
Copy Markdown
Member Author

In my local benchmark, no overhead is observed in free-threaded multithreading and multiple subinterpreters. (only noisy)

import pyperf
from concurrent import interpreters

CODE = """
import collections, gc, threading, time

gc.disable()
window = collections.deque(maxlen=8)

def churn():
    t = threading.Thread(target=lambda: window.append([{} for _ in range(2000)]))
    t.start()
    t.join()

for _ in range(8):
    churn()
op = gc.collect if role == "gc" else churn
ready.put(None)
if background:
    while stop.empty():
        op()
else:
    while (loops := cmd.get()) is not None:
        t0 = time.perf_counter()
        for _ in range(loops):
            op()
        out.put(time.perf_counter() - t0)
"""


class Pool:
    def __init__(self, roles, bg_roles):
        self.ready, self.out, self.stop = (
            interpreters.create_queue() for _ in range(3))
        self.cmds, self.threads = [], []
        for role, background in ([(r, False) for r in roles]
                                 + [(r, True) for r in bg_roles]):
            cmd = interpreters.create_queue()
            interp = interpreters.create()
            interp.prepare_main(role=role, background=background, cmd=cmd,
                                ready=self.ready, out=self.out, stop=self.stop)
            thread = threading.Thread(target=interp.exec, args=(CODE,))
            thread.start()
            self.threads.append(thread)
            if not background:
                self.cmds.append(cmd)
        for _ in self.threads:
            self.ready.get()

    def run(self, loops):
        for cmd in self.cmds:
            cmd.put(loops)
        return sum(self.out.get() for _ in self.cmds) / len(self.cmds)

    def close(self):
        self.stop.put(None)
        for cmd in self.cmds:
            cmd.put(None)
        for thread in self.threads:
            thread.join()


pool = None


def bench(loops, roles, bg_roles):
    global pool
    if pool is None:
        pool = Pool(roles, bg_roles)
    return pool.run(loops)


if __name__ == "__main__":
    runner = pyperf.Runner()
    try:
        # name, measured interpreters, background interpreters
        runner.bench_time_func("gc_collect_1interp", bench, ["gc"], [])
        runner.bench_time_func("gc_collect_4interp", bench, ["gc"] * 4, [])
        runner.bench_time_func("thread_churn", bench, ["churn"] * 2, [])
        runner.bench_time_func("thread_churn_gc_loop", bench, ["churn"] * 2, ["gc"])
    finally:
        if pool is not None:
            pool.close()
import collections
import gc
import threading
import time

import pyperf

window = collections.deque(maxlen=8)
stop = threading.Event()
background = []


def churn():
    t = threading.Thread(target=lambda: window.append([{} for _ in range(2000)]))
    t.start()
    t.join()


def run_until_stopped(op):
    while not stop.is_set():
        op()


def bench(loops, op, bg_ops):
    if not window:
        gc.disable()
        for _ in range(8):
            churn()
        for bg_op in bg_ops:
            thread = threading.Thread(target=run_until_stopped, args=(bg_op,))
            thread.start()
            background.append(thread)
    t0 = time.perf_counter()
    for _ in range(loops):
        op()
    return time.perf_counter() - t0


if __name__ == "__main__":
    runner = pyperf.Runner()
    try:
        # name, measured operation, background threads
        runner.bench_time_func("gc_collect", bench, gc.collect, [])
        runner.bench_time_func("gc_collect_churn", bench, gc.collect, [churn] * 2)
        runner.bench_time_func("thread_churn", bench, churn, [])
        runner.bench_time_func("thread_churn_gc_loop", bench, churn, [gc.collect])
    finally:
        stop.set()
        for thread in background:
            thread.join()

@corona10
corona10 requested a review from colesbury November 18, 2026 14:21
@corona10
corona10 merged commit 484bc16 into python:main Nov 19, 2026
63 checks passed
@miss-islington-app

Copy link
Copy Markdown

Thanks @corona10 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15.
🐍🍒⛏🤖

@bedevere-app

bedevere-app Bot commented Nov 19, 2026

Copy link
Copy Markdown

GH-162269 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Nov 19, 2026
@bedevere-app

bedevere-app Bot commented Nov 19, 2026

Copy link
Copy Markdown

GH-162270 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Nov 19, 2026
corona10 added a commit that referenced this pull request Nov 19, 2026
…2003) (gh-162270)

gh-161751: Fix race in mimalloc abandoned segment visit (gh-162003)
(cherry picked from commit 484bc16)


gh-161751: Fix race in mimalloc abandoned segment visit in free-threaded GC

Co-authored-by: Donghee Na <donghee.na@python.org>
corona10 added a commit that referenced this pull request Nov 19, 2026
…2003) (gh-162269)

gh-161751: Fix race in mimalloc abandoned segment visit (gh-162003)
(cherry picked from commit 484bc16)


gh-161751: Fix race in mimalloc abandoned segment visit in free-threaded GC

Co-authored-by: Donghee Na <donghee.na@python.org>
@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.15 (tier-1) has failed when building commit b61e538.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/2019/builds/908) and take a look at the build logs.
  4. Check if the failure is related to this commit (b61e538) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/2019/builds/908

Failed tests:

  • test_multiprocessing_fork.test_processes

Failed subtests:

  • test_interrupt - test.test_multiprocessing_fork.test_processes.WithProcessesTestProcess.test_interrupt

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 650, in test_interrupt
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 631, in _kill_process
    self.assertEqual(join(), None)
                     ~~~~^^
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 304, in __call__
    return self.func(*args, **kwds)
           ~~~~~~~~~^^^^^^^^^^^^^^^
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/multiprocessing/process.py", line 156, in join
    res = self._popen.wait(timeout)
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line 44, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 627, in handler
    raise RuntimeError(f'join took longer than {timeout} seconds: {p}')
RuntimeError: join took longer than 30.0 seconds: <Process name='Process-158' pid=2391210 parent=2390924 started daemon>


Traceback (most recent call last):
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 650, in test_interrupt
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 631, in _kill_process
    self.assertEqual(join(), None)
                     ~~~~^^
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 304, in __call__
    return self.func(*args, **kwds)
           ~~~~~~~~~^^^^^^^^^^^^^^^
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/multiprocessing/process.py", line 156, in join
    res = self._popen.wait(timeout)
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line 44, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.15.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 627, in handler
    raise RuntimeError(f'join took longer than {timeout} seconds: {p}')
RuntimeError: join took longer than 30.0 seconds: <Process name='Process-1' pid=2391800 parent=2391798 started daemon>

@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.x (tier-1) has failed when building commit 484bc16.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/506/builds/14075) and take a look at the build logs.
  4. Check if the failure is related to this commit (484bc16) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/506/builds/14075

Summary of the results of the build (if available):

Click to see traceback logs
Note: switching to '484bc16723ef534e693bcd13e139ccd710a64da4'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 484bc16723ef gh-161751: Fix race in mimalloc abandoned segment visit (gh-162003)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module

/tmp/ccSgRzzD.s: Assembler messages:
/tmp/ccSgRzzD.s: Fatal error: Programs/_testembed.o: No space left on device
make: *** [Makefile:2132: Programs/_testembed.o] Error 1

@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.14 (tier-1) has failed when building commit 320b591.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1802/builds/1578) and take a look at the build logs.
  4. Check if the failure is related to this commit (320b591) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1802/builds/1578

Summary of the results of the build (if available):

Click to see traceback logs
Note: switching to '320b59196ebd9cbc2f8be18326d65c7e8603dc93'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 320b59196ebd [3.14] gh-161751: Fix race in mimalloc abandoned segment visit (gh-162003) (gh-162270)
Switched to and reset branch '3.14'

configure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)

ar: libpython3.14d.a: error reading Modules/_io/iobase.o: No space left on device
make: *** [Makefile:1180: libpython3.14d.a] Error 1

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [Makefile:3408: clean-retain-profile] Error 1 (ignored)

@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.15 (tier-1) has failed when building commit b61e538.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/2106/builds/589) and take a look at the build logs.
  4. Check if the failure is related to this commit (b61e538) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/2106/builds/589

Summary of the results of the build (if available):

Click to see traceback logs
Note: switching to 'b61e53854c33af7bc226022ed50684efe0bd8728'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at b61e53854c33 [3.15] gh-161751: Fix race in mimalloc abandoned segment visit (gh-162003) (gh-162269)
Switched to and reset branch '3.15'

configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module

/tmp/ccylQOBn.s: Assembler messages:
/tmp/ccylQOBn.s: Fatal error: Python/import.o: No space left on device
make: *** [Makefile:3433: Python/import.o] Error 1

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [Makefile:3527: clean-retain-profile] Error 1 (ignored)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants