Skip to content

fix: refuse restart after stop() - #1336

Open
davidberenstein1957 wants to merge 4 commits into
masterfrom
fix/stop-idempotent
Open

davidberenstein1957 wants to merge 4 commits into
masterfrom
fix/stop-idempotent

Conversation

@davidberenstein1957

@davidberenstein1957 davidberenstein1957 commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Stop idempotency itself (a repeat stop() writing only one CSV row) is covered by the already-merged #1408. This PR adds a _stopped guard, set by stop(), and checked by both start() and start_task(), so calling either after stop() logs the same clear "already stopped, cannot be restarted" error instead of running into whatever the now-torn-down state produces. On master, start_task() after stop() hits self._scheduler_monitor_power.start() on None, since _scheduler is None post-stop, and raises AttributeError; start() after stop() was not previously guarded either. _stopped is initialized in _initialize_runtime_state() and only ever set in stop().

Also adds a regression test, test_stop_releases_the_lock_only_once, covering that a repeat stop() does not release the lock a second time (the lock by then may belong to a different tracker instance).

Related Issue

None. Stop idempotency itself (one CSV row on a double stop()) is covered by the already-merged #1408.

Motivation and Context

start() and start_task() are two separate entry points that a caller can invoke after stop(). Before this PR, neither refused to run after a stop, so a caller could hit stale or torn-down state; in the case of start_task(), that meant an unrelated AttributeError from calling .start() on the now-None _scheduler_monitor_power, instead of a clear error saying the tracker cannot be restarted.

How Has This Been Tested?

tests/test_emissions_tracker.py::TestRestartAfterStop::test_start_after_stop_is_refused calls start() after stop() and asserts the "cannot be restarted" error is logged.

tests/test_emissions_tracker.py::TestRestartAfterStop::test_start_task_after_stop_is_refused calls start_task() after stop() and asserts the same error is logged, with no active task started, instead of an AttributeError. It fails on master (AttributeError: 'NoneType' object has no attribute 'start') and passes with this change.

tests/test_emissions_tracker.py::TestCarbonTracker::test_stop_releases_the_lock_only_once asserts Lock.release() is called exactly once across two stop() calls.

Full file: 38 passed (tests/test_emissions_tracker.py + tests/test_offline_emissions_tracker.py).

Screenshots (if appropriate):

N/A

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

AI Usage Disclosure

  • 🟥 AI-vibecoded
  • 🟠 AI-generated
  • ⭐ AI-assisted
  • ♻️ No AI used

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the docs/how-to/contributing.md document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@codecov

codecov Bot commented Aug 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.75%. Comparing base (e159a68) to head (adab363).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1336      +/-   ##
==========================================
+ Coverage   93.66%   93.75%   +0.08%     
==========================================
  Files          50       50              
  Lines        5591     5600       +9     
==========================================
+ Hits         5237     5250      +13     
+ Misses        354      350       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@davidberenstein1957
davidberenstein1957 marked this pull request as ready for review August 12, 2026 19:14
@davidberenstein1957
davidberenstein1957 requested a review from a team as a code owner August 12, 2026 19:14
davidberenstein1957 added a commit that referenced this pull request Aug 19, 2026
Handlers were installed before acquire() could fail. On the "another
instance is already running" path acquire() raises, the tracker sets
_another_instance_already_running and stop() returns at its early guard
without ever reaching release() -- so the host application's SIGINT and
SIGTERM stayed hijacked for the life of the process.

Install them after open(LOCKFILE, "x") succeeds instead, and drop the
_atexit_hook indirection: atexit.unregister() compares with ==, not
identity, so a bound method unregisters fine.

Also make release() idempotent (moved here from #1336): it edits the same
few lines of release() this branch already rewrites.

The deadlock test now unregisters its atexit hook, so a reverted lock.py
fails the suite instead of wedging the interpreter at exit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
davidberenstein1957 added a commit that referenced this pull request Aug 20, 2026
`Lock` installed SIGINT/SIGTERM handlers and threw away the previous ones, so
the host application's handlers were destroyed and Ctrl-C stopped raising
KeyboardInterrupt. Save the previous handlers, chain to them from
`_handle_exit`, and restore them in `release()`, unregistering the atexit hook
so a released lock is not pinned.

Handlers are installed in `acquire()` after `open(LOCKFILE, "x")` succeeds,
not in `__init__`. On the "another instance is already running" path
`acquire()` raises, the tracker sets `_another_instance_already_running`, and
`stop()` returns at its early guard without ever reaching `release()` -- so
handlers installed in the constructor stayed hijacked for the life of the
process.

The thread lock is reentrant: `_handle_exit` calls `release()`, which takes
`_thread_lock`, so a signal delivered while the same thread was inside
`acquire()`/`release()` deadlocked on a plain `Lock`. `release()` is also
idempotent now (moved here from #1336, since it edits the same few lines this
branch already rewrites), and the `_atexit_hook` indirection is dropped:
`atexit.unregister()` compares with `==`, not identity, so a bound method
unregisters fine.

Tests cover the default and ignored signal dispositions, and the deadlock test
unregisters its atexit hook so a reverted lock.py fails the suite instead of
wedging the interpreter at exit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@benoit-cty

Copy link
Copy Markdown
Contributor

🤖 This review comment was written and posted by Claude Opus 5.5 (AI assistant), at the request of @benoit-cty. Findings were checked by reading the code and running tests locally (merged with current master where relevant), but please double-check before acting on them.

Verdict: 🔧 Request changes (rebase down, or close)

Mostly superseded by #1408 (8171d13, now on master). stop() already sets _start_time = None and returns the cached final_emissions, and issue #1307 is closed. The only new behaviour left here is "refuse to restart after stop".

Must fix:

  1. Conflicts with master, and the remaining logic won't work after a rebase.

    • The _stopped_at guard duplicates fix: make tracker.stop() idempotent #1408, and conflicts with it in stop() (emissions_tracker.py ~L911-925).
    • The restart refusal sits inside if self._start_time is not None: in start(). After fix: make tracker.stop() idempotent #1408 that value is None once stopped, so the refusal would never fire.
    • If you keep this PR, rebase it down to the restart refusal only, and use a dedicated flag (e.g. self._stopped = True set in stop()).
  2. Behaviour choice to confirm with maintainers. On master today, start → stop → start → stop "works", noisily:

    • The second start() logs a suppressed AttributeError: 'NoneType' object has no attribute 'start', because the scheduler is None.
    • The second stop() writes a second CSV row whose energy includes the first run's.

    With this PR, the second run writes nothing and only logs an error. Refusing is defensible, but the error message should say so clearly and suggest creating a new tracker. The alternative is to support restarting properly: re-create the scheduler and reset the counters.

Nit:

@davidberenstein1957

Copy link
Copy Markdown
Collaborator Author

Made the changes in ee7ff84: merged master, reduced to the restart refusal only.

Refusing vs. properly supporting restart is still your call.

@davidberenstein1957 davidberenstein1957 changed the title fix: make tracker.stop() idempotent fix: refuse restart after stop() Sep 28, 2026
benoit-cty pushed a commit that referenced this pull request Oct 26, 2026
`Lock` installed SIGINT/SIGTERM handlers and threw away the previous ones, so
the host application's handlers were destroyed and Ctrl-C stopped raising
KeyboardInterrupt. Save the previous handlers, chain to them from
`_handle_exit`, and restore them in `release()`, unregistering the atexit hook
so a released lock is not pinned.

Handlers are installed in `acquire()` after `open(LOCKFILE, "x")` succeeds,
not in `__init__`. On the "another instance is already running" path
`acquire()` raises, the tracker sets `_another_instance_already_running`, and
`stop()` returns at its early guard without ever reaching `release()` -- so
handlers installed in the constructor stayed hijacked for the life of the
process.

The thread lock is reentrant: `_handle_exit` calls `release()`, which takes
`_thread_lock`, so a signal delivered while the same thread was inside
`acquire()`/`release()` deadlocked on a plain `Lock`. `release()` is also
idempotent now (moved here from #1336, since it edits the same few lines this
branch already rewrites), and the `_atexit_hook` indirection is dropped:
`atexit.unregister()` compares with `==`, not identity, so a bound method
unregisters fine.

Tests cover the default and ignored signal dispositions, and the deadlock test
unregisters its atexit hook so a reverted lock.py fails the suite instead of
wedging the interpreter at exit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
davidberenstein1957 and others added 4 commits October 31, 2026 13:55
A dedicated `_stopped` flag set in stop() makes start() log an error and
return, instead of half-restarting with a None scheduler. Keep the
lock-release-once regression test; idempotent stop() itself is covered
by #1408 on master.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JfRKgUd281URY6fMyNAQmh

Copy link
Copy Markdown
Contributor

Rebased on master and cleaned up. Head is now adab363.

Rebase

  • ee7ff84 merged master into this branch. Because of that, a plain rebase replayed the old _stopped_at commit (334f76d), which conflicted with fix: make tracker.stop() idempotent #1408. I rebuilt the branch as a linear history on master instead:
    • 4fd5ea3: net content of ee7ff84
    • 49ae9d6: 044c0d6, cherry-picked
  • 334f76d is dropped, since ee7ff84 had already reverted it. The net diff is the same as before and David is still the author of both commits.

Review

  • No correctness bugs found.
  • The earlier review requests are all handled in the code:
    • the guard is a dedicated _stopped flag;
    • the duplicate idempotent-stop test is removed;
    • the error message says to create a new tracker.
  • In-repo callers (@track_emissions, the CLI, the examples) all create a new tracker for each run, so none of them are affected.

Commits added

  • 529b6c6 refactor(tracker): the error message was copied in both start() and start_task(). It is now one module constant, _RESTART_AFTER_STOP_ERROR.
  • adab363 docs: docs/tutorials/python-api.md now says that a stopped tracker cannot be restarted and that you should create a new one.

Validation

  • All changed lines in codecarbon/ are covered (test_emissions_tracker.py + test_offline_emissions_tracker.py: 45 passed).
  • pre-commit passes, and uv run task docs builds with the link check passing.
  • Full package suite: 831 passed, 4 failed. The failures are in test_rapl_mmio_scanning / test_rapl_permissions. They also fail on master in my environment, most likely because the tests run as root. They are not related to this PR.

Left for the maintainer

  • Should a tracker refuse to restart after stop() (this PR), or support restarting properly by re-creating the scheduler and resetting the counters? I left the approach unchanged.

Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants