Skip to content

feat(apscheduler): Add cron monitoring for scheduled jobs - #7840

Open
wedamija wants to merge 23 commits into
masterfrom
danf/apscheduler-crons
Open

wedamija wants to merge 23 commits into
masterfrom
danf/apscheduler-crons

Conversation

@wedamija

@wedamija wedamija commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Description

Adds cron monitoring to a new APScheduler integration (3.3+), opt-in like CeleryIntegration(monitor_beat_tasks=True):

sentry_sdk.init(integrations=[APSchedulerIntegration(monitor_jobs=True)])

When a job is due, an in_progress check-in is sent with a monitor config built from its trigger, so Sentry creates the monitor on the first run; ok or error follows with the duration. CronTrigger and IntervalTrigger are supported. Triggers Sentry can't represent exactly, and one-off DateTrigger jobs, are not monitored, rather than sent with a schedule that would cause false missed-check-in alerts. The slug is the job id, or the job name when the id was generated; exclude_jobs skips jobs by regex.

Motivation

Check-ins for a monitor that doesn't exist yet are dropped, so APScheduler users have to create each monitor by hand and add check-in calls to every job. The schedule is already on the trigger, so the SDK can send it and monitors are created and kept in sync automatically.

Testing

tests/integrations/apscheduler, including thread pool, process pool and asyncio schedulers.

Docs: getsentry/sentry-docs#19786

Add an opt-in APSchedulerIntegration(monitor_jobs=True) for APScheduler 3.x.
When a job with a cron or interval trigger is submitted to its executor, send
an in_progress check-in with a monitor config derived from the trigger, so the
monitor is created on first run. A scheduler listener sends the ok/error
check-in when the run finishes or is missed.

The integration is not auto-enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Results 📊

✅ 137486 passed | ⏭️ 7530 skipped | Total: 145016 | Pass Rate: 94.81% | Execution Time: 448m 10s

📊 Comparison with Base Branch

Metric Change
Total Tests 📈 +696
Passed Tests 📈 +622
Failed Tests —
Skipped Tests 📈 +74

All tests are passing successfully.

✅ Patch coverage is 97.28% (250 of 257 changed executable lines covered; target 80%).
Project statement coverage is 90.39% (up 0.07 percentage points from base (d41e392) to head (0012437)).

Changed files with executable lines (4)
File Patch coverage Changed executable lines
sentry_sdk/integrations/apscheduler.py 96.80% 212/219 covered; missed: 210, 252, 353, 355, 356, 357, 358; partial branches: 118, 205, 251, 349, 382
sentry_sdk/crons/decorator.py 100.00% 2/2 covered
sentry_sdk/crons/utils.py 100.00% 32/32 covered
sentry_sdk/integrations/arq.py 100.00% 4/4 covered
Coverage diff
@@            Coverage Diff             @@
##        master     #7840       +/-##
==========================================
+ Coverage    90.32%    90.39%    +0.07%
==========================================
  Files          203       204        +1
  Tracked lines     26905     27149      +244
  Branches     10048     10120       +72
==========================================
+ Hits         24301     24540      +239
- Misses        2604      2609        +5
- Partials      1517      1520        +3

Generated by Coverage Action

wedamija and others added 2 commits October 2, 2026 15:53
Regenerating locally dropped cached releases of other packages. Restore them
and keep only the APScheduler entries.

Co-Authored-By: Claude <noreply@anthropic.com>
Move the APScheduler interval-to-schedule conversion into a private
helper in sentry_sdk.crons so other integrations can reuse it.

Co-Authored-By: Claude <noreply@anthropic.com>
wedamija and others added 3 commits October 2, 2026 17:40
Co-Authored-By: Claude <noreply@anthropic.com>
- Handle the Optional[str] check-in id that capture_checkin now returns.
- Send the monitor config only when the trigger's timezone is an IANA
  name. Sentry rejects names like "UTC+02:00", and without a timezone it
  would read the crontab as UTC.
- Open the check-ins only once the executor accepts the job, so runs
  skipped because of max_instances send none, instead of in_progress
  followed by error.
- Warn once per job when it has no explicit id and its name is used as
  the monitor slug.
- Say that only APScheduler 3.x is supported when the import fails.

Co-Authored-By: Claude <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 21, 2026

Copy link
Copy Markdown

PY-3033

# Conflicts:
#	scripts/populate_tox/package_dependencies.jsonl
#	scripts/populate_tox/releases.jsonl
Comment thread sentry_sdk/integrations/apscheduler.py Resolved
Sentry monitor schedules don't end, so a finite cron or interval trigger
would raise missed check-ins after its last run.
Move arq's timezone name logic into sentry_sdk.crons.utils and use it for
APScheduler too, so fixed-offset timezones with whole-hour offsets are
monitored with Etc/GMT names. tzlocal's "local" zone is rejected.
Jitter delays runs by up to that many seconds, so add it on top of Sentry's
default 1 minute check-in margin.
@wedamija
wedamija marked this pull request as ready for review October 22, 2026 18:58
@wedamija
wedamija requested a review from a team as a code owner October 22, 2026 18:58
@wedamija
wedamija requested a review from sentrivana October 22, 2026 18:58
# Conflicts:
#	scripts/populate_tox/releases.jsonl
A run that dies inside the executor, like a crashed process pool worker,
goes through BaseExecutor._run_job_error, which dispatches no event. Close
that job's oldest pending check-in as an error there.
A job function wrapped in @sentry_sdk.monitor sends its own check-ins, so
monitoring it again reported every run twice. The decorator marks its
wrapper with _sentry_monitored, and the integration skips marked jobs.
With coalesce=False, a submit can carry several overdue run times, which
sent a burst of in_progress/ok pairs. Open one check-in per submit and
close it on the last run time's event.
With TZ set to a file path, tzlocal names the zone after the file's
basename, like "localtime", which Sentry rejects as an invalid monitor
config. Return a zone's name only when zoneinfo or pytz knows it.
APScheduler 4 fails the 3.x imports, which already raises DidNotEnable.
Comment on lines +249 to +252
assert check_ins[0]["monitor_config"]["schedule"] == {
"type": "crontab",
"value": expected,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single weekday values expand to multi-day crontab ranges

For a single weekday expression with no step, _get_crontab_day_of_week substitutes 6 when expr.last is None, then emits a range instead of a single day. For example, day_of_week='0' becomes 1-7 and sat,sun becomes 1-7,1-7, although APScheduler runs those jobs only on the specified weekdays. The resulting monitor schedule can cause false missed-check-in alerts. Preserve the single value when expr.last is None and expr.step is None.

Evidence
  • _get_crontab_day_of_week sets last = 6 whenever expr.last is None, then emits a range when last != first.
  • APScheduler represents a single weekday such as 0 as a RangeExpression with last=None and no step; the conversion therefore emits 1-7 rather than 1.
  • test_cron_trigger_monitor_config expects day_of_week='0' to produce 0 9 * * 1 and sat,sun to produce 0 9 * * 6,7, confirming the intended schedules.

Identified by Warden · code-review · TY7-K25

Comment on lines +114 to +118
def finish_oldest(self, job_id: str, status: str) -> None:
with self.lock:
key = next((key for key in self.pending if key[0] == job_id), None)
pending = None if key is None else self.pending.pop(key)
if pending is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Executor errors can close the wrong concurrent job check-in

When an executor reports a task-level failure for one of multiple overlapping submissions of the same job, _run_job_error provides only the job ID. finish_oldest() therefore closes the earliest pending check-in rather than the failed run's check-in. This can mark another run as failed and leave the failed run's check-in pending because it receives no execution event.

Evidence
  • _patch_run_job_error calls finish_oldest(job_id, ERROR); it does not identify the failed submission's run time.
  • finish_oldest removes the first pending entry for that job ID, while add stores entries by (job_id, run_time).
  • With max_instances > 1, multiple submissions of one job can have pending check-ins simultaneously.
  • The process-pool crash test exercises only one in-flight instance, so it does not cover this mismatch with concurrent submissions.

Identified by Warden · code-review · ATT-VVT

Comment on lines +300 to +301
if getattr(job.func, "_sentry_monitored", False):
return None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decorated bound methods are not skipped by APScheduler monitoring

The duplicate-suppression check inspects job.func directly. For a decorated instance method, APScheduler stores a bound method, while _sentry_monitored is set on the underlying wrapper function. The check therefore misses the marker and emits integration check-ins in addition to the decorator's check-ins. Check job.func.__func__ for bound methods as well.

Evidence
  • BaseExecutor.submit_job calls _begin_check_in() for monitored jobs, and _begin_check_in() skips only when getattr(job.func, "_sentry_monitored", False) is true.
  • The sync and async wrappers in sentry_sdk/crons/decorator.py set _sentry_monitored on the wrapper function.
  • A decorated instance method scheduled as instance.method is a bound method; the marker is on its underlying function, not the bound method object.
  • The existing APScheduler test for decorator suppression uses a free function, so it does not cover this bound-method case.
Also found at 1 additional location
  • tests/integrations/apscheduler/test_apscheduler.py:160-176

Identified by Warden · find-bugs · LUZ-4EM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant