Repository navigation
Conversation
…mers PollingEventSource and TimerEventSource each created a java.util.Timer, and PerResourcePollingEventSource a ScheduledThreadPoolExecutor of its own, so every polling event source and every controller cost a thread that the operator neither sized nor shut down. They now schedule on executors managed by ExecutorServiceManager, which builds them from ConfigurationService: - getScheduledExecutorService() backs the polling event sources - getRetryAndRescheduleExecutorService() backs the retry and reschedule timer of every controller, kept separate so that a slow poll cannot delay a retry Both are sized by concurrentScheduledTaskThreads() (4 by default) and concurrentRetryAndRescheduleThreads() (2 by default), overridable programmatically or through the josdk.scheduled-tasks.concurrent-threads and josdk.retry-and-reschedule.concurrent-threads keys. Both default to daemon threads, discard the tasks scheduled for later on shutdown so they don't hold up the termination of the operator, and drop cancelled tasks eagerly. The previous Executors.newScheduledThreadPool(0) was effectively single threaded and created non daemon threads. An event source resolves its executor on every start rather than at creation time, since the manager replaces its pools when the operator is restarted, and only shuts one down if it created it itself. That also fixes PerResourcePollingEventSource shutting down a user supplied executor and not registering its tasks again after a restart. PollingEventSource gains a constructor taking an EventSourceContext, the one without it is deprecated, and both polling configurations accept an executor to run a single event source on a pool of its own.
…rator replaces its executors
… resource polling, clarify executor ownership in docs
…ed after stop during start
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
When useVirtualThreads() is enabled, the client created by the default ConfigurationService#getKubernetesClient() now uses a virtual thread task executor (informer event dispatching, watch event delivery) instead of fabric8's cached platform thread pool, shut down when the client is closed. User provided clients are used as is.
…ording and desired state aspects - bounded virtual thread executor rejects non-positive concurrency and returns tasks that never started from shutdownNow - event sink aggregation also updates type, reason and action - long supplied event names are shortened with a hash so they stay distinct - events for cluster scoped resources in an overridden namespace carry an eventTime - desired state aspects can look up the actual resource of their dependent, receive the user's bulk dependent, and tolerate a missing controller configuration
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved executor lifecycle, polling startup, and event-validation issues can lose work or prevent events from being recorded.
4 open issues
What changed in this PR
Adds opt-in virtual threads to Java Operator SDK while preserving concurrency limits and Java 17 fallback. The PR also centralizes scheduling and includes event-recording and desired-state fixes.
Changes:
- Adds virtual-thread configuration, executor implementations, client integration, documentation, and tests.
- Moves polling and retry scheduling to separate configurable executors with lifecycle management.
- Updates event aggregation, long event names, and desired-state aspect handling.
| File | Description |
|---|---|
| sample-operators/webpage/src/main/java/io/javaoperatorsdk/operator/sample/WebPageOperator.java | Enables virtual threads in the sample. |
| operator-framework/src/test/java/io/javaoperatorsdk/operator/dependent/multiplemanagedexternaldependenttype/MultipleManagedExternalDependentResourceReconciler.java | Supplies context for shared polling. |
| operator-framework/src/test/java/io/javaoperatorsdk/operator/config/loader/ConfigLoaderTest.java | Tests new configuration keys. |
| operator-framework/src/test/java/io/javaoperatorsdk/operator/baseapi/virtualthreads/VirtualThreadsTestReconciler.java | Records thread type and concurrency. |
| operator-framework/src/test/java/io/javaoperatorsdk/operator/baseapi/virtualthreads/VirtualThreadsIT.java | Tests reconciliation threading and limits. |
| operator-framework/src/test/java/io/javaoperatorsdk/operator/baseapi/virtualthreads/VirtualThreadsCustomResource.java | Provides the integration-test resource. |
| operator-framework/src/main/java/io/javaoperatorsdk/operator/config/loader/ConfigLoader.java | Binds virtual-thread and scheduling settings. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/event/source/timer/TimerEventSourceTest.java | Tests executor ownership and restart. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/event/source/polling/PollingEventSourceTest.java | Tests shared scheduling and cancellation. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/event/source/polling/PerResourcePollingEventSourceTest.java | Tests polling restart behavior. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/dependent/DesiredStateAspectAuditTest.java | Adds desired-state aspect regression cases. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/api/event/EventRecorderAuditTest.java | Adds event-recording regression cases. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/api/event/DefaultEventRecorderTest.java | Tests collision-resistant name shortening. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/api/config/VirtualThreadsTest.java | Tests executor and client threading. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/api/config/ExecutorServiceManagerTest.java | Tests scheduler separation and lifecycle. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/api/config/ConfigurationServiceOverriderTest.java | Tests threading configuration overrides. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/api/config/BoundedVirtualThreadExecutorAuditTest.java | Tests limits and pending-task shutdown. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/timer/TimerEventSource.java | Replaces timers with supplied or owned executors. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/polling/PollingEventSource.java | Adds shared scheduling and executor ownership. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/polling/PollingConfigurationBuilder.java | Adds custom polling executor configuration. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/polling/PollingConfiguration.java | Stores the optional polling executor. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/polling/PerResourcePollingEventSource.java | Resolves executors on startup and cancels tasks. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/polling/PerResourcePollingConfiguration.java | Defers default executor selection. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/EventSources.java | Accepts a retry scheduler supplier. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/EventSourceManager.java | Connects retries to the managed scheduler. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/external/PollingDependentResource.java | Passes context to polling sources. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/BulkDependentResourceReconciler.java | Passes the user dependent to aspects. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/reconciler/DefaultContext.java | Handles reentrant desired-state aspect lookups. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/event/DefaultEventSink.java | Updates fields during named-event aggregation. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/event/DefaultEventRecorder.java | Adds event timestamps and hashed name suffixes. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/VirtualThreads.java | Implements virtual executors and concurrency bounds. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/Utils.java | Adds named daemon scheduler utilities. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/ExecutorServiceManager.java | Selects executors and manages scheduler lifecycle. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/ConfigurationServiceOverrider.java | Exposes threading and scheduler overrides. |
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/ConfigurationService.java | Defines threading defaults and client integration. |
| docs/content/en/docs/documentation/operations/configuration.md | Documents virtual threads and configuration keys. |
| docs/content/en/docs/documentation/eventing.md | Documents shared polling and executor ownership. |
Review effort: Balanced
- remove leftover platform thread executor assignments that overrode the virtual thread based caching executor in ExecutorServiceManager - buffer per resource polling registrations that arrive before the event source started and replay them once the executor is resolved - close the race between execute and shutdownNow of the bounded virtual thread executor with a shared lifecycle lock - reject aggregating events that carry an eventTime when type, reason or action differ - keep the JVM alive with a non-daemon thread while virtual threads are in use - run the startup deadlock test on Java 25 or later only
The event recorder aggregation, long name and cluster scoped event namespace fixes now live in fix/event-recorder-correctness.
…nternal Add a virtual thread backed scheduled executor and related configuration, docs and tests. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…resilience - use the daemon scheduled thread pool for standalone timer and polling executors - resolve the timer executor before marking the event source as running - keep polling a resource after its fetcher failed - exclude the intentional compareTo/equals mismatch of scheduled handles from SpotBugs Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…overriding Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved scheduler contract violations and cancellation/lifecycle races can skip fixed-rate executions, run cancelled work, and leave duplicate or stale polling chains.
7 open issues
Return unstarted scheduled work from shutdownNow() · New Remove cancelled handles from waiting during rearm · New Publish runner before cancellation check · New Preserve overdue deadlines for fixed-rate task catch-up · New Prevent stale replay registrations after deletion · New Invalidate stale polling chains across lifecycle restarts · New Wait for event-producing thread names before consuming them · New
4 resolved since last review
Review effort: Balanced
| cancelWaiting(true); | ||
| timer.shutdownNow(); | ||
| for (var task : worker.shutdownNow()) { | ||
| if (task instanceof Dispatched dispatched) { | ||
| dispatched.handle.cancel(false); |
…ranch Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
There was a problem hiding this comment.
🟡 Changes recommended
Operator construction prematurely starts a non-daemon keep-alive thread, preventing JVM exit if the operator is never started or reconciler registration fails.
7 open issues
If an in-flight fetch returns afterstop()and a subsequentstart(), this check sees the source…Operator.init()obtains this manager, whose constructor callsstart(), so merely constructing… Virtual-thread flag overrides inherited custom executors · NewshutdownNow()omits scheduled tasks that never started: it discards both waiting timers and… Invalidate stale polling chains across lifecycle restarts Remove cancelled handles from waiting during rearm Return unstarted scheduled work from shutdownNow()
8 resolved since last review
A deletion between removing a buffered registration and installing its future finds nothing to… The event count does not guarantee that the thread-name entry is available.CapturingEventHandler… After a fixed-rate task overruns its period,rearm()clamps the delay to zero andarm()resets… Cancellation can occur after theisDone()check but beforerunneris assigned, so… Wait for event-producing thread names before consuming them Prevent stale replay registrations after deletion Preserve overdue deadlines for fixed-rate task catch-up Publish runner before cancellation check
Review effort: Balanced
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…d handles from the waiting set - a per resource poll that outlives a stop and start of the event source no longer reschedules itself next to the polling chain of the new run - a periodic task cancelled while being re-armed is removed from the waiting set of the scheduled executor Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved polling lifecycle races can schedule deleted resources, prevent polling after resource recreation, and create duplicate polling chains after restart.
7 open issues
If an in-flight fetch returns afterstop()and a subsequentstart(), this check sees the source… Add lifecycle-managed non-daemon keep-alive thread · New Virtual-thread flag overrides inherited custom executorsshutdownNow()omits scheduled tasks that never started: it discards both waiting timers and… Invalidate stale polling chains across lifecycle restarts Remove cancelled handles from waiting during rearm Return unstarted scheduled work from shutdownNow()
1 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved defects can cause premature operator exit, incorrect shutdown accounting, cancelled-task retention, lost polling after resource recreation, and silent replacement of custom executors.
8 open issues
If an in-flight fetch returns afterstop()and a subsequentstart(), this check sees the source… Align operator keep-alive documentation with implementation · New Add lifecycle-managed non-daemon keep-alive thread Virtual-thread flag overrides inherited custom executorsshutdownNow()omits scheduled tasks that never started: it discards both waiting timers and… Invalidate stale polling chains across lifecycle restarts Remove cancelled handles from waiting during rearm Return unstarted scheduled work from shutdownNow()
Review effort: Balanced
| - Virtual threads are always daemon threads, so unlike the threads of the platform thread pools they | ||
| don't keep the JVM running: if the `main` method of your application returns after starting the | ||
| operator, make sure something else keeps the JVM alive. |
…s an override changes their settings Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
A startup replay/deletion race can prevent polling for recreated resources, while shutdownNow() omits unstarted tasks from its return value in violation of its contract.
3 open issues
5 resolved since last review
If an in-flight fetch returns afterstop()and a subsequentstart(), this check sees the source… Add lifecycle-managed non-daemon keep-alive threadshutdownNow()omits scheduled tasks that never started: it discards both waiting timers and… Invalidate stale polling chains across lifecycle restarts Remove cancelled handles from waiting during rearm
Review effort: Balanced
…ecutors when started again while running - a per resource registration that races stop() is buffered until the next start instead of leaving a stale entry that blocks polling after a restart - the timer and per resource polling event sources resolve their executor again when started while running, which happens when the operator is started again after a failed start and replaces its executors Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
shutdownNow() omits unstarted scheduled tasks from its returned list, violating the ExecutorService contract and preventing callers from recovering pending work.
2 open issues
1 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved scheduler lifecycle bugs can prevent polling and retries from resuming after restart, interrupt in-flight work, and omit pending tasks from shutdownNow() results.
6 open issues
Stale polling task prevents restart after partial startup failure · New shutdownNow interrupts in-flight polling fetches · New Pending timers are lost when the shared executor is replaced · New Owned executor shutdown interrupts running event handlers · New Align operator keep-alive documentation with implementation Return unstarted scheduled work from shutdownNow()
Review effort: Balanced
…s, and let in-flight work finish when stopping owned executors - PollingEventSource schedules its polling task again when started while running after the operator replaced its executors - TimerEventSource schedules the timers pending on the replaced executor again for their remaining delay - stopping a polling or timer event source that owns its executor no longer interrupts a poll or handler in flight Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>


feat: virtual thread support
Adds a single flag that switches the framework's concurrent work over to virtual threads:
It can also be set with
josdk.use-virtual-threads. It is off by default, so nothing changes forexisting operators.
Why
Reconcilers spend almost all their time blocked on the Kubernetes API server or on external
systems. Virtual threads make that kind of blocking cheap, so concurrency limits can be raised far
above what is reasonable with platform threads, without sizing and tuning thread pools.
What changes when the flag is on
withConcurrentReconciliationThreads(n)still caps how many run at once (enforced with aSemaphore).withConcurrentWorkflowExecutorThreads(n).josdk-keep-alive) alive until the operator stops. This keeps an operator whosemainreturns after starting it from letting the JVM exit.Java 25 only
Virtual threads are supported on Java 25 and later (JEP 491
removed the carrier pinning inside
synchronizedblocks, which the codebase uses extensively; onJava 21 to 24 it can starve the carrier pool and deadlock startup). This is checked at runtime: on
an older JVM a warning is logged and platform threads are used, so one configuration works on every
Java version the operator runs on. The compilation target stays Java 17.
Scheduled executors (the larger part of this PR)
Previously
PollingEventSourceandTimerEventSourceeach created ajava.util.Timer, andPerResourcePollingEventSourcecreated its ownScheduledThreadPoolExecutor. Each cost a threadthat the operator neither sized nor shut down. They now use executors managed by
ExecutorServiceManager, built fromConfigurationService:getScheduledExecutorService()backs the polling event sources. It is sized withconcurrentScheduledTaskThreads()(default 4) orjosdk.scheduled-tasks.concurrent-threads.getRetryAndRescheduleExecutorService()backs the retry and reschedule timer of everycontroller. It is sized with
concurrentRetryAndRescheduleThreads()(default 2) orjosdk.retry-and-reschedule.concurrent-threads. It is kept separate so a slow poll cannot delay aretry.
ConfigurationServiceOverrider. The operator owns the lifecycle ofthese executors and shuts them down on stop. An executor given to a single event source through
its configuration builder is the user's to shut down.
reschedules don't delay termination), remove cancelled tasks eagerly, and let idle threads time
out after 60 seconds.
start()rather than at construction, because theoperator replaces its executors when it is stopped and started again.
VirtualThreadScheduledExecutorThe JDK has no scheduled executor backed by virtual threads, so with the flag on the defaults are a
new package-private
VirtualThreadScheduledExecutor: one daemon platform thread does the timingand hands each fired task to a virtual-thread-per-task executor.
typically blocking calls.
withConcurrentScheduledTaskThreadsandwithConcurrentRetryAndRescheduleThreadsonly apply to platform threads.on
shutdown(), and tasks that have already started are let finish.shutdownNow()interruptsrunning tasks.
ScheduledThreadPoolExecutordoes. Fixed-delay tasks re-arm from completion time.
Other changes
Utilsto an internalExecutorUtilsInternal.ConfigLoaderandConfigurationServiceOverriderexpose the new settings.DefaultContextandBulkDependentResourceReconciler: fixes for re-entrant desired-statecomputation, found while running under virtual threads.
VirtualThreadScheduledExecutorhandles. They are ordered by remainingdelay but must stay distinct by identity, as the JDK's own scheduled futures.
Documentation
operations/configuration.md: new "Virtual Threads" section and the new configuration keys.eventing.md: "Threading of the polling event sources", with sizing and executor ownership.Testing
Unit tests, with the virtual thread ones gated by
@EnabledForJreRange(min = JAVA_25):VirtualThreadScheduledExecutorTest: virtual thread execution, cancellation, fixed delay andfixed rate, overrun catch-up, shutdown semantics, unbounded concurrency.
VirtualThreadsTest,VirtualThreadsJvmLifetimeTest: bounded executor, fallback on old JVMs,JVM stays alive after
mainreturns.ExecutorServiceManagerTest,ConfigurationServiceOverriderTest: lifecycle and overriding.PerResourcePollingEventSourceTest,PollingEventSourceTest,TimerEventSourceTest: restart,shared executors, stop and start ordering, polling resilience.
ExecutorUtilsInternalTest: idle pool threads time out and delayed tasks still run.Integration test:
VirtualThreadsITruns a full operator with the flag on.Known limitations and follow-ups