Skip to content

feat: virtual thread support - #3664

Open
csviri wants to merge 31 commits into
nextfrom
virtual-threads-and-shared-executors
Open

csviri wants to merge 31 commits into
nextfrom
virtual-threads-and-shared-executors

Conversation

@csviri

@csviri csviri commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

feat: virtual thread support

Adds a single flag that switches the framework's concurrent work over to virtual threads:

Operator operator = new Operator(override -> override.withUseVirtualThreads(true));

It can also be set with josdk.use-virtual-threads. It is off by default, so nothing changes for
existing operators.

Why

Reconcilers spend almost all their time blocked on the Kubernetes API server or on external
systems. Virtual threads make that kind of blocking cheap, so concurrency limits can be raised far
above what is reasonable with platform threads, without sizing and tuning thread pools.

What changes when the flag is on

Area Behavior
Reconciliations Run on virtual threads. withConcurrentReconciliationThreads(n) still caps how many run at once (enforced with a Semaphore).
Dependent workflows Same: virtual threads, still capped by withConcurrentWorkflowExecutorThreads(n).
Internal housekeeping The unbounded caching executor (starting informers, etc.) uses a virtual thread per task.
Scheduled tasks Polling event sources and retried or rescheduled reconciliations run on virtual threads. See the next section.
Kubernetes client The default client created by the framework runs its internal task executor (informer event dispatch, watch event delivery) on virtual threads. A client provided by the user is used as is.
JVM lifetime Virtual threads are always daemon threads, so the framework keeps one idle platform thread (josdk-keep-alive) alive until the operator stops. This keeps an operator whose main returns after starting it from letting the JVM exit.

Java 25 only

Virtual threads are supported on Java 25 and later (JEP 491
removed the carrier pinning inside synchronized blocks, which the codebase uses extensively; on
Java 21 to 24 it can starve the carrier pool and deadlock startup). This is checked at runtime: on
an older JVM a warning is logged and platform threads are used, so one configuration works on every
Java version the operator runs on. The compilation target stays Java 17.

Scheduled executors (the larger part of this PR)

Previously PollingEventSource and TimerEventSource each created a java.util.Timer, and
PerResourcePollingEventSource created its own ScheduledThreadPoolExecutor. Each cost a thread
that the operator neither sized nor shut down. They now use executors managed by
ExecutorServiceManager, built from ConfigurationService:

  • getScheduledExecutorService() backs the polling event sources. It is sized with
    concurrentScheduledTaskThreads() (default 4) or josdk.scheduled-tasks.concurrent-threads.
  • getRetryAndRescheduleExecutorService() backs the retry and reschedule timer of every
    controller. It is sized with concurrentRetryAndRescheduleThreads() (default 2) or
    josdk.retry-and-reschedule.concurrent-threads. It is kept separate so a slow poll cannot delay a
    retry.
  • Both can be replaced through ConfigurationServiceOverrider. The operator owns the lifecycle of
    these executors and shuts them down on stop. An executor given to a single event source through
    its configuration builder is the user's to shut down.
  • The default pools use daemon threads, drop tasks scheduled for later on shutdown (so far-out
    reschedules don't delay termination), remove cancelled tasks eagerly, and let idle threads time
    out after 60 seconds.
  • Event sources resolve their executor on every start() rather than at construction, because the
    operator replaces its executors when it is stopped and started again.

VirtualThreadScheduledExecutor

The JDK has no scheduled executor backed by virtual threads, so with the flag on the defaults are a
new package-private VirtualThreadScheduledExecutor: one daemon platform thread does the timing
and hands each fired task to a virtual-thread-per-task executor.

  • The number of scheduled tasks running at once is not limited in this mode, because they are
    typically blocking calls. withConcurrentScheduledTaskThreads and
    withConcurrentRetryAndRescheduleThreads only apply to platform threads.
  • It keeps the semantics of the default platform executor: waiting and periodic tasks are discarded
    on shutdown(), and tasks that have already started are let finish. shutdownNow() interrupts
    running tasks.
  • Fixed-rate tasks make up missed executions after an overrun, as ScheduledThreadPoolExecutor
    does. Fixed-delay tasks re-arm from completion time.

Other changes

  • Executor helpers moved from Utils to an internal ExecutorUtilsInternal.
  • ConfigLoader and ConfigurationServiceOverrider expose the new settings.
  • DefaultContext and BulkDependentResourceReconciler: fixes for re-entrant desired-state
    computation, found while running under virtual threads.
  • The webpage sample operator now runs with virtual threads.
  • spotbugs exclusion for the VirtualThreadScheduledExecutor handles. They are ordered by remaining
    delay but must stay distinct by identity, as the JDK's own scheduled futures.

Documentation

  • operations/configuration.md: new "Virtual Threads" section and the new configuration keys.
  • eventing.md: "Threading of the polling event sources", with sizing and executor ownership.

Testing

Unit tests, with the virtual thread ones gated by @EnabledForJreRange(min = JAVA_25):

  • VirtualThreadScheduledExecutorTest: virtual thread execution, cancellation, fixed delay and
    fixed rate, overrun catch-up, shutdown semantics, unbounded concurrency.
  • VirtualThreadsTest, VirtualThreadsJvmLifetimeTest: bounded executor, fallback on old JVMs,
    JVM stays alive after main returns.
  • ExecutorServiceManagerTest, ConfigurationServiceOverriderTest: lifecycle and overriding.
  • PerResourcePollingEventSourceTest, PollingEventSourceTest, TimerEventSourceTest: restart,
    shared executors, stop and start ordering, polling resilience.
  • ExecutorUtilsInternalTest: idle pool threads time out and delayed tasks still run.

Integration test: VirtualThreadsIT runs a full operator with the flag on.

Known limitations and follow-ups

  • Only Java 25 and later is officially supported for virtual threads.

csviri added 12 commits October 27, 2026 14:18
…mers

PollingEventSource and TimerEventSource each created a java.util.Timer,
and PerResourcePollingEventSource a ScheduledThreadPoolExecutor of its
own, so every polling event source and every controller cost a thread
that the operator neither sized nor shut down.

They now schedule on executors managed by ExecutorServiceManager, which
builds them from ConfigurationService:

- getScheduledExecutorService() backs the polling event sources
- getRetryAndRescheduleExecutorService() backs the retry and reschedule
  timer of every controller, kept separate so that a slow poll cannot
  delay a retry

Both are sized by concurrentScheduledTaskThreads() (4 by default) and
concurrentRetryAndRescheduleThreads() (2 by default), overridable
programmatically or through the josdk.scheduled-tasks.concurrent-threads
and josdk.retry-and-reschedule.concurrent-threads keys. Both default to
daemon threads, discard the tasks scheduled for later on shutdown so
they don't hold up the termination of the operator, and drop cancelled
tasks eagerly. The previous Executors.newScheduledThreadPool(0) was
effectively single threaded and created non daemon threads.

An event source resolves its executor on every start rather than at
creation time, since the manager replaces its pools when the operator is
restarted, and only shuts one down if it created it itself. That also
fixes PerResourcePollingEventSource shutting down a user supplied
executor and not registering its tasks again after a restart.

PollingEventSource gains a constructor taking an EventSourceContext, the
one without it is deprecated, and both polling configurations accept an
executor to run a single event source on a pool of its own.
… resource polling, clarify executor ownership in docs
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
When useVirtualThreads() is enabled, the client created by the default
ConfigurationService#getKubernetesClient() now uses a virtual thread
task executor (informer event dispatching, watch event delivery) instead
of fabric8's cached platform thread pool, shut down when the client is
closed. User provided clients are used as is.
…ording and desired state aspects

- bounded virtual thread executor rejects non-positive concurrency and returns
  tasks that never started from shutdownNow
- event sink aggregation also updates type, reason and action
- long supplied event names are shortened with a hash so they stay distinct
- events for cluster scoped resources in an overridden namespace carry an eventTime
- desired state aspects can look up the actual resource of their dependent,
  receive the user's bulk dependent, and tolerate a missing controller configuration
Copilot AI balanced review requested due to automatic review settings October 27, 2026 13:21
@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: fe183367-762c-48ec-9e17-0f286c723d19

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved executor lifecycle, polling startup, and event-validation issues can lose work or prevent events from being recorded.

4 open issues
What changed in this PR

Adds opt-in virtual threads to Java Operator SDK while preserving concurrency limits and Java 17 fallback. The PR also centralizes scheduling and includes event-recording and desired-state fixes.

Changes:

  • Adds virtual-thread configuration, executor implementations, client integration, documentation, and tests.
  • Moves polling and retry scheduling to separate configurable executors with lifecycle management.
  • Updates event aggregation, long event names, and desired-state aspect handling.
File Description
sample-operators/​webpage/​src/​main/​java/​io/​javaoperatorsdk/​operator/​sample/​WebPageOperator.java Enables virtual threads in the sample.
operator-framework/​src/​test/​java/​io/​javaoperatorsdk/​operator/​dependent/​multiplemanagedexternaldependenttype/​MultipleManagedExternalDependentResourceReconciler.java Supplies context for shared polling.
operator-framework/​src/​test/​java/​io/​javaoperatorsdk/​operator/​config/​loader/​ConfigLoaderTest.java Tests new configuration keys.
operator-framework/​src/​test/​java/​io/​javaoperatorsdk/​operator/​baseapi/​virtualthreads/​VirtualThreadsTestReconciler.java Records thread type and concurrency.
operator-framework/​src/​test/​java/​io/​javaoperatorsdk/​operator/​baseapi/​virtualthreads/​VirtualThreadsIT.java Tests reconciliation threading and limits.
operator-framework/​src/​test/​java/​io/​javaoperatorsdk/​operator/​baseapi/​virtualthreads/​VirtualThreadsCustomResource.java Provides the integration-test resource.
operator-framework/​src/​main/​java/​io/​javaoperatorsdk/​operator/​config/​loader/​ConfigLoader.java Binds virtual-thread and scheduling settings.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​timer/​TimerEventSourceTest.java Tests executor ownership and restart.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PollingEventSourceTest.java Tests shared scheduling and cancellation.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PerResourcePollingEventSourceTest.java Tests polling restart behavior.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​processing/​dependent/​DesiredStateAspectAuditTest.java Adds desired-state aspect regression cases.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​event/​EventRecorderAuditTest.java Adds event-recording regression cases.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​event/​DefaultEventRecorderTest.java Tests collision-resistant name shortening.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​config/​VirtualThreadsTest.java Tests executor and client threading.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​config/​ExecutorServiceManagerTest.java Tests scheduler separation and lifecycle.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​config/​ConfigurationServiceOverriderTest.java Tests threading configuration overrides.
operator-framework-core/​src/​test/​java/​io/​javaoperatorsdk/​operator/​api/​config/​BoundedVirtualThreadExecutorAuditTest.java Tests limits and pending-task shutdown.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​timer/​TimerEventSource.java Replaces timers with supplied or owned executors.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PollingEventSource.java Adds shared scheduling and executor ownership.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PollingConfigurationBuilder.java Adds custom polling executor configuration.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PollingConfiguration.java Stores the optional polling executor.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PerResourcePollingEventSource.java Resolves executors on startup and cancels tasks.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​source/​polling/​PerResourcePollingConfiguration.java Defers default executor selection.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​EventSources.java Accepts a retry scheduler supplier.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​event/​EventSourceManager.java Connects retries to the managed scheduler.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​dependent/​external/​PollingDependentResource.java Passes context to polling sources.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​processing/​dependent/​BulkDependentResourceReconciler.java Passes the user dependent to aspects.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​reconciler/​DefaultContext.java Handles reentrant desired-state aspect lookups.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​event/​DefaultEventSink.java Updates fields during named-event aggregation.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​event/​DefaultEventRecorder.java Adds event timestamps and hashed name suffixes.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​config/​VirtualThreads.java Implements virtual executors and concurrency bounds.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​config/​Utils.java Adds named daemon scheduler utilities.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​config/​ExecutorServiceManager.java Selects executors and manages scheduler lifecycle.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​config/​ConfigurationServiceOverrider.java Exposes threading and scheduler overrides.
operator-framework-core/​src/​main/​java/​io/​javaoperatorsdk/​operator/​api/​config/​ConfigurationService.java Defines threading defaults and client integration.
docs/​content/​en/​docs/​documentation/​operations/​configuration.md Documents virtual threads and configuration keys.
docs/​content/​en/​docs/​documentation/​eventing.md Documents shared polling and executor ownership.

Review effort: Balanced

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri csviri linked an issue Oct 27, 2026 that may be closed by this pull request
csviri and others added 5 commits October 27, 2026 17:22
- remove leftover platform thread executor assignments that overrode the virtual
  thread based caching executor in ExecutorServiceManager
- buffer per resource polling registrations that arrive before the event source
  started and replay them once the executor is resolved
- close the race between execute and shutdownNow of the bounded virtual thread
  executor with a shared lifecycle lock
- reject aggregating events that carry an eventTime when type, reason or action differ
- keep the JVM alive with a non-daemon thread while virtual threads are in use
- run the startup deadlock test on Java 25 or later only
The event recorder aggregation, long name and cluster scoped event namespace
fixes now live in fix/event-recorder-correctness.
…nternal

Add a virtual thread backed scheduled executor and related configuration,
docs and tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…resilience

- use the daemon scheduled thread pool for standalone timer and polling executors
- resolve the timer executor before marking the event source as running
- keep polling a resource after its fetcher failed
- exclude the intentional compareTo/equals mismatch of scheduled handles from SpotBugs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…overriding

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +124 to +128
cancelWaiting(true);
timer.shutdownNow();
for (var task : worker.shutdownNow()) {
if (task instanceof Dispatched dispatched) {
dispatched.handle.cancel(false);
csviri and others added 2 commits October 29, 2026 11:49
…ranch

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

csviri and others added 2 commits October 29, 2026 13:45
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…d handles from the waiting set

- a per resource poll that outlives a stop and start of the event source no longer reschedules itself next to the polling chain of the new run
- a periodic task cancelled while being re-armed is removed from the waiting set of the scheduled executor

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +60 to +62
- Virtual threads are always daemon threads, so unlike the threads of the platform thread pools they
don't keep the JVM running: if the `main` method of your application returns after starting the
operator, make sure something else keeps the JVM alive.
…s an override changes their settings

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

csviri and others added 2 commits October 30, 2026 10:14
…ecutors when started again while running

- a per resource registration that races stop() is buffered until the next start instead of leaving a stale entry that blocks polling after a restart
- the timer and per resource polling event sources resolve their executor again when started while running, which happens when the operator is started again after a failed start and replaces its executors

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

shutdownNow() omits unstarted scheduled tasks from its returned list, violating the ExecutorService contract and preventing callers from recovering pending work.

2 open issues
1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved scheduler lifecycle bugs can prevent polling and retries from resuming after restart, interrupt in-flight work, and omit pending tasks from shutdownNow() results.

6 open issues

Review effort: Balanced

…s, and let in-flight work finish when stopping owned executors

- PollingEventSource schedules its polling task again when started while running after the operator replaced its executors
- TimerEventSource schedules the timers pending on the replaced executor again for their remaining delay
- stopping a polling or timer event source that owns its executor no longer interrupts a poll or handler in flight

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for Virtual Threads

2 participants