[kannanjgithub]

Backport of #13087 to v1.85.x.

A gRPC client using xDS-driven mTLS does two things concurrently:

1. Certificate provider: watches cert files on disk, periodically reloads them, and notifies subscribers when certs change.
2. Channel subscriptions: each gRPC channel that needs mTLS registers a watcher on the cert provider to get the TLS credentials.

The DistributorWatcher sits in the middle and fans out: one cert provider pushes updates to it, and it distributes to N channel watchers.

I think there are two conditions in this that can lead to a ConcurrentModificationException:

1. updateSpiffeTrustMap isn't synchronized (unlike updateCertificate and updateTrustedRoots), so it can race against addWatcher/removeWatcher from another thread.
2. if callbacks mutate their own subscription. A callchain (line numbers per 1.83.0 / 1084e97) for that is:

  DistributorWatcher.onError()                              CertificateProvider.java:126
    → for (watcher : downstreamWatchers)                    CertificateProvider.java:127
      → watcher.onError(errorStatus)                        CertificateProvider.java:128
        → DynamicSslContextProvider.onError()               DynamicSslContextProvider.java:143
          → callback.onException(error)                     DynamicSslContextProvider.java:145
            → SslContextProviderSupplier.onException()      SslContextProviderSupplier.java:79
              → releaseSslContextProvider(toRelease)        SslContextProviderSupplier.java:81
                → TlsContextManagerImpl.releaseClient...()  TlsContextManagerImpl.java:78
                  → ReferenceCountingMap.releaseInternal()  ReferenceCountingMap.java:89
                    → if refcount == 0: value.close()       ReferenceCountingMap.java:95
                      → CertProviderSslContextProvider.close()  CertProviderSslContextProvider.java:216
                        → certHandle.close()                CertProviderSslContextProvider.java:218
                          → Handle.close()                  CertificateProviderStore.java:62
                            → distWatcher.removeWatcher()   CertificateProviderStore.java:64  <-- REENTRANT

We observed ConcurrentModificationException in updateSpiffeTrustMap() in production, consistent with the unsynchronized iteration described above. We also observed startup RPC deadlines during the rollout, but have not established the causal connection between these symptoms.

These all occurred in a service that creates many (~60) channels on startup, while getting the CDS config over xDS. The service uses grpc-java / grpc-xds 1.83.0. When we switched it to use mTLS with SPIFFE certificates, we got the following errors:

DEADLINE_EXCEEDED: CallOptions deadline exceeded after 4.998s.
Name resolution delay 0.000000000 seconds.
[closed=[], open=[[connecting_and_lb_delay=4999ms, was_still_waiting]]]

  io.grpc.StatusException: UNKNOWN
    at io.grpc.xds.internal.security.DynamicSslContextProvider.onError(DynamicSslContextProvider.java:145)
    at io.grpc.xds.internal.security.certprovider.CertificateProvider$DistributorWatcher.onError(CertificateProvider.java:128)
    at io.grpc.xds.internal.security.certprovider.FileWatcherCertificateProvider.checkAndReloadCertificates(FileWatcherCertificateProvider.java:149)
  Caused by: java.util.ConcurrentModificationException
    at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1606)
    at java.base/java.util.HashMap$KeyIterator.next(HashMap.java:1629)
    at io.grpc.xds.internal.security.certprovider.CertificateProvider$DistributorWatcher.updateSpiffeTrustMap(CertificateProvider.java:120)
    at io.grpc.xds.internal.security.certprovider.FileWatcherCertificateProvider.checkAndReloadCertificates(FileWatcherCertificateProvider.java:144)

java.util.ConcurrentModificationException
  at java.util.HashMap$HashIterator.nextNode(HashMap.java:1606)
  at java.util.HashMap$KeyIterator.next(HashMap.java:1629)
  at CertificateProvider$DistributorWatcher.onError(CertificateProvider.java:127)
  at FileWatcherCertificateProvider.checkAndReloadCertificates(FileWatcherCertificateProvider.java:149)