Repository navigation
feat(spanner): add built-in tracing to Cloud Trace - #15064
uditdasari wants to merge 3 commits into
Conversation
Setting SPANNER_ENABLE_TRACING=true (or SpannerOptions.Builder#setEnableBuiltInTracing(true)) makes the client export its traces to Cloud Trace and turns on end to end tracing, without any OpenTelemetry setup in the application. It is off by default. This adds com.google.cloud.opentelemetry:exporter-trace and io.opentelemetry:opentelemetry-sdk-trace as compile dependencies for all applications, also when built-in tracing is off. They can conflict with other versions of these libraries that an application already uses.
There was a problem hiding this comment.
Code Review
This pull request introduces built-in tracing capabilities to the Google Cloud Spanner Java client using OpenTelemetry. It adds configuration options to SpannerOptions to enable built-in tracing and set a sampling rate, either programmatically or via environment variables. A new BuiltInTracingProvider class is introduced to manage and cache OpenTelemetry instances per project and sampling rate. The Spanner client implementation and RPC layers are updated to use this new tracing mechanism. Feedback on the changes suggests wrapping the Runtime.getRuntime().addShutdownHook call in BuiltInTracingProvider within a try-catch block to handle potential IllegalStateExceptions if the JVM is already shutting down.
| if (shutdownHook == null) { | ||
| shutdownHook = new Thread(this::shutdown); | ||
| Runtime.getRuntime().addShutdownHook(shutdownHook); | ||
| } |
There was a problem hiding this comment.
Registering a shutdown hook via Runtime.getRuntime().addShutdownHook can throw an IllegalStateException if the JVM is already in the process of shutting down. Since this call is outside the try-catch block, it can cause unexpected crashes for callers initializing or using a client during JVM shutdown. Wrapping the registration in a try-catch block for IllegalStateException resolves this safely.
if (shutdownHook == null) {
Thread hook = new Thread(this::shutdown);
try {
Runtime.getRuntime().addShutdownHook(hook);
shutdownHook = hook;
} catch (IllegalStateException e) {
// JVM is shutting down, ignore
}
}Use exporter-trace 0.33.0 so that its dependencies match the versions in the shared dependencies. Do not use built-in tracing when the client uses a universe domain other than googleapis.com. Only log the sampling rate when built-in tracing is enabled, and log which OpenTelemetry instance is used when built-in tracing is not used.
Built-in tracing now exports spans with the OpenTelemetry OTLP gRPC exporter to the Cloud Telemetry (OTLP) API instead of the Cloud Trace exporter, as recommended by Cloud Trace. The OpenTelemetry artifacts are already part of the shared dependencies, so no new third-party dependency is added. Spans carry the gcp.project_id resource attribute and the service name from OTEL_SERVICE_NAME. Built-in tracing is now also used when the configured OpenTelemetry instance has no enabled tracer, for example when an application uses OpenTelemetry only for metrics.
This adds built-in tracing. Setting
SPANNER_ENABLE_TRACING=true(orSpannerOptions.Builder#setEnableBuiltInTracing(true)) makes the client export its traces to Cloud Trace and turns on end to end tracing, without any OpenTelemetry setup in the application. It is off by default, because Cloud Trace charges for the spans that are exported.What it does
telemetry.googleapis.com, which Cloud Trace recommends instead of the Cloud Trace exporter. The exporter uses the credentials of the client, and the spans carry thegcp.project_idresource attribute with the project of the client. The service name is taken fromOTEL_SERVICE_NAMEif that is set.traceparentandx-goog-spanner-end-to-end-tracingare sent and Spanner adds its server side spans to the same trace.OTEL_TRACES_SAMPLER_ARGenvironment variable or withsetBuiltInTracingSamplingRate(double). The startup log of the client shows whether built-in tracing is enabled, and the sampling rate when it is enabled.ConnectionSpannerOptions.Builder#build()does not callSpannerOptions.Builder#build().traceparentheader and API tracing (if enabled separately) use the built-in instance. Metrics are not affected.When it is not used
Tracer#isEnabled()), for example anOpenTelemetryset inSpannerOptionswith a tracer provider that exports spans, or a global OpenTelemetry registered by the application or by the OpenTelemetry Java agent. That configuration is used instead, end to end tracing is still turned on, and a log line says which instance is used. An OpenTelemetry instance that is only used for metrics does not disable built-in tracing.googleapis.com. The Telemetry API and Cloud Trace are not available in other universes.Requirements
The project of the client needs the Telemetry API and the Cloud Trace API enabled, and the credentials need the
telemetry.traces.writepermission, which is included inroles/cloudtrace.agent,roles/telemetry.tracesWriterandroles/telemetry.writer. User credentials also need a quota project.Behavior for existing applications
SpannerOptions.enableOpenCensusTraces()after it has created a client or a connection, that call now throws, because the tracing framework is already set to OpenTelemetry. Calling it before creating any client still works, and then built-in tracing is not used.Dependencies
io.opentelemetry:opentelemetry-sdk-trace,io.opentelemetry:opentelemetry-exporter-otlpandio.opentelemetry:opentelemetry-exporter-sender-grpc-managed-channelare now compile dependencies. Their versions are managed by the OpenTelemetry BOM in the shared dependencies. The OkHttp sender of the OTLP exporter is excluded, so the exporter uses gRPC. These dependencies are added for all applications, also when built-in tracing is off, and can conflict with other versions of these libraries that an application already uses. This should be called out in the release notes.Testing
BuiltInTracingTestruns a client against a mock server and checks that the request carries a sampledtraceparentand the end to end tracing header, that the exported spans have the same trace id and thegcp.project_idresource attribute, that clients of the same project share one instance and other projects get their own, that an OpenTelemetry instance with tracing is used instead of built-in tracing, that a metrics only OpenTelemetry instance (set in the options or global) does not disable built-in tracing, and that built-in tracing is not used without credentials, in another universe domain, or with OpenCensus. It also checks the request headers and the service name.BuiltInTracingConnectionTestchecks that the Connection API spans and the Spanner spans are exported by built-in tracing.SpannerOptionsnow also stubgetTracingOpenTelemetry().OTEL_SDK_DISABLED=truebuilt-in tracing is used.