Repository navigation
Conversation
In the cluster CI jobs, nginx sends HTTP requests to one of three servers, and the servers keep users and roles in Keeper. A server learns about a new user or role a moment after another server created it. Four client-v2 tests create a user or role and then connect as it at once, so they fail when the second request goes to a server that is behind: - ClientTests.testOperationCustomRoles and testClientCustomRoles: UNKNOWN_ROLE. testOperationCustomRoles failed on the latest main run. - HttpTransportTests.testPasswordAuthentication: "Failed to authenticate". Its last step expects a ClientException, so on a server that is behind it could also pass for the wrong reason. - HttpTransportTests.testAuthHeaderIsKeptFromUser: same pattern, has not failed yet. Send X-ClickHouse-Replica-Tag (ClickHouseHttpProto.HEADER_REPLICA_TAG) with one value on every request of these tests. nginx already routes on this header (hash $http_x_clickhouse_replica_tag), and ClickHouse Cloud uses it for the same purpose. A single server ignores it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Client V2 CoverageCoverage Report
Class Coverage
|
|
JDBC V2 CoverageCoverage Report
Class Coverage
|
JDBC V1 CoverageCoverage Report
Class Coverage
|
Client V1 CoverageCoverage Report
Class Coverage
|
chernser
reviewed
Oct 21, 2026
| * stays on one server. | ||
| */ | ||
| private Client.Builder newClientOnSameServer() { | ||
| return newClient().httpHeader(ClickHouseHttpProto.HEADER_REPLICA_TAG, "client_tests_access"); |
Contributor
There was a problem hiding this comment.
Pinning replica makes this tests effectively same as we run for single node. Then it should be skipped for cloud and cluster or another solution should be applied.
And we cannot skip this tests because they verify important feature.
Author
There was a problem hiding this comment.
Good point.
I can only assume this is a timing issue. The tests create users and roles and it needs to wait a few seconds for that information to be propagated throughout the cluster.
Shall we add some retries to the tests themselves?
Contributor
There was a problem hiding this comment.
We can but I would start with talking to Cloud Team about this issues.
For that we need to fetch query_log for tests and understand in what time queries landed on different replicas.
chernser
requested changes
Oct 21, 2026
chernser
left a comment
chernser
left a comment
Contributor
There was a problem hiding this comment.
please skip user related tests for cluster and cloud or please suggest another solution.
Current one just makes test single node ones.
Thanks!
ccoltx
marked this pull request as draft
October 21, 2026 05:40
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
client-v2 + CH clusterjobs fail intermittently onmainand on PR branches: 11 of the last 40 runs. Every one of those failures came from one of four tests:ClientTests.testOperationCustomRolesmainrun, 2026-10-20)UNKNOWN_ROLE: There is no role ...HttpTransportTests.testPasswordAuthenticationFailed to authenticateClientTests.testClientCustomRolestestOperationCustomRolesRowBinaryFormatWriterTest.writeMissingFieldsTest/writeNumbersTestmain, 2026-10-17)UNKNOWN_STATUS_OF_INSERT: timeout while waiting for quorum. Not fixed here, see below.Cause of the first three: the cluster job runs 3 servers behind nginx. nginx sends each HTTP request to any server, and the servers keep users and roles in Keeper (
<user_directories><replicated>). A server learns about a new user or role a moment after another server created it. Row data doesn't have this problem, because the cluster profile setsinsert_quorum=3. Users and roles have no such guarantee. These tests create a role or user and then connect as it right away. When that request goes to a server that is behind, it fails.Fix: these tests send
X-ClickHouse-Replica-Tag(ClickHouseHttpProto.HEADER_REPLICA_TAG) with one value on every request, so all of their requests go to the same server:hash $http_x_clickhouse_replica_tag consistent). It was added for this purpose in d16ec2d.Changed tests:
ClientTests.testOperationCustomRolesandtestClientCustomRoles: the admin and user clients use a newnewClientOnSameServer(). The version check still usesnewClient().HttpTransportTests.testPasswordAuthentication: all three clients. The last step expects aClientException, and a login that fails on a server that is behind also throws one, so that step could pass for the wrong reason before.HttpTransportTests.testAuthHeaderIsKeptFromUser: same pattern. It hasn't failed yet.Test code only. No production code changes.
Not changed
RowBinaryFormatWriterTestquorum timeouts: in both failures, one replica didn't acknowledge an insert withininsert_quorum_timeout(10 minutes), and the test then hung for 1800 s. Both tables had unique names and plain numeric columns, so neither a reused table nor an experimental type explains it. The job uploads only surefire and failsafe reports, not server logs, so the cause can't be seen yet. A follow-up could upload the ClickHouse container logs when the cluster job fails.HttpTransportTests.testSSLAuthentication: it has the same pattern but goes over HTTPS, which nginx passes through as TCP, so a header can't steer it. It hasn't failed in the last 40 runs.ClientTests.testRuntimeCredentialChange: it already handles this race with random user names and a retry loop.Test
🤖 Generated with Claude Code
Note
Low Risk
Integration test stabilization only; no production code paths or runtime client API changes.
Overview
Fixes intermittent client-v2 + cluster failures where tests create a user or role and immediately authenticate as it. Behind nginx, successive HTTP calls can hit different replicas before replicated access-control metadata catches up, causing
UNKNOWN_ROLEor authentication errors.ClientTests adds
newClientOnSameServer(), which setsX-ClickHouse-Replica-Tag(ClickHouseHttpProto.HEADER_REPLICA_TAG) so setup and login stay on one node.testOperationCustomRolesandtestClientCustomRolesuse it for admin and user clients (version skips still usenewClient()).HttpTransportTests applies the same header via a
SAME_SERVER_TAGconstant on every client intestPasswordAuthentication(including the branch that expectsClientException) andtestAuthHeaderIsKeptFromUser, so negative auth cases fail for the intended reason rather than replica lag.Test-only changes; no production client behavior changes.
Reviewed by Cursor Bugbot for commit ff95785. Bugbot is set up for automated code reviews on this repo. Configure here.