Repository navigation
Conversation
The File Writer fills in ${...} variables in the S3 custom headers for
every message, but the pooled S3Connection kept the headers of the
message that created it. The pool key is only the credentials and the
region, so every later message reused the first message's values.
The File Writer now hands the borrowed connection the current message's
headers before writing. They live on the connection, not in its
S3SchemeProperties, because every connection in a pool shares the
properties of the message that created the pool. A borrowed connection
belongs to one sender until it is returned, so this is safe with several
queue or processing threads. Other schemes and the File Reader are
unchanged.
Reported as NextGen Connect issue 5116.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Finnegan's Owner <44065187+pacmano1@users.noreply.github.com>
|
Can we test this in CI? s3mock or similar? |
|
Moto works for an end-to-end test. With #506's SDK, s3mock doesn't: it only reads the bucket from the path, and the SDK has no environment switch for path style. MinIO is gone from Docker Hub, and current LocalStack needs an account token. So a Moto job in the integration harness can cover this without touching AWS. I only tried it on #506's SDK, not main's, so I'd add it as its own PR once #506 is in. Real AWS through GitHub OIDC is the other option, but it needs an AWS account the project is comfortable with. |
Closes #511.
The File Writer fills in
${...}variables in S3 custom headers for every message, but the pooledS3Connectionkept the headers of the message that created the pool, so every later write sent the first message's values.Change
FileDispatcher.send: after borrowing a connection, an S3 connection gets this message's headers.S3Connection: keeps them in its own field (setCustomHeaders), whichgetCustomHeaders()uses when set. Not inschemeProps, because every connection in a pool shares the first message'sS3SchemeProperties.Thread safety: Donkey clones the connector properties for each message, including a copy of the header map, and the pool hands a connection to one sender until it's returned, so the headers set on a borrowed connection belong to that sender's message.
Tests
FileDispatcherTest: two sends through one pooledS3Connectionwith a mocked S3 client, checkingx-test-idon each upload request. Fails without the fix (expected:<[1, 2]> but was:<[1, 1]>), passes with it.How to verify
Needs an S3 bucket and credentials the engine can use, for example
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKENandAWS_REGIONin the engine's environment, with Use Default Credential Provider Chain on.$c('clinic', String(msg));; File Writer in S3 mode, S3 Bucket<bucket>/s3-headers, file name${message.messageId}.txt, template${message.rawData}; in Amazon S3 Advanced Settings, Custom HTTP Headersx-amz-meta-clinic=${clinic}andx-test-id=${message.messageId}.clinic Aandclinic B.aws s3api head-object --bucket <bucket> --key s3-headers/2.txt --query Metadata{"clinic": "clinic A", "x-test-id": "1"}{"clinic": "clinic B", "x-test-id": "2"}Live-tested against real S3 on Java 21, on top of #506 (AWS SDK 2.55.12), the SDK this will ship with. Before:
2.txtcarried message 1's values, with Keep Connection Open on and off (also reproduced on main). After: every object carried its own message's values, with Keep Connection Open on and off, and with the destination queue on and 2 threads (6 messages).Co-authored with Claude Code.