Repository navigation
Upgrade the AWS SDK to 2.55.12, realign Jackson on 2.18.11 and upgrade Swagger to 2.2.41 - #506
tonygermano wants to merge 3 commits into
Conversation
The AWS SDK modules were on 2.15.28 while netty-nio-client was already on 2.20.140. The SDK supports only matching versions across its modules, and the mix did not link: netty-nio-client 2.20.140 references members of utils and http-client-spi that 2.15.28 lacks (for example ProxySystemSetting.HTTPS_PROXY_HOST, SdkHttpRequest.forEachHeader and SdkHttpConfigurationOption.TCP_KEEPALIVE), and s3 2.15.28 references the arns module, which never shipped, so parsing an S3 ARN failed with NoClassDefFoundError. Align on 2.55.12, the newest release past the 14-day minimum age as of 2026-10-22. Since 2.17 the SDK parses JSON with its own shaded copy of jackson-core (third-party-jackson-core) and no longer depends on com.fasterxml.jackson at all, which is what lets the next commit realign Jackson. 2.20.140, the version netty-nio-client was on, bundles jackson-core 2.15.2, which has three advisories (CVE-2026-18401, CVE-2026-68494 and CVE-2026-89425); 2.55.12 bundles 2.21.7, which has none. New runtime modules: arns, checksums, checksums-spi, crt-core, endpoints-spi, http-auth, http-auth-aws, http-auth-aws-eventstream, http-auth-spi, identity-spi, json-utils, retries, retries-spi, third-party-jackson-core and utils-lite, placed with the rest of the SDK. No other library changes: the SDK's POMs ask for newer netty, httpcore, slf4j and reactive-streams, but every member reference resolves against the versions already shipped. The only non-AWS additions to the verification metadata are two BOM POMs, junit-bom (imported by the SDK's bom-internal) and jetty-bom (imported by netty-nio-client's POM); no jars come from them. apache5-client is left out. Since 2.46 it is the service clients' default HTTP client, but our S3 connector sets ApacheHttpClient explicitly, and without apache5-client on the classpath a client built without an HTTP client still gets ApacheHttpClient, as before. The client still ships only regions and utils; Region.regions() and Region.id(), the only calls it makes, need nothing else. The region list grows from 31 to 53. Changes that user scripts calling the SDK directly may notice, compared with 2.15.28: - Since 2.30, S3 clients send a CRC32 checksum on upload and validate checksums on download. AWS S3 supports this, including from our S3 connector, but some S3-compatible stores reject it; set requestChecksumCalculation(WHEN_REQUIRED) on the client builder or AWS_REQUEST_CHECKSUM_CALCULATION=when_required to turn it off. - Since 2.53, the SDK core needs service client modules of 2.28.1 or later, so older AWS service jars added to custom-lib stop working. - Since 2.25, the Netty async client takes its proxy settings from environment variables and system properties by default, as the Apache client already did. - Removed classes and members: JavaSystemSetting.USER_REGION, core.util.json.JacksonUtils, the services.s3.checksums classes and the metadata classes of retired services. - Breaking for compiled code and subclasses only: HttpCredentialsProvider became an interface, the STS credential providers' getUpdatedCredentials returns AwsSessionCredentials, NoOpSigner is final, and the JSON protocol factories use the shaded JsonFactory. Every software.amazon.awssdk member reference in the distribution now resolves, except those into the optional aws-crt native library, which was not shipped before either. Checksums regenerated with a cold cache per CONTRIBUTING.md. CVE: CVE-2026-18401 CVE: CVE-2026-68494 CVE: CVE-2026-89425 Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Tony Germano <tony@germano.name>
jackson-core was bumped alone to 2.18.8 for a security fix, leaving the rest of the family on 2.14.3. The family could not follow, because jackson-databind 2.16 removed APIs that swagger-core 2.0.10 and AWS SDK 2.15.28 called. The previous commit took the AWS SDK off Jackson. This one upgrades Swagger and realigns all of Jackson in one step, since neither Swagger release works with the other's Jackson. Jackson 2.18.11 rather than 2.18.8: databind 2.14.3 has eleven open advisories, and 2.18.8 still has nine in databind and two in core. 2.18.11 is the first release with none in any shipped module. The 2.19 line is no longer maintained: 2.19.4, its last release, has sixteen open advisories across databind and core. Swagger 2.2.41 rather than 2.2.28, the first release built against Jackson 2.18: with 2.2.28 the generated /api/openapi.json loses DashboardStatus.childStatuses, a property the API returns. 2.2.41 is the first release that keeps it. It is built against Jackson 2.19.2, but every com.fasterxml.jackson member reference it makes resolves against 2.18.11. No new runtime dependencies: its POM adds jackson-jaxrs-json-provider and newer javassist, which none of its classes reference. Its only unresolved references are to three Bean Validation 2.0 constraints (Email, NotBlank, NotEmpty) that validation-api 1.1.0 lacks; these are only reached for models annotated with them, and none can be. The generated OpenAPI document changes in these ways: - byte[] properties are a base64 string (type string, format byte) instead of an array of strings - CodeTemplateLibrarySaveResult's Throwable is named librariesCause, after its getter, instead of cause - form parameters of the channel, statistics and login operations gain their descriptions - twenty generated operationIds of overloaded methods swap their _1 suffix, because Swagger now reads the overloads in the opposite order; the next commit pins the previous IDs Every Jackson and Swagger member reference in the distribution resolves, apart from the Bean Validation constraints above and log4j-core's optional XML and YAML layouts, which were unresolved before. Checksums regenerated with a cold cache per CONTRIBUTING.md. CVE: CVE-2026-19032 CVE: CVE-2026-54512 CVE: CVE-2026-54513 CVE: CVE-2026-54514 CVE: CVE-2026-54515 CVE: CVE-2026-59889 CVE: CVE-2026-68497 CVE: CVE-2026-77310 CVE: CVE-2026-83557 CVE: CVE-2026-89407 CVE: CVE-2026-89425 CVE: CVE-2026-91776 CVE: CVE-2026-91777 Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Tony Germano <tony@germano.name>
Swagger derives an operationId from the Java method name and gives an overloaded name a _1 suffix in the order it reads the methods. Swagger 2.2.41 reads them in the opposite order from 2.0.10, so ten pairs of operations swapped IDs: getEvents, for example, moved from GET /events to POST /events/_search. A client generated from the spec would map the same method name to a different endpoint. Set operationId explicitly on the 20 affected operations, keeping the IDs the previous spec published. The generated spec now matches the previous one in every operationId, and is otherwise unchanged from the upgrade commit. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Tony Germano <tony@germano.name>
pacmano1
left a comment
There was a problem hiding this comment.
None of the "Still to test by hand" boxes were ticked. An SDK jump of almost six years (2.15.28 from November 2020 to 2.55.12 from October 2026) shouldn't merge without touching real AWS, so I ran them against a real bucket, queue and KMS key, on main and on this branch. All fine on this branch:
- S3: File Writer and Reader, including an access point ARN. Uploads carry CRC32. The ARN only works in the colon form (
arn:aws:s3:us-west-2:<account>:accesspoint:<name>/<prefix>). The slash form the console shows fails with "Invalid ARN", because the connector cuts the host at the first/. Worth a line in the release notes. - KMS: a channel script calling
KmsClientto encrypt and decrypt with a real key, and the File Reader reading an object encrypted with that key. - Region dropdown: 53 regions in the real Administrator, and a region only the new SDK knows (
ca-west-1) now selects instead of falling back to Custom.
Please tick the boxes and point them here.
operationIds. Main never had a fixed set: Swagger 2.0.10 takes Java's method order as is, and across four boots of main (Java 17, 21 twice, 25) 8 of the 16 overloaded names moved, getAlertInfo even between the two Java 21 boots. 2.2.41 sorts classes and methods itself, so this branch gave the same IDs every time, pinned or not. Your pins copy one run of main; my first Java 21 boot of main had setChannelEnabled and setChannelInitialState the other way round. Please reword "All operationIds are unchanged" to say main's moved between boots and this fixes them. Pinning the other six overloaded names (exportMessagesServer, getMessages, removeMessages, sendTestEmail, testConnection, testWrite) is optional; it would stop a future Swagger change from moving them.
Aside for @gibson9583, since this breaks sqs-source-connector:
- 1.0.2 fails at channel start on this branch:
NoClassDefFoundError: software/amazon/awssdk/protocols/query/interceptor/QueryParametersToBodyInterceptor. That's the "service client modules of 2.28.1 or later" note above. - Built with
-Daws.sdk.version=2.55.12, it sends and receives on a real queue. It needs two test-only changes first:server/pom.xmlneedsaws-json-protocolas provided, because SQS moved to the JSON protocol in 2.21, andSqsQueueInspectorWireTestfakes a query-protocol XML reply, so it needs a JSON one. - That build doesn't run on 4.6 or 4.7 (
NoClassDefFoundError: software/amazon/awssdk/awscore/AwsClient), so it needs its own release line, with the engine bounds set inplugin.xmland the catalog.
|
@pacmano1 I checked the boxes for the AWS tests and updated the PR description to point to your commit. Thanks for doing that. I'm not an AWS user. I don't understand what the additional ask is, but you can push changes that you feel are necessary to this PR. I was just trying to upgrade these problem libraries while breaking as little as possible. If you feel something needs mentioning in the release notes, please propose the exact text that should go in the release notes, and I will add it to the PR description (or you can). |
|
Edited the description: reworded the three operationId lines and added the access point ARN form to the release notes. Ignore my bit about pinning the other six; 2.2.41 sorts the methods, so they can't move. |
Supersedes #493, which bumps jackson-core alone to 2.18.11.
Summary
A comment in
libs.versions.tomlsays the Jackson family can't be upgraded until the AWS SDK and Swagger are. This PR does all three in an order that keeps every library working with the Jackson version it gets:Jackson 2.18.11 and the newer copy of jackson-core bundled in the AWS SDK together clear 15 CVEs. Each commit lists its CVEs in its trailers.
The AWS SDK upgrade changes some behavior that channel scripts calling the SDK directly could notice. See Changes for channel scripts.
Why
jackson-databind 2.16 removed APIs that swagger-core 2.0.10 and AWS SDK 2.15.28 call, so the family was held on 2.14.3. Only jackson-core was bumped, for a security fix. databind 2.14.3 now has eleven open advisories.
The AWS SDK mix was already broken on its own:
ProxySystemSetting.HTTPS_PROXY_HOSTandSdkHttpRequest.forEachHeader.arnsmodule, which wasn't shipped. Parsing an S3 ARN failed withNoClassDefFoundError.Commits
third-party-jackson-core), so it no longer depends oncom.fasterxml.jacksonat all. In 2.55.12 that copy is jackson-core 2.21.7, which has no advisories. 2.20.140, the version netty-nio-client was on, bundles 2.15.2, which has three.arns,checksums,checksums-spi,crt-core,endpoints-spi,http-auth,http-auth-aws,http-auth-aws-eventstream,http-auth-spi,identity-spi,json-utils,retries,retries-spi,third-party-jackson-coreandutils-lite.verification-metadata.xmlare two BOM POMs:junit-bom, imported by the SDK'sbom-internal, andjetty-bom, imported by netty-nio-client's POM. No jars come from them.apache5-clientis left out. Since 2.46 it's the default HTTP client of the service clients, but our S3 connector setsApacheHttpClientexplicitly. Withoutapache5-clienton the classpath, a client built without an HTTP client still gets
ApacheHttpClient, as before.regionsandutils. Those cover the only calls it makes (Region.regions()andRegion.id()). The region list grows from 31 to 53.DashboardStatus.childStatusesin/api/openapi.json. The first release built for Jackson 2.18 (2.2.28) drops that property. 2.2.41 is built against Jackson 2.19.2, but every Jackson member it references exists in 2.18.11.
<name>_1. 2.2.41 reads the methods in the opposite order from 2.0.10, so ten pairs swapped IDs. For example,getEventsmoved fromGET /eventstoPOST /events/_search.operationIdexplicitly, using the names from one run of the previous spec.Checksums in
verification-metadata.xmlwere regenerated with a cold cache, as CONTRIBUTING.md describes.Changes for channel scripts
Channel scripts can call the AWS SDK directly. These changes between 2.15.28 and 2.55.12 could affect them:
S3 checksums (since 2.30): S3 clients send a CRC32 checksum on upload and validate checksums on download. AWS S3 supports this, but some S3-compatible stores reject it. Our S3 connector now sends and valid
ates checksums too. It only talks to AWS endpoints, so it shouldn't break this way, but S3 reads and writes are on the manual test list. To turn it off, set
requestChecksumCalculation(RequestChecksumCalculation .WHEN_REQUIRED)on the client builder orAWS_REQUEST_CHECKSUM_CALCULATION=when_requiredin the environment.Service client versions (since 2.53): the SDK core needs service client modules of 2.28.1 or later. Older AWS service jars added to custom-lib (DynamoDB, SQS and so on) stop working and should be updated t
o 2.55.12.
Netty proxy settings (since 2.25): the async Netty client now takes its proxy settings from environment variables and system properties by default, as the Apache client already did.
Removed APIs, which break any caller, scripts included:
JavaSystemSetting.USER_REGION;core.util.json.JacksonUtils;services.s3.checksumsclasses;Changed APIs that break only compiled Java code (such as extensions) or classes that extend the SDK's. Rhino resolves calls at run time, so scripts that only call these aren't affected.
HttpCredentialsProvideris now an interface instead of an abstract class.getUpdatedCredentialsreturnsAwsSessionCredentials.NoOpSigneris final.JsonFactory.The credential interfaces moved to
identity-spi, but existing calls still work.Changes to the OpenAPI document
The API's responses don't change: payloads are written by XStream, not Jackson. The generated spec does change, and each change makes it describe those payloads correctly:
byte[]properties (Attachment.content,RawMessage.rawBytes,EncryptionSettings.secretKey) are nowtype: string, format: byte. The old spec said they were arrays of strings. The server has alwayssent a base64 string, e.g.
"rawBytes":"aGk=".CodeTemplateLibrarySaveResultnames its exceptionlibrariesCauseinstead ofcause.librariesCauseis the name the server sends.oasdiff breakingflags the first two changes. It reports 18 errors for the type change and 2 warnings for the removedcauseproperty. They only affect clients generated from the old spec, and those clients couldn't read the real responses for these fields anyway.
On main, the operationIds of overloaded methods could change between restarts, because Swagger 2.0.10 followed Java's method order. 2.2.41 sorts the methods itself, so they're now fixed. The 20 pinned here keep the names one run of main produced.
Testing
Claude checked:
./gradlew clean build dist: all 722 tests pass.software.amazon.awssdk,com.fasterxml.jacksonandio.swaggerresolves, except:/api/openapi.json,/api/openapi.yamland Swagger UI load.regionsandutilson the classpath.internalpackages. That gave the changes listed above. Claude checked the "removed method" hits by hand: methods that only moved to a superclass aren't listed.To compare the specs yourself:
oasdiff changelogon the old and new specs (strip theexampleskeys first, because they reference external files);diff <(jq -S . old.json) <(jq -S . new.json).Still to test by hand:See this commit for test results on the following:Release notes
arnsmodule that S3 ARN parsing needs, and its bundled JSON parser has no known advisories. Channel scripts that call the SDK directly should review Changes for channel scripts, in particular the S3 checksum default and the 2.28.1 minimum for other AWS service jars. S3 access point ARNs now work in the File connector's S3 Bucket field, in the colon formarn:aws:s3:<region>:<account>:accesspoint:<name>/<prefix>. The slash form the AWS console shows (...:accesspoint/<name>) fails with "Invalid ARN".byte[]fields as base64 strings and namesCodeTemplateLibrarySaveResult.librariesCausecorrectly. Clients generated from the spec may need tobe regenerated.
🤖 Drafted with Claude Code