feat(storage): add context-owned S3 object storage

.github/workflows/example-gate.yml

@@ -7,6 +7,11 @@ on:
 jobs:
   examples:
     runs-on: ubuntu-latest
+    env:
+      TEAQL_S3_ENDPOINT: http://127.0.0.1:19010
+      TEAQL_S3_BUCKET: teaql-java-example-ci
+      TEAQL_S3_ACCESS_KEY: teaql-java-example-ci
+      TEAQL_S3_SECRET_KEY: teaql-java-example-ci-fixture-only
     steps:
       - uses: actions/checkout@v4
       - name: Install example verification tools
@@ -16,4 +21,27 @@ jobs:
           distribution: temurin
           java-version: '21'
           cache: maven
+      - name: Start dedicated S3 example fixture
+        run: |
+          docker run -d --name teaql-java-example-s3 \
+            -p 127.0.0.1:19010:9000 \
+            -e RUSTFS_ACCESS_KEY="$TEAQL_S3_ACCESS_KEY" \
+            -e RUSTFS_SECRET_KEY="$TEAQL_S3_SECRET_KEY" \
+            -e RUSTFS_CONSOLE_ENABLE=false \
+            rustfs/rustfs@sha256:41fe89380f4120a337790c02af192c3fe7bb55c3edc2e6e9357b487b47c6ab21 /data
+          for attempt in {1..30}; do
+            if curl -sS -o /dev/null --max-time 2 "$TEAQL_S3_ENDPOINT/"; then
+              exit 0
+            fi
+            if [[ "$(docker inspect --format '{{.State.Running}}' teaql-java-example-s3)" != true ]]; then
+              docker logs teaql-java-example-s3
+              exit 1
+            fi
+            sleep 1
+          done
+          docker logs teaql-java-example-s3
+          exit 1
       - run: ./scripts/verify-examples.sh
+      - name: Stop dedicated S3 example fixture
+        if: always()
+        run: docker rm -f teaql-java-example-s3 || true

README.md

@@ -320,6 +320,7 @@ Request JSON is not allowed to replace those trusted capabilities.
 - [Runtime design](RUNTIME_DESIGN.md)
 - [Logging design](LOG_DESIGN.md)
 - [Shared load state](examples/shared-load-state/README.md)
+- [Binary object storage and temporary S3 URLs](examples/object-storage/README.md)
 - [Extension points](https://github.com/teaql/teaql-conformance/blob/main/design/runtime-customization-p0-p1-matrix.md)
 
 ## Safe troubleshooting

examples/object-storage/README.md

@@ -0,0 +1,64 @@
+# Context owned object storage example
+
+Tracking: [Java #211](https://github.com/teaql/teaql-java/issues/211),
+[shared contract #122](https://github.com/teaql/teaql-conformance/issues/122).
+
+The low-level runtime facade stores binary objects through an optional S3
+provider. It requires no generated domain model. All TeaQL dependencies in
+this example resolve from this repository's Maven reactor.
+
+```java
+context.putAttribute(ObjectStorageProvider.class.getName(), provider);
+var storage = context.objectStorage("store payment receipt attachment");
+var key = new ObjectStorage.Key("receipts/opaque-reference.pdf");
+storage.put(key, inputStream, ObjectStorage.PutOptions.create(contentLength));
+var link = storage.getReadUrl(key, ObjectStorage.ReadUrlOptions.defaults());
+// Return link.url() to the authorized caller; never log the signed URL.
+try (var content = storage.get(key)) {
+    content.stream().transferTo(destination);
+}
+```
+
+Trusted startup constructs one `S3ObjectStorage` with shared SDK clients and
+installs it on each context. `S3ObjectStorage.connect` accepts an endpoint
+(null for AWS), region, an SDK credentials provider, bucket and path-style flag.
+The SDK credentials provider can use the default chain or session credentials.
+The provider owns its client/presigner; close it at application shutdown, not
+after every operation. The default factory uses the SDK Apache 5 HTTP client
+with an owned connection pool. Runtime telemetry observes operations automatically.
+
+Keys are durable; URLs default to five minutes and are reusable bearer links.
+Expiration is an upper bound, subject to credential expiry and provider policy.
+Only the application decides who may obtain a URL. A signed URL does not prove
+that the object exists. Keys, custom metadata and URLs are masked in diagnostic
+string representations; call explicit accessors only when actually needed.
+
+Uploads spool to a private temporary file with bounded memory before a retryable
+single PUT. Declared length must match exactly; the limit is 5 GiB. The facade
+closes the supplied stream. Downloads are direct streams and must be closed.
+`getBytes(key, maximumBytes)` enforces a size limit. Create-only uses atomic
+`If-None-Match`; replace must be explicit. SQL transactions do not include S3.
+
+## Verify against a dedicated local S3 service
+
+Configure `TEAQL_S3_ENDPOINT`, `TEAQL_S3_BUCKET`, `TEAQL_S3_ACCESS_KEY` and
+`TEAQL_S3_SECRET_KEY` for a test-only bucket. The example creates that bucket if
+absent; the production provider never creates buckets. Each run uses unique
+keys and deletes only those keys. Do not point it at production storage.
+
+```bash
+bash examples/object-storage/verify.sh
+bash examples/object-storage/verify.sh
+# Optional repeated lifecycle probe in one JVM:
+TEAQL_S3_ITERATIONS=10 bash examples/object-storage/verify.sh
+```
+
+The script fails if configuration is absent. It runs the core contract and
+HTTP error tests and enables the live example explicitly. It tests binary/empty
+data, metadata, duplicate and concurrent create-only, explicit replacement,
+bounded reads, signed URL download and expiry, missing bucket and repeated
+delete. Plain `mvn test` excludes live execution unless `-Dteaql.s3.live=true`;
+that exclusion is not live-provider evidence.
+
+The provider SDK version is pinned in `teaql-object-storage-s3/pom.xml`.
+AWS-hosted S3 and other compatible providers require separate qualification.

examples/object-storage/pom.xml

@@ -0,0 +1,10 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <parent><groupId>io.teaql</groupId><artifactId>teaql-java-parent</artifactId><version>1.555-RELEASE</version><relativePath>../../pom.xml</relativePath></parent>
+  <artifactId>teaql-object-storage-example</artifactId>
+  <dependencies>
+    <dependency><groupId>io.teaql</groupId><artifactId>teaql-runtime</artifactId></dependency>
+    <dependency><groupId>io.teaql</groupId><artifactId>teaql-object-storage-s3</artifactId><version>${project.version}</version></dependency>
+    <dependency><groupId>junit</groupId><artifactId>junit</artifactId><scope>test</scope></dependency>
+  </dependencies>
+</project>

examples/object-storage/src/test/java/io/teaql/examples/storage/ObjectStorageExampleTest.java

@@ -0,0 +1,115 @@
+package io.teaql.examples.storage;
+
+import io.teaql.core.storage.*;
+import io.teaql.core.storage.ObjectStorage.*;
+import io.teaql.runtime.DefaultUserContext;
+import io.teaql.storage.s3.S3ObjectStorage;
+import java.net.URI;
+import java.net.http.*;
+import java.time.Duration;
+import java.util.*;
+import java.util.concurrent.*;
+import org.junit.Test;
+import static org.junit.Assert.*;
+import software.amazon.awssdk.auth.credentials.*;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.s3.*;
+
+/** Opt-in live test; verify.sh always requires configuration, never silently skips. */
+public class ObjectStorageExampleTest {
+    @Test public void binaryStorageFromContext() throws Exception {
+        org.junit.Assume.assumeTrue("Live S3 requires examples/object-storage/verify.sh", Boolean.getBoolean("teaql.s3.live"));
+        int iterations = Integer.getInteger("teaql.s3.iterations", 1);
+        if (iterations < 1 || iterations > 100) throw new IllegalArgumentException("iterations must be 1..100");
+        for (int run = 0; run < iterations; run++) verifyOnce();
+    }
+    private void verifyOnce() throws Exception {
+        URI endpoint = URI.create(required("TEAQL_S3_ENDPOINT"));
+        String bucket = required("TEAQL_S3_BUCKET");
+        var credentials = StaticCredentialsProvider.create(AwsBasicCredentials.create(
+                required("TEAQL_S3_ACCESS_KEY"), required("TEAQL_S3_SECRET_KEY")));
+        // Test infrastructure only. The runtime provider never creates a bucket.
+        try (var admin = S3Client.builder().endpointOverride(endpoint).region(Region.US_EAST_1)
+                .credentialsProvider(credentials).forcePathStyle(true).build()) {
+            try { admin.headBucket(b -> b.bucket(bucket)); }
+            catch (software.amazon.awssdk.services.s3.model.NoSuchBucketException missing) { admin.createBucket(b -> b.bucket(bucket)); }
+            catch (software.amazon.awssdk.services.s3.model.S3Exception failure) {
+                if (failure.statusCode() != 404) throw failure;
+                admin.createBucket(b -> b.bucket(bucket));
+            }
+        }
+        try (var provider = S3ObjectStorage.connect(endpoint, Region.US_EAST_1, credentials, bucket, true)) {
+            var context = new DefaultUserContext(null);
+            context.putAttribute(ObjectStorageProvider.class.getName(), provider);
+            var storage = context.objectStorage("verify binary object storage");
+            var key = new Key("java/" + UUID.randomUUID() + "/二进制 space+%.bin");
+            var emptyKey = new Key(key.value() + ".empty");
+            var raceKey = new Key(key.value() + ".race");
+            byte[] bytes = new byte[256 * 1024];
+            new Random(42).nextBytes(bytes);
+            var options = new PutOptions(bytes.length, "application/octet-stream", Map.of("fixture", "binary"), WriteMode.CREATE_ONLY);
+            try {
+                assertFalse(storage.exists(key));
+                expect(StorageException.Code.NOT_FOUND, () -> storage.get(key));
+                assertEquals(bytes.length, storage.putBytes(key, bytes, options).size());
+                assertEquals("binary", storage.head(key).orElseThrow().customMetadata().get("fixture"));
+                assertArrayEquals(bytes, storage.getBytes(key, bytes.length));
+                expect(StorageException.Code.CONFLICT, () -> storage.putBytes(key, new byte[0], PutOptions.create(0)));
+                assertArrayEquals(bytes, storage.getBytes(key, bytes.length));
+                expect(StorageException.Code.TOO_LARGE, () -> storage.getBytes(key, bytes.length - 1));
+                var url = storage.getReadUrl(key, new ReadUrlOptions(Duration.ofSeconds(30), Disposition.ATTACHMENT, "sample.bin"));
+                assertFalse(url.toString().contains("X-Amz"));
+                var http = HttpClient.newHttpClient();
+                var downloaded = http.send(HttpRequest.newBuilder(url.url()).GET().build(), HttpResponse.BodyHandlers.ofByteArray());
+                assertEquals(200, downloaded.statusCode());
+                assertArrayEquals(bytes, downloaded.body());
+                assertTrue(downloaded.headers().firstValue("Content-Disposition").orElseThrow().contains("sample.bin"));
+                var expiring = storage.getReadUrl(key, new ReadUrlOptions(Duration.ofSeconds(1), null, null));
+                Thread.sleep(2200);
+                assertEquals(403, http.send(HttpRequest.newBuilder(expiring.url()).GET().build(), HttpResponse.BodyHandlers.discarding()).statusCode());
+                storage.putBytes(emptyKey, new byte[0], PutOptions.create(0));
+                assertArrayEquals(new byte[0], storage.getBytes(emptyKey, 0));
+                expect(StorageException.Code.INVALID_INPUT, () -> storage.putBytes(emptyKey, new byte[]{1}, PutOptions.create(0)));
+                storage.putBytes(key, new byte[]{4, 5}, new PutOptions(2, null, Map.of(), WriteMode.REPLACE));
+                assertArrayEquals(new byte[]{4, 5}, storage.getBytes(key, 2));
+                ExecutorService threads = Executors.newFixedThreadPool(2);
+                try {
+                    var tasks = new ArrayList<Callable<Boolean>>();
+                    for (int i = 0; i < 2; i++) tasks.add(() -> {
+                        var other = new DefaultUserContext(null);
+                        other.putAttribute(ObjectStorageProvider.class.getName(), provider);
+                        try { other.objectStorage("verify atomic create").putBytes(raceKey, new byte[]{7}, PutOptions.create(1)); return true; }
+                        catch (StorageException error) { assertEquals(StorageException.Code.CONFLICT, error.code()); return false; }
+                    });
+                    var results = threads.invokeAll(tasks);
+                    assertEquals(1, (results.get(0).get() ? 1 : 0) + (results.get(1).get() ? 1 : 0));
+                } finally { threads.shutdownNow(); }
+                // A missing bucket must not become a false exists result.
+                try (var wrong = S3ObjectStorage.connect(endpoint, Region.US_EAST_1, credentials, "missing-" + UUID.randomUUID(), true)) {
+                    var other = new DefaultUserContext(null);
+                    other.putAttribute(ObjectStorageProvider.class.getName(), wrong);
+                    expect(StorageException.Code.UNAVAILABLE, () -> other.objectStorage("verify missing bucket").exists(key));
+                }
+            } finally {
+                storage.delete(key); storage.delete(key); storage.delete(emptyKey); storage.delete(raceKey);
+            }
+            assertFalse(storage.exists(key));
+            System.out.println("PASS Java S3: binary, empty, metadata, URL download/expiry, atomic create, replace, bounded reads, missing bucket, repeated delete");
+        }
+    }
+    static String required(String name) {
+        String value = System.getenv(name);
+        if (value == null || value.isBlank()) throw new IllegalStateException("Required test setting: " + name);
+        return value;
+    }
+    static void expect(StorageException.Code code, Runnable work) {
+        try { work.run(); fail("Expected " + code); }
+        catch (StorageException error) {
+            if (code != error.code()) {
+                AssertionError failure = new AssertionError("Expected " + code + " but got " + error.code());
+                failure.initCause(error); // Sanitized wrapper only; retains the exact provider call site.
+                throw failure;
+            }
+        }
+    }
+}

examples/object-storage/verify.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+set -euo pipefail
+repo="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+: "${TEAQL_S3_ENDPOINT:?dedicated local test endpoint required}"
+: "${TEAQL_S3_BUCKET:?dedicated test bucket required}"
+: "${TEAQL_S3_ACCESS_KEY:?test access key required}"
+: "${TEAQL_S3_SECRET_KEY:?test secret key required}"
+cd "$repo"
+mvn -B -pl examples/object-storage -am -Dteaql.s3.live=true -Dteaql.s3.iterations="${TEAQL_S3_ITERATIONS:-1}" \
+  -Dtest=ObjectStorageExampleTest,ObjectStorageContractTest,S3ObjectStorageTest \
+  -Dsurefire.failIfNoSpecifiedTests=false test

pom.xml

@@ -76,6 +76,8 @@
         <module>teaql-dm8</module>
         <module>teaql-android</module>
         <module>teaql-business-id-jdbc</module>
+        <module>teaql-object-storage-s3</module>
+        <module>examples/object-storage</module>
         <module>examples/business-id-runtime</module>
         <module>examples/security-foundations</module>
         <module>examples/round-trip-reference-runtime</module>

scripts/verify-examples.sh

@@ -4,14 +4,15 @@ set -euo pipefail
 repo="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 verification_dir="$(mktemp -d)"
 trap 'rm -rf -- "$verification_dir"' EXIT
-expected=(business-id-runtime conformance order-management round-trip-reference-runtime school-management security-foundations shared-load-state standard-order trace-chain)
+expected=(business-id-runtime conformance object-storage order-management round-trip-reference-runtime school-management security-foundations shared-load-state standard-order trace-chain)
 mapfile -t actual < <(find "$repo/examples" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | sort)
 if [[ "${actual[*]}" != "${expected[*]}" ]]; then
   echo "example inventory changed; update scripts/verify-examples.sh: ${actual[*]}" >&2
   exit 1
 fi
 
 cd "$repo"
+bash examples/object-storage/verify.sh
 mvn -q -DskipTests install
 mvn -q -pl teaql-sqlite \
   -Dtest=DerivedQueryTraceSqliteTest,LikeIntentPrivacySqliteTest,TypedIntentPrivacySqliteTest test

teaql-core/src/main/java/io/teaql/core/UserContext.java

@@ -230,6 +230,11 @@ default Object extension(String name) {
         return null;
     }
 
+    /** Low-level binary object operations, with explicit execution intent. */
+    default io.teaql.core.storage.ObjectStorage objectStorage(String comment) {
+        return io.teaql.core.storage.ObjectStorage.forContext(this, comment);
+    }
+
     default <T> T capability(Class<T> capabilityType) {
         return null;
     }