feat(rs-client): Add automatic resumable uploads by lcian · Pull Request #700 · getsentry/objectstore

lcian · 2026-10-21T11:17:46Z

Changelog Entry

Session::put now uses resumable uploads automatically for buffers and paths to regular files of 256 MiB or more, and resends only what the server hasn't stored after a failed write
- Change the threshold with Usecase::with_resumable_threshold_bytes or per request with .resumable_threshold_bytes(...); None disables automatic resumable uploads.
- Set ClientBuilder::spool_dir to choose where payloads compressed on the fly are spooled (defaults to the system temp dir).
- The resumable-upload-api feature only gates the manual resumable upload API. This new behavior is enabled regardless of that feature flag.

Details

Session::put now uses resumable uploads automatically for eligible uploads, matching the Python client (#667). An upload is eligible if:

It's a buffer or a path to a regular file (put, put_path)
Its input size, before client compression, is at least the resumable threshold (default 256 MiB)
It's sent standalone. Uploads queued through many() keep their existing batch or direct routing and ignore the threshold.

Open file handles (put_file), streams, generic readers, non-regular paths and inputs of unknown size use a direct upload.
The API and behavior mirrors the Python client:

let usecase = Usecase::from_static("debug_files")
    .with_resumable_threshold_bytes(Some(256 * 1024 * 1024)); // None disables; Some(0) includes every supported payload

session.put(payload).resumable_threshold_bytes(None).send().await?; // per-request override

ClientBuilder::new(url).spool_dir("/var/tmp/objectstore"); // optional

Path and reader lifetime. Each attempt opens an independent reader. Source paths must remain readable, unchanged, and must not be renamed, removed, or replaced until the upload completes. Compressed payloads spool to named temporary files and retries reopen them. The HTTP layer may retain old readers after the upload returns or is dropped, delaying handle closure and temporary-storage reclamation.

For reviewers

The Error variants ResumableUploadUnavailable, UploadOffsetMismatch and the chunk-granularity variant are no longer feature-gated. Since Error is #[non_exhaustive] (feat(rs-client)!: Mark Error as non-exhaustive #706), this is additive.
Most of the logic is in put.rs (routing), put/payload.rs (preparation and readers) and put/automatic.rs (recovery loop).

Closes FS-561

Resume eligible buffer and file uploads from authoritative server progress, with configurable thresholds and temporary-file compression. Keep automatic uploads available without the manual protocol feature.

Separate payload preparation and reader lifetime from protocol recovery. Stop and join each request reader before returning its result, report local IO errors directly, and keep progress probes in a dedicated retry loop.

Unify direct fallbacks, make sizing optional, and limit resumables to standalone uploads.

# Conflicts: # Cargo.lock # clients/rust/src/error.rs Co-Authored-By: Lorenzo Cian <17258265+lcian@users.noreply.github.com>

github-actions · 2026-10-22T15:30:10Z

Semver Impact of This PR

This PR lands in these workspaces:

clients/rust: 🟡 Minor (new features)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).

New Features

Session::put now uses resumable uploads automatically for buffers and paths to regular files of 256 MiB or more, and resends only what the server hasn't stored after a failed write by lcian with sentry-junior in #700

Change the threshold with Usecase::with_resumable_threshold_bytes or per request with .resumable_threshold_bytes(...); None disables automatic resumable uploads.

Set ClientBuilder::spool_dir to choose where payloads compressed on the fly are spooled (defaults to the system temp dir).

The resumable-upload-api feature only gates the manual resumable upload API. This new behavior is enabled regardless of that feature flag.

_{🤖 This preview updates automatically when you update the PR.}

…ated docs Co-Authored-By: Lorenzo Cian <17258265+lcian@users.noreply.github.com>

Keep both test dependencies and support gzip when spooling resumable payloads.

Limit automatic resumables to buffers and stable paths. Reopen source paths and named spool files per attempt to remove reader coordination.

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

jan-auer · 2026-10-29T09:32:01Z

            name: Arc::new(UsecaseName::new(name)?),
            compression: Some(Compression::Zstd),
            expiration_policy: ExpirationPolicy::default(),
+            resumable_threshold_bytes: Some(256 * 1024 * 1024),


This means automatic resumable uploads are enabled by default. This is something we should call out explicitly in the changelog.

Let's check with Ingest team what their thoughts on the behavior are.

Fixes the changelog preview added in #737, which fails at startup on every PR ([run on #700](https://github.com/getsentry/objectstore/actions/runs/40873702019): "likely failed because of a workflow file issue"). Craft's reusable preview job requests `contents: read`, `pull-requests: write` and `statuses: write`. A called workflow can't get more permissions than its caller, and our caller only granted the first two. This adds `statuses: write`, matching the old per-client preview workflows. The preview runs from the base branch (`pull_request_target`), so it starts working on other PRs once this merges. #skip-changelog --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Lorenzo Cian <17258265+lcian@users.noreply.github.com>

jan-auer

Approved provided the below comments are addressed.

jan-auer · 2026-10-30T09:35:23Z

+            Self::Temporary { file, .. } => {
+                let file = Arc::clone(file);
+                File::from_std(
+                    tokio::task::spawn_blocking(move || file.reopen())


I think we should re-consider this, possibly in a follow-up PR. I know that the reopen here is required due to ownership problems with the stream being static. There's a second issue, though: This reopen requires us to use a named temporary file, which isn't cleaned up automatically when the process crashes. Therefore, we could be leaking files in a temp directory.

We should explore writing a custom stream that can read from a shared file reference (or use some interior mutability) so we can switch to a regular temp file.

This doesn't have to block merging this PR, though.

After reading the docs of https://docs.rs/tempfile/latest/tempfile/struct.NamedTempFile.html#method.new I want to address this 😅

See https://github.com/getsentry/objectstore/pull/746/changes

jan-auer · 2026-10-30T09:36:05Z

+                self.key.clone(),
+                self.metadata,
+            );
+            if let Ok(Some(upload)) = create.send().await {


Can we cache rejections on the client, so we don't re-attempt to create resumable uploads if we know a server blocks it?

Currently we cannot distinguish between "this particular upload creation request was rejected" or "the server doesn't support resumable uploads" because both return 501 not implemented.
We would need a way (e.g header) to distinguish those.
But in general they should always be supported, so I'm not sure this is necessary.

Keep optional resumable exports together and move the dead-code allowance into the module.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c6b8bef. Configure here.}

feat(rust): Add automatic resumable uploads

96bf619

Resume eligible buffer and file uploads from authoritative server progress, with configurable thresholds and temporary-file compression. Keep automatic uploads available without the manual protocol feature.

This comment has been minimized.

Sign in to view

lcian changed the title ~~feat(rust): Add automatic resumable uploads~~ feat(rust-client): Add automatic resumable uploads Oct 21, 2026

lcian and others added 3 commits October 21, 2026 14:48

ref(rust-client): Simplify automatic upload recovery

64aae59

Separate payload preparation and reader lifetime from protocol recovery. Stop and join each request reader before returning its result, report local IO errors directly, and keep progress probes in a dedicated retry loop.

ref(rust-client): Address automatic upload API feedback

e0e4113

Unify direct fallbacks, make sizing optional, and limit resumables to standalone uploads.

Merge branch 'main' into lcian/feat/rust-automatic-resumable-uploads

5695815

# Conflicts: # Cargo.lock # clients/rust/src/error.rs Co-Authored-By: Lorenzo Cian <17258265+lcian@users.noreply.github.com>

sentry-junior Bot changed the title ~~feat(rust-client): Add automatic resumable uploads~~ feat(rs-client): Add automatic resumable uploads Oct 22, 2026

sentry-junior Bot and others added 7 commits October 23, 2026 11:30

ref(rs-client): Simplify automatic upload preparation and trim duplic…

b06ec0b

…ated docs Co-Authored-By: Lorenzo Cian <17258265+lcian@users.noreply.github.com>

ref(rs-client): Poll the upload receiver directly

34fd381

chore: Merge main and resolve Rust client conflicts

a478425

Keep both test dependencies and support gzip when spooling resumable payloads.

docs(rs-client): Keep upload documentation at the right level

e8a70b3

ref(rs-client): Separate resumable API from protocol implementation

593f8b0

ref(rs-client): Use independent readers for resumable paths

7300c71

Limit automatic resumables to buffers and stable paths. Reopen source paths and named spool files per attempt to remove reader coordination.

ref(rs-client): Share resumable constructors and keep fields private

2af4aa9

This comment has been minimized.

Sign in to view

cursor Bot reviewed Oct 28, 2026

View reviewed changes

Comment thread clients/rust/src/put.rs Resolved

lcian added 9 commits October 28, 2026 15:38

ref(rs-client): Simplify direct upload body conversion

0241bd9

ref(rs-client): Stream upload files with an async generator

37a16ae

ref(rs-client): Follow existing file streaming pattern

30c7a1a

ref(rs-client): Use consistent internal constructor names

a0a007d

docs(rs-client): Simplify automatic upload module comment

f3f93c9

ref(rs-client): Return offsets and inline resumable recovery

7322630

ref(rs-client): Rename offset helper to get_offset

60764f3

ref(rs-client): Clarify prepared HTTP body conversion name

188afab

ref(rs-client): Fall back for unsupported upload payloads

3f64519

This comment has been minimized.

Sign in to view

docs(rs-client): Explain unreachable payload fallback branches

15bc2d0

cursor Bot reviewed Oct 28, 2026

View reviewed changes

Comment thread clients/rust/src/put/payload.rs Resolved

lcian marked this pull request as ready for review October 28, 2026 16:03

lcian requested a review from a team as a code owner October 28, 2026 16:03

This comment has been minimized.

Sign in to view

jan-auer reviewed Oct 29, 2026

View reviewed changes

lcian marked this pull request as draft October 29, 2026 12:25

lcian marked this pull request as ready for review October 29, 2026 12:26

chore: Merge main into automatic resumable uploads branch

27ec7ec

sentry-junior Bot and others added 5 commits October 29, 2026 12:40

chore: Trigger changelog preview

042a359

ref(rs-client): Simplify automatic upload errors and code layout

2db1f26

ref(rs-client): Flatten the resumable upload module

3f07540

ref(rs-client): Avoid boxed readers when spooling uploads

c07e591

ref(rs-client): Encapsulate spool settings and align visibility

9ca153c

jan-auer approved these changes Oct 30, 2026

View reviewed changes

lcian added 2 commits October 30, 2026 12:05

fix(rs-client): Mint fallback tokens after upload preparation

ddafbf3

Keep optional resumable exports together and move the dead-code allowance into the module.

chore: Merge main and update dependencies

c6b8bef

cursor Bot reviewed Oct 30, 2026

View reviewed changes

Comment thread clients/rust/src/put/automatic.rs Resolved

lcian mentioned this pull request Oct 30, 2026

ref(rs-client): Use anonymous files for compressed resumable uploads #746

Draft

lcian added 3 commits October 30, 2026 13:48

chore: Merge main into automatic resumable uploads

a31e2d6

feat(rs-client): Add optional tracing for upload fallback

c744554

docs(rs-client): Clarify resumable uploads and feature flags

5c27eb2

Conversation

lcian commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog Entry

Details

Uh oh!

This comment has been minimized.

github-actions Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Semver Impact of This PR

New Features

Uh oh!

This comment has been minimized.

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jan-auer Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jan-auer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jan-auer Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

lcian Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

lcian Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

jan-auer Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

lcian Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lcian commented Oct 21, 2026 •

edited

Loading

github-actions Bot commented Oct 22, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading