Repository navigation
Conversation
…e_utils
- Add explicit S3 path detection using dataframe_or_path.startswith('s3://')
- Use S3Downloader.read_bytes() with io.BytesIO() wrapper for S3 paths
- Remove s3fs dependency from sagemaker-mlops pyproject.toml
- Preserves existing functionality for DataFrames and local file paths
Testing:
- All 43 existing unit tests pass - no regressions
- Added test_s3fs_equivalence.py proving S3Downloader.read_bytes produces
IDENTICAL results to s3fs for the same S3 input
- Equivalence test validates: same DataFrame shape, columns, dtypes, and values
- See EQUIVALENCE_PROOF.md for detailed analysis
pravali96
had a problem deploying
to
manual-approval
November 19, 2026 20:56 — with
GitHub Actions
Error
pravali96
had a problem deploying
to
manual-approval
November 19, 2026 20:57 — with
GitHub Actions
Error
pravali96
had a problem deploying
to
manual-approval
November 19, 2026 20:58 — with
GitHub Actions
Error
Follow-up to the s3fs removal in prepare_fg_from_dataframe_or_file. Three issues with the previous commit: 1. The S3 read dropped the caller's credentials. S3Downloader.read_bytes was called without a session, so read_bytes fell back to `sagemaker_session or Session()` and built a fresh default-credential-chain session, while the FeatureGroup was built from the caller's session resolved ~40 lines later. A caller passing an explicit cross-account session read S3 as the ambient identity. Session resolution now happens before any data is read and is passed through to read_bytes. 2. pathlib.Path inputs were rejected. The signature accepts Union[str, Path, DataFrame] but the branch tested only `str`, so a Path fell through to the "Invalid type" error. The branch now accepts both and the error message lists the real accepted types. 3. EQUIVALENCE_PROOF.md and test_s3fs_equivalence.py were committed to the repository root. The script was a standalone main() that manipulated sys.path and asserted pd.read_csv(BytesIO(x)) == pd.read_csv(BytesIO(x)) -- the same expression on both sides, so it exercised neither s3fs nor the new branch. Both files are removed and replaced with real unit tests. Tests added to the existing sagemaker-mlops/tests/unit/sagemaker/mlops/feature_store/test_feature_utils.py: - an s3:// path is fetched via S3Downloader and the downloaded bytes (not the URI) are what reach load_feature_definitions - the caller's session is threaded into read_bytes, for both an explicit session and one resolved from role + region - a local path never touches S3Downloader - pathlib.Path is accepted Three of the four fail against the previous commit's source and pass against this one. Full file: 47 passed, 24 failed; the 24 failures are identical at 5fe6647 (before any s3fs work), so they are pre-existing and unrelated. Note for reviewers: dropping s3fs is a behavioural change for callers that passed an s3:// path together with fsspec `storage_options` in **kwargs -- those kwargs were previously forwarded to s3fs and will now raise against a BytesIO buffer. --- X-AI-Prompt: Remove s3fs from the SageMaker SDK; investigate why it was introduced and what used it, then fix the session-threading and Path gaps in PR aws#6596 and replace the stray proof files with real unit tests. X-AI-Tool: Claude Code
pravali96
had a problem deploying
to
manual-approval
November 19, 2026 21:04 — with
GitHub Actions
Error
pravali96
had a problem deploying
to
manual-approval
November 19, 2026 21:06 — with
GitHub Actions
Error
pravali96
had a problem deploying
to
manual-approval
November 19, 2026 21:06 — with
GitHub Actions
Error
Preserves existing behaviour for s3:// input after the s3fs removal.
pandas infers compression from the file extension only when read_csv is given
a path. The s3:// branch hands it an in-memory buffer instead, which has no
filename, so inference was lost and a gzipped object failed:
pd.read_csv(io.BytesIO(gzip.compress(csv)))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1
s3://bucket/data.csv.gz worked before this change (pandas saw the URL and
inferred gzip from the suffix), so this would have been a silent regression.
Compression is now inferred from the S3 key with pandas' own
infer_compression, covering .gz/.bz2/.zip/.xz/.zst. An explicit compression=
from the caller still wins.
The docstring now mentions that the path may be an s3:// URI -- stating what
already worked, not promoting S3 input to a documented feature; the function
remains unexported and absent from the docs site, as before.
Tests (added to the existing test_feature_utils.py):
- a gzipped S3 object parses; fails without the inference, verified
- an explicit compression= kwarg is not overridden by inference
Class: 21 passed. File: 49 passed, 24 failed -- those 24 are a local
config.yaml missing SchemaVersion (jsonschema validation in
sagemaker/core/config/config.py), identical on master and unrelated to s3fs.
Known difference, not addressed here: read_bytes buffers the whole object
before parsing where s3fs streamed, so peak memory is higher on large CSVs
and chunksize=/iterator= no longer stream the download.
---
X-AI-Prompt: Remove s3fs from sagemaker-mlops without changing the customer experience; keep s3:// input working as before and don't promote S3 to a first-class feature.
X-AI-Tool: Claude Code
pravali96
had a problem deploying
to
manual-approval
November 19, 2026 21:56 — with
GitHub Actions
Error
pravali96
had a problem deploying
to
manual-approval
November 19, 2026 21:57 — with
GitHub Actions
Error
pravali96
had a problem deploying
to
manual-approval
November 19, 2026 22:03 — with
GitHub Actions
Error
storage_options configures the fsspec filesystem pandas would have opened an
s3:// URI with -- for S3 that was s3fs.S3FileSystem(**storage_options), giving
callers anon, requester_pays, client_kwargs/endpoint_url and friends.
The object is now fetched with S3Downloader before parsing, so read_csv
receives a buffer and rejects the argument outright:
ValueError: storage_options passed with file object or non-fsspec file path
Nothing in this package passes storage_options; it can only arrive from a
caller, since **kwargs are forwarded to read_csv and the docstring points at
the pandas read_csv reference. Surfacing a pandas error about file objects is
not useful, so the argument is dropped with a warning via core's existing
pop_out_unused_kwarg helper ("storage_options supplied in kwargs will be
ignored") and the read proceeds on the session's credentials.
Only the s3:// branch drops it -- for a local path storage_options remains
pandas' business and is still forwarded.
Note this does remove capability rather than relocate it: anon,
requester_pays and a custom endpoint_url have no S3Downloader equivalent.
Callers needing specific credentials should pass a configured session=.
Tests:
- storage_options on an s3:// path warns and the read still succeeds; fails
with the pandas ValueError if the pop is removed, verified
- storage_options on a local path still reaches read_csv untouched
Class: 23 passed. File: 51 passed, 24 failed -- the 24 are a local
config.yaml missing SchemaVersion, identical on master and unrelated.
Re-verified end to end against a dev account (us-west-2): s3:// plain and
gzipped reads on both a default and a non-default bucket, plus local path,
Path and DataFrame inputs. 8/8.
---
X-AI-Prompt: Follow the SDK's own convention for a no-longer-applicable kwarg -- warn and drop storage_options on the s3:// path instead of letting pandas raise, with tests.
X-AI-Tool: Claude Code
pravali96
requested a deployment
to
manual-approval
November 19, 2026 23:03 — with
GitHub Actions
Waiting
pravali96
requested a deployment
to
manual-approval
November 19, 2026 23:03 — with
GitHub Actions
Waiting
pravali96
had a problem deploying
to
manual-approval
November 19, 2026 23:03 — with
GitHub Actions
Error
pravali96
marked this pull request as ready for review
November 19, 2026 23:23
pravali96
requested a deployment
to
manual-approval
November 19, 2026 23:23 — with
GitHub Actions
Waiting
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
s3fsneeds aiobotocore, which pinsbotocore<1.43.118, while this package requiresbotocore>=1.43.127.pipresolves that by backtracking tos3fs 0.4.2(2020) — fsspec warns on load that it's "very old and known to cause severe performance issues." The dependency was forcing a six-year-old package into every install.Description of changes:
Removes s3fs from sagemaker-mlops/pyproject.toml and keeps s3:// input working exactly as before — no new feature, no promotion:
Net diff across three commits: pyproject.toml −1, feature_utils.py +41/−13, test_feature_utils.py +158.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.