Skip to content

feature: add endpoint autoscaling to sagemaker.serve and fix managed instance scaling for fine-tuned deployments - #6591

Open
lhnealreilly wants to merge 6 commits into
aws:masterfrom
lhnealreilly:fix/model-customization-managed-instance-scaling
Open

lhnealreilly wants to merge 6 commits into
aws:masterfrom
lhnealreilly:fix/model-customization-managed-instance-scaling

Conversation

@lhnealreilly

@lhnealreilly lhnealreilly commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Issue

Autoscaling a SageMaker endpoint from the SDK has three gaps:

  1. No autoscaling API. Autoscaling lives in Application Auto Scaling (and CloudWatch, for alarm-driven policies), and the SDK has no interface to it, so users hand-roll register_scalable_target / put_scaling_policy / put_metric_alarm calls with boto3 (model.deploy to allow for auto scale configuration #1880). migration.md lists endpoint auto-scaling as "Via ModelBuilder.deploy", but deploy(managed_instance_scaling=...) only sets an inference-component endpoint's instance bounds: nothing changes a variant's instance count, a component's copy count or a serverless endpoint's provisioned concurrency with traffic.
  2. Model customization deployments ignore managed_instance_scaling. deploy() sets it on the endpoint configuration of a regular deployment (_deploy_core_endpoint), but a fine-tuned deployment goes through _deploy_model_customization, which builds the production variant from name, instance type and count only. Its inference components then cannot add instances, and managed instance scaling can only be added later with a new endpoint configuration.
  3. sagemaker-core drops falsy shape fields. _serialize_shape keeps a field only when its serialized value is truthy (if serialize_result := serialize(v)), so 0, False and "" set on a shape never reach the API. For managed instance scaling, MinInstanceCount: 0, the setting scale to zero requires, is dropped and CreateEndpointConfig fails with "Instance count needs to be specified for enabled managed instance scaling".

Change

sagemaker.serve.autoscaling (new)

from sagemaker.serve.autoscaling import (
    Alarm, CustomMetric, ScheduledAction, StepAdjustment, StepScalingPolicy, TargetTrackingPolicy,
    describe_autoscaling, disable_autoscaling, enable_autoscaling, resume_autoscaling,
    suspend_autoscaling,
)

# One target-tracking policy on the default metric.
enable_autoscaling(endpoint_name="my-endpoint", min_capacity=1, max_capacity=4, target_value=4)

# Any mix of policies and schedules.
gpu = CustomMetric("GPUUtilization", namespace="/aws/sagemaker/Endpoints")
enable_autoscaling(
    endpoint_name="my-endpoint",
    min_capacity=1,
    max_capacity=8,
    policies=[
        TargetTrackingPolicy(target_value=4),
        TargetTrackingPolicy(target_value=70, metric=gpu, name="gpu"),
        StepScalingPolicy(
            steps=[StepAdjustment(scaling_adjustment=2, lower_bound=0)],
            alarm=Alarm(metric=gpu, threshold=90, evaluation_periods=2),
            name="gpu-burst",
        ),
    ],
    scheduled_actions=[
        ScheduledAction(name="business-hours", schedule="cron(0 8 ? * MON-FRI *)",
                        min_capacity=2, timezone="America/New_York"),
    ],
)

# An inference component that scales to zero copies.
enable_autoscaling(inference_component_name="my-ic", min_capacity=0, max_capacity=4,
                   target_value=4, scale_to_zero=True)

suspend_autoscaling(endpoint_name="my-endpoint")
resume_autoscaling(endpoint_name="my-endpoint")
describe_autoscaling(endpoint_name="my-endpoint")
disable_autoscaling(endpoint_name="my-endpoint")

What scales follows the endpoint: a real-time or asynchronous endpoint's production variant (sagemaker:variant:DesiredInstanceCount), an inference component (sagemaker:inference-component:DesiredCopyCount), or a serverless endpoint's provisioned concurrency (sagemaker:variant:DesiredProvisionedConcurrency).

  • Target tracking on a predefined metric by short name (concurrent-requests, the default for a real-time endpoint; invocations; provisioned-concurrency-utilization for a serverless endpoint) or full type, on any CloudWatch metric through CustomMetric (dimensions default to the scaled resource's own), or on metric math by passing a CustomizedMetricSpecification dict. An asynchronous endpoint defaults to ApproximateBacklogSizePerInstance, as Autoscale an asynchronous endpoint recommends.
  • Step scaling with the CloudWatch alarm that drives it: the module creates the alarm, points it at the policy and tags it.
  • Scheduled actions (at, rate, cron) with time zone and start and end times.
  • Scale to zero for inference components, following Scale an endpoint to zero instances (a step policy on a NoCapacityInvocationFailures alarm; the minimum is registered as 1 until the alarm exists and lowered to 0 last, so a failed alarm write cannot leave a component that scales in to 0 and never back out), and for asynchronous endpoints (a step policy on HasBacklogWithoutCapacity).
  • Suspend and resume scale-in, scale-out and scheduled scaling.
  • Declarative and scoped. enable_autoscaling manages the set it is given: calling it again changes those policies and schedules and removes the ones no longer given. Everything it creates is named sagemaker-sdk-..., and the scalable target and alarms it creates are tagged; disable_autoscaling deletes only those, deregisters the target only when this module registered it and nothing else is on it, and deletes an alarm only when it carries the tag. Policies, schedules, alarms and targets created any other way stay.
  • Refused before any write: an endpoint that is not InService, an on-demand serverless variant (SageMaker scales it itself), several variants with none named, variant scaling on an endpoint that hosts inference components, a component on an endpoint without managed instance scaling (copies could only fill the instances the endpoint has), an adapter component (it names the base component whose copies it runs on), invocations on a base component with adapters, scale to zero on a real-time variant or provisioned concurrency, scale to zero on a component without a managed-instance-scaling minimum of 0 or with adapters, and invalid bounds, metrics, steps or schedules.
  • Reads page through describe_scalable_targets, describe_scaling_policies and describe_scheduled_actions, and Application Auto Scaling is eventually consistent, so wait=True (the default) reads the change back before returning.

Predictive scaling is not included: PutScalingPolicy with PolicyType=PredictiveScaling on a SageMaker variant returns AccessDeniedException: Account is not authorized.

Model customization deployments

_deploy_model_customization reads managed_instance_scaling from the deploy kwargs, with the same dict and default as the regular path (MinInstanceCount falls back to the initial instance count), validates MaxInstanceCount, and sets it on the new endpoint configuration. When the endpoint already exists, or the model is deployed on the endpoint's variant (a restricted model package, or a Nova model without inference-component resources), the setting does not apply and is logged as ignored.

model_builder.deploy(
    endpoint_name="my-lora-endpoint",
    inference_component_name="my-lora-endpoint-adapter",
    managed_instance_scaling={"MinInstanceCount": 1, "MaxInstanceCount": 4},
)

sagemaker-core shape serialization

_serialize_shape now keeps falsy scalars (0, False, "") and still drops None/Unassigned and empty dicts and lists, so a nested shape with nothing set (such as the empty StoppingCondition the tuner builds when no stopping condition is given) is still omitted from the request. No SDK code constructs a shape with a "", 0 or False literal field, so the change affects only values callers set explicitly, which now reach the API as given.

Docs

migration.md maps hand-rolled boto3 autoscaling to the new module, and docs/api/sagemaker_serve.rst documents it.

Testing

Unit:

  • sagemaker-serve/tests/unit/test_autoscaling.py: 71 tests, 100% line coverage of autoscaling.py. A stateful fake of the SageMaker, Application Auto Scaling and CloudWatch clients covers each target kind and policy type, custom metrics and metric math, step alarms, scheduled actions, the declarative removal of entries no longer given (and the ones left alone, including entries on later pages of the describe calls), both scale-to-zero paths and the inference-component write order, a failed alarm write, every refusal (asserting no write), suspend and resume, turning off with and without ownership, and the read-back retry.
  • sagemaker-serve/tests/unit/test_model_customization_managed_instance_scaling.py: 9 tests: the setting reaches the new endpoint configuration, the minimum default, a missing or too-small maximum, nothing set unless requested, an existing endpoint, a restricted package and a Nova model left alone with a warning, and a minimum of 0 serialized as 0.
  • sagemaker-core/tests/unit/generated/test_utils.py: falsy scalar fields are kept; unset fields, empty nested shapes and empty dicts are omitted.
  • Full suites, compared with master in the same environment: sagemaker-core 4153 passed (master: 4150) and sagemaker-serve 2337 passed (master: 2262); the 9 and 59 failures are the same tests on both, which reach AWS endpoints and fail without credentials.
  • black, flake8, pylint (10.00 on the new module) and pydocstyle.

Integration (sagemaker-serve/tests/integ/test_autoscaling_integration.py, new, slow_test). Each test creates its own CPU endpoint from the public hub's pytorch-ic-mobilenet-v2 and deletes everything afterwards:

  • Real-time variant: two target-tracking policies (one on CPUUtilization), a step policy with its alarm and a scheduled action; suspend and resume; a second call with one policy removes the rest and the alarm; turning off deregisters.
  • Inference component: an endpoint configuration with MinInstanceCount: 0 created through EndpointConfig.create (it reads back as 0); scale to zero (both policies, the alarm, minimum 0); back to a minimum of 1 (step policy and alarm removed); turning off.
  • Asynchronous endpoint: scale to zero on the backlog metric with the HasBacklogWithoutCapacity policy and alarm; turning off removes the alarm.
  • Serverless endpoint with provisioned concurrency: target tracking on provisioned-concurrency utilization; turning off.

All four pass against SageMaker in us-west-2 and leave no endpoint, configuration, model, component, alarm, scalable target or S3 object behind. test_model_customization_deployment.py::test_deploy_from_training_job now also deploys with managed_instance_scaling={"MinInstanceCount": 1, "MaxInstanceCount": 2} and checks the endpoint configuration it creates.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

deploy(managed_instance_scaling=...) set managed instance scaling on the endpoint configuration of a regular deployment, but _deploy_model_customization built the production variant without it, so a fine-tuned model's inference components could not add instances. Set it on the new endpoint configuration with the same dict and default as the regular path, warn when the endpoint already exists, and refuse MinInstanceCount 0, which sagemaker-core's shape serialization drops.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

The PR plumbs managed_instance_scaling through the model-customization (fine-tuned) deploy path (_deploy_model_customization), building a ProductionVariantManagedInstanceScaling on the new endpoint config, defaulting MinInstanceCount to the instance count, warning when an endpoint already exists, and rejecting MinInstanceCount: 0 with a clear error. The change is small, consistent with the existing regular deploy path, and comes with focused unit tests. I found no correctness or security issues; only a minor scope observation below.

Reviewed commit c2eb5d9. Automated review; verify before acting.

@@ -6586,14 +6614,32 @@ def _deploy_model_customization(
# Check if endpoint exists

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Low: There is another EndpointConfig.create in this same method on the restricted-model-package / model-on-variant branch (base line ~6504, not shown in this diff) that still ignores managed_instance_scaling. A user passing managed_instance_scaling for a restricted-package fine-tuned model would silently get an endpoint config without it, with no warning. If that path is in scope for the same argument, consider applying the setting there too (or emitting the same ignored-warning); if it's intentionally out of scope, a brief note would help future readers.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Handled in 1006cdc. A restricted model package (and a Nova model without inference-component resources) is deployed on the endpoint's variant, where managed instance scaling does not apply, so that branch now logs that managed_instance_scaling is ignored instead of dropping it silently. Unit tests cover both branches.

Add enable_autoscaling, disable_autoscaling and describe_autoscaling: target-tracking autoscaling of a production variant's instance count or an inference component's copy count, with scale to zero for inference components (a step-scaling policy and a NoCapacityInvocationFailures alarm, the minimum lowered to 0 last). What they create is tagged, and turning off deletes only that. sagemaker-core's _serialize_shape now drops only None, as the dict and list serializers do, so a MinInstanceCount of 0 reaches the API; the model customization deploy path accepts it. Unit and integration tests, migration.md and the API docs.
@lhnealreilly lhnealreilly changed the title fix: honor managed_instance_scaling on model customization deployments feature: add endpoint autoscaling to sagemaker.serve and fix managed instance scaling for fine-tuned deployments Nov 18, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR adds a new sagemaker.serve.autoscaling module (enable_autoscaling / disable_autoscaling / describe_autoscaling) that wraps Application Auto Scaling and CloudWatch for variant instance-count and inference-component copy-count target tracking, including scale-to-zero. It also fixes _serialize_shape in sagemaker-core to preserve falsy values (so MinInstanceCount: 0 reaches the API), and wires managed_instance_scaling into the model-customization deploy path. The implementation is careful (tag-scoped teardown, min-of-1-until-alarm ordering, eventual-consistency read-back) and comes with extensive unit and integration tests. Overall it looks solid; my comments are about blast radius of the serialize change and one remaining deploy path that still ignores managed_instance_scaling.

Reviewed commit 51d2864. Automated review; verify before acting.

# Drop only Unassigned/None; preserve valid falsy values like False, 0, "".
for k, v in vars(value).items():
if serialize_result := serialize(v):
if (serialize_result := serialize(v)) is not None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Medium: This change is correct and brings _serialize_shape in line with _serialize_dict/_serialize_list (which already keep falsy values), but note it is a global behavior change across every shape serialized by sagemaker-core, not just ProductionVariantManagedInstanceScaling. Previously any shape field set to "", 0, or False was silently dropped; now it is emitted to the API. For numeric/boolean fields this is the desired fix, but for optional string fields that some callers set to "" (rather than leaving Unassigned) this now sends an empty string where the API previously saw the field absent, which some operations reject. The added regression tests cover the managed-instance-scaling case and an empty-string S3 URI, but consider whether any existing resource/shape relies on the old falsy-drop behavior for empty strings before release.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed it is global. I checked the blast radius: an AST scan of every sagemaker-*/src module for calls to a sagemaker.core.shapes class with a "", 0 or False literal keyword finds none, so the SDK itself never relied on the old drop. What changes is that a value a caller sets explicitly now reaches the API as given, which is what _serialize_dict and _serialize_list already do (test_serialize_preserves_falsy_dict_values). The full sagemaker-core and sagemaker-serve unit suites show no new failures against master, and the integration test creates a MinInstanceCount: 0 endpoint configuration through EndpointConfig.create and reads it back as 0.

@@ -6586,14 +6605,32 @@ def _deploy_model_customization(
# Check if endpoint exists

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Low: The managed_instance_scaling wiring here covers the inference-component / LoRA customization path, but the restricted-model-package branch earlier in this method (the EndpointConfig.create around line 6504, taken for restricted packages deployed model-on-variant) still builds its production variant from name/instance type/count only and ignores managed_instance_scaling. If a user passes managed_instance_scaling for a restricted package deployment it is silently dropped with no warning. Worth either handling it there too or documenting the gap.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same fix as the comment above: both model-on-variant branches warn now (1006cdc).

min_instance_count=managed_instance_scaling.get(
"MinInstanceCount", initial_instance_count
),
max_instance_count=managed_instance_scaling.get("MaxInstanceCount"),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Low: _model_customization_managed_instance_scaling always sets max_instance_count=managed_instance_scaling.get("MaxInstanceCount"), which is None when the caller omits MaxInstanceCount. The non-customization deploy path (around line 4150) instead only sets MaxInstanceCount when present. With the serialize fix a None is dropped so the payloads end up equivalent, but managed instance scaling requires a max; consider validating/defaulting MaxInstanceCount (as you default MinInstanceCount) so an omitted max fails with a clear SDK-side error rather than a server-side rejection.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1006cdc: _model_customization_managed_instance_scaling raises ValueError when MaxInstanceCount is missing, not an integer, below 1 or below MinInstanceCount, before anything is created. Unit test added.

…gets for sagemaker.serve.autoscaling

TargetTrackingPolicy on predefined metrics, CustomMetric or metric math; StepScalingPolicy with the CloudWatch alarm the module creates; ScheduledAction; suspend_autoscaling and resume_autoscaling; serverless provisioned concurrency and asynchronous endpoints (backlog metric, scale to zero on HasBacklogWithoutCapacity). enable_autoscaling manages the set it is given and removes SDK-created entries no longer given; ownership is by the sagemaker-sdk- prefix and tags. Model customization deploys validate MaxInstanceCount and warn when managed_instance_scaling does not apply.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR adds a new sagemaker.serve.autoscaling module (declarative Application Auto Scaling/CloudWatch wrappers with target-tracking, step-scaling, scheduled actions and scale-to-zero), propagates managed_instance_scaling to the fine-tuned model-customization deploy path, and fixes _serialize_shape in sagemaker-core to keep falsy values (0, False, "") so MinInstanceCount: 0 reaches the API. The core serialization fix is correct and consistent with the existing dict/list serializers, and is well covered by both unit and integration tests. The code is thorough and the test suite is extensive. Only minor issues found.

Note on scope: the _serialize_shape change is broad — it now preserves every falsy serialized shape field (and empty nested shapes serialize to {} rather than being dropped) across all of sagemaker-core, not just managed instance scaling. This matches the dict/list serializers and the new test_serialize_drops_unset_shape_fields confirms Unassigned is still dropped, so it looks intentional and safe, but it is worth being aware of for any API that relied on empty sub-objects being omitted.

Findings not attached to a changed line:

  • sagemaker-serve/tests/unit/test_model_customization_managed_instance_scaling.py:3145 🟡 Low: @_patches is applied twice to test_a_nova_model_on_the_variant_warns_that_it_is_ignored, so the test method is wrapped with 28 patch decorators (every target patched twice) and receives 28 mocks via *mocks. This is almost certainly unintentional. Because unittest.mock injects patched mocks bottom-up, the doubling also makes mocks[0] fragile — it only coincidentally refers to _is_nova_model. Remove the duplicate decorator.

Reviewed commit 1006cdc. Automated review; verify before acting.

_serialize_shape keeps falsy scalars (0, False, "") but still drops empty dicts and lists, so a nested shape with nothing set, such as an empty StoppingCondition, is omitted as before.
@lhnealreilly

lhnealreilly commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

On the _serialize_shape scope note from the review of 1006cdc: the empty-nested-shape point was right. Before this PR a nested shape with nothing set serialized to {} and was dropped as falsy; with the broader change it would have been sent, and tuner.py builds an empty StoppingCondition() when the trainer has no stopping condition. 52ef7a5 narrows the fix: falsy scalars (0, False, "") are kept, while None/Unassigned and empty dicts and lists are still dropped, so empty sub-objects stay omitted as before. New unit test test_serialize_still_omits_empty_nested_shapes_and_lists; full sagemaker-core and sagemaker-serve unit suites show no new failures against master, and the inference-component integration test (a MinInstanceCount: 0 endpoint configuration through EndpointConfig.create) passes against SageMaker.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR adds a new sagemaker.serve.autoscaling module (target-tracking/step/scheduled Application Auto Scaling for variants, inference components and serverless provisioned concurrency), fixes sagemaker-core's _serialize_shape so falsy scalars like MinInstanceCount: 0 reach the API, and extends the model-customization deploy path to honor managed_instance_scaling. The work is thorough and well tested (large unit + integration suites). The _serialize_shape fix is correct and consistent with the existing _serialize_dict/_serialize_list behavior. I found only minor issues: a duplicated test decorator, a potentially misleading error path, and missing pagination in _read that could matter for the declarative delete logic.

Reviewed commit 52ef7a5. Automated review; verify before acting.

def test_a_nova_model_on_the_variant_warns_that_it_is_ignored(self, *mocks):
mocks[0].return_value = True # _is_nova_model
with patch.object(ModelBuilder, "_deploy_nova_model") as deploy_nova, patch(
"sagemaker.serve.model_builder.logger"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Low: test_a_nova_model_on_the_variant_warns_that_it_is_ignored is decorated with @_patches twice. Since _patches stacks 14 patch() decorators, applying it twice patches the same 14 targets a second time and injects 28 mock arguments. It happens to still work because both applications start with _is_nova_model (so mocks[0] is correct), but the duplicate is almost certainly accidental and confusing — drop one @_patches.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the duplicate @_patches in 31183f9.

and not others
)
if deregister:
# Deregistering deletes the target's policies and scheduled actions too.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Low: If a caller passes only metric=... (no target_value, no policies), neither branch here builds a policy, and execution falls through to the raise ValueError("Pass target_value, policies or scheduled_actions.") below — a message that does not hint that metric was given but silently ignored because target_value was omitted. Consider raising a clearer error when metric is supplied without target_value.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 31183f9: metric without target_value (and without policies) now raises ValueError("metric needs a target_value (or pass policies)."). Unit test added.

return {
"PolicyName": f"{NAME_PREFIX}target-tracking{suffix}",
"PolicyType": "TargetTrackingScaling",
"TargetTrackingScalingPolicyConfiguration": configuration,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Low: _read calls describe_scaling_policies / describe_scheduled_actions / describe_scalable_targets without handling pagination. These APIs return at most MaxResults (50) per page with a NextToken. On a target that also has many non-SDK policies/actions, a managed policy on a later page would be missed by the declarative delete in enable_autoscaling (_managed(_policy_names(current)) - wanted) and never cleaned up. Paginating these reads would make the declarative behavior robust.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 31183f9: _read pages through describe_scalable_targets, describe_scaling_policies and describe_scheduled_actions with the botocore paginators. A unit test with one entry per page checks that SDK-created policies and schedules behind others are still found and removed, and the integration tests pass with the paginated reads.

…hout a target

_read collects every page of describe_scalable_targets, describe_scaling_policies and describe_scheduled_actions, so SDK-created entries behind many others are still found. enable_autoscaling(metric=...) without target_value raises a clear error. Drop a duplicate test decorator and a misplaced noqa.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR adds a new sagemaker.serve.autoscaling module (declarative enable_autoscaling/disable_autoscaling/suspend/resume/describe over Application Auto Scaling + CloudWatch), fixes the model-customization deploy path to honor managed_instance_scaling, and fixes _serialize_shape in sagemaker-core to preserve falsy scalar fields (so MinInstanceCount: 0 reaches the API). The implementation is careful (ownership tagging, declarative diffing, scale-to-zero ordering, eventual-consistency read-back) and is backed by thorough unit and integration tests. The _serialize_shape change is consistent with the already-updated _serialize_dict/_serialize_list. I found no high-severity issues; the change looks clean. A couple of low-severity notes below.

Reviewed commit 31183f9. Automated review; verify before acting.

# still omitted from the request.
for k, v in vars(value).items():
if serialize_result := serialize(v):
serialize_result = serialize(v)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Low: This change makes _serialize_shape emit any field whose serialized value is a non-None, non-empty-dict, non-empty-list value, including 0, False and "". That is the intended fix (and is now consistent with _serialize_dict/_serialize_list), but it is a behavior-widening change for all shapes, not just managed instance scaling: fields previously set to an empty string or 0 and silently dropped will now be sent to the API. Worth confirming no existing callers relied on the old drop-on-falsy behavior (e.g. default "" fields on other shapes) — the added regression tests only cover a few shapes.



def _policy_names(state) -> set:
return {p["PolicyName"] for p in state["policies"]}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Low: _wait_for uses a fixed 6 attempts × 5s = up to 30s of blocking time.sleep when wait=True (the default). Application Auto Scaling writes can occasionally take longer to read back, in which case the function logs a warning and returns a stale state that may not reflect the requested change, with no way for the caller to distinguish "settled" from "gave up". Consider surfacing the not-settled outcome more explicitly (e.g. in the returned dict) so callers can react rather than silently proceeding on stale state.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR adds a new public module sagemaker.serve.autoscaling (Application Auto Scaling for endpoints, inference components and serverless provisioned concurrency), wires managed_instance_scaling into the model-customization deploy path, and fixes _serialize_shape in sagemaker-core so falsy scalar shape fields (e.g. MinInstanceCount: 0) survive serialization. The code is well-structured, carefully documented, and backed by extensive unit tests plus integration tests. Overall the change looks correct and well-tested; the _serialize_shape fix is consistent with the already-updated _serialize_dict/_serialize_list logic. I only have minor observations, noted below.

Reviewed commit 5584393. Automated review; verify before acting.

Takes the same ``{"MinInstanceCount": ..., "MaxInstanceCount": ...}`` dict as
``deploy(managed_instance_scaling=...)`` on the other deploy paths; ``MinInstanceCount``
defaults to the initial instance count.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Low: _model_customization_managed_instance_scaling requires MaxInstanceCount and raises ValueError when it is missing, whereas the existing _deploy_core_endpoint path (around line 4147-4159 in this file) treats MaxInstanceCount as optional and simply omits it when not provided. The two deploy paths now accept the same managed_instance_scaling={...} dict but validate it differently. Requiring MaxInstanceCount is arguably the more correct behavior (the API needs it for enabled managed scaling), but the divergence is worth reconciling so the same input behaves the same on both paths.

This branch was previously deployed

1 inactive deployment
auto-approve — 55843930 Deployed Nov 18, 2026 by lhnealreilly via wait-for-approval #897
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants