Repository navigation
feature: add endpoint autoscaling to sagemaker.serve and fix managed instance scaling for fine-tuned deployments - #6591
Conversation
deploy(managed_instance_scaling=...) set managed instance scaling on the endpoint configuration of a regular deployment, but _deploy_model_customization built the production variant without it, so a fine-tuned model's inference components could not add instances. Set it on the new endpoint configuration with the same dict and default as the regular path, warn when the endpoint already exists, and refuse MinInstanceCount 0, which sagemaker-core's shape serialization drops.
There was a problem hiding this comment.
🤖 AI code review
The PR plumbs managed_instance_scaling through the model-customization (fine-tuned) deploy path (_deploy_model_customization), building a ProductionVariantManagedInstanceScaling on the new endpoint config, defaulting MinInstanceCount to the instance count, warning when an endpoint already exists, and rejecting MinInstanceCount: 0 with a clear error. The change is small, consistent with the existing regular deploy path, and comes with focused unit tests. I found no correctness or security issues; only a minor scope observation below.
Reviewed commit c2eb5d9. Automated review; verify before acting.
| @@ -6586,14 +6614,32 @@ def _deploy_model_customization( | |||
| # Check if endpoint exists | |||
There was a problem hiding this comment.
🟡 Low: There is another EndpointConfig.create in this same method on the restricted-model-package / model-on-variant branch (base line ~6504, not shown in this diff) that still ignores managed_instance_scaling. A user passing managed_instance_scaling for a restricted-package fine-tuned model would silently get an endpoint config without it, with no warning. If that path is in scope for the same argument, consider applying the setting there too (or emitting the same ignored-warning); if it's intentionally out of scope, a brief note would help future readers.
There was a problem hiding this comment.
Handled in 1006cdc. A restricted model package (and a Nova model without inference-component resources) is deployed on the endpoint's variant, where managed instance scaling does not apply, so that branch now logs that managed_instance_scaling is ignored instead of dropping it silently. Unit tests cover both branches.
Add enable_autoscaling, disable_autoscaling and describe_autoscaling: target-tracking autoscaling of a production variant's instance count or an inference component's copy count, with scale to zero for inference components (a step-scaling policy and a NoCapacityInvocationFailures alarm, the minimum lowered to 0 last). What they create is tagged, and turning off deletes only that. sagemaker-core's _serialize_shape now drops only None, as the dict and list serializers do, so a MinInstanceCount of 0 reaches the API; the model customization deploy path accepts it. Unit and integration tests, migration.md and the API docs.
There was a problem hiding this comment.
🤖 AI code review
This PR adds a new sagemaker.serve.autoscaling module (enable_autoscaling / disable_autoscaling / describe_autoscaling) that wraps Application Auto Scaling and CloudWatch for variant instance-count and inference-component copy-count target tracking, including scale-to-zero. It also fixes _serialize_shape in sagemaker-core to preserve falsy values (so MinInstanceCount: 0 reaches the API), and wires managed_instance_scaling into the model-customization deploy path. The implementation is careful (tag-scoped teardown, min-of-1-until-alarm ordering, eventual-consistency read-back) and comes with extensive unit and integration tests. Overall it looks solid; my comments are about blast radius of the serialize change and one remaining deploy path that still ignores managed_instance_scaling.
Reviewed commit 51d2864. Automated review; verify before acting.
| # Drop only Unassigned/None; preserve valid falsy values like False, 0, "". | ||
| for k, v in vars(value).items(): | ||
| if serialize_result := serialize(v): | ||
| if (serialize_result := serialize(v)) is not None: |
There was a problem hiding this comment.
🟠 Medium: This change is correct and brings _serialize_shape in line with _serialize_dict/_serialize_list (which already keep falsy values), but note it is a global behavior change across every shape serialized by sagemaker-core, not just ProductionVariantManagedInstanceScaling. Previously any shape field set to "", 0, or False was silently dropped; now it is emitted to the API. For numeric/boolean fields this is the desired fix, but for optional string fields that some callers set to "" (rather than leaving Unassigned) this now sends an empty string where the API previously saw the field absent, which some operations reject. The added regression tests cover the managed-instance-scaling case and an empty-string S3 URI, but consider whether any existing resource/shape relies on the old falsy-drop behavior for empty strings before release.
There was a problem hiding this comment.
Agreed it is global. I checked the blast radius: an AST scan of every sagemaker-*/src module for calls to a sagemaker.core.shapes class with a "", 0 or False literal keyword finds none, so the SDK itself never relied on the old drop. What changes is that a value a caller sets explicitly now reaches the API as given, which is what _serialize_dict and _serialize_list already do (test_serialize_preserves_falsy_dict_values). The full sagemaker-core and sagemaker-serve unit suites show no new failures against master, and the integration test creates a MinInstanceCount: 0 endpoint configuration through EndpointConfig.create and reads it back as 0.
| @@ -6586,14 +6605,32 @@ def _deploy_model_customization( | |||
| # Check if endpoint exists | |||
There was a problem hiding this comment.
🟡 Low: The managed_instance_scaling wiring here covers the inference-component / LoRA customization path, but the restricted-model-package branch earlier in this method (the EndpointConfig.create around line 6504, taken for restricted packages deployed model-on-variant) still builds its production variant from name/instance type/count only and ignores managed_instance_scaling. If a user passes managed_instance_scaling for a restricted package deployment it is silently dropped with no warning. Worth either handling it there too or documenting the gap.
There was a problem hiding this comment.
Same fix as the comment above: both model-on-variant branches warn now (1006cdc).
| min_instance_count=managed_instance_scaling.get( | ||
| "MinInstanceCount", initial_instance_count | ||
| ), | ||
| max_instance_count=managed_instance_scaling.get("MaxInstanceCount"), |
There was a problem hiding this comment.
🟡 Low: _model_customization_managed_instance_scaling always sets max_instance_count=managed_instance_scaling.get("MaxInstanceCount"), which is None when the caller omits MaxInstanceCount. The non-customization deploy path (around line 4150) instead only sets MaxInstanceCount when present. With the serialize fix a None is dropped so the payloads end up equivalent, but managed instance scaling requires a max; consider validating/defaulting MaxInstanceCount (as you default MinInstanceCount) so an omitted max fails with a clear SDK-side error rather than a server-side rejection.
There was a problem hiding this comment.
Done in 1006cdc: _model_customization_managed_instance_scaling raises ValueError when MaxInstanceCount is missing, not an integer, below 1 or below MinInstanceCount, before anything is created. Unit test added.
…gets for sagemaker.serve.autoscaling TargetTrackingPolicy on predefined metrics, CustomMetric or metric math; StepScalingPolicy with the CloudWatch alarm the module creates; ScheduledAction; suspend_autoscaling and resume_autoscaling; serverless provisioned concurrency and asynchronous endpoints (backlog metric, scale to zero on HasBacklogWithoutCapacity). enable_autoscaling manages the set it is given and removes SDK-created entries no longer given; ownership is by the sagemaker-sdk- prefix and tags. Model customization deploys validate MaxInstanceCount and warn when managed_instance_scaling does not apply.
There was a problem hiding this comment.
🤖 AI code review
This PR adds a new sagemaker.serve.autoscaling module (declarative Application Auto Scaling/CloudWatch wrappers with target-tracking, step-scaling, scheduled actions and scale-to-zero), propagates managed_instance_scaling to the fine-tuned model-customization deploy path, and fixes _serialize_shape in sagemaker-core to keep falsy values (0, False, "") so MinInstanceCount: 0 reaches the API. The core serialization fix is correct and consistent with the existing dict/list serializers, and is well covered by both unit and integration tests. The code is thorough and the test suite is extensive. Only minor issues found.
Note on scope: the _serialize_shape change is broad — it now preserves every falsy serialized shape field (and empty nested shapes serialize to {} rather than being dropped) across all of sagemaker-core, not just managed instance scaling. This matches the dict/list serializers and the new test_serialize_drops_unset_shape_fields confirms Unassigned is still dropped, so it looks intentional and safe, but it is worth being aware of for any API that relied on empty sub-objects being omitted.
Findings not attached to a changed line:
sagemaker-serve/tests/unit/test_model_customization_managed_instance_scaling.py:3145🟡 Low:@_patchesis applied twice totest_a_nova_model_on_the_variant_warns_that_it_is_ignored, so the test method is wrapped with 28 patch decorators (every target patched twice) and receives 28 mocks via*mocks. This is almost certainly unintentional. Becauseunittest.mockinjects patched mocks bottom-up, the doubling also makesmocks[0]fragile — it only coincidentally refers to_is_nova_model. Remove the duplicate decorator.
Reviewed commit 1006cdc. Automated review; verify before acting.
_serialize_shape keeps falsy scalars (0, False, "") but still drops empty dicts and lists, so a nested shape with nothing set, such as an empty StoppingCondition, is omitted as before.
|
On the |
There was a problem hiding this comment.
🤖 AI code review
This PR adds a new sagemaker.serve.autoscaling module (target-tracking/step/scheduled Application Auto Scaling for variants, inference components and serverless provisioned concurrency), fixes sagemaker-core's _serialize_shape so falsy scalars like MinInstanceCount: 0 reach the API, and extends the model-customization deploy path to honor managed_instance_scaling. The work is thorough and well tested (large unit + integration suites). The _serialize_shape fix is correct and consistent with the existing _serialize_dict/_serialize_list behavior. I found only minor issues: a duplicated test decorator, a potentially misleading error path, and missing pagination in _read that could matter for the declarative delete logic.
Reviewed commit 52ef7a5. Automated review; verify before acting.
| def test_a_nova_model_on_the_variant_warns_that_it_is_ignored(self, *mocks): | ||
| mocks[0].return_value = True # _is_nova_model | ||
| with patch.object(ModelBuilder, "_deploy_nova_model") as deploy_nova, patch( | ||
| "sagemaker.serve.model_builder.logger" |
There was a problem hiding this comment.
🟡 Low: test_a_nova_model_on_the_variant_warns_that_it_is_ignored is decorated with @_patches twice. Since _patches stacks 14 patch() decorators, applying it twice patches the same 14 targets a second time and injects 28 mock arguments. It happens to still work because both applications start with _is_nova_model (so mocks[0] is correct), but the duplicate is almost certainly accidental and confusing — drop one @_patches.
There was a problem hiding this comment.
Removed the duplicate @_patches in 31183f9.
| and not others | ||
| ) | ||
| if deregister: | ||
| # Deregistering deletes the target's policies and scheduled actions too. |
There was a problem hiding this comment.
🟡 Low: If a caller passes only metric=... (no target_value, no policies), neither branch here builds a policy, and execution falls through to the raise ValueError("Pass target_value, policies or scheduled_actions.") below — a message that does not hint that metric was given but silently ignored because target_value was omitted. Consider raising a clearer error when metric is supplied without target_value.
There was a problem hiding this comment.
Done in 31183f9: metric without target_value (and without policies) now raises ValueError("metric needs a target_value (or pass policies)."). Unit test added.
| return { | ||
| "PolicyName": f"{NAME_PREFIX}target-tracking{suffix}", | ||
| "PolicyType": "TargetTrackingScaling", | ||
| "TargetTrackingScalingPolicyConfiguration": configuration, |
There was a problem hiding this comment.
🟡 Low: _read calls describe_scaling_policies / describe_scheduled_actions / describe_scalable_targets without handling pagination. These APIs return at most MaxResults (50) per page with a NextToken. On a target that also has many non-SDK policies/actions, a managed policy on a later page would be missed by the declarative delete in enable_autoscaling (_managed(_policy_names(current)) - wanted) and never cleaned up. Paginating these reads would make the declarative behavior robust.
There was a problem hiding this comment.
Done in 31183f9: _read pages through describe_scalable_targets, describe_scaling_policies and describe_scheduled_actions with the botocore paginators. A unit test with one entry per page checks that SDK-created policies and schedules behind others are still found and removed, and the integration tests pass with the paginated reads.
…hout a target _read collects every page of describe_scalable_targets, describe_scaling_policies and describe_scheduled_actions, so SDK-created entries behind many others are still found. enable_autoscaling(metric=...) without target_value raises a clear error. Drop a duplicate test decorator and a misplaced noqa.
There was a problem hiding this comment.
🤖 AI code review
This PR adds a new sagemaker.serve.autoscaling module (declarative enable_autoscaling/disable_autoscaling/suspend/resume/describe over Application Auto Scaling + CloudWatch), fixes the model-customization deploy path to honor managed_instance_scaling, and fixes _serialize_shape in sagemaker-core to preserve falsy scalar fields (so MinInstanceCount: 0 reaches the API). The implementation is careful (ownership tagging, declarative diffing, scale-to-zero ordering, eventual-consistency read-back) and is backed by thorough unit and integration tests. The _serialize_shape change is consistent with the already-updated _serialize_dict/_serialize_list. I found no high-severity issues; the change looks clean. A couple of low-severity notes below.
Reviewed commit 31183f9. Automated review; verify before acting.
| # still omitted from the request. | ||
| for k, v in vars(value).items(): | ||
| if serialize_result := serialize(v): | ||
| serialize_result = serialize(v) |
There was a problem hiding this comment.
🟡 Low: This change makes _serialize_shape emit any field whose serialized value is a non-None, non-empty-dict, non-empty-list value, including 0, False and "". That is the intended fix (and is now consistent with _serialize_dict/_serialize_list), but it is a behavior-widening change for all shapes, not just managed instance scaling: fields previously set to an empty string or 0 and silently dropped will now be sent to the API. Worth confirming no existing callers relied on the old drop-on-falsy behavior (e.g. default "" fields on other shapes) — the added regression tests only cover a few shapes.
|
|
||
|
|
||
| def _policy_names(state) -> set: | ||
| return {p["PolicyName"] for p in state["policies"]} |
There was a problem hiding this comment.
🟡 Low: _wait_for uses a fixed 6 attempts × 5s = up to 30s of blocking time.sleep when wait=True (the default). Application Auto Scaling writes can occasionally take longer to read back, in which case the function logs a warning and returns a stale state that may not reflect the requested change, with no way for the caller to distinguish "settled" from "gave up". Consider surfacing the not-settled outcome more explicitly (e.g. in the returned dict) so callers can react rather than silently proceeding on stale state.
There was a problem hiding this comment.
🤖 AI code review
This PR adds a new public module sagemaker.serve.autoscaling (Application Auto Scaling for endpoints, inference components and serverless provisioned concurrency), wires managed_instance_scaling into the model-customization deploy path, and fixes _serialize_shape in sagemaker-core so falsy scalar shape fields (e.g. MinInstanceCount: 0) survive serialization. The code is well-structured, carefully documented, and backed by extensive unit tests plus integration tests. Overall the change looks correct and well-tested; the _serialize_shape fix is consistent with the already-updated _serialize_dict/_serialize_list logic. I only have minor observations, noted below.
Reviewed commit 5584393. Automated review; verify before acting.
| Takes the same ``{"MinInstanceCount": ..., "MaxInstanceCount": ...}`` dict as | ||
| ``deploy(managed_instance_scaling=...)`` on the other deploy paths; ``MinInstanceCount`` | ||
| defaults to the initial instance count. | ||
|
|
There was a problem hiding this comment.
🟡 Low: _model_customization_managed_instance_scaling requires MaxInstanceCount and raises ValueError when it is missing, whereas the existing _deploy_core_endpoint path (around line 4147-4159 in this file) treats MaxInstanceCount as optional and simply omits it when not provided. The two deploy paths now accept the same managed_instance_scaling={...} dict but validate it differently. Requiring MaxInstanceCount is arguably the more correct behavior (the API needs it for enabled managed scaling), but the divergence is worth reconciling so the same input behaves the same on both paths.
Issue
Autoscaling a SageMaker endpoint from the SDK has three gaps:
register_scalable_target/put_scaling_policy/put_metric_alarmcalls with boto3 (model.deploy to allow for auto scale configuration #1880).migration.mdlists endpoint auto-scaling as "Via ModelBuilder.deploy", butdeploy(managed_instance_scaling=...)only sets an inference-component endpoint's instance bounds: nothing changes a variant's instance count, a component's copy count or a serverless endpoint's provisioned concurrency with traffic.managed_instance_scaling.deploy()sets it on the endpoint configuration of a regular deployment (_deploy_core_endpoint), but a fine-tuned deployment goes through_deploy_model_customization, which builds the production variant from name, instance type and count only. Its inference components then cannot add instances, and managed instance scaling can only be added later with a new endpoint configuration.sagemaker-coredrops falsy shape fields._serialize_shapekeeps a field only when its serialized value is truthy (if serialize_result := serialize(v)), so0,Falseand""set on a shape never reach the API. For managed instance scaling,MinInstanceCount: 0, the setting scale to zero requires, is dropped andCreateEndpointConfigfails with "Instance count needs to be specified for enabled managed instance scaling".Change
sagemaker.serve.autoscaling(new)What scales follows the endpoint: a real-time or asynchronous endpoint's production variant (
sagemaker:variant:DesiredInstanceCount), an inference component (sagemaker:inference-component:DesiredCopyCount), or a serverless endpoint's provisioned concurrency (sagemaker:variant:DesiredProvisionedConcurrency).concurrent-requests, the default for a real-time endpoint;invocations;provisioned-concurrency-utilizationfor a serverless endpoint) or full type, on any CloudWatch metric throughCustomMetric(dimensions default to the scaled resource's own), or on metric math by passing aCustomizedMetricSpecificationdict. An asynchronous endpoint defaults toApproximateBacklogSizePerInstance, as Autoscale an asynchronous endpoint recommends.at,rate,cron) with time zone and start and end times.NoCapacityInvocationFailuresalarm; the minimum is registered as 1 until the alarm exists and lowered to 0 last, so a failed alarm write cannot leave a component that scales in to 0 and never back out), and for asynchronous endpoints (a step policy onHasBacklogWithoutCapacity).enable_autoscalingmanages the set it is given: calling it again changes those policies and schedules and removes the ones no longer given. Everything it creates is namedsagemaker-sdk-..., and the scalable target and alarms it creates are tagged;disable_autoscalingdeletes only those, deregisters the target only when this module registered it and nothing else is on it, and deletes an alarm only when it carries the tag. Policies, schedules, alarms and targets created any other way stay.InService, an on-demand serverless variant (SageMaker scales it itself), several variants with none named, variant scaling on an endpoint that hosts inference components, a component on an endpoint without managed instance scaling (copies could only fill the instances the endpoint has), an adapter component (it names the base component whose copies it runs on),invocationson a base component with adapters, scale to zero on a real-time variant or provisioned concurrency, scale to zero on a component without a managed-instance-scaling minimum of 0 or with adapters, and invalid bounds, metrics, steps or schedules.describe_scalable_targets,describe_scaling_policiesanddescribe_scheduled_actions, and Application Auto Scaling is eventually consistent, sowait=True(the default) reads the change back before returning.Predictive scaling is not included:
PutScalingPolicywithPolicyType=PredictiveScalingon a SageMaker variant returnsAccessDeniedException: Account is not authorized.Model customization deployments
_deploy_model_customizationreadsmanaged_instance_scalingfrom the deploy kwargs, with the same dict and default as the regular path (MinInstanceCountfalls back to the initial instance count), validatesMaxInstanceCount, and sets it on the new endpoint configuration. When the endpoint already exists, or the model is deployed on the endpoint's variant (a restricted model package, or a Nova model without inference-component resources), the setting does not apply and is logged as ignored.sagemaker-coreshape serialization_serialize_shapenow keeps falsy scalars (0,False,"") and still dropsNone/Unassignedand empty dicts and lists, so a nested shape with nothing set (such as the emptyStoppingConditionthe tuner builds when no stopping condition is given) is still omitted from the request. No SDK code constructs a shape with a"",0orFalseliteral field, so the change affects only values callers set explicitly, which now reach the API as given.Docs
migration.mdmaps hand-rolled boto3 autoscaling to the new module, anddocs/api/sagemaker_serve.rstdocuments it.Testing
Unit:
sagemaker-serve/tests/unit/test_autoscaling.py: 71 tests, 100% line coverage ofautoscaling.py. A stateful fake of the SageMaker, Application Auto Scaling and CloudWatch clients covers each target kind and policy type, custom metrics and metric math, step alarms, scheduled actions, the declarative removal of entries no longer given (and the ones left alone, including entries on later pages of the describe calls), both scale-to-zero paths and the inference-component write order, a failed alarm write, every refusal (asserting no write), suspend and resume, turning off with and without ownership, and the read-back retry.sagemaker-serve/tests/unit/test_model_customization_managed_instance_scaling.py: 9 tests: the setting reaches the new endpoint configuration, the minimum default, a missing or too-small maximum, nothing set unless requested, an existing endpoint, a restricted package and a Nova model left alone with a warning, and a minimum of 0 serialized as 0.sagemaker-core/tests/unit/generated/test_utils.py: falsy scalar fields are kept; unset fields, empty nested shapes and empty dicts are omitted.masterin the same environment:sagemaker-core4153 passed (master: 4150) andsagemaker-serve2337 passed (master: 2262); the 9 and 59 failures are the same tests on both, which reach AWS endpoints and fail without credentials.black,flake8,pylint(10.00 on the new module) andpydocstyle.Integration (
sagemaker-serve/tests/integ/test_autoscaling_integration.py, new,slow_test). Each test creates its own CPU endpoint from the public hub'spytorch-ic-mobilenet-v2and deletes everything afterwards:CPUUtilization), a step policy with its alarm and a scheduled action; suspend and resume; a second call with one policy removes the rest and the alarm; turning off deregisters.MinInstanceCount: 0created throughEndpointConfig.create(it reads back as 0); scale to zero (both policies, the alarm, minimum 0); back to a minimum of 1 (step policy and alarm removed); turning off.HasBacklogWithoutCapacitypolicy and alarm; turning off removes the alarm.All four pass against SageMaker in us-west-2 and leave no endpoint, configuration, model, component, alarm, scalable target or S3 object behind.
test_model_customization_deployment.py::test_deploy_from_training_jobnow also deploys withmanaged_instance_scaling={"MinInstanceCount": 1, "MaxInstanceCount": 2}and checks the endpoint configuration it creates.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.