Skip to content

Merge latest changes from main to 'Documentation' branch - #192

Open
rsareddy0329 wants to merge 256 commits into
documentationfrom
main
Open

rsareddy0329 wants to merge 256 commits into
documentationfrom
main

Conversation

@rsareddy0329

Copy link
Copy Markdown
Collaborator

PR Approval Steps

For Requester

  1. Description
    • Check the PR title and description for clarity. It should describe the changes made and the reason behind them.
    • Ensure that the PR follows the contribution guidelines, if applicable.
  2. Security requirements
    • Ensure that a Pull Request (PR) does not expose passwords and other sensitive information by using git-secrets and upload relevant evidence: https://github.com/awslabs/git-secrets
    • Ensure commit has GitHub Commit Signature
  3. Manual review
    1. Click on the Files changed tab to see the code changes. Review the changes thoroughly:
      • Code Quality: Check for coding standards, naming conventions, and readability.
      • Functionality: Ensure that the changes meet the requirements and that all necessary code paths are tested.
      • Security: Check for any security issues or vulnerabilities.
      • Documentation: Confirm that any necessary documentation (code comments, README updates, etc.) has been updated.
  4. Check for Merge Conflicts:
    • Verify if there are any merge conflicts with the base branch. GitHub will usually highlight this. If there are conflicts, you should resolve them.

For Reviewer

  1. Go through For Requester section to double check each item.
  2. Request Changes or Approve the PR:
    1. If the PR is ready to be merged, click Review changes and select Approve.
    2. If changes are required, select Request changes and provide feedback. Be constructive and clear in your feedback.
  3. Merging the PR
    1. Check the Merge Method:
      1. Decide on the appropriate merge method based on your repository's guidelines (e.g., Squash and merge, Rebase and merge, or Merge).
    2. Merge the PR:
      1. Click the Merge pull request button.
      2. Confirm the merge by clicking Confirm merge.

mollyheamazon and others added 19 commits July 22, 2025 14:06
* Add unique time string to integ test

* Update syntax
* Training CLI & SDK: example notebook and README update

* Update training cli example notebook

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
* Update inferenece SDK examples

* Update readme
* Enable Hyperpod telemetry

* Enable Hyperpod telemetry

* Enable Hyperpod telemetry

* Enable Hyperpod telemetry

* Enable Hyperpod telemetry

* Enable Hyperpod telemetry

* CLI: Enable Telemetry

* CLI: Enable Telemetry

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
* update help text to avoid truncation

* update volume flag to support hostPath and pvc, before e2e testing

* clean up and e2e working

* Minor updates after PR

* update

* Added unit tests for volume, all cli unit tests passed
Co-authored-by: pintaoz <pintaoz@amazon.com>
* Update inference config and integ tests

* Update integ tests for new canaries
* Manual release v3.0.2

* Update changelog

---------

Co-authored-by: pintaoz <pintaoz@amazon.com>
* Update readme for volume flag

* Add schema pattern check to pytorch-job template, unit test added, all test passed locally
…8s (#138)

* Add k8s version validation check between server and client version according to the supported versioning constraints by k8s

* Fix unit test cases

* Move regex to a constant.

**Description**
- Removed an integration test case that was being mocked.
- Moved a regex to a constant.

**Testing Done**
Unit test cases pass no changes made to integration test cases and they should not be affected.

* Add k8s version validation check between server and client version according to the supported versioning constraints by k8s

* Add ref link for version comptability contraints

**Description**
Added a link to k8s documentation mentioning the constraints that rule the version compatibility of client and server.

**Testing Done**
No breaking changes.
* Fix SDK training test: Add wait time before refresh

* Fix training tests in canaries
@rsareddy0329
rsareddy0329 requested a review from a team as a code owner August 5, 2025 23:05
rsareddy0329 and others added 10 commits August 6, 2025 13:51
* Update documentation-with-new-changes branch with latest changes from main (#190)

* Fix training test (#184)

* Fix SDK training test: Add wait time before refresh

* Fix training tests in canaries

* Update logging information for submitting and deleting training job (#189)

Co-authored-by: pintaoz <pintaoz@amazon.com>

---------

Co-authored-by: Zhaoqi <zhaoqiwang.baruch@gmail.com>
Co-authored-by: pintaoz-aws <167920275+pintaoz-aws@users.noreply.github.com>
Co-authored-by: pintaoz <pintaoz@amazon.com>

* Documentation Fixes (#191)

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* update documentation with new changes branch with latest changes (#194)

* Fix training test (#184)

* Fix SDK training test: Add wait time before refresh

* Fix training tests in canaries

* Update logging information for submitting and deleting training job (#189)

Co-authored-by: pintaoz <pintaoz@amazon.com>

---------

Co-authored-by: Zhaoqi <zhaoqiwang.baruch@gmail.com>
Co-authored-by: pintaoz-aws <167920275+pintaoz-aws@users.noreply.github.com>
Co-authored-by: pintaoz <pintaoz@amazon.com>

* Documentation Fixes (#195)

* Documentation Fixes

* Documentation Fixes

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* Documentation Fixes (#197)

* Documentation Fixes

* Documentation Fixes

* Documentation Fixes

* Documentation Fixes

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* Documentation Fixes (#198)

* Documentation Fixes

* Documentation Fixes

* Documentation Fixes

* Documentation Fixes

* Documentation Fixes

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* Documentation fixes (#199)

* Documentation Fixes

* Documentation Fixes

* Documentation Fixes

* Documentation Fixes

* Documentation Fixes

* Documentation Fixes

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

---------

Co-authored-by: Zhaoqi <zhaoqiwang.baruch@gmail.com>
Co-authored-by: pintaoz-aws <167920275+pintaoz-aws@users.noreply.github.com>
Co-authored-by: pintaoz <pintaoz@amazon.com>
Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
…s to view SDK config code (#188)

Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>
* Add instance type support for ml.p6e-gb200.36xlarge

Updated support for ml.p6-b200.48xlarge as well

* Add ml.p6e-gb200.36xlarge to efa plugin
…holder value (#206)

Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>
Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
* Add labels to the top level metadata (#158)

Co-authored-by: pintaoz <pintaoz@amazon.com>

* Implemented GPU Quota Allocation Feature.

Co-authored-by: aleszewi <aleszewi@amazon.com>

* Revert "Implemented GPU Quota Allocation Feature."

This reverts commit 790b8f1df59494a982463aaed9e5b3f2afa44123.

* Fix: Template issue - pick user defined template version (#154)

* Fix: Template issue - pick user defined template version

* Fix: Template issue - pick user defined template version & add topology labels in 1.1

* Fix: Template issue - pick user defined template version & add topology labels in 1.1

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* Fix: Add __init__ to the new schema (#163)

* Fix: Template issue - pick user defined template version

* Fix: Template issue - pick user defined template version & add topology labels in 1.1

* Fix: Template issue - pick user defined template version & add topology labels in 1.1

* Fix: Add __init__ to load the new schema

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* Add labels and annotations to top level metadata v1.1 (#165)

* Add labels to top level metadata v1.1

* Move topology labels to annotations

* Update topology parameter names

* Add unit test

---------

Co-authored-by: pintaoz <pintaoz@amazon.com>

* Added GPU quota allocation.

Co-authored-by: aleszewi <aleszewi@amazon.com>

* Changed neuron key to neurondevice. (#177)

Co-authored-by: Marta Aleszewicz <aleszewi@amazon.com>

* fix: Renamed memory-in-gib to memory for consistency. (#179)

cr: https://code.amazon.com/reviews/CR-214599587

Co-authored-by: Marta Aleszewicz <aleszewi@amazon.com>

* Add validation to topology labels (#178)

* Add validation to topology labels

* Add validation to topology labels

* Add validation to topology labels

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* Add integ tests for topology annotations (#180)

* Add labels to top level metadata v1.1

* Move topology labels to annotations

* Update topology parameter names

* Add unit test

* Topology integ tests

* Add invalid test case

* Add empty test case

---------

Co-authored-by: pintaoz <pintaoz@amazon.com>

* Add integration tests for gpu quota allocation feature (#184)

* add integration tests for gpu quota allocation feature

* add valueError assertions for invalid test cases

* Updating the CHANGELOG and minor version

---------

Co-authored-by: pintaoz-aws <167920275+pintaoz-aws@users.noreply.github.com>
Co-authored-by: pintaoz <pintaoz@amazon.com>
Co-authored-by: Marta Aleszewicz <aleszewi@amazon.com>
Co-authored-by: rsareddy0329 <rsareddy0329@gmail.com>
Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
Co-authored-by: mx26pol <martacalif@gmail.com>
Co-authored-by: satish Kumar <satishgollaprolu@gmail.com>
…gs to hyp-jumpstart-endpoint (#213)

* Update generate_click_command inject logic to not expose unwanted flags to hyp-jumpstart-endpoint

* Update unit tests for bug fix, change --label_selector to --label-selector
* Update generate_click_command inject logic to not expose unwanted flags to hyp-jumpstart-endpoint

* Update unit tests for bug fix, change --label_selector to --label-selector

* Update README, example notebooks and documentation to 1)remove model_version, 2)add --model-volume-mount-name 3)remove tar.gz from --model-location 4)update unique mount_path for --volume

* Update README, example notebooks and documentation to remove tls-config for jumpstart

* minor update to remove tar.gz from --model-location for documentation
#219)

* add metadata_name argument to js and custom endpoint to match with SDK

* fix integ
rsareddy0329 and others added 30 commits August 10, 2026 16:34
* Bug fix in Hyperpod recipe rendering

* Bug fix in Hyperpod recipe rendering

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
* docs: add v3.10.0 changelog

* chore: bump project version to 3.10.0
…monSets (#451)

Under node DiskPressure, HMA was evicted moments before a GPU failure went
undetected and unremediated (P438078245). Setting priorityClassName to the
built-in system-node-critical makes HMA the last pod evicted (after training
and other monitoring pods), preserving health monitoring during node pressure.
Mirrors what EKSNodeMonitoringAgent already does.

Applied to both the health-monitoring-agent (NVIDIA) and
health-monitoring-agent-non-nvidia (Trainium/Inferentia) DaemonSets.
Configurable via .Values.priorityClassName, defaulting to system-node-critical.
* fix: update instance type to g6e for deepseek integ tests

* fix: use smaller model so tests run on g5.8xlarge

---------

Co-authored-by: Syed Jafri <syedjfr@amazon.com>
Bumps the inference-operator Helm chart to 2.5.0 (operator v3.5), updates the
umbrella chart dependency pin, and adds the weights-cache node affinity field
to the inference-operator CRDs.
* fix: conditional aws-hyperpod namespace creation

* fix: deep health check namespace reference

* chore: allow conditional creation of mpi-operator chart

* chore: address review changes
* Add v3.11.0 changelog and version bump

* Release 3.11 - update setup.py

---------

Co-authored-by: Syed Jafri <syedjfr@amazon.com>
- Chart 2.5.0 -> 2.6.0, appVersion 3.5 -> 3.6, image tag v3.5 -> v3.6
- Regenerated IEC/JSM CRDs: add spec/status.inferenceGateway opt-in
  (CR-295860367) and IEC spec.kubernetes.podTemplateAnnotations
  (CR-299953131). Additive, backward compatible.

Co-authored-by: Xuan Lu <xua@amazon.com>
* feat(helm): bump inference-operator to v3.7, add priorityClassName, CloudWatch logging, tolerations, and MP mode

* fix: restore trailing newline in values.yaml

* feat(helm): add fluentbit ConfigMap, registryDomain resolver, bump Chart to v2.7.0/appVersion 3.7, bump fsx to 1.17.0
Bump the HyperPod gpu-operator subchart to v26.7.0 to clear out-of-SLA
container CVEs on the v26.3.1 stack. Pairs with the ECR replication of
the v26.7.0 family; toolkit stays default-enabled for coexistence.

- operator + validator v26.3.1 -> v26.7.0
- device-plugin + gfd v0.19.0 -> v0.20.0
- container-toolkit v1.19.0 -> v1.20.0
- mig-manager v0.14.0 -> v0.15.0

Co-authored-by: Stephen Via <svia@amazon.com>
…with bug fixes and CVE fixes (#468)

Bug Fixes

* Telemetry-only health check timeouts no longer trigger node reboots
* Raised the driver-issue timeout threshold from 3 to 5 consecutive checks
* Added a 20s component timeout for the GPU fabric manager check

New Configuration

* XID_63_ACTION_DISABLED (default false). When set to true, Xid 63 (GPU row
  remap pending) is recorded as a warning with no reboot action. Detection and
  metrics are unaffected; only the remediation action is suppressed.

Security

* Go dependency updates for CVE remediation.
* fix: Stabilize integration cluster tests

* fix: Isolate recipe test storage

* fix: Support Kubernetes 37 models
fix(inference-operator): update Helm to EKS add-on migration script.
…#449)

The kubeflow training-operator Deployment has no tolerations, so it cannot
be scheduled onto nodes carrying the HyperPod deep health check taint
sagemaker.amazonaws.com/node-health-status=Unschedulable:NoSchedule.

On clusters where every node is tainted during an on-start deep health
check, the operator pod stays Pending and PyTorchJob-based checks such as
MultiNodeTraining never get reconciled.

Other controllers and device plugins in this chart already tolerate the
same taint (mpi-operator, neuron-device-plugin, nvidia-device-plugin,
aws-efa-k8s-device-plugin). This makes training-operators consistent.
`hyp create` rendered standard project template manifests (k8s.jinja /
cfn_params.jinja) with a plain `jinja2.Template`. Rendering now uses
`jinja2.sandbox.ImmutableSandboxedEnvironment` so template expressions
cannot access unsafe Python attributes or callables.

The recipe rendering path (recipe_utils._render_k8s_template) already used
a sandboxed environment; this aligns the standard-template path in
_create_standard_template with it.

Adds tests asserting that a disallowed template expression is blocked and
that benign templates still render.

Co-authored-by: Mufaddal Rohawala <mufi@amazon.com>
hyp list cluster-stack and HpClusterStack.list() return all CloudFormation stacks in the account and Region, but the help text, output header, and documentation described them as HyperPod-only.

Clarify the behavior in the CLI help, SDK docstring, output header, README, Read the Docs, and example notebooks. Also fix the invalid `hyp list hyp-cluster` usage example and add regression tests covering mixed HyperPod and unrelated stacks plus help/behavior consistency.

Refs #348
Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
…default (#479)

* feat: add hyperpod-kubernetes-resource-monitor subchart, disabled by default

Adds the HyperPod Kubernetes Resource Monitoring Agent as the
hyperpod-kubernetes-resource-monitor subchart (enabled: false in the parent
values).

* feat: name hyperpod-kubernetes-resource-monitor resources hyperpod-kubernetes-resource-monitoring-agent

Use fixed resource names, like health-monitoring-agent: the Deployment,
ServiceAccount, ClusterRole and ClusterRoleBinding are named
hyperpod-kubernetes-resource-monitoring-agent, whatever the release name.
The values key (hyperpod-kubernetes-resource-monitor) and the
app=cluster-monitoring-agent label are unchanged.

---------

Co-authored-by: Xin Wang <xwnamz@amazon.com>
* fix: remove NCCL RAS settings the resource monitor ignores

The agent's NCCL RAS job monitor takes each job's endpoints from the
pod IPs of its training pods. It never reads ncclRas.nodes, autoDiscover
or nodeSelector, and it does not use retryDelay or outputDir, so setting
them had no effect. Remove them from the values and the Deployment env.

maxRetries is the number of connection attempts per endpoint in each
monitoring cycle; the comment now says so.

Older agent images read the removed env vars with the same defaults the
chart set, so their behavior does not change.

* fix: pin hyperpod-kubernetes-resource-monitor to image 1.0.508.0_1.0.327.0

The new image fixes the job-level NCCL RAS state, which was always
no_data, and stops querying the training operator's pod-manager pods
as NCCL RAS endpoints.

---------

Co-authored-by: Xin Wang <xwnamz@amazon.com>
Turn the hyperpod-kubernetes-resource-monitor subchart on by default, like
health-monitoring-agent. Set hyperpod-kubernetes-resource-monitor.enabled=false
to skip it. The SageMaker console already sends the value explicitly, so
console installs are unchanged.

Co-authored-by: Xin Wang <xwnamz@amazon.com>
- Chart 2.7.0 -> 2.8.3, appVersion 3.7 -> 3.8, image tag v3.7 -> v3.8; parent chart dependency bumped to 2.8.3
- Regenerated IEC/JSM CRDs: add IEC spec.dynamoSpec (managed NVIDIA Dynamo), status.selfHealCount, and path-safe pattern and maxLength on invocationEndpoint and loadBalancer.healthCheckPath. Descriptions reworded. Backward compatible.
- Manager memory limit 256Mi -> 512Mi.
- RBAC: grant the operator access to nvidia.com dynamographdeployments.
- Subcharts: aws-mountpoint-s3-csi-driver 1.14.1 -> 1.15.0 (with sidecar image pins), aws-fsx-csi-driver 1.17.0 -> 1.19.0; Chart.lock regenerated.

Signed-off-by: Ophelia Yang <ziyng@amazon.com>
hyperpod-resource-ownership records who created each RayJob, HyperPodPyTorchJob,
InferenceEndpointConfig and JumpStartModel (the created-by annotation) and lets only the
creator, the operators and cluster admins update or delete the ones marked OwnerOnly,
including through their status and scale subresources. The owner can change
ownership-type; created-by is immutable.

hyperpod-compute-configuration installs the ComputeConfiguration CRD and its admission
webhook. An admin defines named compute shapes in a namespace; a RayJob,
HyperPodPyTorchJob or InferenceEndpointConfig that references one by label gets its
compute filled in and checked, and namespaces labelled
compute.configuration.sagemaker.amazonaws.com/enforcement=enabled require every such
workload to use one.

Both webhook images come from the regional HyperPod ECR repository for the 14 supported
Regions, by tag or by digest; an image override must be pinned by digest. Both require
cert-manager and Kubernetes 1.30+.

Co-authored-by: Meghana Janapareddy <meghajan@amazon.com>

This branch is waiting to be deployed

1 inactive (outdated) and 1 waiting deployments
manual-approval — b063e884 Waiting Nov 22, 2026 by papriwal via wait-for-approval #855
auto-approve — 1ffbd651 Deployed Sep 3, 2025 by zhaoqizqwang
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.