Skip to content

Hkrma nccl ras settings - #480

Merged
papriwal merged 2 commits into
aws:mainfrom
emeraldbay:hkrma-nccl-ras-settings
Nov 16, 2026
Merged

papriwal merged 2 commits into
aws:mainfrom
emeraldbay:hkrma-nccl-ras-settings

Conversation

@emeraldbay

Copy link
Copy Markdown
Collaborator

What's changing and why?

Follow-up to #479 for the hyperpod-kubernetes-resource-monitor subchart. The subchart stays disabled by default.

  • image.tag: 1.0.498.0_1.0.325.0 -> 1.0.508.0_1.0.327.0. The new agent image:
    • reports the job-level NCCL RAS state (healthy, degraded, unhealthy) from each monitoring cycle
    • no longer queries the training operator's pod-manager pods (<job>-podmanagers-0) as NCCL RAS endpoints
  • Removes monitoring.ncclRas.nodes, autoDiscover, nodeSelector, retryDelay and outputDir, and their NCCL_RAS_* env vars.

Before/After UX

Before:
With the subchart enabled, the agent image is 1.0.498.0_1.0.325.0;

After:
With the subchart enabled, the agent image is 1.0.508.0_1.0.327.0; the job-level NCCL RAS state reflects the endpoints. The five values are gone; a values file that still sets them renders the same chart (Helm ignores unused values).

How was this change tested?

  • helm dependency update + helm lint HyperPodHelmChart, subchart off and on: 0 failed.
  • helm template (default): 0 objects from the subchart.
  • helm template --set hyperpod-kubernetes-resource-monitor.enabled=true --set hyperpod-kubernetes-resource-monitor.region=ca-central-1: image 843976229209.dkr.ecr.ca-central-1.amazonaws.com/hyper pod-cluster-monitoring-agent:1.0.508.0_1.0.327.0; NCCL RAS env is NCCL_RAS_PORT, NCCL_RAS_TIMEOUT, NCCL_RAS_MAX_RETRIES, NCCL_RAS_MONITOR_INTERVAL only.
  • Same chart change and agent unit tests in the agent's own package.
  • Installed from this branch on a HyperPod EKS cluster (us-east-1, 2 x p5.48xlarge + 2 x c5.4xlarge) with the subchart enabled: pod Running, /healthz /readyz /health 200, HyperPodKRMAConfig
    status Healthy; a 2-node, 16-GPU HyperPodPyTorchJob with NCCL_RAS_ADDR set to the pod IP reported job-level NCCL RAS state healthy with 2 endpoints (pod-manager pod skipped).
  • helm template --set hyperpod-kubernetes-resource-monitor.enabled=true --set hyperpod-kubernetes-resource-monitor.region=ap-southeast-3: image 971422672635.dkr.ecr.ap-southeast-3.amazonaws.com/h yperpod-cluster-monitoring-agent:1.0.508.0_1.0.327.0.

Are unit tests added?

No. Helm chart only; no Python code changes.

Are integration tests added?

No

Reviewer Guidelines

‼️ Merge Requirements: PRs with failing integration tests cannot be merged without justification.

One of the following must be true:

  • All automated PR checks pass
  • Failed tests include local run results/screenshots proving they work
  • Changes are documentation-only

Xin Wang added 2 commits November 11, 2026 04:32
The agent's NCCL RAS job monitor takes each job's endpoints from the
pod IPs of its training pods. It never reads ncclRas.nodes, autoDiscover
or nodeSelector, and it does not use retryDelay or outputDir, so setting
them had no effect. Remove them from the values and the Deployment env.

maxRetries is the number of connection attempts per endpoint in each
monitoring cycle; the comment now says so.

Older agent images read the removed env vars with the same defaults the
chart set, so their behavior does not change.
…327.0

The new image fixes the job-level NCCL RAS state, which was always
no_data, and stops querying the training operator's pod-manager pods
as NCCL RAS endpoints.
@emeraldbay
emeraldbay requested a review from a team as a code owner November 14, 2026 17:36
@emeraldbay
emeraldbay deployed to manual-approval November 14, 2026 17:36 — with GitHub Actions Active
@huaziyao

Copy link
Copy Markdown
Collaborator

LGTM

@papriwal
papriwal merged commit 76b6540 into aws:main Nov 16, 2026
5 of 6 checks passed

This branch was successfully deployed

1 active deployment
manual-approval — f598b47d Deployed Nov 14, 2026 by emeraldbay via wait-for-approval #845
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants