Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions helm_chart/HyperPodHelmChart/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -100,3 +100,11 @@ dependencies:
version: "0.1.0"
repository: "file://charts/hyperpod-kubernetes-resource-monitor"
condition: hyperpod-kubernetes-resource-monitor.enabled
- name: hyperpod-resource-ownership
version: "0.1.0"
repository: "file://charts/hyperpod-resource-ownership"
condition: hyperpod-resource-ownership.enabled
- name: hyperpod-compute-configuration
version: "0.1.0"
repository: "file://charts/hyperpod-compute-configuration"
condition: hyperpod-compute-configuration.enabled
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
apiVersion: v2
name: hyperpod-compute-configuration
description: The HyperPod ComputeConfiguration CRD and its admission webhook
type: application

version: 0.1.0
appVersion: "0.1.0"
# 1.30 is the first version with ValidatingAdmissionPolicy in admissionregistration.k8s.io/v1,
# which the chart's policies use.
# The -0 suffix lets EKS versions such as v1.33.13-eks-cfb47f5 match.
kubeVersion: ">=1.30.0-0"

keywords:
- kubernetes
- hyperpod
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
Thank you for installing {{ .Chart.Name }}.

Your release is named {{ .Release.Name }}.
{{- if (.Values.crd | default dict).enable }}

The ComputeConfiguration CRD has been installed.

To verify the installation:

kubectl get customresourcedefinitions computeconfigurations.compute.configuration.sagemaker.amazonaws.com
kubectl get computeconfigurations --all-namespaces
{{- else }}

crd.enable is false, so this release did not install the ComputeConfiguration CRD.
{{- end }}
{{- if .Values.webhook.enable }}

The compute configuration admission webhook is installed (image {{ include "hyperpod-compute-configuration.webhookImage" . }}).
It handles RayJob, HyperPodPyTorchJob and InferenceEndpointConfig creates and
updates that carry the compute.configuration.sagemaker.amazonaws.com/name label, in every
namespace except kube-system:
- The mutating webhook fills in compute and never blocks a workload (failurePolicy Ignore).
- The validating webhook runs only in namespaces labelled
{{ include "hyperpod-compute-configuration.enforcementLabel" . }}=enabled and rejects workloads
it can't check (failurePolicy Fail).
- A ValidatingAdmissionPolicy rejects labelled objects of other kinds in every namespace except
kube-system, unless a controller created them (they have an owner, or come from
webhook.workloadControllers).
- A second ValidatingAdmissionPolicy requires every RayJob, HyperPodPyTorchJob and
InferenceEndpointConfig in enforced namespaces to use a compute configuration, and accepts them
only in API version v1 (in any namespace when they reference a compute configuration).
Enforcement covers RayJob, HyperPodPyTorchJob and InferenceEndpointConfig. Other kinds (such as
RayCluster, Kubeflow PyTorchJob or Pods) are admitted as usual.
The webhook can read compute configurations and namespaces (read-only ClusterRole).{{ if .Values.certManager.enable }}
Its TLS certificate is issued by cert-manager, which must be installed in the cluster.{{ end }}

To verify the webhook:

kubectl -n {{ include "hyperpod-compute-configuration.namespaceName" . }} rollout status deployment/{{ include "hyperpod-compute-configuration.webhookName" . }}
kubectl -n {{ include "hyperpod-compute-configuration.namespaceName" . }} get certificate {{ include "hyperpod-compute-configuration.webhookName" . }}-cert
kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration {{ include "hyperpod-compute-configuration.webhookName" . }}
kubectl -n {{ include "hyperpod-compute-configuration.namespaceName" . }} logs deployment/{{ include "hyperpod-compute-configuration.webhookName" . }} | grep "Webhook version"
{{- end }}

To learn more about the release, try:

$ helm status {{ .Release.Name }} -n {{ .Release.Namespace }}
$ helm get all {{ .Release.Name }} -n {{ .Release.Namespace }}

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{{- if and .Values.webhook.enable .Values.certManager.enable }}
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: {{ include "hyperpod-compute-configuration.webhookName" . }}-cert
namespace: {{ include "hyperpod-compute-configuration.namespaceName" . }}
labels:
{{- include "hyperpod-compute-configuration.webhookLabels" . | nindent 4 }}
spec:
secretName: {{ include "hyperpod-compute-configuration.webhookName" . }}-tls
dnsNames:
- {{ include "hyperpod-compute-configuration.webhookName" . }}.{{ include "hyperpod-compute-configuration.namespaceName" . }}.svc
- {{ include "hyperpod-compute-configuration.webhookName" . }}.{{ include "hyperpod-compute-configuration.namespaceName" . }}.svc.cluster.local
issuerRef:
kind: Issuer
name: {{ include "hyperpod-compute-configuration.webhookName" . }}-selfsigned
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{{- if and .Values.webhook.enable .Values.certManager.enable }}
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: {{ include "hyperpod-compute-configuration.webhookName" . }}-selfsigned
namespace: {{ include "hyperpod-compute-configuration.namespaceName" . }}
labels:
{{- include "hyperpod-compute-configuration.webhookLabels" . | nindent 4 }}
spec:
selfSigned: {}
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
{{- /*
Generated from the ComputeConfiguration Go types. Do not edit by hand.
*/}}
{{- if (.Values.crd | default dict).enable }}
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
{{- if (.Values.crd | default dict).keep }}
"helm.sh/resource-policy": keep
{{- end }}
labels:
app.kubernetes.io/component: compute-configurations
name: computeconfigurations.compute.configuration.sagemaker.amazonaws.com
spec:
group: compute.configuration.sagemaker.amazonaws.com
names:
kind: ComputeConfiguration
listKind: ComputeConfigurationList
plural: computeconfigurations
shortNames:
- computeconfig
singular: computeconfiguration
scope: Namespaced
versions:
- additionalPrinterColumns:
- jsonPath: .spec.targetKinds
name: Targets
type: string
- jsonPath: .spec.resources.gpus
name: GPUs
type: integer
- jsonPath: .spec.resources.vcpus
name: vCPUs
type: string
- jsonPath: .spec.resources.memory
name: Memory
type: string
- jsonPath: .spec.resources.efa
name: EFA
type: integer
- jsonPath: .spec.resources.worker.gpus
name: Worker GPUs
type: integer
- jsonPath: .spec.resources.worker.vcpus
name: Worker vCPUs
type: string
- jsonPath: .spec.resources.worker.memory
name: Worker Memory
type: string
- jsonPath: .metadata.creationTimestamp
name: Age
type: date
name: v1alpha1
schema:
openAPIV3Schema:
description: |-
ComputeConfiguration is an admin-defined, named compute shape
that workloads of the target kind can reference.
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: |-
ComputeConfigurationSpec defines the compute shape and
the workload kind it applies to.
properties:
description:
description: |-
Description is a human-readable description of this
configuration.
maxLength: 1024
type: string
instanceType:
description: |-
InstanceType is the instance type this configuration
targets, for example ml.p5.48xlarge. The admission webhook applies it as
the node.kubernetes.io/instance-type node selector. If it isn't set, the
configuration doesn't manage the instance type.
maxLength: 64
pattern: ^ml\.[a-z0-9-]+\.[a-z0-9]+$
type: string
resources:
description: |-
Resources is the compute shape applied to the workload.
Set either vcpus and memory (one shape for every pod), or head and
worker (Ray only, a shape per role), not both.
properties:
efa:
description: EFA is the number of EFA devices. Omitted means 0.
format: int32
minimum: 0
type: integer
gpus:
description: |-
GPUs is the number of accelerator devices (NVIDIA GPUs
or Neuron devices, per the workload's instance type). Omitted means 0.
format: int32
minimum: 0
type: integer
head:
description: Head is the Ray head pod's compute shape.
properties:
efa:
default: 0
description: EFA is the number of EFA devices.
format: int32
minimum: 0
type: integer
gpus:
default: 0
description: |-
GPUs is the number of accelerator devices
(NVIDIA GPUs or Neuron devices, per the workload's instance type).
format: int32
minimum: 0
type: integer
memory:
description: |-
Memory is the memory quantity, for example 64Gi.
Must be greater than 0.
pattern: ^([1-9][0-9]*(\.[0-9]+)?|0\.[0-9]*[1-9][0-9]*)(k|M|G|T|P|Ki|Mi|Gi|Ti|Pi)?$
type: string
vcpus:
description: |-
VCPUs is the CPU quantity, for example 8 or 500m.
Must be greater than 0.
pattern: ^([1-9][0-9]*(\.[0-9]+)?|0\.[0-9]*[1-9][0-9]*)m?$
type: string
required:
- memory
- vcpus
type: object
memory:
description: |-
Memory is the memory quantity, for example 64Gi.
Must be greater than 0.
pattern: ^([1-9][0-9]*(\.[0-9]+)?|0\.[0-9]*[1-9][0-9]*)(k|M|G|T|P|Ki|Mi|Gi|Ti|Pi)?$
type: string
vcpus:
description: |-
VCPUs is the CPU quantity, for example 8 or 500m.
Must be greater than 0.
pattern: ^([1-9][0-9]*(\.[0-9]+)?|0\.[0-9]*[1-9][0-9]*)m?$
type: string
worker:
description: Worker is the Ray worker pods' compute shape.
properties:
efa:
default: 0
description: EFA is the number of EFA devices.
format: int32
minimum: 0
type: integer
gpus:
default: 0
description: |-
GPUs is the number of accelerator devices
(NVIDIA GPUs or Neuron devices, per the workload's instance type).
format: int32
minimum: 0
type: integer
memory:
description: |-
Memory is the memory quantity, for example 64Gi.
Must be greater than 0.
pattern: ^([1-9][0-9]*(\.[0-9]+)?|0\.[0-9]*[1-9][0-9]*)(k|M|G|T|P|Ki|Mi|Gi|Ti|Pi)?$
type: string
vcpus:
description: |-
VCPUs is the CPU quantity, for example 8 or 500m.
Must be greater than 0.
pattern: ^([1-9][0-9]*(\.[0-9]+)?|0\.[0-9]*[1-9][0-9]*)m?$
type: string
required:
- memory
- vcpus
type: object
type: object
x-kubernetes-validations:
- message: vcpus and memory are required (for Ray, set head and worker
instead)
rule: has(self.head) || has(self.worker) || (has(self.vcpus) &&
has(self.memory))
- message: head and worker must be set together
rule: has(self.head) == has(self.worker)
- message: head and worker can't be combined with vcpus, memory, gpus
or efa; set those inside head and worker instead
rule: '!(has(self.head) || has(self.worker)) || !(has(self.vcpus)
|| has(self.memory) || has(self.gpus) || has(self.efa))'
targetKinds:
description: |-
TargetKinds are the workload kinds this configuration
applies to, each as <group>/<Kind>: ray.io/RayJob,
sagemaker.amazonaws.com/HyperPodPyTorchJob,
inference.sagemaker.aws.amazon.com/InferenceEndpointConfig or
kubeflow.org/PyTorchJob.
items:
enum:
- ray.io/RayJob
- sagemaker.amazonaws.com/HyperPodPyTorchJob
- inference.sagemaker.aws.amazon.com/InferenceEndpointConfig
- kubeflow.org/PyTorchJob
type: string
maxItems: 16
minItems: 1
type: array
x-kubernetes-list-type: set
required:
- resources
- targetKinds
type: object
x-kubernetes-validations:
- message: head and worker resources are only supported when every target
kind is a Ray kind (ray.io/...)
rule: '!has(self.resources.head) || self.targetKinds.all(k, k.startsWith(''ray.io/''))'
status:
description: |-
ComputeConfigurationStatus is reserved for the admission
webhook; nothing writes it today.
type: object
required:
- spec
type: object
x-kubernetes-validations:
- message: name must be at most 63 characters (it is used as a label value
on workloads)
rule: self.metadata.name.size() <= 63
served: true
storage: true
subresources:
status: {}
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{{- if ((.Values.rbac | default dict).helpers | default dict).enable }}
# Full access to ComputeConfigurations.
# Not bound to anyone: bind it with a RoleBinding (one namespace) or ClusterRoleBinding (all).
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: compute-configurations
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/name: {{ .Chart.Name }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
name: {{ .Chart.Name }}-computeconfiguration-admin
rules:
- apiGroups:
- compute.configuration.sagemaker.amazonaws.com
resources:
- computeconfigurations
verbs:
- '*'
- apiGroups:
- compute.configuration.sagemaker.amazonaws.com
resources:
- computeconfigurations/status
verbs:
- get
{{- end }}
Loading
Loading