Skip to content

CKS: upgradeKubernetesCluster fails on control node when binaries ISO ships headlamp.yaml instead of dashboard.yaml #14244

Open

Description

@mw-0

problem

Since 4.23, create-kubernetes-binaries-iso.sh packages the Headlamp dashboard as headlamp.yaml
instead of dashboard.yaml. The cluster create path (k8s-control-node.yml) handles both files,
but upgrade-kubernetes.sh still hardcodes:

/opt/bin/kubectl apply -f ${BINARIES_DIR}/dashboard.yaml

The script runs with bash -e, so on the first control node it exits non-zero after kubeadm
upgrade apply and the kubelet swap have already succeeded. The worker retries
(cloud.kubernetes.cluster.upgrade.retries) and then fails the job, leaving the control node
on the new version but cordoned, the workers on the old version, the cluster in Alert,
and the DB still recording the old version.

Management server log:
SSH execution of command sudo ./upgrade-kubernetes.sh.sh 1.36.0 true false false true
has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
Failed to upgrade Kubernetes cluster : single, unable to upgrade Kubernetes node on VM : single-control-...

Running the script by hand on the control node shows:
error: the path "/mnt/k8sdisk//dashboard.yaml" does not exist

ISO contents:
10-kubeadm.conf autoscaler.yaml cni cri-tools docker headlamp.yaml k8s kubelet.service
manifest.yaml network.yaml provider.yaml snapshot-crds.yaml

versions

ACS 4.23.0.0 (also present on main at 0a5bf30)
Kubernetes upgrade 1.35.8 -> 1.36.0, ISO built with the 4.23 create-kubernetes-binaries-iso.sh
Hypervisor: kvm Network: isolated..

The steps to reproduce the bug

  1. Build and register a Kubernetes version ISO with the 4.23 create-kubernetes-binaries-iso.sh (contains headlamp.yaml)
  2. Deploy a CKS cluster on the previous minor version
  3. Call upgradeKubernetesCluster to the new version
  4. Job fails on the first control node; kubectl get nodes shows control on new version, SchedulingDisabled

What to do about it?

Make upgrade-kubernetes.sh use the same fallback as k8s-control-node.yml: apply headlamp.yaml
if present, else dashboard.yaml, else warn. I'll open a PR. Workaround: patch
script/upgrade-kubernetes.sh inside the management server jar and restart.
I already have tested a fix and will put in a PR.

Activity

  1. boring-cyborg commented on Sep 24, 2026

    @boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template!

  2. added a commit that references this issue on Sep 24, 2026
    9f7e9f1
  3. added this to the 24.0 milestone on Oct 8, 2026
  4. nvazquez commented on Oct 30, 2026

    @nvazquez
    Contributor

    PR in progress: #14245

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions