Skip to content

Latest commit

 

History

History
226 lines (159 loc) · 8.79 KB

setup.md

File metadata and controls

226 lines (159 loc) · 8.79 KB

Setup Guide

This guide covers setting up a HyperFleet environment for running E2E tests locally.

Table of Contents

Deployment Options

Clone the infrastructure repository:

git clone https://github.com/openshift-hyperfleet/hyperfleet-infra/
cd hyperfleet-infra/terraform

Choose one of the following deployment options based on your needs:

  • Kind (local): Fast setup, no cloud dependencies, uses port-forwarding
  • GCP: Cloud environment, requires GCP access, slower setup, uses a LoadBalancer service

Both options deploy the desire delivery stack next to the adapters: Redis (the desire store), the applier, and the remote adapter cl-desire. The remote-transport suites need it (test cases). See Desire delivery in the hyperfleet-infra README.

The tests reach the API through the gateway, svc/hyperfleet-gateway, which listens on plain HTTP on port 8000. svc/hyperfleet-api is ClusterIP-only and serves TLS to the gateway, so do not point the tests at it.

Authentication mode

The E2E runner defaults to AUTH_MODE=EDGE+API. Set AUTH_MODE to the same value used to deploy HyperFleet. Supported values are NONE, API, EDGE, and EDGE+API.

The E2E runner uses TENANT_MODEL=onprem by default. Set TENANT_MODEL to the value that you used to deploy HyperFleet. Supported values are onprem and oracle. The tenant model selects the tenant claims in human tokens.

In authenticated modes, normal API operations run as a human. At startup, each test process mints a JWT for hyperfleet-e2e@example.com. The JWT comes from the mock OIDC issuer in NAMESPACE (svc/hyperfleet-mock-oidc, release hyperfleet-mock-oidc). The process mints a new token shortly before the old token expires.

The suite reaches the issuer through the Kubernetes API server service proxy. Each mint is one HTTP request.

The tenant of the default human is the run ID (RUN_ID). Because of this, reruns in one namespace do not see each other's resources.

The adapters and the machine-auth specs in e2e/auth use machine tokens. These specs mint their own Kubernetes TokenRequest tokens for hyperfleet-e2e-runner and for a temporary alternate ServiceAccount. For which identity a spec should use, see API Identities.

Mode Token sent by E2E for API operations Machine token sent by test adapters
NONE No token Chart default; auth is disabled
API Bearer <mock OIDC JWT> Chart default Bearer
EDGE Bearer <mock OIDC JWT> ServiceAccount <token>
EDGE+API Bearer <mock OIDC JWT> ServiceAccount <token>

The schemes select different Authorino authentication methods. ServiceAccount triggers Kubernetes TokenReview; Bearer triggers human OIDC JWT validation. This prevents a machine token from entering the human-token path. See the infra authentication explanation.

The expected created_by and deleted_by value depends on the mode:

  • API and EDGE+API: the email of the human.
  • EDGE: system@hyperfleet.local. In this mode, the API does not validate tokens and records its system default.

HYPERFLEET_IDENTITY_EXPECTEDIDENTITY overrides the value.

Authenticated runs need:

  • The mock OIDC issuer in NAMESPACE. Deploy it with OIDC_ISSUER_MODE=mock. This is the hyperfleet-infra default for e2e-kind and e2e-gcp.
  • A kubeconfig with these permissions in NAMESPACE:
    • Create and get services/proxy. The token request and the discovery check in Human token mint failed use it.
    • Create TokenRequests for ServiceAccounts.
    • Create the temporary alternate ServiceAccount.

Before any spec runs, each test process also lists clusters as the default human. If the mint fails or the API does not answer 200, the suite stops. See Human token mint failed and Default human rejected.

Option 1: Kind (Local)

1. Deploy HyperFleet to Kind cluster:

# option 1: Export namespace and helmfile env
export NAMESPACE=<your-dev-namespace> ; export HELMFILE_ENV=e2e-kind ; make local-up-kind
# option 2: Set in command line
NAMESPACE=<your-dev-namespace> HELMFILE_ENV=e2e-kind make local-up-kind

2. Port-forward the gateway:

export API_LOCAL_PORT=8000
kubectl port-forward -n ${NAMESPACE} svc/hyperfleet-gateway ${API_LOCAL_PORT}:8000

3. Configure environment variables:

export HYPERFLEET_API_URL=http://localhost:${API_LOCAL_PORT}
export NAMESPACE=<your-dev-namespace>

4. Verify deployment:

# Check Helm releases
helm list -n ${NAMESPACE}

# Verify all pods are running
kubectl get pods -n ${NAMESPACE}

# Test API connectivity
curl -f -X GET ${HYPERFLEET_API_URL}/api/hyperfleet/v1/clusters/

Option 2: GCP

1. Deploy HyperFleet to GCP cluster:

Note: Make sure your terraform files are up to date. See hyperfleet-infra/CONTRIBUTING.md for details.

  • terraform/envs/gke/dev.tfbackend
  • terraform/envs/gke/dev.tfvars

See hyperfleet-infra/README.md for infrastructure deployment details.

# option 1: Export namespace and helmfile env
export NAMESPACE=<your-dev-namespace> ; export HELMFILE_ENV=e2e-gcp ; make local-up-gcp
# option 2: Set in command line
NAMESPACE=<your-dev-namespace> HELMFILE_ENV=e2e-gcp make local-up-gcp

2. Wait for the gateway's external IP:

In e2e-gcp, svc/hyperfleet-gateway is a LoadBalancer.

# Wait for the external IP to be assigned (may take 1-2 minutes)
kubectl get svc hyperfleet-gateway -n ${NAMESPACE} -w

With AUTH_MODE=EDGE or EDGE+API, Authorino denies requests whose host it does not know. For a dynamically assigned LoadBalancer IP, re-apply the gateway with AUTHORINO_HOSTS set to that IP before running tests. CI setup performs this second apply automatically.

3. Configure environment variables:

export API_EXTERNAL_IP=$(kubectl get svc hyperfleet-gateway -n ${NAMESPACE} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export HYPERFLEET_API_URL=http://${API_EXTERNAL_IP}:8000
export NAMESPACE=<your-dev-namespace>

4. Verify deployment:

# Check Helm releases
helm list -n ${NAMESPACE}

# Verify all pods are running
kubectl get pods -n ${NAMESPACE}

# Test API connectivity
curl -f -X GET ${HYPERFLEET_API_URL}/api/hyperfleet/v1/clusters/

Configure Test Settings

Every live run starts with a read-only drift check: hyperfleet-e2e runs helm get values hyperfleet-api --all --namespace "$NAMESPACE" and compares the deployed config.entities with its descriptor table. Set NAMESPACE to the namespace that holds the API release and make sure helm and KUBECONFIG can read it. A mismatch fails the suite before any spec runs; see debugging.md. make list-tests and --dry-run skip the check.

Override Image Settings (Optional)

If your deployment uses custom image settings, update env/env.local in this repo to match your infrastructure deployment settings:

  • Kind deployments: Match settings from env.kind
  • GCP deployments: Match settings from env.gcp

Update env/env.local:

# env/env.local
export IMAGE_REGISTRY=<registry>
export <COMPONENT>_IMAGE_REPO=<repo>
export <COMPONENT>_IMAGE_TAG=<tag>

Source the configuration:

source env/env.local

This configuration is required for running tier2 tests.

Troubleshooting

Infrastructure Setup Issues

For additional help with infrastructure deployment and configuration, see:

For test-specific troubleshooting (timeouts, API errors, namespace mismatches), see the Runbook Troubleshooting section.


Next Steps: Once your environment is set up, see the Runbook for running tests and troubleshooting.