--- name: Debug Cluster description: "Provides systematic debugging approaches for HyperShift hosted-cluster issues. Auto-applies when debugging cluster problems, investigating stuck deletions, or troubleshooting control plane issues." --- # HyperShift Cluster Debugging Guide This skill provides structured debugging workflows for common HyperShift hosted-cluster issues. ## When to Use This Skill This skill automatically applies when: - Investigating hosted-cluster deletion issues - Debugging stuck resources or finalizers - Troubleshooting control plane problems - Analyzing NodePool lifecycle issues - Reviewing operator logs for cluster problems ## Provider-Specific Troubleshooting For provider-specific issues and detailed troubleshooting steps, refer to these subskills: - **AWS**: [aws-troubleshooting.md](.\/aws-troubleshooting.md) - AWS-specific CAPI issues, infrastructure cleanup, and reinstallation The main skill below provides provider-agnostic debugging workflows. When you encounter provider-specific issues, consult the relevant subskill for detailed resolution steps. ## Key Components to Understand ### Resource Hierarchy - **HostedCluster (HC)**: Main cluster resource in the management cluster - **HostedControlPlane (HCP)**: Control plane representation of the HC in HCP namespace - **NodePool (NP)**: Worker node pool resources - **CAPI Resources**: Cluster API resources (Cluster, Machine, etc.) in HCP namespace ### Operators - **hypershift-operator (HO)**: Manages HC and NP resources - **control-plane-operator (CPO)**: Manages HCP and control plane components - **hosted-cluster-config-operator (HCCO)**: Manages configuration and in-cluster resources for hosted clusters ### Namespaces - **HC namespace**: Where HostedCluster and NodePool resources live (e.g., `default`, `clusters`) - **HCP namespace**: Where control plane pods and CAPI resources run (e.g., `clusters-`) ## Common Debugging Scenarios ### Scenario: Hosted Cluster Stuck Deleting When a hosted-cluster is stuck in deleting state, follow this systematic debugging process: #### 1. Node Pools Deletion Check and verify NodePool deletion is progressing: ```bash # Check NodePool resources in HC namespace kubectl get nodepool -n # Check CAPI cluster resource status in HCP namespace kubectl get cluster -n -o yaml # Check CAPI provider pod logs kubectl logs -n deployment\/capi-provider # Check CAPI machines status in HCP namespace kubectl get machines -n kubectl describe machines -n # Review HyperShift operator logs for NodePool issues kubectl logs -n hypershift deployment\/operator --tail=100 | grep -i nodepool kubectl logs -n hypershift deployment\/operator --tail=100 | grep -i ``` **What to look for:** - Finalizers blocking NodePool deletion - CAPI machines that aren't terminating - Provider errors preventing machine deletion - HO logs showing reconciliation errors #### 2. HostedControlPlane Resource Deletion Verify HCP resource and pods are being cleaned up: ```bash # Check HCP resource status kubectl get hostedcontrolplane -n -o yaml # Check pods in HCP namespace kubectl get pods -n # Check for stuck pods kubectl get pods -n --field-selector=status.phase!=Running # Review control-plane-operator logs kubectl logs -n deployment\/control-plane-operator --tail=100 ``` **What to look for:** - HCP finalizers blocking deletion - Pods with finalizers or in Terminating state - CPO logs showing errors in resource cleanup - PVC or other resources preventing namespace deletion #### 3. HCP Namespace Deletion Investigate why the HCP namespace isn't being removed: ```bash # Check namespace status kubectl get namespace -o yaml # List all remaining resources in namespace kubectl api-resources --verbs=list --namespaced -o name | \ xargs -n 1 kubectl get --show-kind --ignore-not-found -n # Check for resources with finalizers kubectl get all -n -o json | \ jq '.items[] | select(.metadata.finalizers != null) | {kind: .kind, name: .metadata.name, finalizers: .metadata.finalizers}' # Review HO logs for namespace cleanup kubectl logs -n hypershift deployment\/operator --tail=100 | grep -i namespace ``` **What to look for:** - Resources with finalizers preventing deletion - API resources that HO should have cleaned up - Webhook or admission controller errors - Namespace stuck in Terminating state #### 4. HostedCluster Resource Deletion Final check on the HostedCluster resource itself: ```bash # Check HostedCluster status kubectl get hostedcluster -n

-o yaml # Check HostedCluster finalizers kubectl get hostedcluster -n

-o jsonpath='{.metadata.finalizers}' # Review HO logs for HostedCluster deletion kubectl logs -n hypershift deployment\/operator --tail=200 | grep -i "hostedcluster.*" ``` **What to look for:** - Finalizers blocking HostedCluster deletion - HO errors in reconciliation loop - Dependencies that haven't been cleaned up - Cloud resources that failed to delete #### Quick Debugging Checklist When investigating cluster deletion issues: - [ ] Check resource status and conditions - [ ] Review relevant operator logs (HO, CPO) - [ ] Inspect finalizers on stuck resources - [ ] Verify CAPI resources are reconciling - [ ] Check for events indicating failures - [ ] Look for provider errors - [ ] Verify namespace cleanup progress - [ ] Check for webhook or admission errors #### Common Issues and Resolutions ##### Issue: NodePool won't delete - **Cause**: CAPI machines stuck due to provider errors - **Resolution**: Check provider credentials, investigate machine deletion errors in HO logs ##### Issue: Machines stuck in "Deleting" phase with "WaitingForInfrastructureDeletion" - **Cause**: Cluster resource deleted before machine resources, blocking CAPI controller reconciliation - **Root Cause**: CAPI machine controllers require the cluster resource to be "ready" to proceed with deletion - **Symptoms**: - Machines show `InfrastructureReady: 1 of 2 completed` - CAPI logs show cluster resource is not ready - Instances still running but controller can't terminate them - **Resolution**: Provider-specific cleanup is required. See provider-specific troubleshooting: - **AWS**: See [aws-troubleshooting.md](.\/aws-troubleshooting.md) for detailed resolution steps ##### Issue: HCP namespace stuck in Terminating - **Cause**: Resources with finalizers or failing webhooks - **Resolution**: List all resources in namespace, remove finalizers from orphaned resources, check webhook availability ##### Issue: HostedCluster stuck with finalizers - **Cause**: Dependencies not fully cleaned up (NodePools, HCP, cloud resources) - **Resolution**: Verify all dependent resources are deleted, check HO logs for reconciliation errors ##### Issue: Control plane pods not terminating - **Cause**: PVCs or other resources with protection finalizers - **Resolution**: Check PVC status, review storage class finalizers, inspect CPO logs ### Scenario: HyperShift CRDs Missing or Corrupted When HyperShift CRDs are accidentally deleted or corrupted (e.g., after using `hypershift destroy infra`), you'll need to reinstall HyperShift. **⚠️ WARNING: HyperShift reinstallation should be a last resort.** Only proceed if CRDs are genuinely missing or corrupted and cannot be recovered through other means. Reinstallation will cause downtime and may impact existing hosted-clusters. #### Symptoms: - Commands like `kubectl get hostedclusters` fail with: `error: the server doesn't have a resource type "hostedclusters"` - HyperShift operator logs show errors about missing CRDs - CRDs for HostedCluster, NodePool, or CAPI resources are missing #### Check for Missing CRDs: ```bash # Check if critical HyperShift CRDs exist kubectl get crd hostedclusters.hypershift.openshift.io kubectl get crd nodepools.hypershift.openshift.io # Count HyperShift CRDs (should be ~9) kubectl get crd | grep hypershift | wc -l # Count CAPI CRDs (should be ~50) kubectl get crd | grep cluster.x-k8s.io | wc -l ``` #### Resolution: Reinstall HyperShift **🤖 AI Assistant Note:** When this scenario is encountered, Claude should guide and suggest the reinstallation steps to the user but NEVER execute the reinstallation commands itself. The user must explicitly run these commands. Provide clear instructions and explanations, but do not use the Bash tool to perform the actual reinstallation. ##### Step 1: Gather Required Parameters ```bash # You'll need these for reinstallation: # - OIDC storage provider configuration (provider-specific, see below) # - Provider credentials (if applicable) # - Any custom configuration flags used in original installation ``` **Provider-specific parameters:** - **AWS**: See [aws-troubleshooting.md](.\/aws-troubleshooting.md) for AWS-specific OIDC S3 parameters ##### Step 2: Completely Uninstall HyperShift ```bash hypershift install render | kubectl delete -f - ``` This will: - Remove the HyperShift operator deployment - Delete all HyperShift CRDs (HostedCluster, NodePool, etc.) - Clean up RBAC resources, webhooks, and other components ##### Step 3: Reinstall HyperShift ```bash hypershift install \ [provider-specific-flags] \ --enable-defaulting-webhook true ``` Add any other flags that were part of your original installation. **Provider-specific installation:** - **AWS**: See [aws-troubleshooting.md](.\/aws-troubleshooting.md) for AWS-specific installation example ##### Step 4: Verify Installation ```bash # Check operator is running kubectl get deploy -n hypershift kubectl get pods -n hypershift # Verify CRDs are installed kubectl get crd | grep hostedcluster kubectl get crd | grep nodepool kubectl get crd | grep cluster.x-k8s.io | wc -l # Test API accessibility kubectl get hostedclusters -A # Check operator logs for errors kubectl logs -n hypershift deployment\/operator --tail=50 # Verify controllers are running kubectl logs -n hypershift deployment\/operator --tail=100 | grep "Starting workers" ``` **Expected Results After Reinstallation:** - Operator deployment: 2\/2 READY - HostedCluster CRD: Present - NodePool CRD: Present - CAPI CRDs: ~50 installed - HyperShift CRDs: ~9 total - Controllers: HostedCluster, NodePool, and other controllers with workers started - Webhooks: Mutating webhook configured - API: `kubectl get hostedclusters -A` returns successfully (even if no clusters exist) **Important Notes:** - Reinstallation does NOT affect existing hosted-clusters if their resources still exist - If CRDs were deleted, any existing HostedCluster\/NodePool resources are gone - You may need to recreate hosted-clusters if their definitions were lost - Ensure you use the same configuration flags as the original installation ## General Debugging Tips ### Checking Finalizers ```bash # List all finalizers on a resource kubectl get -n -o jsonpath='{.metadata.finalizers}' # Remove a specific finalizer (use with caution!) kubectl patch -n -p '{"metadata":{"finalizers":null}}' --type=merge ``` ### Operator Logs ```bash # HyperShift operator logs with context kubectl logs -n hypershift deployment\/operator --tail=500 --timestamps # Control plane operator logs kubectl logs -n deployment\/control-plane-operator --tail=500 --timestamps # Follow logs in real-time kubectl logs -n hypershift deployment\/operator -f ``` ### Resource Events ```bash # Get events for a specific resource kubectl describe -n # Get all events in a namespace, sorted by time kubectl get events -n --sort-by='.lastTimestamp' ``` ### Conditions and Status ```bash # Check resource conditions kubectl get -n -o jsonpath='{.status.conditions}' | jq . # Check specific condition kubectl get hostedcluster -n -o jsonpath='{.status.conditions[?(@.type=="Available")]}' ``` ## Additional Resources - HyperShift operator code: `hypershift-operator\/controllers\/hostedcluster\/` - Control plane operator code: `control-plane-operator\/controllers\/` - API definitions: `api\/hypershift\/v1beta1\/` - E2E test examples: `test\/e2e\/`