--- name: spectrocloud-appliance-mode description: Build and deploy Spectro Cloud Palette Edge clusters using Appliance Mode. Covers CanvOS builds, provider images, ISOs, content bundles, 2-node deployments, and CI\/CD integration patterns. --- # Spectro Cloud Appliance Mode Appliance Mode creates immutable edge deployments using Kairos-based images built with CanvOS. ## When to Use | Use Case | Mode | |----------|------| | Immutable, locked-down edge | Appliance | | 2-node HA clusters | Appliance (only option) | | Existing OS\/golden images | Agent | | Quick testing on VMs | Agent | ## Key Concepts - **CanvOS**: Earthly-based build system for edge artifacts - **Provider Image**: Container image with OS + specific K8s version - **Installer ISO**: Bootable image with Palette agent + user-data - **Content Bundle**: Pre-cached images for offline deployments ## Prerequisites - Linux AMD64 build machine (4+ CPU, 8GB+ RAM, 150GB+ storage) - Git, Docker, Earthly ```bash # Install Earthly curl -fsSL https:\/\/releases.earthly.dev\/earthly-linux-amd64 -o earthly sudo mv earthly \/usr\/local\/bin\/ && sudo chmod +x \/usr\/local\/bin\/earthly earthly bootstrap ``` ## Build Process **Ask the user:** 1. Registry for provider images? (ttl.sh for testing, or specify) 2. K8s distribution? (k3s, rke2, kubeadm, kubeadm-fips) 3. 2-node deployment? (requires K3s + TWO_NODE=true) 4. Need content bundles for offline? ### Step 0: Determine K8s Version **Query CanvOS for supported versions before building.** Default to n-1 minor for stability. ```bash # Get latest CanvOS tag CANVOS_TAG=$(gh api repos\/spectrocloud\/CanvOS\/tags --jq '.[0].name') echo "Latest CanvOS: $CANVOS_TAG" # Fetch supported K8s versions curl -s "https:\/\/raw.githubusercontent.com\/spectrocloud\/CanvOS\/$CANVOS_TAG\/k8s_version.json" > k8s_versions.json # Get n-1 minor for chosen distribution (default: k3s) DISTRO="k3s" # or: rke2, kubeadm, kubeadm-fips K8S_VERSION=$(jq -r --arg d "$DISTRO" '.[$d] | sort_by(split(".") | map(tonumber)) | reverse | group_by(split(".")[0:2] | join(".")) | sort_by(.[0] | split(".") | map(tonumber)) | reverse | .[1][0] \/\/ .[0][0]' k8s_versions.json) echo "Recommended $DISTRO version: $K8S_VERSION" # Show all available for reference jq -r --arg d "$DISTRO" '.[$d] | sort_by(split(".") | map(tonumber)) | reverse | .[0:5] | join(", ")' k8s_versions.json ``` **Use this K8S_VERSION in Step 1's .arg file.** ### Step 1: Clone and Configure ```bash # Clone fresh or update existing repo if [ -d "CanvOS" ]; then cd CanvOS && git fetch --tags && git pull else git clone https:\/\/github.com\/spectrocloud\/CanvOS.git && cd CanvOS fi LATEST_TAG=$(git describe --tags --abbrev=0) echo "Using CanvOS $LATEST_TAG" git checkout "$LATEST_TAG" # Use K8S_VERSION from Step 0, or set manually cat << EOF > .arg OS_DISTRIBUTION=ubuntu OS_VERSION=24.04 K8S_DISTRIBUTION=${DISTRO:-k3s} K8S_VERSION=${K8S_VERSION:-1.32.9} IMAGE_REGISTRY=ttl.sh IMAGE_REPO=my-edge-images CUSTOM_TAG=demo ARCH=amd64 FIPS_ENABLED=false TWO_NODE=false EOF ``` ### Step 2: Create user-data **Bridge networking is recommended** for VM-based edge deployments (allows containers to get IPs on host network). ```bash cat << 'EOF' > user-data #cloud-config install: reboot: true users: - name: kairos passwd: kairos groups: [sudo, admin] sudo: ALL=(ALL) NOPASSWD:ALL stages: initramfs: - name: "Setup bridge networking" files: - path: \/etc\/systemd\/network\/20-dhcp.network content: | [Match] Name=en* [Network] Bridge=br0 LinkLocalAddressing=no permissions: 0644 owner: 0 group: 0 - path: \/etc\/systemd\/network\/bridge0.netdev content: | [NetDev] Name=br0 Kind=bridge permissions: 0644 owner: 0 group: 0 - path: \/etc\/systemd\/network\/bridge0.network content: | [Match] Name=br0 [Network] DHCP=yes permissions: 0644 owner: 0 group: 0 stylus: site: paletteEndpoint: api.spectrocloud.com edgeHostToken: projectName: EOF ``` **For proxy, static IP, bonds**: See `references\/user-data-examples.yaml` ### Step 3: Build ```bash docker login # Skip for ttl.sh earthly +iso # Build ISO first - start imaging nodes while provider images push earthly --push +build-provider-images # Output: build\/palette-edge-installer.iso ``` ### Step 4: Version the ISO **Critical**: The default ISO name has no version info. Rename it before uploading: ```bash # Create versioned ISO name with date and K8s version ISO_NAME="palette-edge-$(grep K8S_VERSION .arg | cut -d= -f2)-$(date +%Y%m%d-%H%M).iso" mv build\/palette-edge-installer.iso "build\/$ISO_NAME" echo "Built: $ISO_NAME" ``` **Before imaging any VM**, verify you're using the correct ISO: ```bash # Check ISO build date (should match your recent build) ls -la build\/*.iso # If uploading to Proxmox, include version in the storage name # e.g., "palette-edge-k3s-1.30.5-20241222-1430.iso" ``` ## 2-Node Deployment 2-node HA uses Postgres + Kine backend. **Appliance-mode only.** ### Requirements - **K3s only** (not kubeadm\/rke2) - Ubuntu 22.04 recommended - Cannot expand to 3+ nodes later ### 2-Node .arg ```bash cat << 'EOF' > .arg OS_DISTRIBUTION=ubuntu OS_VERSION=22.04 K8S_DISTRIBUTION=k3s K8S_VERSION=1.33.3 IMAGE_REGISTRY=ttl.sh IMAGE_REPO=my-edge-2node CUSTOM_TAG=two-node-demo ARCH=amd64 TWO_NODE=true EOF ``` ### Build & Deploy ```bash earthly +iso # Build ISO first - start imaging nodes while provider images push earthly --push +provider-image ``` When creating cluster: toggle "Two-Node Mode", select exactly 2 edge hosts. ### 2-Node Failover Behavior - One node is leader (writes), other is follower - Follower detects leader failure and self-promotes (~30s failover) - On recovery, nodes compare timestamps; most recent becomes leader ## Content Bundles Pre-cache images for offline deployments: ```bash # Install Palette CLI curl -LO https:\/\/github.com\/spectrocloud\/palette-cli\/releases\/latest\/download\/palette-linux-amd64 chmod +x palette-linux-amd64 && sudo mv palette-linux-amd64 \/usr\/local\/bin\/palette # Build bundle palette content build --arch amd64 --project-id \ --cluster-profile-ids , --output .\/content-bundle.tar.zst # Include in ISO mkdir -p content && cp content-bundle.tar.zst content\/ earthly +iso ``` ## Deploy ### Pre-Deploy Checklist - [ ] ISO name includes version\/date (not generic `palette-edge-installer.iso`) - [ ] Provider image tag matches ISO build (check `.arg` used) - [ ] Old ISOs cleaned up from hypervisor storage - [ ] Old edge hosts deleted from Palette (prevents duplicate ID errors) ### Proxmox Upload ```bash # Upload versioned ISO to Proxmox ISO_PATH="\/path\/to\/palette-edge-k3s-1.30.5-20241222.iso" scp "$ISO_PATH" root@172.18.0.4:\/var\/lib\/vz\/template\/iso\/ # List ISOs on Proxmox to verify and clean up old ones ssh root@172.18.0.4 "ls -la \/var\/lib\/vz\/template\/iso\/palette-*.iso" # Remove stale ISOs (keep only latest) ssh root@172.18.0.4 "rm \/var\/lib\/vz\/template\/iso\/palette-edge-OLD*.iso" ``` ### VM Creation **Sizing considerations:** | Use Case | Disk Size | Notes | |----------|-----------|-------| | Basic edge | 100GB | ~2.5GB free after OS for storage pools | | With Piraeus file pool | 200GB+ | File pools use root partition space | | With separate data disk | 100GB + data disk | Attach second disk for storage pool | **Boot order is critical:** ``` # Proxmox boot order format: boot: order=scsi0;ide2;net0 ``` - Disk first (`scsi0`), then CD-ROM (`ide2`) - Empty disk falls through to CD on first boot - After install, boots from disk (avoids reinstall loop) - **Wrong order** = reinstall loop or hang **VM creation steps:** 1. Create VM: 4+ CPU, 8GB+ RAM, disk sized for use case 2. Attach the **versioned** ISO (verify name before attaching) 3. Set boot order: `order=scsi0;ide2;net0` 4. Boot - installation is automatic 5. Verify in Palette: Clusters > Edge Hosts > Registered ## Automated Builds via SSH For environments with a dedicated build machine, automate the entire process: ```bash BUILD_HOST="ubuntu@subtle-bug.maas" # Or your build machine # Clone, configure, and build in one session ssh $BUILD_HOST << 'ENDSSH' cd ~ if [ -d "CanvOS" ]; then cd CanvOS && git fetch --tags && git pull else git clone https:\/\/github.com\/spectrocloud\/CanvOS.git && cd CanvOS fi LATEST_TAG=$(git describe --tags --abbrev=0) echo "Using CanvOS $LATEST_TAG" git checkout "$LATEST_TAG" cat << 'ARGFILE' > .arg OS_DISTRIBUTION=ubuntu OS_VERSION=22.04 K8S_DISTRIBUTION=k3s K8S_VERSION=1.33.3 IMAGE_REGISTRY=ttl.sh IMAGE_REPO=my-edge CUSTOM_TAG=2node ARCH=amd64 TWO_NODE=true ARGFILE cat << 'USERDATA' > user-data #cloud-config install: reboot: true users: - name: kairos passwd: kairos groups: [sudo, admin] sudo: ALL=(ALL) NOPASSWD:ALL stages: initramfs: - name: "Setup bridge networking" files: - path: \/etc\/systemd\/network\/20-dhcp.network content: | [Match] Name=en* [Network] Bridge=br0 LinkLocalAddressing=no - path: \/etc\/systemd\/network\/bridge0.netdev content: | [NetDev] Name=br0 Kind=bridge - path: \/etc\/systemd\/network\/bridge0.network content: | [Match] Name=br0 [Network] DHCP=yes stylus: site: paletteEndpoint: api.spectrocloud.com edgeHostToken: projectName: USERDATA earthly +iso # Build ISO first - start imaging nodes while provider images push earthly --push +provider-image ENDSSH # Copy ISO back with version name scp $BUILD_HOST:~\/CanvOS\/build\/palette-edge-installer.iso \ ".\/palette-edge-k3s-1.33.3-$(date +%Y%m%d).iso" ``` ## CI\/CD Integration See `references\/cicd-workflow.md` for GitHub Actions and GitLab CI examples. ## Troubleshooting | Issue | Solution | |-------|----------| | Registry errors | Verify `docker login`, ttl.sh needs no login but expires in 24h | | ISO boot hangs | Check UEFI\/BIOS mode, EFI partition size | | Reinstall loop | Boot order wrong - must be disk first, then CD-ROM. See fix below. | | Not registering | Check user-data, network, logs: `journalctl -u spectro-stylus-agent.service -f` | | Re-imaging | Delete old edge host from Palette first | | Wrong K8s version | Verify ISO name matches expected build, check for stale ISOs | | Stale ISO used | List ISOs on hypervisor, delete old ones, re-upload versioned ISO | | Storage pool too small | 100GB disk leaves ~2.5GB free - increase disk or add data disk | | Cluster stuck Provisioning, no nodes | VMs likely in install loop - check boot order and power cycle | | user-data changes not applied | Regenerate userdata: `kairos-agent notify agent.bootstrap` (see below) | ### User-Data and \/oem Directory All `#cloud-config` files in `\/oem` are merged to generate `\/run\/stylus\/userdata`. If you modify files in `\/oem`, regenerate userdata: ```bash # On the edge node - regenerates \/run\/stylus\/userdata from \/oem files kairos-agent notify agent.bootstrap ``` **Use cases:** - Debugging user-data issues: check `\/run\/stylus\/userdata` for merged result - Adding config post-install: place files in `\/oem\/*.yaml`, then regenerate - Troubleshooting registration: verify stylus config in `\/run\/stylus\/userdata` ### Fixing Install Loop (Boot Order) **Symptom**: VMs keep reinstalling from ISO instead of booting from installed disk. Cluster stays in "Provisioning" with no nodes appearing for 30+ minutes. **Cause**: Boot order is `ide2;scsi0` (CD-ROM first) instead of `scsi0;ide2` (disk first). **Fix**: ```bash # 1. Change boot order via Proxmox API curl -k -X PUT "https:\/\/:8006\/api2\/json\/nodes\/proxmox\/qemu\/\/config" \ -H "Cookie: PVEAuthCookie=${TICKET}" \ -H "CSRFPreventionToken: ${CSRF}" \ --data-urlencode "boot=order=scsi0;ide2;net0" # 2. MUST power off completely (not reboot\/reset!) curl -k -X POST "...\/qemu\/\/status\/stop" ... # 3. Wait for VM to stop, then power on curl -k -X POST "...\/qemu\/\/status\/start" ... ``` **CRITICAL**: Reboot and reset do NOT reliably apply boot order changes. You MUST do a full power off then power on. ## BYOOS Pack Version **Always use the latest BYOOS pack version** when creating cluster profiles. As of Jan 2025, this is **2.1.0**. Query to confirm latest: ```bash curl -s "https:\/\/api.spectrocloud.com\/v1\/packs?filters=metadata.name=edge-native-byoi&limit=50" \ -H "ApiKey: $API_KEY" | jq -r '[.items[] | select(.spec.registryUid == "5eecc89d0b150045ae661cef")] | sort_by(.spec.version | split(".") | map(tonumber)) | reverse | .[0].spec.version' ``` **Registry note**: BYOOS exists in two registries. Use **Public Repo** (type=`spectro`) for Terraform\/API: - Public Repo UID: `5eecc89d0b150045ae661cef` - Type in Terraform: `type = "spectro"` (NOT "oci") ## Provider Image Tag **Don't assume the tag format** - CanvOS generates tags based on `.arg` values but **includes Kairos version, not OS version**: - Expected: `k3s-1.33.5-ubuntu-22.04-2node` - Actual: `k3s-1.33.5-v4.8.1-2node` (Kairos version) Always check the actual tag after build: ```bash # Check local images after build docker images | grep $IMAGE_REPO # Check registry (for Docker Hub) curl -s "https:\/\/hub.docker.com\/v2\/repositories\/$IMAGE_REPO\/tags" | jq '.results[].name' # Check ttl.sh (use crane or skopeo) crane ls ttl.sh\/$IMAGE_REPO ``` **CRITICAL**: Use the EXACT tag from the build output in: 1. BYOOS pack `options.system.uri` in the cluster profile 2. Ensure K8s pack version matches what was built into the image ## Quick Reference | Item | Value | |------|-------| | CanvOS | `https:\/\/github.com\/spectrocloud\/CanvOS` | | ISO output | `build\/palette-edge-installer.iso` (rename with version!) | | Versioned ISO | `palette-edge--.iso` | | Image tag | Check `docker images` after build - use exact tag | | Proxmox boot order | `boot: order=scsi0;ide2;net0` | | SSH access | kairos \/ kairos | ## Additional Resources - `references\/networking-examples.yaml` - Bridge and bond configurations - `references\/user-data-examples.yaml` - Advanced user-data options - `references\/cicd-workflow.md` - CI\/CD pipeline examples ## Links - [EdgeForge Workflow](https:\/\/docs.spectrocloud.com\/clusters\/edge\/edgeforge-workflow\/) - [Two-Node Architecture](https:\/\/docs.spectrocloud.com\/clusters\/edge\/architecture\/two-node\/) - [CanvOS GitHub](https:\/\/github.com\/spectrocloud\/CanvOS) - [Edge Config Examples](https:\/\/github.com\/spectrocloud\/edge-config-examples)