# spectrocloud-appliance-mode

> Build and deploy Spectro Cloud Palette Edge clusters using Appliance Mode. Covers CanvOS builds, provider images, ISOs, content bundles, 2-node deployments, and CI/CD integration patterns.

- Author: github-actions[bot]
- Repository: nctiggy/claude-skills
- Version: 20260115185028
- Stars: 1
- Forks: 0
- Last Updated: 2026-02-06
- Source: https://github.com/nctiggy/claude-skills
- Web: https://mule.run/skillshub/@@nctiggy/claude-skills~spectrocloud-appliance-mode:20260115185028

---

---
name: spectrocloud-appliance-mode
description: Build and deploy Spectro Cloud Palette Edge clusters using Appliance Mode. Covers CanvOS builds, provider images, ISOs, content bundles, 2-node deployments, and CI/CD integration patterns.
---

# Spectro Cloud Appliance Mode

Appliance Mode creates immutable edge deployments using Kairos-based images built with CanvOS.

## When to Use

| Use Case | Mode |
|----------|------|
| Immutable, locked-down edge | Appliance |
| 2-node HA clusters | Appliance (only option) |
| Existing OS/golden images | Agent |
| Quick testing on VMs | Agent |

## Key Concepts

- **CanvOS**: Earthly-based build system for edge artifacts
- **Provider Image**: Container image with OS + specific K8s version
- **Installer ISO**: Bootable image with Palette agent + user-data
- **Content Bundle**: Pre-cached images for offline deployments

## Prerequisites

- Linux AMD64 build machine (4+ CPU, 8GB+ RAM, 150GB+ storage)
- Git, Docker, Earthly

```bash
# Install Earthly
curl -fsSL https://releases.earthly.dev/earthly-linux-amd64 -o earthly
sudo mv earthly /usr/local/bin/ && sudo chmod +x /usr/local/bin/earthly
earthly bootstrap
```

## Build Process

**Ask the user:**
1. Registry for provider images? (ttl.sh for testing, or specify)
2. K8s distribution? (k3s, rke2, kubeadm, kubeadm-fips)
3. 2-node deployment? (requires K3s + TWO_NODE=true)
4. Need content bundles for offline?

### Step 0: Determine K8s Version

**Query CanvOS for supported versions before building.** Default to n-1 minor for stability.

```bash
# Get latest CanvOS tag
CANVOS_TAG=$(gh api repos/spectrocloud/CanvOS/tags --jq '.[0].name')
echo "Latest CanvOS: $CANVOS_TAG"

# Fetch supported K8s versions
curl -s "https://raw.githubusercontent.com/spectrocloud/CanvOS/$CANVOS_TAG/k8s_version.json" > k8s_versions.json

# Get n-1 minor for chosen distribution (default: k3s)
DISTRO="k3s"  # or: rke2, kubeadm, kubeadm-fips
K8S_VERSION=$(jq -r --arg d "$DISTRO" '.[$d] |
  sort_by(split(".") | map(tonumber)) | reverse |
  group_by(split(".")[0:2] | join(".")) |
  sort_by(.[0] | split(".") | map(tonumber)) | reverse | .[1][0] // .[0][0]' k8s_versions.json)
echo "Recommended $DISTRO version: $K8S_VERSION"

# Show all available for reference
jq -r --arg d "$DISTRO" '.[$d] | sort_by(split(".") | map(tonumber)) | reverse | .[0:5] | join(", ")' k8s_versions.json
```

**Use this K8S_VERSION in Step 1's .arg file.**

### Step 1: Clone and Configure

```bash
# Clone fresh or update existing repo
if [ -d "CanvOS" ]; then
  cd CanvOS && git fetch --tags && git pull
else
  git clone https://github.com/spectrocloud/CanvOS.git && cd CanvOS
fi
LATEST_TAG=$(git describe --tags --abbrev=0)
echo "Using CanvOS $LATEST_TAG"
git checkout "$LATEST_TAG"

# Use K8S_VERSION from Step 0, or set manually
cat << EOF > .arg
OS_DISTRIBUTION=ubuntu
OS_VERSION=24.04
K8S_DISTRIBUTION=${DISTRO:-k3s}
K8S_VERSION=${K8S_VERSION:-1.32.9}
IMAGE_REGISTRY=ttl.sh
IMAGE_REPO=my-edge-images
CUSTOM_TAG=demo
ARCH=amd64
FIPS_ENABLED=false
TWO_NODE=false
EOF
```

### Step 2: Create user-data

**Bridge networking is recommended** for VM-based edge deployments (allows containers to get IPs on host network).

```bash
cat << 'EOF' > user-data
#cloud-config
install:
  reboot: true

users:
  - name: kairos
    passwd: kairos
    groups: [sudo, admin]
    sudo: ALL=(ALL) NOPASSWD:ALL

stages:
  initramfs:
    - name: "Setup bridge networking"
      files:
        - path: /etc/systemd/network/20-dhcp.network
          content: |
            [Match]
            Name=en*
            [Network]
            Bridge=br0
            LinkLocalAddressing=no
          permissions: 0644
          owner: 0
          group: 0
        - path: /etc/systemd/network/bridge0.netdev
          content: |
            [NetDev]
            Name=br0
            Kind=bridge
          permissions: 0644
          owner: 0
          group: 0
        - path: /etc/systemd/network/bridge0.network
          content: |
            [Match]
            Name=br0
            [Network]
            DHCP=yes
          permissions: 0644
          owner: 0
          group: 0

stylus:
  site:
    paletteEndpoint: api.spectrocloud.com
    edgeHostToken: <registration-token>
    projectName: <project-name>
EOF
```

**For proxy, static IP, bonds**: See `references/user-data-examples.yaml`

### Step 3: Build

```bash
docker login <registry>  # Skip for ttl.sh
earthly +iso  # Build ISO first - start imaging nodes while provider images push
earthly --push +build-provider-images
# Output: build/palette-edge-installer.iso
```

### Step 4: Version the ISO

**Critical**: The default ISO name has no version info. Rename it before uploading:

```bash
# Create versioned ISO name with date and K8s version
ISO_NAME="palette-edge-$(grep K8S_VERSION .arg | cut -d= -f2)-$(date +%Y%m%d-%H%M).iso"
mv build/palette-edge-installer.iso "build/$ISO_NAME"
echo "Built: $ISO_NAME"
```

**Before imaging any VM**, verify you're using the correct ISO:
```bash
# Check ISO build date (should match your recent build)
ls -la build/*.iso

# If uploading to Proxmox, include version in the storage name
# e.g., "palette-edge-k3s-1.30.5-20241222-1430.iso"
```

## 2-Node Deployment

2-node HA uses Postgres + Kine backend. **Appliance-mode only.**

### Requirements
- **K3s only** (not kubeadm/rke2)
- Ubuntu 22.04 recommended
- Cannot expand to 3+ nodes later

### 2-Node .arg

```bash
cat << 'EOF' > .arg
OS_DISTRIBUTION=ubuntu
OS_VERSION=22.04
K8S_DISTRIBUTION=k3s
K8S_VERSION=1.33.3
IMAGE_REGISTRY=ttl.sh
IMAGE_REPO=my-edge-2node
CUSTOM_TAG=two-node-demo
ARCH=amd64
TWO_NODE=true
EOF
```

### Build & Deploy
```bash
earthly +iso  # Build ISO first - start imaging nodes while provider images push
earthly --push +provider-image
```

When creating cluster: toggle "Two-Node Mode", select exactly 2 edge hosts.

### 2-Node Failover Behavior
- One node is leader (writes), other is follower
- Follower detects leader failure and self-promotes (~30s failover)
- On recovery, nodes compare timestamps; most recent becomes leader

## Content Bundles

Pre-cache images for offline deployments:

```bash
# Install Palette CLI
curl -LO https://github.com/spectrocloud/palette-cli/releases/latest/download/palette-linux-amd64
chmod +x palette-linux-amd64 && sudo mv palette-linux-amd64 /usr/local/bin/palette

# Build bundle
palette content build --arch amd64 --project-id <uid> \
  --cluster-profile-ids <id1>,<id2> --output ./content-bundle.tar.zst

# Include in ISO
mkdir -p content && cp content-bundle.tar.zst content/
earthly +iso
```

## Deploy

### Pre-Deploy Checklist
- [ ] ISO name includes version/date (not generic `palette-edge-installer.iso`)
- [ ] Provider image tag matches ISO build (check `.arg` used)
- [ ] Old ISOs cleaned up from hypervisor storage
- [ ] Old edge hosts deleted from Palette (prevents duplicate ID errors)

### Proxmox Upload
```bash
# Upload versioned ISO to Proxmox
ISO_PATH="/path/to/palette-edge-k3s-1.30.5-20241222.iso"
scp "$ISO_PATH" root@172.18.0.4:/var/lib/vz/template/iso/

# List ISOs on Proxmox to verify and clean up old ones
ssh root@172.18.0.4 "ls -la /var/lib/vz/template/iso/palette-*.iso"

# Remove stale ISOs (keep only latest)
ssh root@172.18.0.4 "rm /var/lib/vz/template/iso/palette-edge-OLD*.iso"
```

### VM Creation

**Sizing considerations:**
| Use Case | Disk Size | Notes |
|----------|-----------|-------|
| Basic edge | 100GB | ~2.5GB free after OS for storage pools |
| With Piraeus file pool | 200GB+ | File pools use root partition space |
| With separate data disk | 100GB + data disk | Attach second disk for storage pool |

**Boot order is critical:**
```
# Proxmox boot order format:
boot: order=scsi0;ide2;net0
```
- Disk first (`scsi0`), then CD-ROM (`ide2`)
- Empty disk falls through to CD on first boot
- After install, boots from disk (avoids reinstall loop)
- **Wrong order** = reinstall loop or hang

**VM creation steps:**
1. Create VM: 4+ CPU, 8GB+ RAM, disk sized for use case
2. Attach the **versioned** ISO (verify name before attaching)
3. Set boot order: `order=scsi0;ide2;net0`
4. Boot - installation is automatic
5. Verify in Palette: Clusters > Edge Hosts > Registered

## Automated Builds via SSH

For environments with a dedicated build machine, automate the entire process:

```bash
BUILD_HOST="ubuntu@subtle-bug.maas"  # Or your build machine

# Clone, configure, and build in one session
ssh $BUILD_HOST << 'ENDSSH'
cd ~
if [ -d "CanvOS" ]; then
  cd CanvOS && git fetch --tags && git pull
else
  git clone https://github.com/spectrocloud/CanvOS.git && cd CanvOS
fi
LATEST_TAG=$(git describe --tags --abbrev=0)
echo "Using CanvOS $LATEST_TAG"
git checkout "$LATEST_TAG"

cat << 'ARGFILE' > .arg
OS_DISTRIBUTION=ubuntu
OS_VERSION=22.04
K8S_DISTRIBUTION=k3s
K8S_VERSION=1.33.3
IMAGE_REGISTRY=ttl.sh
IMAGE_REPO=my-edge
CUSTOM_TAG=2node
ARCH=amd64
TWO_NODE=true
ARGFILE

cat << 'USERDATA' > user-data
#cloud-config
install:
  reboot: true
users:
  - name: kairos
    passwd: kairos
    groups: [sudo, admin]
    sudo: ALL=(ALL) NOPASSWD:ALL
stages:
  initramfs:
    - name: "Setup bridge networking"
      files:
        - path: /etc/systemd/network/20-dhcp.network
          content: |
            [Match]
            Name=en*
            [Network]
            Bridge=br0
            LinkLocalAddressing=no
        - path: /etc/systemd/network/bridge0.netdev
          content: |
            [NetDev]
            Name=br0
            Kind=bridge
        - path: /etc/systemd/network/bridge0.network
          content: |
            [Match]
            Name=br0
            [Network]
            DHCP=yes
stylus:
  site:
    paletteEndpoint: api.spectrocloud.com
    edgeHostToken: <TOKEN>
    projectName: <PROJECT>
USERDATA

earthly +iso  # Build ISO first - start imaging nodes while provider images push
earthly --push +provider-image
ENDSSH

# Copy ISO back with version name
scp $BUILD_HOST:~/CanvOS/build/palette-edge-installer.iso \
  "./palette-edge-k3s-1.33.3-$(date +%Y%m%d).iso"
```

## CI/CD Integration

See `references/cicd-workflow.md` for GitHub Actions and GitLab CI examples.

## Troubleshooting

| Issue | Solution |
|-------|----------|
| Registry errors | Verify `docker login`, ttl.sh needs no login but expires in 24h |
| ISO boot hangs | Check UEFI/BIOS mode, EFI partition size |
| Reinstall loop | Boot order wrong - must be disk first, then CD-ROM. See fix below. |
| Not registering | Check user-data, network, logs: `journalctl -u spectro-stylus-agent.service -f` |
| Re-imaging | Delete old edge host from Palette first |
| Wrong K8s version | Verify ISO name matches expected build, check for stale ISOs |
| Stale ISO used | List ISOs on hypervisor, delete old ones, re-upload versioned ISO |
| Storage pool too small | 100GB disk leaves ~2.5GB free - increase disk or add data disk |
| Cluster stuck Provisioning, no nodes | VMs likely in install loop - check boot order and power cycle |
| user-data changes not applied | Regenerate userdata: `kairos-agent notify agent.bootstrap` (see below) |

### User-Data and /oem Directory

All `#cloud-config` files in `/oem` are merged to generate `/run/stylus/userdata`. If you modify files in `/oem`, regenerate userdata:

```bash
# On the edge node - regenerates /run/stylus/userdata from /oem files
kairos-agent notify agent.bootstrap
```

**Use cases:**
- Debugging user-data issues: check `/run/stylus/userdata` for merged result
- Adding config post-install: place files in `/oem/*.yaml`, then regenerate
- Troubleshooting registration: verify stylus config in `/run/stylus/userdata`

### Fixing Install Loop (Boot Order)

**Symptom**: VMs keep reinstalling from ISO instead of booting from installed disk. Cluster stays in "Provisioning" with no nodes appearing for 30+ minutes.

**Cause**: Boot order is `ide2;scsi0` (CD-ROM first) instead of `scsi0;ide2` (disk first).

**Fix**:
```bash
# 1. Change boot order via Proxmox API
curl -k -X PUT "https://<PROXMOX>:8006/api2/json/nodes/proxmox/qemu/<VMID>/config" \
  -H "Cookie: PVEAuthCookie=${TICKET}" \
  -H "CSRFPreventionToken: ${CSRF}" \
  --data-urlencode "boot=order=scsi0;ide2;net0"

# 2. MUST power off completely (not reboot/reset!)
curl -k -X POST ".../qemu/<VMID>/status/stop" ...

# 3. Wait for VM to stop, then power on
curl -k -X POST ".../qemu/<VMID>/status/start" ...
```

**CRITICAL**: Reboot and reset do NOT reliably apply boot order changes. You MUST do a full power off then power on.

## BYOOS Pack Version

**Always use the latest BYOOS pack version** when creating cluster profiles. As of Jan 2025, this is **2.1.0**.

Query to confirm latest:
```bash
curl -s "https://api.spectrocloud.com/v1/packs?filters=metadata.name=edge-native-byoi&limit=50" \
  -H "ApiKey: $API_KEY" | jq -r '[.items[] | select(.spec.registryUid == "5eecc89d0b150045ae661cef")] |
  sort_by(.spec.version | split(".") | map(tonumber)) | reverse | .[0].spec.version'
```

**Registry note**: BYOOS exists in two registries. Use **Public Repo** (type=`spectro`) for Terraform/API:
- Public Repo UID: `5eecc89d0b150045ae661cef`
- Type in Terraform: `type = "spectro"` (NOT "oci")

## Provider Image Tag

**Don't assume the tag format** - CanvOS generates tags based on `.arg` values but **includes Kairos version, not OS version**:
- Expected: `k3s-1.33.5-ubuntu-22.04-2node`
- Actual: `k3s-1.33.5-v4.8.1-2node` (Kairos version)

Always check the actual tag after build:

```bash
# Check local images after build
docker images | grep $IMAGE_REPO

# Check registry (for Docker Hub)
curl -s "https://hub.docker.com/v2/repositories/$IMAGE_REPO/tags" | jq '.results[].name'

# Check ttl.sh (use crane or skopeo)
crane ls ttl.sh/$IMAGE_REPO
```

**CRITICAL**: Use the EXACT tag from the build output in:
1. BYOOS pack `options.system.uri` in the cluster profile
2. Ensure K8s pack version matches what was built into the image

## Quick Reference

| Item | Value |
|------|-------|
| CanvOS | `https://github.com/spectrocloud/CanvOS` |
| ISO output | `build/palette-edge-installer.iso` (rename with version!) |
| Versioned ISO | `palette-edge-<K8S_VERSION>-<YYYYMMDD-HHMM>.iso` |
| Image tag | Check `docker images` after build - use exact tag |
| Proxmox boot order | `boot: order=scsi0;ide2;net0` |
| SSH access | kairos / kairos |

## Additional Resources

- `references/networking-examples.yaml` - Bridge and bond configurations
- `references/user-data-examples.yaml` - Advanced user-data options
- `references/cicd-workflow.md` - CI/CD pipeline examples

## Links

- [EdgeForge Workflow](https://docs.spectrocloud.com/clusters/edge/edgeforge-workflow/)
- [Two-Node Architecture](https://docs.spectrocloud.com/clusters/edge/architecture/two-node/)
- [CanvOS GitHub](https://github.com/spectrocloud/CanvOS)
- [Edge Config Examples](https://github.com/spectrocloud/edge-config-examples)