--- name: automation description: automation skill metadata: short-description: automation skill category: utilities source: claude-code-templates --- # Automation # Automation and Orchestration Comprehensive guide to IT automation, scripting, configuration management, orchestration tools, and reducing operational toil. ## Table of Contents - [Automation Strategy](#automation-strategy) - [Scripting Best Practices](#scripting-best-practices) - [Configuration Management](#configuration-management) - [Orchestration Tools](#orchestration-tools) - [CI\/CD for Infrastructure](#cicd-for-infrastructure) - [Runbook Automation](#runbook-automation) - [Self-Healing Systems](#self-healing-systems) - [Toil Reduction](#toil-reduction) ## Automation Strategy ### When to Automate ```yaml Automation Decision Framework: High Priority (Automate Immediately): - Task performed > 10 times per week - Task is error-prone when done manually - Task requires precision (e.g., exact timing, complex steps) - Task performed outside business hours - Task blocks other work Examples: - Server provisioning - Deployment process - Backup operations - Certificate renewal - Log rotation Medium Priority (Automate Within 3 Months): - Task performed 2-10 times per week - Task is time-consuming (> 30 minutes) - Task has multiple steps that could be forgotten - Task involves multiple systems Examples: - User account creation - Monthly reporting - Security patching - Capacity planning data collection Low Priority (Consider Automation): - Task performed < 2 times per week - Task is simple and quick (< 5 minutes) - Task requires human judgment - Automation effort > 10x manual effort Examples: - One-off investigations - Ad-hoc queries - Executive requests Don't Automate: - Task requires critical thinking - Task changes frequently (automation would break) - Task is performed once per year - Legal\/compliance requires human oversight Examples: - Incident response decision-making - Architecture reviews - Contract negotiations ``` ### Automation Maturity Model ```yaml Level 0: Manual - All tasks performed manually - Documentation may exist - High error rate, inconsistent results - Tribal knowledge Level 1: Documented - Detailed runbooks exist - Step-by-step procedures - Anyone can follow documentation - Still fully manual execution Level 2: Scripted - Scripts automate individual tasks - Scripts run on-demand - Reduces errors, saves time - Scripts may live on individual machines Example: backup_database.sh Level 3: Scheduled Automation - Scripts run on schedule (cron, systemd timers) - Central script repository (Git) - Logging and error handling - Notifications on failure Example: Daily backups via cron Level 4: Event-Driven Automation - Automation triggered by events (alerts, webhooks) - Self-healing (auto-restart failed services) - Integration with monitoring systems - Closed-loop automation Example: Auto-scale on high CPU alert Level 5: Intelligent Automation - Machine learning predicts issues - Proactive remediation before user impact - Continuous optimization - Full observability and feedback loops Example: Predictive auto-scaling based on traffic patterns ``` ### ROI Calculation for Automation ```python # Automation ROI Calculator def calculate_automation_roi( manual_time_hours, frequency_per_month, hourly_cost, automation_development_hours, automation_maintenance_hours_per_month ): """ Calculate ROI for automation project Args: manual_time_hours: Time to perform task manually (hours) frequency_per_month: How often task is performed per month hourly_cost: Cost per hour of engineer time ($) automation_development_hours: Time to develop automation (hours) automation_maintenance_hours_per_month: Ongoing maintenance (hours\/month) Returns: dict with ROI metrics """ # Monthly cost without automation monthly_manual_cost = manual_time_hours * frequency_per_month * hourly_cost # Annual manual cost annual_manual_cost = monthly_manual_cost * 12 # Upfront automation cost automation_dev_cost = automation_development_hours * hourly_cost # Monthly automation maintenance cost monthly_automation_cost = automation_maintenance_hours_per_month * hourly_cost # Annual automation cost annual_automation_cost = monthly_automation_cost * 12 # Annual savings annual_savings = annual_manual_cost - annual_automation_cost # Break-even time (months) if annual_savings > 0: break_even_months = automation_dev_cost \/ (annual_savings \/ 12) else: break_even_months = float('inf') # 3-year ROI three_year_savings = (annual_savings * 3) - automation_dev_cost roi_percentage = (three_year_savings \/ automation_dev_cost) * 100 if automation_dev_cost > 0 else 0 return { 'monthly_manual_cost': monthly_manual_cost, 'annual_manual_cost': annual_manual_cost, 'automation_dev_cost': automation_dev_cost, 'annual_automation_cost': annual_automation_cost, 'annual_savings': annual_savings, 'break_even_months': break_even_months, 'three_year_savings': three_year_savings, 'roi_percentage': roi_percentage, 'recommendation': 'Automate' if break_even_months < 12 else 'Consider alternatives' } # Example: Database backup automation result = calculate_automation_roi( manual_time_hours=2, # 2 hours to backup manually frequency_per_month=30, # Daily backups hourly_cost=75, # Engineer costs $75\/hour automation_development_hours=16, # 2 days to develop automation automation_maintenance_hours_per_month=1 # 1 hour\/month maintenance ) print("=== Automation ROI Analysis ===") print(f"Monthly Manual Cost: ${result['monthly_manual_cost']:,.2f}") print(f"Annual Manual Cost: ${result['annual_manual_cost']:,.2f}") print(f"Automation Development Cost: ${result['automation_dev_cost']:,.2f}") print(f"Annual Automation Cost: ${result['annual_automation_cost']:,.2f}") print(f"Annual Savings: ${result['annual_savings']:,.2f}") print(f"Break-even Time: {result['break_even_months']:.1f} months") print(f"3-Year Savings: ${result['three_year_savings']:,.2f}") print(f"3-Year ROI: {result['roi_percentage']:.0f}%") print(f"Recommendation: {result['recommendation']}") # Output: # === Automation ROI Analysis === # Monthly Manual Cost: $4,500.00 # Annual Manual Cost: $54,000.00 # Automation Development Cost: $1,200.00 # Annual Automation Cost: $900.00 # Annual Savings: $53,100.00 # Break-even Time: 0.3 months # 3-Year Savings: $158,100.00 # 3-Year ROI: 13175% # Recommendation: Automate ``` ## Scripting Best Practices ### Shell Scripting (Bash) ```bash #!\/bin\/bash # # Script: server_health_check.sh # Description: Perform comprehensive server health check # Author: Ops Team # Date: 2025-01-15 # Version: 1.0 # Strict error handling set -euo pipefail # Exit on error, undefined var, pipe failure IFS=$'\n\t' # Set internal field separator # Configuration readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" readonly LOG_FILE="\/var\/log\/health_check.log" readonly EMAIL_TO="ops-team@example.com" readonly DISK_THRESHOLD=80 readonly MEMORY_THRESHOLD=85 readonly CPU_THRESHOLD=90 # Colors for output readonly RED='\033[0;31m' readonly GREEN='\033[0;32m' readonly YELLOW='\033[1;33m' readonly NC='\033[0m' # No Color # Logging function log() { local level=$1 shift local message="$@" local timestamp=$(date '+%Y-%m-%d %H:%M:%S') echo "[${timestamp}] [${level}] ${message}" | tee -a "${LOG_FILE}" } # Error handling error_exit() { log "ERROR" "$1" exit 1 } # Cleanup on exit cleanup() { log "INFO" "Cleaning up..." # Remove temp files, etc. } trap cleanup EXIT # Check if running as root check_root() { if [[ $EUID -ne 0 ]]; then error_exit "This script must be run as root" fi } # Check disk space check_disk_space() { log "INFO" "Checking disk space..." local status=0 while read -r line; do local usage=$(echo "$line" | awk '{print $5}' | sed 's\/%\/\/') local mount=$(echo "$line" | awk '{print $6}') if [[ $usage -ge $DISK_THRESHOLD ]]; then log "WARN" "Disk usage on ${mount} is ${usage}% (threshold: ${DISK_THRESHOLD}%)" status=1 else log "INFO" "Disk usage on ${mount} is ${usage}% - OK" fi done < <(df -h | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{print $0}') return $status } # Check memory usage check_memory() { log "INFO" "Checking memory usage..." local total_mem=$(free -m | awk '\/^Mem:\/{print $2}') local used_mem=$(free -m | awk '\/^Mem:\/{print $3}') local usage_pct=$((used_mem * 100 \/ total_mem)) if [[ $usage_pct -ge $MEMORY_THRESHOLD ]]; then log "WARN" "Memory usage is ${usage_pct}% (threshold: ${MEMORY_THRESHOLD}%)" return 1 else log "INFO" "Memory usage is ${usage_pct}% - OK" return 0 fi } # Check CPU load check_cpu() { log "INFO" "Checking CPU load..." local load_avg=$(uptime | awk -F'load average:' '{print $2}' | awk '{print $1}' | sed 's\/,\/\/') local cpu_count=$(nproc) local load_pct=$(echo "scale=2; ($load_avg \/ $cpu_count) * 100" | bc) local load_pct_int=${load_pct%.*} if [[ $load_pct_int -ge $CPU_THRESHOLD ]]; then log "WARN" "CPU load is ${load_pct}% (threshold: ${CPU_THRESHOLD}%)" return 1 else log "INFO" "CPU load is ${load_pct}% - OK" return 0 fi } # Check critical services check_services() { log "INFO" "Checking critical services..." local services=("sshd" "nginx" "postgresql") local status=0 for service in "${services[@]}"; do if systemctl is-active --quiet "$service"; then log "INFO" "Service ${service} is running - OK" else log "ERROR" "Service ${service} is NOT running" status=1 fi done return $status } # Check network connectivity check_network() { log "INFO" "Checking network connectivity..." local hosts=("8.8.8.8" "1.1.1.1") local status=0 for host in "${hosts[@]}"; do if ping -c 3 -W 5 "$host" &>\/dev\/null; then log "INFO" "Network connectivity to ${host} - OK" else log "ERROR" "Cannot reach ${host}" status=1 fi done return $status } # Generate report generate_report() { local overall_status=$1 cat < \/tmp\/health_report.txt Server Health Check Report ========================== Hostname: $(hostname) Date: $(date) Status: $([[ $overall_status -eq 0 ]] && echo "HEALTHY" || echo "ISSUES DETECTED") Details: $(tail -n 50 "${LOG_FILE}") EOF # Email report if issues detected if [[ $overall_status -ne 0 ]]; then mail -s "ALERT: Health Check Failed on $(hostname)" "${EMAIL_TO}" < \/tmp\/health_report.txt fi } # Main execution main() { log "INFO" "Starting server health check..." local overall_status=0 check_root check_disk_space || overall_status=1 check_memory || overall_status=1 check_cpu || overall_status=1 check_services || overall_status=1 check_network || overall_status=1 if [[ $overall_status -eq 0 ]]; then log "INFO" "All health checks passed" else log "WARN" "Some health checks failed" fi generate_report $overall_status log "INFO" "Health check complete" exit $overall_status } # Run main function main "$@" ``` ### Python Automation Scripts ```python #!\/usr\/bin\/env python3 """ User account management automation Automates user provisioning across multiple systems """ import argparse import logging import sys import subprocess import json from typing import Dict, List, Optional from pathlib import Path import yaml # Configure logging logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', handlers=[ logging.FileHandler('\/var\/log\/user_management.log'), logging.StreamHandler(sys.stdout) ] ) logger = logging.getLogger(__name__) class UserManagementError(Exception): """Custom exception for user management errors""" pass class UserManager: """Manages user accounts across systems""" def __init__(self, config_file: str): self.config = self._load_config(config_file) self.dry_run = False def _load_config(self, config_file: str) -> Dict: """Load configuration from YAML file""" try: with open(config_file, 'r') as f: return yaml.safe_load(f) except Exception as e: logger.error(f"Failed to load config: {e}") raise UserManagementError(f"Config load failed: {e}") def _run_command(self, cmd: List[str], check: bool = True) -> subprocess.CompletedProcess: """Execute shell command with error handling""" logger.debug(f"Executing command: {' '.join(cmd)}") if self.dry_run: logger.info(f"[DRY RUN] Would execute: {' '.join(cmd)}") return subprocess.CompletedProcess(cmd, 0, '', '') try: result = subprocess.run( cmd, check=check, capture_output=True, text=True, timeout=30 ) return result except subprocess.CalledProcessError as e: logger.error(f"Command failed: {e.stderr}") raise UserManagementError(f"Command failed: {e}") except subprocess.TimeoutExpired: logger.error(f"Command timed out: {' '.join(cmd)}") raise UserManagementError("Command timeout") def user_exists(self, username: str) -> bool: """Check if user exists""" result = self._run_command(['id', username], check=False) return result.returncode == 0 def create_user( self, username: str, full_name: str, email: str, groups: Optional[List[str]] = None, ssh_public_key: Optional[str] = None ) -> None: """Create user account with specified attributes""" logger.info(f"Creating user: {username}") if self.user_exists(username): logger.warning(f"User {username} already exists") return # Create user cmd = ['useradd', '-m', '-s', '\/bin\/bash'] if full_name: cmd.extend(['-c', full_name]) if groups: cmd.extend(['-G', ','.join(groups)]) cmd.append(username) self._run_command(cmd) # Set up SSH key if ssh_public_key: self._setup_ssh_key(username, ssh_public_key) # Send welcome email (pseudo-code) # self._send_welcome_email(username, email) logger.info(f"User {username} created successfully") def _setup_ssh_key(self, username: str, public_key: str) -> None: """Set up SSH public key for user""" logger.info(f"Setting up SSH key for {username}") # Create .ssh directory ssh_dir = Path(f"\/home\/{username}\/.ssh") ssh_dir.mkdir(mode=0o700, exist_ok=True) # Write authorized_keys authorized_keys = ssh_dir \/ "authorized_keys" authorized_keys.write_text(public_key + "\n") authorized_keys.chmod(0o600) # Set ownership self._run_command(['chown', '-R', f'{username}:{username}', str(ssh_dir)]) def delete_user(self, username: str, remove_home: bool = False) -> None: """Delete user account""" logger.info(f"Deleting user: {username}") if not self.user_exists(username): logger.warning(f"User {username} does not exist") return cmd = ['userdel'] if remove_home: cmd.append('-r') cmd.append(username) self._run_command(cmd) logger.info(f"User {username} deleted successfully") def modify_user_groups(self, username: str, groups: List[str]) -> None: """Modify user's group membership""" logger.info(f"Modifying groups for {username}") if not self.user_exists(username): raise UserManagementError(f"User {username} does not exist") # Set supplementary groups cmd = ['usermod', '-G', ','.join(groups), username] self._run_command(cmd) logger.info(f"Groups updated for {username}") def list_users(self) -> List[Dict[str, str]]: """List all users on the system""" logger.info("Listing all users") users = [] with open('\/etc\/passwd', 'r') as f: for line in f: parts = line.strip().split(':') if int(parts[2]) >= 1000: # Only regular users users.append({ 'username': parts[0], 'uid': parts[2], 'home': parts[5], 'shell': parts[6] }) return users def bulk_create_users(self, users_file: str) -> None: """Create multiple users from CSV file""" logger.info(f"Bulk creating users from {users_file}") import csv with open(users_file, 'r') as f: reader = csv.DictReader(f) for row in reader: try: self.create_user( username=row['username'], full_name=row.get('full_name', ''), email=row.get('email', ''), groups=row.get('groups', '').split(',') if row.get('groups') else None, ssh_public_key=row.get('ssh_public_key') ) except Exception as e: logger.error(f"Failed to create user {row['username']}: {e}") def main(): parser = argparse.ArgumentParser(description='User account management automation') parser.add_argument('--config', default='\/etc\/user_mgmt\/config.yaml', help='Config file path') parser.add_argument('--dry-run', action='store_true', help='Dry run mode') subparsers = parser.add_subparsers(dest='command', help='Command to execute') # Create user create_parser = subparsers.add_parser('create', help='Create user') create_parser.add_argument('username', help='Username') create_parser.add_argument('--full-name', help='Full name') create_parser.add_argument('--email', help='Email address') create_parser.add_argument('--groups', help='Comma-separated groups') create_parser.add_argument('--ssh-key', help='SSH public key') # Delete user delete_parser = subparsers.add_parser('delete', help='Delete user') delete_parser.add_argument('username', help='Username') delete_parser.add_argument('--remove-home', action='store_true', help='Remove home directory') # List users list_parser = subparsers.add_parser('list', help='List users') # Bulk create bulk_parser = subparsers.add_parser('bulk-create', help='Bulk create users') bulk_parser.add_argument('csv_file', help='CSV file with user data') args = parser.parse_args() # Initialize manager manager = UserManager(args.config) manager.dry_run = args.dry_run # Execute command try: if args.command == 'create': groups = args.groups.split(',') if args.groups else None manager.create_user( args.username, args.full_name or '', args.email or '', groups, args.ssh_key ) elif args.command == 'delete': manager.delete_user(args.username, args.remove_home) elif args.command == 'list': users = manager.list_users() for user in users: print(f"{user['username']}\t{user['uid']}\t{user['home']}") elif args.command == 'bulk-create': manager.bulk_create_users(args.csv_file) else: parser.print_help() sys.exit(1) except UserManagementError as e: logger.error(f"Operation failed: {e}") sys.exit(1) if __name__ == '__main__': main() ``` ## Configuration Management ### Ansible Playbooks ```yaml # playbook.yml - Web server setup playbook --- - name: Configure web servers hosts: webservers become: yes vars: nginx_port: 80 app_user: webapp app_path: \/var\/www\/myapp tasks: # System updates - name: Update apt cache apt: update_cache: yes cache_valid_time: 3600 when: ansible_os_family == "Debian" # Install packages - name: Install required packages apt: name: - nginx - python3-pip - git - ufw state: present # Create application user - name: Create application user user: name: "{{ app_user }}" shell: \/bin\/bash create_home: yes system: no # Configure firewall - name: Configure UFW ufw: rule: allow port: "{{ item }}" proto: tcp loop: - 22 - 80 - 443 - name: Enable UFW ufw: state: enabled policy: deny # Deploy application - name: Create application directory file: path: "{{ app_path }}" state: directory owner: "{{ app_user }}" group: "{{ app_user }}" mode: '0755' - name: Clone application repository git: repo: 'https:\/\/github.com\/example\/myapp.git' dest: "{{ app_path }}" version: main become_user: "{{ app_user }}" notify: restart nginx # Configure Nginx - name: Deploy Nginx configuration template: src: templates\/nginx.conf.j2 dest: \/etc\/nginx\/sites-available\/myapp owner: root group: root mode: '0644' notify: restart nginx - name: Enable Nginx site file: src: \/etc\/nginx\/sites-available\/myapp dest: \/etc\/nginx\/sites-enabled\/myapp state: link notify: restart nginx - name: Remove default Nginx site file: path: \/etc\/nginx\/sites-enabled\/default state: absent notify: restart nginx # Install application dependencies - name: Install Python dependencies pip: requirements: "{{ app_path }}\/requirements.txt" virtualenv: "{{ app_path }}\/venv" virtualenv_command: python3 -m venv become_user: "{{ app_user }}" # Configure systemd service - name: Deploy systemd service file template: src: templates\/myapp.service.j2 dest: \/etc\/systemd\/system\/myapp.service owner: root group: root mode: '0644' notify: restart myapp - name: Enable and start application service systemd: name: myapp enabled: yes state: started daemon_reload: yes # Monitoring - name: Install node_exporter include_role: name: node_exporter handlers: - name: restart nginx service: name: nginx state: restarted - name: restart myapp service: name: myapp state: restarted # Nginx template (templates\/nginx.conf.j2) --- server { listen {{ nginx_port }}; server_name {{ ansible_hostname }}; location \/ { proxy_pass http:\/\/127.0.0.1:8000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } location \/static { alias {{ app_path }}\/static; } } ``` ### Ansible Inventory ```ini # inventory\/production.ini [webservers] web01.example.com ansible_host=10.0.10.20 web02.example.com ansible_host=10.0.10.21 web03.example.com ansible_host=10.0.10.22 [databases] db01.example.com ansible_host=10.0.30.10 db02.example.com ansible_host=10.0.30.11 [monitoring] mon01.example.com ansible_host=10.0.99.10 [all:vars] ansible_user=ansible ansible_become=yes ansible_python_interpreter=\/usr\/bin\/python3 [webservers:vars] nginx_port=80 environment=production ``` ### Running Ansible ```bash # Check syntax ansible-playbook playbook.yml --syntax-check # Dry run (check mode) ansible-playbook -i inventory\/production.ini playbook.yml --check # Run playbook ansible-playbook -i inventory\/production.ini playbook.yml # Run specific tags ansible-playbook -i inventory\/production.ini playbook.yml --tags "nginx,app" # Limit to specific hosts ansible-playbook -i inventory\/production.ini playbook.yml --limit web01.example.com # Run ad-hoc command ansible webservers -i inventory\/production.ini -m shell -a "uptime" ansible webservers -i inventory\/production.ini -m apt -a "name=nginx state=latest" --become ``` ## Orchestration Tools ### Terraform for Infrastructure Orchestration See [infrastructure.md](infrastructure.md#infrastructure-as-code) for detailed Terraform examples. ### Kubernetes Operators (Advanced Automation) ```yaml # Custom Kubernetes Operator for database backups # backup-operator.yaml apiVersion: v1 kind: ServiceAccount metadata: name: backup-operator namespace: operators --- apiVersion: rbac.authorization.k8s.io\/v1 kind: ClusterRole metadata: name: backup-operator rules: - apiGroups: [""] resources: ["pods", "secrets"] verbs: ["get", "list", "create"] - apiGroups: ["batch"] resources: ["cronjobs", "jobs"] verbs: ["get", "list", "create", "update", "delete"] --- apiVersion: rbac.authorization.k8s.io\/v1 kind: ClusterRoleBinding metadata: name: backup-operator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: backup-operator subjects: - kind: ServiceAccount name: backup-operator namespace: operators --- apiVersion: apps\/v1 kind: Deployment metadata: name: backup-operator namespace: operators spec: replicas: 1 selector: matchLabels: app: backup-operator template: metadata: labels: app: backup-operator spec: serviceAccountName: backup-operator containers: - name: operator image: myregistry\/backup-operator:latest env: - name: WATCH_NAMESPACE value: "" - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: OPERATOR_NAME value: "backup-operator" ``` ## CI\/CD for Infrastructure ### GitOps Workflow ```yaml # GitOps workflow for infrastructure changes Step 1: Developer makes change - Edit Terraform\/Ansible files - Commit to feature branch - Open pull request Step 2: Automated validation (CI) - terraform fmt -check (code formatting) - terraform validate (syntax check) - tflint (linting) - terraform plan (dry run) - Security scan (Checkov, tfsec) - Cost estimation (Infracost) Step 3: Peer review - Code review by team member - Review terraform plan output - Approve or request changes Step 4: Merge to main - PR approved and merged - Triggers deployment pipeline Step 5: Automated deployment (CD) - terraform apply (auto-approved for specific changes) - Or manual approval for high-risk changes - Post-deployment validation - Notify team in Slack Step 6: Monitoring - Monitor for issues - Auto-rollback on failure - Update status page ``` ### GitHub Actions for Infrastructure ```yaml # .github\/workflows\/terraform.yml name: Terraform CI\/CD on: pull_request: branches: [main] paths: - 'terraform\/**' push: branches: [main] paths: - 'terraform\/**' env: TF_VERSION: 1.6.0 AWS_REGION: us-east-1 jobs: validate: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions\/checkout@v3 - name: Setup Terraform uses: hashicorp\/setup-terraform@v2 with: terraform_version: ${{ env.TF_VERSION }} - name: Terraform Format Check run: terraform fmt -check -recursive working-directory: .\/terraform - name: Terraform Init run: terraform init working-directory: .\/terraform - name: Terraform Validate run: terraform validate working-directory: .\/terraform - name: TFLint uses: terraform-linters\/setup-tflint@v3 with: tflint_version: latest - name: Run TFLint run: tflint --recursive working-directory: .\/terraform plan: runs-on: ubuntu-latest needs: validate if: github.event_name == 'pull_request' steps: - name: Checkout code uses: actions\/checkout@v3 - name: Setup Terraform uses: hashicorp\/setup-terraform@v2 with: terraform_version: ${{ env.TF_VERSION }} - name: Configure AWS Credentials uses: aws-actions\/configure-aws-credentials@v2 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: ${{ env.AWS_REGION }} - name: Terraform Init run: terraform init working-directory: .\/terraform - name: Terraform Plan run: terraform plan -out=tfplan working-directory: .\/terraform - name: Comment PR with Plan uses: actions\/github-script@v6 with: script: | const fs = require('fs'); const plan = fs.readFileSync('.\/terraform\/tfplan', 'utf8'); github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, body: `## Terraform Plan\n\`\`\`\n${plan}\n\`\`\`` }); apply: runs-on: ubuntu-latest needs: validate if: github.event_name == 'push' && github.ref == 'refs\/heads\/main' steps: - name: Checkout code uses: actions\/checkout@v3 - name: Setup Terraform uses: hashicorp\/setup-terraform@v2 with: terraform_version: ${{ env.TF_VERSION }} - name: Configure AWS Credentials uses: aws-actions\/configure-aws-credentials@v2 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: ${{ env.AWS_REGION }} - name: Terraform Init run: terraform init working-directory: .\/terraform - name: Terraform Apply run: terraform apply -auto-approve working-directory: .\/terraform - name: Notify Slack uses: 8398a7\/action-slack@v3 with: status: ${{ job.status }} text: 'Terraform applied successfully' webhook_url: ${{ secrets.SLACK_WEBHOOK }} if: always() ``` ## Runbook Automation ### ChatOps Integration ```python # Slack bot for runbook automation from slack_bolt import App from slack_bolt.adapter.socket_mode import SocketModeHandler import subprocess import logging app = App(token="xoxb-your-token") logging.basicConfig(level=logging.INFO) # Command: restart service @app.command("\/restart-service") def restart_service_command(ack, command, respond): ack() service_name = command['text'] # Validate service name (whitelist) allowed_services = ['nginx', 'postgresql', 'redis'] if service_name not in allowed_services: respond(f"❌ Service '{service_name}' is not allowed. Allowed: {', '.join(allowed_services)}") return # Confirm with user respond(f"Restarting service: {service_name}...") try: # Execute restart command result = subprocess.run( ['ansible', 'webservers', '-m', 'service', '-a', f'name={service_name} state=restarted'], capture_output=True, text=True, timeout=30 ) if result.returncode == 0: respond(f"✅ Service {service_name} restarted successfully\n```{result.stdout}```") else: respond(f"❌ Failed to restart {service_name}\n```{result.stderr}```") except Exception as e: respond(f"❌ Error: {str(e)}") # Command: check server health @app.command("\/health-check") def health_check_command(ack, command, respond): ack() hostname = command['text'] or 'all' respond(f"Running health check on {hostname}...") try: result = subprocess.run( ['ansible', hostname, '-m', 'shell', '-a', '\/opt\/scripts\/health_check.sh'], capture_output=True, text=True, timeout=60 ) respond(f"```{result.stdout}```") except Exception as e: respond(f"❌ Error: {str(e)}") # Interactive approval flow @app.action("approve_deployment") def handle_approval(ack, body, client): ack() user = body["user"]["id"] deployment_id = body["actions"][0]["value"] # Execute deployment result = execute_deployment(deployment_id) client.chat_postMessage( channel=body["channel"]["id"], text=f"<@{user}> approved deployment {deployment_id}. Status: {result}" ) def execute_deployment(deployment_id): # Deployment logic here return "Success" if __name__ == "__main__": handler = SocketModeHandler(app, "xapp-your-app-token") handler.start() ``` ## Self-Healing Systems ### Auto-Remediation Framework ```python # Self-healing automation framework import time import logging from prometheus_client import start_http_server, Counter, Gauge import requests logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) # Metrics remediation_counter = Counter('remediations_total', 'Total remediations', ['type', 'status']) system_health = Gauge('system_health', 'System health status (1=healthy, 0=unhealthy)') class SelfHealingAgent: """Auto-remediation agent that monitors and fixes common issues""" def __init__(self): self.checks = [ self.check_disk_space, self.check_service_health, self.check_memory_usage, ] def check_disk_space(self): """Check disk space and clean up if needed""" import shutil stat = shutil.disk_usage('\/') usage_pct = (stat.used \/ stat.total) * 100 if usage_pct > 85: logger.warning(f"Disk usage high: {usage_pct:.1f}%") self.cleanup_disk() remediation_counter.labels(type='disk_cleanup', status='executed').inc() return False return True def cleanup_disk(self): """Clean up disk space""" import subprocess logger.info("Cleaning up disk space...") # Clean apt cache subprocess.run(['apt-get', 'clean'], check=False) # Clean old logs subprocess.run(['find', '\/var\/log', '-name', '*.gz', '-mtime', '+30', '-delete'], check=False) # Clean temp files subprocess.run(['find', '\/tmp', '-mtime', '+7', '-delete'], check=False) logger.info("Disk cleanup complete") def check_service_health(self): """Check if critical services are running""" import subprocess services = ['nginx', 'postgresql'] all_healthy = True for service in services: result = subprocess.run( ['systemctl', 'is-active', service], capture_output=True, text=True ) if result.returncode != 0: logger.warning(f"Service {service} is not running") self.restart_service(service) all_healthy = False return all_healthy def restart_service(self, service_name): """Restart a failed service""" import subprocess logger.info(f"Restarting service: {service_name}") try: subprocess.run(['systemctl', 'restart', service_name], check=True, timeout=30) logger.info(f"Service {service_name} restarted successfully") remediation_counter.labels(type='service_restart', status='success').inc() except Exception as e: logger.error(f"Failed to restart {service_name}: {e}") remediation_counter.labels(type='service_restart', status='failed').inc() def check_memory_usage(self): """Check memory usage and kill memory hogs if needed""" import psutil mem = psutil.virtual_memory() usage_pct = mem.percent if usage_pct > 90: logger.warning(f"Memory usage critical: {usage_pct:.1f}%") self.kill_memory_hogs() remediation_counter.labels(type='memory_cleanup', status='executed').inc() return False return True def kill_memory_hogs(self): """Kill processes using excessive memory""" import psutil logger.info("Identifying memory hogs...") # Get processes sorted by memory usage processes = [] for proc in psutil.process_iter(['pid', 'name', 'memory_percent']): try: processes.append(proc.info) except (psutil.NoSuchProcess, psutil.AccessDenied): pass processes.sort(key=lambda x: x['memory_percent'], reverse=True) # Kill top memory consumer (with safety checks) for proc in processes[:3]: # Check top 3 # Don't kill critical processes if proc['name'] in ['systemd', 'init', 'sshd']: continue if proc['memory_percent'] > 20: # Using more than 20% memory logger.warning(f"Killing memory hog: {proc['name']} (PID {proc['pid']})") try: psutil.Process(proc['pid']).kill() break except Exception as e: logger.error(f"Failed to kill process: {e}") def run(self, interval=60): """Run continuous health checks""" logger.info("Starting self-healing agent...") while True: try: all_healthy = True for check in self.checks: if not check(): all_healthy = False system_health.set(1 if all_healthy else 0) time.sleep(interval) except Exception as e: logger.error(f"Error in health check loop: {e}") time.sleep(interval) if __name__ == '__main__': # Start Prometheus metrics server start_http_server(9101) # Run agent agent = SelfHealingAgent() agent.run(interval=60) ``` ## Toil Reduction ### Measuring Toil ```yaml Toil Definition (Google SRE): - Manual: Requires human intervention - Repetitive: Done over and over - Automatable: Can be automated - Tactical: Interrupt-driven, reactive - No enduring value: Doesn't improve system - Scales linearly: Grows with service Examples of Toil: - Manually restarting services - Copying files between servers - Running deployment scripts manually - Responding to recurring alerts - Manual user provisioning - Generating reports manually Target: < 50% of time on toil (Google SRE recommendation) ``` ### Toil Reduction Strategy ```yaml Step 1: Identify Toil (Weekly) - Track time spent on tasks - Categorize: Toil vs Engineering work - Prioritize by time impact Tool: Time tracking spreadsheet Columns: Task, Time Spent, Frequency, Automatable?, Priority Step 2: Quantify Impact - Calculate time spent per month - Estimate automation effort - Calculate ROI (see ROI calculator above) Step 3: Automate High-Impact Toil - Start with highest ROI - Build automation incrementally - Test thoroughly - Document automation Step 4: Measure Improvement - Track toil percentage monthly - Celebrate wins - Share automations across team Step 5: Prevent New Toil - Question new manual processes - Design for automation from start - Code review includes automation check ``` This comprehensive automation guide provides everything needed to reduce toil and improve operational efficiency through automation. ## Usage Invoke this skill with: ``` $automation [arguments] ``` Or let Codex auto-select based on your prompt.