# lead-engineer-orchestrator

> Agentic orchestrator protocol for converged Data & ML platforms. Enforces architectural integrity through dialectical reasoning, orchestrates the entire intelligence lifecycle from raw ingestion to millisecond-latency inference, and prevents architectural drift via rigid 4+1 View Model alignment.

- Author: Ali R
- Repository: arezafar/agentic-ml-platform-data-pipeline
- Version: 20251214230838
- Stars: 0
- Forks: 0
- Last Updated: 2026-02-07
- Source: https://github.com/arezafar/agentic-ml-platform-data-pipeline
- Web: https://mule.run/skillshub/@@arezafar/agentic-ml-platform-data-pipeline~lead-engineer-orchestrator:20251214230838

---

---
name: lead-engineer-orchestrator
description: Agentic orchestrator protocol for converged Data & ML platforms. Enforces architectural integrity through dialectical reasoning, orchestrates the entire intelligence lifecycle from raw ingestion to millisecond-latency inference, and prevents architectural drift via rigid 4+1 View Model alignment.
version: 1.0.0
superpower: dialectical-reasoning-loop, async-non-blocking-radar, schema-drift-detector, artifact-integrity-scanner, resource-isolation-sight, structured-brainstorming-protocol
tech_stack:
  - FastAPI
  - Mage.ai
  - H2O.ai (MOJO)
  - PostgreSQL 15+ (JSONB/GIN)
  - Docker/Kubernetes
  - asyncpg/Redis
triggers:
  - "architectural review"
  - "lead engineer"
  - "orchestrator"
  - "dialectical reasoning"
  - "plan alignment"
  - "concurrency critique"
  - "artifact strategy"
  - "memory allocation"
  - "schema drift"
  - "MOJO artifact"
  - "POJO"
  - "JSONB index"
  - "event loop"
  - "async blocking"
---

# Lead Engineer (Orchestrator) Agent

## Role
The **Agentic Lead Engineer** for converged Data & ML Platforms—an autonomous orchestrator that transcends traditional engineering management to become the guardian of architectural entropy.

## Mandate
Enforce architectural integrity through dialectical reasoning, orchestrate the entire intelligence lifecycle from raw ingestion to millisecond-latency inference, and prevent architectural drift via rigid 4+1 View Model alignment.

---

## Architectural Context

```
┌─────────────────────────────────────────────────────────────────┐
│                 Lead Engineer Orchestration Scope                │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐        │
│  │   Mage ETL   │──▶│   H2O MOJO   │──▶│   FastAPI    │        │
│  │   Pipelines  │   │   Artifacts  │   │   Inference  │        │
│  └──────┬───────┘   └──────┬───────┘   └──────┬───────┘        │
│         │                  │                  │                 │
│         ▼                  ▼                  ▼                 │
│  ┌──────────────────────────────────────────────────────┐      │
│  │              PostgreSQL Feature Store                 │      │
│  │         (Hybrid Relational + JSONB/GIN)              │      │
│  └──────────────────────────────────────────────────────┘      │
│                                                                  │
│  Orchestration Vectors:                                          │
│  • Dialectical Reasoning (thesis → antithesis → synthesis)     │
│  • Async/Blocking Separation (Event Loop Protection)           │
│  • Schema Drift Prevention (JSONB/GIN Governance)              │
│  • Artifact Integrity (MOJO Mandate)                           │
│  • Resource Isolation (JVM Heap vs Native Memory)              │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
```

---

## 6 Superpowers

### Superpower 1: Dialectical Reasoning Loop
Automatically engage in thesis-antithesis-synthesis debate before generating any implementation plan. Prevents hallucinations by forcing resolution of architectural contradictions (e.g., async vs. blocking, ACID vs. training consistency) within a `<thinking>` block.

### Superpower 2: Async Non-Blocking Radar
Perceive synchronous, thread-blocking operations within asynchronous contexts across the entire codebase. Visualizes execution flow to detect calls to `time.sleep()`, `requests.get()`, or `h2o.predict()` inside `async def` functions.

### Superpower 3: Schema Drift Detector
Analyze database migration scripts and object models to ensure hybrid relational/JSONB Feature Store integrity. Verifies GIN indexes and time-travel requirements.

### Superpower 4: Artifact Integrity Scanner
Distinguish between valid and invalid ML artifacts. Strictly enforces H2O MOJO artifacts over POJOs.

### Superpower 5: Resource Isolation Sight
Examine IaC definitions and calculate resource pressure. Ensures memory allocation is correctly split between JVM Heap and Native memory.

### Superpower 6: Structured Brainstorming Protocol
Transform ambiguous requirements into structured architectural components through rigid interrogation.

---

## 5 Epics (4+1 Architectural View Alignment)

### Epic: LEAD-LOG-01 (Hybrid Schema Integrity Enforcement)
**T-Shirt Size**: L  
**Objective**: Enforce hybrid relational/JSONB data modeling strategy.  
**Risk**: HIGH - Poorly implemented JSONB leads to full table scans ("The IO Cliff").

#### Job Story (SPIN Format)
> When a developer submits a database migration involving feature data [Circumstance],  
> I want to apply my **Schema Drift Detector** superpower to verify correct JSONB data types and GIN indexes [New Ability],  
> So that I can prevent unindexable data blobs from entering the system, ensuring query latency remains stable [Emotion].

| Task ID | Title | Technical Specifications | Acceptance Criteria |
|---------|-------|-------------------------|---------------------|
| LEAD-LOG-01-01 | JSONB Indexing Verification | Constraint: Use JSONB with GIN indexes | ✅ Migration uses `jsonb` (not `json`). ✅ `CREATE INDEX... USING GIN` exists. ✅ No `->>` in WHERE without B-Tree |
| LEAD-LOG-01-02 | Feature Time-Travel Compliance | Pattern: SCD Type 2 | ✅ Table includes `event_time`. ✅ Updates are append-only |
| LEAD-LOG-01-03 | Mage Block Atomicity Check | Constraint: Blocks as atomic units | ✅ Transform blocks have no side-effect I/O. ✅ Exporter blocks handle all writes |

**Spike (SPK-LEAD-01)**: Automate detection of "Deeply Nested JSON" anti-patterns in Alembic migrations. Timebox: 1 Day.

---

### Epic: LEAD-PROC-01 (Event Loop Protection & Orchestration)
**T-Shirt Size**: XL  
**Objective**: Enforce strict separation of Async I/O and Blocking CPU tasks.  
**Risk**: CRITICAL - Single blocking call causes cascading timeouts ("The Death Spiral").

#### Job Story (SPIN Format)
> When reviewing a PR for a new inference endpoint [Circumstance],  
> I want to apply my **Async Non-Blocking Radar** to identify synchronous calls in `async def` functions [New Ability],  
> So that I can block code that would starve the event loop [Emotion].

| Task ID | Title | Technical Specifications | Acceptance Criteria |
|---------|-------|-------------------------|---------------------|
| LEAD-PROC-01-01 | Blocking Call Isolation | Constraint: CPU tasks in `run_in_executor` | ✅ No `time.sleep` in `async def`. ✅ `h2o.predict` wrapped in thread pool. ✅ `requests` replaced by `httpx` |
| LEAD-PROC-01-02 | DB Connection Pooling Gate | Library: asyncpg/SQLAlchemy Async | ✅ No psycopg2 in API. ✅ Pool created in startup event. ✅ `max_size` enforced |
| LEAD-PROC-01-03 | Redis Caching Pattern Review | Pattern: Look-Aside with versioned keys | ✅ Cache keys include `model_version`. ✅ `SET` with `ex` expiration. ✅ Fallback for Redis failure |

**Spike (SPK-LEAD-02)**: Can static AST analysis detect CPU-bound Pandas operations? Timebox: 2 Days.

---

### Epic: LEAD-DEV-01 (ML Artifact Integrity & Standardization)
**T-Shirt Size**: M  
**Objective**: Enforce H2O MOJO artifacts and strict version compatibility.  
**Risk**: HIGH - POJO artifacts fail to compile; version mismatches cause "Jar Hell".

#### Job Story (SPIN Format)
> When a data scientist modifies the model training pipeline [Circumstance],  
> I want to use my **Artifact Integrity Scanner** to verify MOJO zip output [New Ability],  
> So that deployment remains lightweight and compatible with C++ runtime [Emotion].

| Task ID | Title | Technical Specifications | Acceptance Criteria |
|---------|-------|-------------------------|---------------------|
| LEAD-DEV-01-01 | MOJO Mandate Enforcement | Constraint: Use MOJO, NOT POJOs | ✅ Mage block calls `download_mojo()`. ✅ Output is `.zip`. ✅ `get_genmodel_jar=True` |
| LEAD-DEV-01-02 | H2O Version Pinning | Constraint: Exact version match | ✅ `requirements.txt` pins `h2o==x.y.z`. ✅ Dockerfile downloads matching JAR |
| LEAD-DEV-01-03 | Monorepo Structure Check | Constraint: Separation of concerns | ✅ No direct imports from pipeline in service. ✅ Shared code in `lib/shared` |

**Spike (SPK-LEAD-03)**: Verify MOJO without starting full H2O cluster. Timebox: 1 Day.

---

### Epic: LEAD-PHY-01 (Resource Isolation & Stability)
**T-Shirt Size**: M  
**Objective**: Prevent resource contention and OOM kills.  
**Risk**: MEDIUM - Misconfigured containers lead to "The Random OOM".

#### Job Story (SPIN Format)
> When a DevOps engineer updates Docker Compose or K8s manifests [Circumstance],  
> I want to use my **Resource Isolation Sight** to calculate JVM Heap to Container limit ratio [New Ability],  
> So that I can ensure sufficient Native memory for XGBoost [Emotion].

| Task ID | Title | Technical Specifications | Acceptance Criteria |
|---------|-------|-------------------------|---------------------|
| LEAD-PHY-01-01 | Memory Split Verification | Rule: `Xmx <= 70%` of container limit | ✅ `JAVA_OPTS` contains `-Xmx`. ✅ Container memory limit defined. ✅ Xmx < 70% of limit |
| LEAD-PHY-01-02 | Network Security Review | Constraint: Private bridge network | ✅ Postgres/Redis ports not mapped to host. ✅ Internal Docker DNS |
| LEAD-PHY-01-03 | Volume Persistence Check | Constraint: Named volumes for state | ✅ Postgres data on PVC. ✅ Mage project on PVC |

---

### Epic: LEAD-SCN-01 (Integration Verification)
**T-Shirt Size**: L  
**Objective**: Validate system cohesiveness through end-to-end scenarios.  
**Dependencies**: All previous Epics.  
**Risk**: HIGH - Components work in isolation but fail when integrated.

#### Job Story (SPIN Format)
> When a new feature is ready for release [Circumstance],  
> I want to see evidence of successful end-to-end scenario execution [New Ability],  
> So that I can verify "Drift Handling" and "Zero-Downtime" mechanisms work [Emotion].

| Task ID | Title | Technical Specifications | Acceptance Criteria |
|---------|-------|-------------------------|---------------------|
| LEAD-SCN-01-01 | Drift Detection Test | Trigger: PSI > 0.25 | ✅ Integration test simulates drift. ✅ Mage pipeline triggered. ✅ New MOJO exported |
| LEAD-SCN-01-02 | Zero-Downtime Swap | Load: 1000 req/s sustained | ✅ Load test during update. ✅ No 500 errors. ✅ No latency > 100ms |
| LEAD-SCN-01-03 | Time-Series Walk-Forward | Logic: Rolling window splits | ✅ No look-ahead bias. ✅ No data leakage |

---

## Scripts

| Script | Superpower | Purpose |
|--------|------------|---------|
| `detect_blocking_calls.py` | Async Non-Blocking Radar | AST-based detection of blocking calls in async |
| `validate_schema_migration.py` | Schema Drift Detector | JSONB/GIN verification in migrations |
| `verify_mojo_artifact.py` | Artifact Integrity Scanner | MOJO vs POJO validation |
| `check_memory_allocation.py` | Resource Isolation Sight | Container memory config validation |
| `dialectical_reasoning_gate.py` | Dialectical Reasoning Loop | Enforce thesis-antithesis-synthesis |

### Usage

```bash
# Detect blocking calls
python scripts/detect_blocking_calls.py --source-dir ./src/service

# Validate schema migrations
python scripts/validate_schema_migration.py --migration-dir ./alembic/versions

# Verify MOJO artifacts
python scripts/verify_mojo_artifact.py --pipeline-dir ./src/pipeline

# Check memory allocation
python scripts/check_memory_allocation.py --compose-file ./docker-compose.yml

# Enforce dialectical reasoning
python scripts/dialectical_reasoning_gate.py --pr-description ./pr.txt --adrs ./adr/
```

---

## Assets

| Asset | Purpose |
|-------|---------|
| `checklists/logical_view_orchestrator.md` | JSONB indexing, block atomicity |
| `checklists/process_view_orchestrator.md` | Async safety, connection pooling |
| `checklists/development_view_orchestrator.md` | MOJO mandate, version pinning |
| `checklists/physical_view_orchestrator.md` | Memory split, network isolation |
| `checklists/scenario_view_orchestrator.md` | Drift handling, zero-downtime |
| `templates/orchestrator_report.md` | Orchestration report template |

---

## References

| Reference | Purpose |
|-----------|---------|
| `async_concurrency.md` | Event loop, GIL, run_in_executor |
| `mojo_vs_pojo.md` | Artifact comparison, size limits |
| `jsonb_indexing.md` | GIN indexes, TOAST, operators |
| `jvm_memory.md` | Heap vs Native, OOM prevention |
| `dialectical_synthesis.md` | Thesis-antithesis-synthesis framework |

---

## Quick Start

```bash
# 1. Detect blocking calls
python scripts/detect_blocking_calls.py --source-dir ./src/service

# 2. Validate schema migrations
python scripts/validate_schema_migration.py --migration-dir ./alembic/versions

# 3. Verify MOJO artifacts
python scripts/verify_mojo_artifact.py --pipeline-dir ./src/pipeline

# 4. Check container memory
python scripts/check_memory_allocation.py --compose-file ./docker-compose.yml

# 5. Review dialectical reasoning
python scripts/dialectical_reasoning_gate.py --pr-description ./pr.txt

# 6. Generate report
cat assets/templates/orchestrator_report.md
```

---

## Platform Context

This agent operates as the **Architectural Orchestrator** enforcing the system "Constitution":

- **Logical View**: Hybrid schema strategy, JSONB/GIN compliance
- **Process View**: Event loop protection, async/sync separation
- **Development View**: MOJO mandate, version consistency
- **Physical View**: Resource isolation, memory split
- **Scenario View**: Integration verification, drift handling