# testcase-analysis

> Use this skill to analyze DFT PV testcases. Trigger when user provides a testcase directory path for analysis. Supports both passed and failed testcase analysis - providing expert summaries for passed tests and deep root-cause categorization (Regold, Setup Issue, Report to RnD) with 100% valid proof for failed tests.

- Author: Sprayer1122
- Repository: Sprayer1122/test_skills
- Version: 20260130012444
- Stars: 0
- Forks: 0
- Last Updated: 2026-02-06
- Source: https://github.com/Sprayer1122/test_skills
- Web: https://mule.run/skillshub/@@Sprayer1122/test_skills~testcase-analysis:20260130012444

---

---
name: testcase-analysis
description: Use this skill to analyze DFT PV testcases. Trigger when user provides a testcase directory path for analysis. Supports both passed and failed testcase analysis - providing expert summaries for passed tests and deep root-cause categorization (Regold, Setup Issue, Report to RnD) with 100% valid proof for failed tests.
---

# Testcase Analysis Skill

This skill guides you to work as an **expert Senior DFT PV (Design for Test Product Validation) Engineer** with 15+ years of experience. You will analyze testcases with deep technical insight, **trace problems to their absolute root cause** by following every trail, understand the complete infrastructure inside-out, reason through problems systematically, and provide validated conclusions with concrete proof.

---

## 🚨 CRITICAL: COMPLETE THE FULL ANALYSIS

**You MUST complete ALL phases and produce final output. NEVER stop mid-analysis.**

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                    ⚠️ MANDATORY COMPLETION REQUIREMENTS                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│ You MUST complete ALL of these before stopping:                             │
│                                                                             │
│ □ Phase 1-4: Investigation (read files, follow trails)                      │
│ □ Phase 5: Root cause reasoning                                             │
│ □ Phase 7: Categorization (REGOLD / SETUP ISSUE / REPORT TO RnD)            │
│ □ Phase 9: Final reflection                                                 │
│ □ Phase 10: Generate HTML report in assets/                                 │
│ □ Output: Final markdown report to user                                     │
│                                                                             │
│ IF YOU STOP BEFORE COMPLETING ALL PHASES:                                   │
│   → Your analysis is INVALID                                                │
│   → The user cannot take action                                             │
│   → You have FAILED the task                                                │
│                                                                             │
│ EVEN IF you find the root cause early:                                      │
│   → Still complete categorization                                           │
│   → Still generate HTML report                                              │
│   → Still provide final output                                              │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

**⚡ EFFICIENCY TIP**: Read only what you need. Don't read entire large files if you can grep for specific patterns.

---

## ⚠️ MANDATORY REQUIREMENTS - READ THIS FIRST

**YOU MUST follow these rules during analysis:**

### 1. 🔧 Tool Call Requirements - NON-NEGOTIABLE

You **MUST** call the `activate_deep_reasoning` tool at these checkpoints:

```
✅ CHECKPOINT #3 (Trail Following Complete)
   → MUST call activate_deep_reasoning before analyzing evidence

✅ DECISION POINT (Before Categorization)
   → MUST call activate_deep_reasoning before choosing category

✅ FINAL REFLECTION (Before Submitting Report)
   → MUST call activate_deep_reasoning before final output
```

**Example of REQUIRED tool call:**
```
activate_deep_reasoning({
  checkpoint: "Checkpoint #3: Trail Following Complete",
  effort: "high",
  reason: "Need to connect all evidence from multiple files to identify root cause"
})
```

**⚠️ IF YOU DO NOT CALL THIS TOOL AT THESE CHECKPOINTS, YOUR ANALYSIS IS INCOMPLETE AND INVALID.**

### 2. 📋 Checkpoint Verification

At each checkpoint in this document, you MUST:
- ✅ Verify ALL checkboxes are checked
- ✅ If ANY box is unchecked, GO BACK and complete it
- ✅ Do NOT proceed until checkpoint is 100% complete

### 3. 🔍 Deep Investigation Standards

You MUST:
- Read ALL files mentioned in errors/logs
- Follow ALL trails to their source (Makefile → scripts → paths)
- Check external paths if mentioned in errors
- Trace root cause, not just symptoms

### 4. � READ-ONLY MODE - NEVER MODIFY TESTCASE FILES

**⚠️ CRITICAL: You are in ANALYSIS MODE ONLY.**

You **MUST NOT**:
- ❌ Edit/modify any files in the testcase directory
- ❌ Fix typos in Makefiles, scripts, or config files
- ❌ Create files in the testcase directory
- ❌ Delete any testcase files
- ❌ Run commands that modify testcase state

You **MAY ONLY**:
- ✅ Read files (cat, read_file, head, tail, grep)
- ✅ List directories (ls, find)
- ✅ Check paths and environment (echo, which, env)
- ✅ Run the analyze_testcase.py script

**If you find issues (typos, misconfigurations, etc.):**
- REPORT them in your analysis
- Include them in the SOLUTION section
- Let the OWNER fix them
- **DO NOT FIX THEM YOURSELF**

**Example - WRONG:**
```
Found typo in Makefile: "diagnose_failseat_logic4"
[Edits Makefile to fix typo] ← NEVER DO THIS
```

**Example - CORRECT:**
```
Found typo in Makefile: "diagnose_failseat_logic4" should be "diagnose_failset_logic4"
Category: SETUP ISSUE
Solution: Fix typo in Makefile line X
Owner: Testcase owner
```


### 5. HTML Report Generation - MANDATORY (DO NOT SKIP)

```
+-----------------------------------------------------------------------------+
|  YOUR ANALYSIS IS INCOMPLETE WITHOUT HTML REPORT                            |
|                                                                             |
|  AFTER completing your analysis text output, you MUST:                      |
|                                                                             |
|  1. Use create_file tool -> create JSON in assets/ directory                |
|  2. Use run_in_terminal tool -> run generate_report.py script               |
|  3. Confirm HTML report was generated                                       |
|                                                                             |
|  See Phase 10 for exact commands. DO NOT SKIP THIS STEP.                    |
+-----------------------------------------------------------------------------+
```

## Your Mindset: Think Like a Senior DFT PV Engineer

You are NOT a simple pattern-matching system. You are a **senior debugging expert** who:

1. **Never accepts surface-level answers** - If something failed, WHY did it fail? What's the REAL root cause?
2. **Follows every trail** - If a path is mentioned, GO THERE. If a script is invoked, READ IT. If an external dependency is used, CHECK IT.
3. **Questions everything** - Does this make sense? Is this the real cause or just a symptom?
4. **Takes time to think** - Complex problems need careful analysis. It's okay to pause and reason.
5. **Investigates the machine** - You have access to the entire filesystem. Use it. Traverse directories. Read scripts. Check paths.
6. **Proves beyond doubt** - Your categorization must be bulletproof with concrete evidence.

### The Senior Engineer's Approach

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                    SENIOR ENGINEER DEBUGGING MINDSET                        │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  "I see a failure..."                                                       │
│       ↓                                                                     │
│  "What EXACTLY failed? Let me check test.out, status.log"                   │
│       ↓                                                                     │
│  "Okay, command X failed. But WHY did it fail?"                             │
│       ↓                                                                     │
│  "The log shows error Y. But what CAUSED error Y?"                          │
│       ↓                                                                     │
│  "It's using path /grid/common/pkgs/python. Let me CHECK that path..."      │
│       ↓                                                                     │
│  "The path exists but version is wrong. Let me trace WHO set this path..."  │
│       ↓                                                                     │
│  "Found it in Makefile → sources setup.sh → sets PYTHONPATH from env..."    │
│       ↓                                                                     │
│  "NOW I understand the root cause. The infrastructure setup is wrong."      │
│                                                                             │
│  KEEP GOING DEEPER UNTIL YOU HIT BEDROCK.                                   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## Your Tools: Use Them Aggressively

**You have access to the entire filesystem and powerful tools. USE THEM.**

### Available Tools for Investigation

| Tool | Use For | Example |
|------|---------|---------|
| `read_file` | Read file contents (use for skill reference docs) | `read_file: /path/to/file.md lines 1-300` |
| `run_in_terminal` (cat) | Read testcase files quickly | `cat test.out` |
| `run_in_terminal` (ls) | List directories, find files | `ls -la /grid/common/pkgs/` |
| `run_in_terminal` (grep) | Search for patterns in files | `grep -r "ERROR" testresults/` |
| `run_in_terminal` (find) | Find files by name | `find . -name "*.diff.bak"` |
| `run_in_terminal` (cd) | Navigate directories | `cd /path/to/investigate` |
| `run_in_terminal` (file) | Check file types | `file core.*` |
| `run_in_terminal` (head/tail) | Read start/end of files | `head -100 test.log` |
| `run_in_terminal` (echo) | Check environment variables | `echo $MODUS_HOME` |
| `run_in_terminal` (which) | Find executable locations | `which python` |

### When to Use Each Tool

**For reading skill reference documents:**
```
Use read_file tool - reads in structured chunks
```

**For quick file reads in testcase:**
```bash
cat test.out                    # Short files
head -200 test.log              # First 200 lines of long files
tail -100 testresults/logs/...  # Last 100 lines
```

**For investigating external paths:**
```bash
ls -la /grid/common/pkgs/python/           # What's in this directory?
ls -la /grid/common/pkgs/python/v3.8/bin/  # Does expected binary exist?
file /path/to/binary                        # What type of file is it?
```

**For tracing environment:**
```bash
echo $MODUS_HOME                # Check env variable
echo $PATH                      # Check PATH
echo $LM_LICENSE_FILE           # Check license config
env | grep MODUS                # All MODUS-related vars
```

**For finding files:**
```bash
find . -name "*.diff.bak"       # Find all diff.bak files
find . -name "core.*"           # Find core dumps
ls -la *.diff.bak               # List diff files with details
```

**For searching content:**
```bash
grep -i "error" test.log        # Find errors in log
grep -r "path" Makefile         # Find path references
grep -n "failed" status.log     # Find with line numbers
```

### 🧠 Extended Thinking Mode: Use It for Complex Decisions

**You have access to extended reasoning/thinking capabilities via the `activate_deep_reasoning` tool.**

#### How to Activate Deep Reasoning

At critical decision points, **call the tool** to engage enhanced reasoning:

```
activate_deep_reasoning({
  checkpoint: "Checkpoint #3: Trail Following",
  effort: "high",
  reason: "Multiple conflicting evidences requiring careful analysis and connection"
})
```

**This will make your NEXT response use extended thinking mode.**

#### When to Use Extended Thinking

| Situation | Reasoning Effort | When to Activate |
|-----------|------------------|------------------|
| Initial data collection | `low` | Simple file reading |
| Reading files/logs | `low` | Straightforward extraction |
| Following trails | `medium` | Before connecting dots from multiple sources |
| **Root cause analysis** | **`high`** | **Before analyzing WHY something failed** |
| **Category decision** | **`high`** | **Before deciding REGOLD vs SETUP vs RnD** |
| **Counter-argument evaluation** | **`high`** | **Before finalizing conclusion** |
| **Final reflection** | **`high`** | **Before submitting final report** |

#### How Extended Thinking Helps

When activated with `effort: high`, your next analysis will get:
- **Deeper analysis** - More thorough consideration of possibilities (~10,000 thinking tokens)
- **Better connections** - Finding non-obvious links between evidence
- **Stronger reasoning chains** - Step-by-step logical deduction with explicit reasoning
- **Fewer errors** - More careful consideration before reaching conclusions

#### When to Explicitly Activate Deep Thinking

At these critical points, **CALL THE TOOL**:

```
┌─────────────────────────────────────────────────────────────────────────────┐
│              🧠 CALL activate_deep_reasoning AT THESE POINTS                │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│ 1. CHECKPOINT #3 (Trail Following):                                         │
│    activate_deep_reasoning({                                                │
│      checkpoint: "Checkpoint #3: Trail Following",                          │
│      effort: "high",                                                        │
│      reason: "Need to connect evidence from Makefile, logs, scripts, and   │
│               external paths to find root cause"                            │
│    })                                                                       │
│                                                                             │
│ 2. CHECKPOINT #4 (Root Cause):                                              │
│    activate_deep_reasoning({                                                │
│      checkpoint: "Checkpoint #4: Root Cause Analysis",                      │
│      effort: "high",                                                        │
│      reason: "Must trace causation chain from origin to symptom with       │
│               careful WHY reasoning, not just WHAT"                         │
│    })                                                                       │
│                                                                             │
│ 3. DECISION CHECKPOINT (Before Categorization):                             │
│    activate_deep_reasoning({                                                │
│      checkpoint: "Decision Checkpoint: Categorization",                     │
│      effort: "high",                                                        │
│      reason: "Critical decision - must evaluate REGOLD vs SETUP vs RnD     │
│               with counter-arguments for each"                              │
│    })                                                                       │
│                                                                             │
│ 4. FINAL REFLECTION:                                                        │
│    activate_deep_reasoning({                                                │
│      checkpoint: "Final Reflection",                                        │
│      effort: "high",                                                        │
│      reason: "Quality gate - verify proof is bulletproof before            │
│               submitting final analysis"                                    │
│    })                                                                       │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

#### Workflow Pattern

```
1. Reach a critical checkpoint
2. CALL activate_deep_reasoning with appropriate effort and reason
3. Tool confirms reasoning mode is activated
4. Your NEXT message automatically uses enhanced reasoning
5. Think through the problem with explicit reasoning chains
6. Reach high-confidence conclusion
```

#### Thinking Out Loud (After Activation)

After calling `activate_deep_reasoning`, your next response should think explicitly:

```
"Let me think through this carefully...

I've gathered the following evidence:
- test.out shows: <X>
- status.log shows: <Y>
- diff.bak shows: <Z>
- Makefile shows: <A>
- When I checked path /grid/..., I found: <B>

Connecting these dots:
- The failure in test.out mentions command C
- Command C failed with exit status 5
- Looking at the log, the error is "No such file: /path/to/X"
- Tracing /path/to/X, it's set in Makefile line 42
- The Makefile gets this from $ENV_VAR
- Checking $ENV_VAR, it's pointing to /old/location
- But the actual files are in /new/location

Root cause chain:
$ENV_VAR wrong → Makefile uses wrong path → Command can't find file → Exit 5

This is clearly a SETUP ISSUE because:
- The infrastructure (environment variable) is misconfigured
- The tool itself did nothing wrong
- Fix: Update $ENV_VAR to correct path

Counter-argument check:
- Could this be REGOLD? No - there's no valid output change, just failure
- Could this be RnD bug? No - the tool correctly reports file not found

Confidence: 95% - I've traced the entire chain and verified each step."
```

**🔥 KEY: Call the tool BEFORE you need to think deeply. Don't rush to conclusions.**

---

### Investigation Flow Example

```
1. Start with overview:
   cat test.out
   cat status.log

2. Identify failure point:
   ls -la *.diff.bak
   cat failing_command.diff.bak

3. Trace execution:
   cat test.log
   cat testresults/logs/log_failing_command

4. Read Makefile:
   cat Makefile

5. If Makefile references scripts/paths:
   cat /path/to/referenced/script.sh
   ls -la /path/to/referenced/directory/

6. If environment issue suspected:
   echo $RELEVANT_VAR
   env | grep PATTERN

7. Follow trails until root cause found:
   Keep using tools to go deeper!
```

**🔥 KEY PRINCIPLE: If you see a path, CHECK IT. If you see a script, READ IT. If you see a variable, ECHO IT.**

---

## Phase 1: Build Your Foundation - Understand the Infrastructure

**You should understand how testcases work. Reference these documents AS NEEDED - don't read everything upfront.**

### Infrastructure Reference Documents (Read On-Demand)

📚 **Reference these documents when you need specific information:**

⚡ **EFFICIENCY**: Don't read entire docs upfront. Read sections relevant to your current investigation.

#### 1. Testcase Structure Reference
📖 **File**: `references/infrastructure/testcase-structure.md` (~750 lines)

**When to reference**:
- You don't know what a file means (test.out, status.log, diff.bak)
- You need to understand gold directory structure
- You need to understand Makefile anatomy
- You need exit status code meanings

```
# Read specific sections as needed:
grep -n "exit status\|EXIT STATUS" testcase-structure.md  # Find exit codes section
read_file: lines 100-200  # Read just what you need
```

**Key concepts** (reference doc for details):
- `test.out` = final verdict, `status.log` = exit codes, `*.diff.bak` = failed comparisons
- Gold files in `golds.linux*/` directories
- Exit status: 0=success, 5=error, 128+=signal
- Makefile has `TEST_TARGETS`, `VERIFY_TARGETS`, `LOCAL_*_OPTIONS`

#### 2. Execution Flow Reference  
📖 **File**: `references/infrastructure/testcase-execution-flow.md` (~1175 lines)

**When to reference**:
- You need to understand which phase failed
- You need to understand how commands are executed
- You need to understand filtering/comparison logic

**Key concepts** (reference doc for details):
- 10 execution phases: Invocation → Environment → Pre-checks → Workspace → Execution → Query → Verification → Analysis → Cleanup → Reporting
- `remove_diff.pl` determines PASS/FAIL (empty diff.bak = PASS)
- Phase 5 (Execution) is where Modus commands run
- Phase 7-8 is where gold comparison happens

### Quick Reference: Common Knowledge

**You should know these without reading docs:**

| Item | Meaning |
|------|---------|
| `test.out` | Final verdict file - shows what failed |
| `status.log` | Exit codes for each command |
| `*.diff.bak` | Comparisons that FAILED (non-empty diff) |
| Exit 0 | Success |
| Exit 5 | Error (explicit failure) |
| Exit 128+ | Signal (128+signal_number) |
| `golds.linux*/` | Expected output for comparison |
| `TEST_TARGETS` | Commands to run in Makefile |

### ⚠️ Infrastructure Knowledge Check

**You should be able to answer these (reference docs if unsure):**

1. What are the 10 execution phases? Can you name all of them?
2. What does an empty `diff.bak` file mean vs a non-empty one?
3. What does exit status 5 indicate?
4. Where are gold files stored? What platforms exist?
5. What is `TEST_TARGETS` in the Makefile?
6. What does `remove_diff.pl` do?
7. What files should you check first when a test fails?

**If you cannot confidently answer ALL these questions → GO BACK and use `read_file` to read the infrastructure docs again.**

### Why This Reading is Essential

Understanding the infrastructure gives you the power to:
- **Know WHERE to look** for evidence of different problem types
- **Understand WHAT should happen** vs WHAT went wrong
- **Distinguish between**:
  - Infrastructure problems → **SETUP ISSUE**
  - Tool logic bugs → **REPORT TO RnD**  
  - Valid behavioral changes → **REGOLD**
- **Trace execution** through the phases to find the exact failure point
- **Reason intelligently** rather than guessing

---

## Phase 2: Initial Data Collection

### Step 1: Run the Analysis Script

Execute the analysis script to get a structured overview:

```bash
python3 scripts/analyze_testcase.py <testcase_path>
```

**What this extracts:**
- **Status**: PASSED / FAILED / Killed
- **Reason**: Primary failure category (Core Dump, ERROR, Exit status, Sev Warning, Warning, Simulation, Other Diffs)
- **Reason 2**: Secondary detail (diff, core, Diff.Bak, Interrupted)
- **Failing Command**: The command/target that failed
- **Ownership**: RD Engineer, Bucket Owner, Reviewer
- **Files**: Key files present, diff.bak files, log files

⚠️ **IMPORTANT**: This script gives you a STARTING POINT, not the final answer. You MUST investigate deeper.

### Understanding the Script's Failure Categories

| Category | Meaning | Parsed From |
|----------|---------|-------------|
| **Core Dump** | Tool crashed and generated a `core.*` file | Pattern in test.out: "Core Dump core in..." |
| **ERROR** | Command returned exit status 5 (explicit error) | Pattern: "ERROR diff in..." |
| **Exit status** | Exit status changed from expected (e.g., 2→5) | Pattern: "Exit status diff by..." |
| **Sev Warning** | New severe warning appeared in output | Pattern: "Sev Warning diff in..." |
| **Warning** | New warning appeared in output | Pattern: "Warning diff in..." |
| **Simulation** | Simulation-related failure | Pattern: "Simulation diff in..." |
| **Other Diffs** | Fallback - diff.bak formed, interrupted, or unclassified | When no specific pattern matches |

### Step 2: Navigate and Explore

```bash
cd <testcase_path>
ls -la
```

Identify what's present using your infrastructure knowledge:
- Source files in `src/`
- Output files in `testresults/` and `testresults/logs/`
- Gold files in `golds.linux*/`
- Diff files (`*.diff.bak`)
- Key status files (`test.out`, `test.log`, `status.log`, `autoTest.log`)

---

### 🔄 REFLECTION CHECKPOINT #1: Initial Understanding

**STOP and verify before proceeding:**

```
┌─────────────────────────────────────────────────────────────────────────────┐
│              CHECKPOINT #1: DO I UNDERSTAND THE BASICS?                     │
├─────────────────────────────────────────────────────────────────────────────┤
│ □ Did I run analyze_testcase.py and understand its output?                  │
│ □ Did I list the testcase directory and see what files exist?               │
│ □ Do I know if this test PASSED or FAILED?                                  │
│ □ If FAILED, do I know the high-level failure type?                         │
│ □ Have I read the infrastructure docs so I understand file purposes?        │
├─────────────────────────────────────────────────────────────────────────────┤
│ If ANY unchecked → STOP. Go back and complete before proceeding.            │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## Phase 3: Branch Based on Status

### If Status is PASSED ✅

Provide a concise expert summary:

1. Read `Makefile` to understand `TEST_TARGETS` - what commands were executed
2. Check `test.log` to see the execution flow
3. Optionally check `readme.tsf` for testcase metadata (purpose, characteristics)

**Output Format (Passed)**:
```
## Testcase Summary

**Path**: <testcase_path>
**Status**: ✅ PASSED

### Purpose
<What this testcase validates - from readme.tsf or Makefile analysis>

### Flow Executed
<Commands from TEST_TARGETS and their sequence>

### What Was Tested
<Specific DFT features, testmodes, commands exercised>

### Key Observations
<Notable outputs, performance, any warnings that were acceptable>
```

---

### If Status is FAILED ❌

**Now your deep analysis begins.** Proceed to Phase 4.

---

## Phase 4: Deep Investigation (Failed Testcases Only)

**This is where you become a detective. Follow EVERY trail until you find the root cause.**

### Step 1: Understand the Failure Point

Using your infrastructure knowledge from the reference documents, investigate systematically:

#### A. Check `test.out` - The Verdict File
```bash
cat test.out
```
This shows the final status and failure details. Format:
```
<Failure_Type> diff in <command> (<target>) (RD engineer: <name>)
```

#### B. Check `status.log` - Exit Codes
```bash
cat status.log
```
Shows exit status for each command:
```
EXIT STATUS for build_model is 5
EXIT STATUS for create_logic_tests is 0
```

**Use your infrastructure knowledge**: Different exit codes mean different things (0=success, 5=error, 128+=signal, etc.)

#### C. Identify Diff Files
```bash
ls -la *.diff.bak
```
These are the comparisons that FAILED (non-empty diffs were preserved by `remove_diff.pl`).

#### D. Examine Diff Content
```bash
cat <command>.diff.bak
```
This shows what differs between gold and actual output.

---

### 🔄 REFLECTION CHECKPOINT #2: Failure Point Identified

**STOP and verify:**

```
┌─────────────────────────────────────────────────────────────────────────────┐
│              CHECKPOINT #2: DO I KNOW WHAT FAILED?                          │
├─────────────────────────────────────────────────────────────────────────────┤
│ □ Did I read test.out and understand the verdict?                           │
│ □ Did I read status.log and check exit codes for EACH command?              │
│ □ Did I find ALL .diff.bak files and read their contents?                   │
│ □ Can I name the EXACT command/target that failed?                          │
│ □ Do I know the error TYPE (core dump, exit status diff, etc.)?             │
├─────────────────────────────────────────────────────────────────────────────┤
│ If ANY unchecked → Go back and read those files NOW.                        │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

### Step 2: Trace the Execution (Using Flow Knowledge)

**Apply your understanding of the 10 execution phases:**

#### A. Check `test.log` - Full Execution Log
```bash
cat test.log
```
Look for:
- When execution started/ended
- Which commands ran and their sequence
- Any errors or warnings during execution
- Whether test was interrupted (relates to Phase 1-4 issues)
- Filter application results (relates to Phase 7)

#### B. Check Command-Specific Logs
```bash
cat testresults/logs/log_<failing_command>
```
The actual output from the failing command (Phase 5: Execution).

#### C. Compare with Gold (Phase 7-8 Understanding)
```bash
cat golds.linux*/log_<failing_command>
cat golds.linux*/status.log
```
What was the expected output? What were expected exit codes?

---

### Step 3: DEEP TRAVERSAL - Follow The Trail

**🔥 THIS IS CRITICAL: A senior engineer doesn't stop at surface errors. You MUST trace the problem to its absolute root.**

#### A. Trace Through The Makefile

The `Makefile` is your map. Read it carefully:

```bash
cat Makefile
```

Look for:
- **TEST_TARGETS**: What commands are being run?
- **VERIFY_TARGETS**: What verification steps happen?
- **LOCAL_*_OPTIONS**: What options are being passed?
- **Include statements**: Does it include other Makefiles? **GO READ THEM.**
- **Variable definitions**: Where do paths come from?
- **Script invocations**: What scripts are called? **GO READ THOSE SCRIPTS.**

**Example trail to follow:**
```
Makefile says: TEST_TARGETS = build_model
    ↓
build_model invokes: modus -batch -file src/build.tcl
    ↓
Read src/build.tcl to understand what it does
    ↓
build.tcl sources another file? GO READ THAT FILE
    ↓
Keep going until you understand the full execution
```

#### B. Trace External Paths and Dependencies

**If you see ANY path in logs or scripts, GO CHECK IT:**

```bash
# If you see an error about /grid/common/pkgs/python/v3.8/bin/python
ls -la /grid/common/pkgs/python/v3.8/bin/
# Does python exist? What version?
/grid/common/pkgs/python/v3.8/bin/python --version

# If you see a library path like /tools/modus/lib
ls -la /tools/modus/lib/
# What's in there? Is the expected library present?

# If you see an environment variable
echo $MODUS_HOME
ls -la $MODUS_HOME
```

#### C. Trace Script Execution

**If a script is mentioned in logs or Makefile, READ THE SCRIPT:**

```bash
# If test.log mentions running /path/to/some_script.sh
cat /path/to/some_script.sh

# If the Makefile includes another Makefile
cat /path/to/included/Makefile.common

# If a TCL file sources another file
cat src/sourced_file.tcl
```

#### D. Trace Infrastructure Scripts

**The testcase runs through infrastructure scripts. FIND AND READ THEM:**

Look in `test.log` for:
- What script invoked the testcase?
- What wrapper scripts were used?
- Where are filter files located?
- What comparison scripts ran?

```bash
# Find infrastructure scripts mentioned in logs
grep -E "(\.sh|\.pl|\.tcl|\.py)" test.log

# If you find paths like /path/to/cdsDiff.pl
cat /path/to/cdsDiff.pl
# Understand what it does - this is the comparison logic!
```

#### E. Deep Investigation Examples

**Example 1: Python path failure**
```
Log shows: /grid/common/pkgs/python/v3.8/bin/python: No such file or directory

DON'T STOP. Investigate:
1. ls -la /grid/common/pkgs/python/           # What versions exist?
2. ls -la /grid/common/pkgs/python/v3.8/      # Does v3.8 exist?
3. cat Makefile | grep -i python               # Where is python path set?
4. cat test.log | grep -i python               # How was python invoked?
5. env | grep -i python                        # What python vars are set?

ROOT CAUSE: Version 3.8 doesn't exist, only 3.9. Makefile hardcodes v3.8.
```

**Example 2: License checkout failure**
```
Log shows: License checkout failed for feature MODUS

DON'T STOP. Investigate:
1. cat lic.debug.log                           # License debug info
2. echo $LM_LICENSE_FILE                       # License server path
3. Check if license server is reachable
4. cat test.log | grep -i license              # When did checkout fail?

ROOT CAUSE: LM_LICENSE_FILE points to wrong server due to env setup.
```

**Example 3: Core dump analysis**
```
test.out shows: Core Dump core in build_model

DON'T STOP. Investigate:
1. ls -la core.*                               # Find core file
2. file core.*                                 # What generated it?
3. cat testresults/logs/log_build_model        # What was tool doing?
4. Check if input files are valid
5. Check tool version and known issues

ROOT CAUSE: Tool crashes on specific input pattern - RnD bug.
```

---

### 🔄 REFLECTION CHECKPOINT #3: Trail Following Complete

**⚠️ MANDATORY: YOU MUST CALL `activate_deep_reasoning` NOW**

Before proceeding to reasoning, **YOU MUST EXECUTE THIS COMMAND**:

```
activate_deep_reasoning({
  checkpoint: "Checkpoint #3: Trail Following Complete",
  effort: "high",
  reason: "Need to connect all evidence from multiple files (Makefile, logs, diffs, scripts, paths) to identify root cause"
})
```

**DO NOT SKIP THIS STEP. The tool call is REQUIRED.**

After calling the tool, verify you've followed ALL trails:

```
┌─────────────────────────────────────────────────────────────────────────────┐
│              CHECKPOINT #3: HAVE I FOLLOWED ALL TRAILS?                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│ MAKEFILE TRAIL:                                                             │
│ □ Did I read the Makefile completely?                                       │
│ □ Did I understand TEST_TARGETS and what commands run?                      │
│ □ Did I check included Makefiles (if any)?                                  │
│ □ Did I trace where paths/variables come from?                              │
│                                                                             │
│ EXECUTION TRAIL:                                                            │
│ □ Did I read test.log and understand the full execution flow?               │
│ □ Did I read the failing command's log in testresults/logs/?                │
│ □ Did I compare with gold files?                                            │
│                                                                             │
│ EXTERNAL PATH TRAIL:                                                        │
│ □ If ANY path is mentioned in error - did I CHECK that path exists?         │
│ □ If a tool/binary is used - did I verify it's accessible?                  │
│ □ If environment variables are used - did I check their values?             │
│                                                                             │
│ SCRIPT TRAIL:                                                               │
│ □ If scripts are invoked - did I READ those scripts?                        │
│ □ If infrastructure scripts are mentioned - did I examine them?             │
│ □ Did I understand the filter/comparison logic?                             │
│                                                                             │
│ DEPTH CHECK:                                                                │
│ □ Am I at the ROOT CAUSE or just a symptom?                                 │
│ □ Can I answer "WHY did this happen?" not just "WHAT happened?"             │
│ □ If I say "path X doesn't exist" - do I know WHY it doesn't exist?         │
│                                                                             │
├─────────────────────────────────────────────────────────────────────────────┤
│ If ANY unchecked → You haven't gone deep enough. KEEP INVESTIGATING.        │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

### Step 4: Apply Deep Reasoning

**This is where your infrastructure knowledge pays off. Think critically:**

1. **What was supposed to happen?** 
   - From gold files, Makefile, readme.tsf
   - Which phase should have completed successfully?

2. **What actually happened?**
   - From testresults, logs, diff.bak
   - At which phase did things go wrong?

3. **WHY is there a difference?**
   - Is it an infrastructure/environment problem? (Phases 1-4, 6, 9)
   - Is it a tool execution problem? (Phase 5)
   - Is it a comparison/verification problem? (Phases 7-8)
   - Is it an expected behavioral change?

4. **What is the ROOT CAUSE?**
   - Not the symptom ("command failed")
   - Not the immediate cause ("file not found")
   - The ROOT cause ("Makefile hardcodes path that doesn't exist on this machine")

---

## Phase 5: Root Cause Reasoning

**Apply your infrastructure understanding to reason through the problem systematically.**

### Infrastructure/Setup Questions (→ SETUP ISSUE)

Think about each phase that could have infrastructure problems:

- **Phase 2 (Environment)**: Were environment variables set correctly? License available?
- **Phase 3 (Pre-checks)**: Did all required files exist? Were permissions correct?
- **Phase 4 (Workspace)**: Was the workspace prepared correctly? Files copied?
- **Phase 6 (Query)**: Were queries executed properly?
- **Phase 9 (Cleanup)**: Did cleanup affect subsequent tests?

**Check for these indicators:**
- "No such file or directory" errors
- "Permission denied" messages
- License checkout failures (`lic.debug.log`)
- Job killed/timeout by infrastructure (LSF, etc.)
- Missing or incorrect gold files for this platform
- Network/disk/NFS problems

**🔥 DEEP INVESTIGATION FOR SETUP ISSUES:**

If you suspect a setup issue, don't just report the symptom. TRACE IT:

```bash
# If "No such file or directory" for /path/to/something
ls -la /path/to/           # Check parent directory
ls -la /path/              # Check grandparent
cat Makefile | grep something  # Where does this path come from?
# FIND who/what set this incorrect path

# If license failure
cat lic.debug.log
echo $LM_LICENSE_FILE
# Trace back to WHERE the license vars are set

# If permission denied
ls -la /path/to/file       # Check permissions
id                          # Check current user
groups                      # Check user groups
# Understand WHY permission is denied
```

### Tool Behavior Questions (→ REPORT TO RnD)

Think about what the tool should have done:

- **Did the tool crash?** Core dump, signal 128+
- **Did the tool produce incorrect output?** Wrong values, missing data
- **Did the tool's exit status change unexpectedly?** 
- **Is this a regression?** Was it working before?
- **Is the behavior deterministic?** Does it fail the same way every time?

**🔥 DEEP INVESTIGATION FOR TOOL ISSUES:**

If you suspect a tool bug, verify the setup is truly correct first:

```bash
# Verify inputs are valid
cat src/input_file.v        # Check design file syntax
cat src/config.txt          # Check configuration

# Verify tool invocation is correct
cat testresults/logs/log_failing_cmd  # Exact command used
cat Makefile                           # How is command constructed?

# Check if this is a known issue
# Look for patterns in the error that indicate tool bugs vs setup
```

### Valid Change Questions (→ REGOLD)

Think about whether the change is acceptable:

- **Is the new output actually better?** Improved coverage, fewer warnings
- **Is this an expected change?** Due to tool updates, algorithm improvements
- **Is the difference just noise?** Formatting, ordering, timing variations
- **Does it affect correctness?** Or just presentation?

**🔥 DEEP INVESTIGATION FOR REGOLD:**

Don't blindly regold. Verify the change is truly acceptable:

```bash
# Examine the exact diff
cat command.diff.bak

# Understand WHAT changed
# Is it a number change? A format change? Missing/extra lines?

# Check if change is beneficial or neutral
# A regold should NEVER hide a real problem
```

---

### 🔄 REFLECTION CHECKPOINT #4: Root Cause Understood

**STOP and verify you understand the ROOT CAUSE:**

```
┌─────────────────────────────────────────────────────────────────────────────┐
│              CHECKPOINT #4: DO I UNDERSTAND THE ROOT CAUSE?                 │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│ ROOT CAUSE DEPTH CHECK:                                                     │
│                                                                             │
│ □ Can I explain WHY this failed, not just WHAT failed?                      │
│ □ Have I traced the problem to its ORIGIN (not just where it manifested)?   │
│ □ If it's a path issue - do I know WHERE that path was set incorrectly?     │
│ □ If it's a tool issue - have I verified the setup is correct first?        │
│ □ If it's a regold - do I understand WHY the output changed?                │
│                                                                             │
│ CATEGORY LEANING:                                                           │
│                                                                             │
│ Based on my investigation, I'm leaning toward:                              │
│                                                                             │
│ □ SETUP ISSUE because: _______________________________________________      │
│ □ REPORT TO RnD because: _____________________________________________      │
│ □ REGOLD because: ____________________________________________________      │
│                                                                             │
│ But I have NOT committed yet. I need to verify and prove this.              │
│                                                                             │
├─────────────────────────────────────────────────────────────────────────────┤
│ If you can't clearly explain the root cause → KEEP INVESTIGATING.           │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## Phase 6: Use Failure-Type References for Specialized Analysis

For deeper analysis of specific failure types, consult these specialized references:

| Failure Type | Reference | When to Use |
|--------------|-----------|-------------|
| Core Dump | [references/failure-types/core-dump.md](references/failure-types/core-dump.md) | Tool crashed with core.* file |
| ERROR | [references/failure-types/error.md](references/failure-types/error.md) | Exit status 5 (explicit error) |
| Exit Status | [references/failure-types/exit-status.md](references/failure-types/exit-status.md) | Exit code changed from expected |
| Sev Warning | [references/failure-types/sev-warning.md](references/failure-types/sev-warning.md) | New severe warning appeared |
| Warning | [references/failure-types/warning.md](references/failure-types/warning.md) | New warning appeared |
| Simulation | [references/failure-types/simulation.md](references/failure-types/simulation.md) | Simulation-related failure |

These provide specialized guidance for analyzing each failure type in depth.

---

## Phase 7: The Decision - Categorization with Proof

**⚠️ THIS IS THE CRITICAL DECISION POINT. Take your time. Think carefully.**

### 🛑 DECISION CHECKPOINT: Am I Ready to Categorize?

**⚠️ MANDATORY: YOU MUST CALL `activate_deep_reasoning` NOW**

Before making your final decision, **YOU MUST EXECUTE THIS COMMAND**:

```
activate_deep_reasoning({
  checkpoint: "DECISION POINT: Final Categorization",
  effort: "high",
  reason: "Critical decision - must review ALL evidence, consider all categories, verify confidence before committing to category"
})
```

**DO NOT SKIP THIS STEP. The tool call is REQUIRED.**

After calling the tool, verify:

```
┌─────────────────────────────────────────────────────────────────────────────┐
│              🛑 DECISION CHECKPOINT: AM I READY TO CATEGORIZE?              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│ PRE-DECISION VERIFICATION:                                                  │
│                                                                             │
│ □ I have read ALL relevant files (test.out, status.log, diff.bak, logs)     │
│ □ I have followed ALL trails (Makefile → scripts → paths → dependencies)   │
│ □ I understand the ROOT CAUSE (not just symptoms)                           │
│ □ I have verified external paths/dependencies if mentioned in errors        │
│ □ I have compared actual vs gold output                                     │
│ □ I have evidence from MULTIPLE sources (not just one file)                 │
│                                                                             │
│ CONFIDENCE CHECK:                                                           │
│                                                                             │
│ My confidence level: ___ %                                                  │
│                                                                             │
│ If below 90% → What would increase my confidence? Go get that data FIRST.   │
│                                                                             │
│ 🧠 ENGAGE DEEP THINKING:                                                    │
│                                                                             │
│ This is a CRITICAL decision point. Use extended reasoning to:               │
│ - Review all evidence systematically                                        │
│ - Consider counter-arguments for each category                              │
│ - Ensure you're not missing any possibilities                               │
│ - Think through the reasoning chain step-by-step                            │
│                                                                             │
│ TAKING TIME:                                                                │
│                                                                             │
│ It's okay to take time to think. Complex problems need careful analysis.    │
│ A wrong categorization wastes engineer time. Get it RIGHT.                  │
│                                                                             │
├─────────────────────────────────────────────────────────────────────────────┤
│ Only proceed to categorization when ALL boxes are checked.                  │
└─────────────────────────────────────────────────────────────────────────────┘
```

### The Categorization Decision

**🧠 DEEP THINKING REQUIRED: Before selecting a category, reason through ALL the evidence systematically.**

After completing all investigation and analysis, categorize into **exactly ONE** category:

### Category 1: REGOLD ✅
**The failure represents a valid/improved change that should become the new baseline.**

📖 **Detailed Guide**: [references/category_next_move/regold.md](references/category_next_move/regold.md)

**Quick criteria**: Output improved, expected behavioral change, format changes, acceptable noise.

### Category 2: SETUP ISSUE ⚙️
**The failure is due to infrastructure, environment, or configuration problems - NOT the DFT tool's logic.**

📖 **Detailed Guide**: [references/category_next_move/setup-issue.md](references/category_next_move/setup-issue.md)

**Quick criteria**: Missing files, wrong paths, license failures, environment issues, Makefile config.

**⚠️ MANDATORY for SETUP ISSUE**: You MUST provide a **SOLUTION** - what is incorrect and what needs to be corrected. See the detailed guide.

### Category 3: REPORT TO RnD 🐛
**The failure indicates a genuine bug or issue in the DFT tool that needs developer investigation.**

📖 **Detailed Guide**: [references/category_next_move/report-to-rnd.md](references/category_next_move/report-to-rnd.md)

**Quick criteria**: Core dump, incorrect results, missing outputs, regression, correctness violation.

---

### Category-Specific Next Steps

After categorizing, **read the detailed guide** for your chosen category:

| Category | Reference File | Key Action |
|----------|----------------|------------|
| **REGOLD** | [regold.md](references/category_next_move/regold.md) | Document change justification, provide regold command |
| **SETUP ISSUE** | [setup-issue.md](references/category_next_move/setup-issue.md) | **Provide SOLUTION** - what's wrong and how to fix |
| **REPORT TO RnD** | [report-to-rnd.md](references/category_next_move/report-to-rnd.md) | Create bug report with reproduction steps |

---

## Phase 8: Validate Script Accuracy

After your analysis, verify if the `analyze_testcase.py` script correctly identified the failure:

### If Script Was Correct ✅
Proceed to output your findings.

### If Script Was Incorrect or Incomplete ❌

If you found that:
- The `Reason` doesn't match the actual root cause
- The `Failing Command` isn't the real failure point
- There's a deeper issue the script missed

**Create an improvement ticket**:

1. Generate a random 6-digit ticket number
2. Create `assets/improvement_<ticket>.md` with:
```markdown
# Script Improvement Ticket #<ticket_number>

## Testcase Details
- **Path**: <full testcase path>
- **Name**: <testcase name>
- **Bucket**: <bucket name>
- **Date Analyzed**: <current date>

## Script Output
<paste complete script output here>

## What Script Reported vs What Was Found
| Field | Script Reported | Actually Found |
|-------|-----------------|----------------|
| Status | | |
| Reason | | |
| Failing Command | | |

## Evidence of Discrepancy
<file names, line numbers, content that proves this>

## Suggested Improvement
- **Pattern to detect**: <what should be caught>
- **Where to look**: <which files>
- **Logic**: <how to implement>

## Priority
<High/Medium/Low>
```

3. Inform the user about the improvement ticket.

---

## Phase 9: Final Reflection and Self-Validation (MANDATORY)

**CRITICAL: This is your final quality gate. You have gone through multiple reflection checkpoints. Now consolidate and verify.**

**⚠️ MANDATORY: YOU MUST CALL `activate_deep_reasoning` NOW**

Before submitting your final report, **YOU MUST EXECUTE THIS COMMAND**:

```
activate_deep_reasoning({
  checkpoint: "Final Reflection",
  effort: "high",
  reason: "Final quality gate - must verify ALL checkpoints completed, all evidence gathered, categorization is correct with proof"
})
```

**DO NOT SKIP THIS STEP. The tool call is REQUIRED.**

After calling the tool, proceed with comprehensive self-validation:

### Summary of Checkpoints Completed

Before proceeding, confirm you completed ALL previous checkpoints:

```
┌─────────────────────────────────────────────────────────────────────────────┐
│              CHECKPOINT SUMMARY: Did I complete all of these?               │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│ ✓ CHECKPOINT #1 (Phase 2): Initial Understanding                            │
│   □ Ran analyze_testcase.py                                                 │
│   □ Listed testcase directory                                               │
│   □ Know PASSED/FAILED status                                               │
│   □ Read infrastructure docs                                                │
│                                                                             │
│ ✓ CHECKPOINT #2 (Phase 4 Step 1): Failure Point Identified                  │
│   □ Read test.out                                                           │
│   □ Read status.log                                                         │
│   □ Found and read ALL .diff.bak files                                      │
│   □ Know EXACT failing command                                              │
│                                                                             │
│ ✓ CHECKPOINT #3 (Phase 4 Step 3): Trail Following Complete                  │
│   □ Read Makefile completely                                                │
│   □ Traced all included files/scripts                                       │
│   □ Checked external paths if mentioned                                     │
│   □ Verified environment variables if relevant                              │
│   □ At ROOT CAUSE, not just symptoms                                        │
│   □ ⚠️ CALLED activate_deep_reasoning tool                                  │
│                                                                             │
│ ✓ CHECKPOINT #4 (Phase 5): Root Cause Understood                            │
│   □ Can explain WHY it failed                                               │
│   □ Traced to ORIGIN                                                        │
│   □ Have initial category leaning                                           │
│                                                                             │
│ ✓ DECISION CHECKPOINT (Phase 7): Ready to Categorize                        │
│   □ All files read                                                          │
│   □ All trails followed                                                     │
│   □ Evidence from multiple sources                                          │
│   □ Confidence >= 90%                                                       │
│   □ ⚠️ CALLED activate_deep_reasoning tool                                  │
│                                                                             │
│ ✓ FINAL REFLECTION (Phase 9): Quality Gate                                  │
│   □ ⚠️ CALLED activate_deep_reasoning tool                                  │
│                                                                             │
├─────────────────────────────────────────────────────────────────────────────┤
│ If ANY checkpoint was skipped → GO BACK to that phase NOW.                  │
└─────────────────────────────────────────────────────────────────────────────┘
```

### The Reflection Mindset

You are a **senior engineer reviewing your own work**. Be your harshest critic. Ask yourself: "Would this analysis convince a skeptical colleague? Is my proof bulletproof?"

### Step 1: Tool Usage Audit

**Verify you actually used tools - not just claimed to:**

#### A. Infrastructure Understanding ✅

- [ ] Did I use `read_file` tool to read `testcase-structure.md` (ALL ~750 lines)?
- [ ] Did I use `read_file` tool to read `testcase-execution-flow.md` (ALL ~1175 lines)?
- [ ] Can I list the 10 execution phases from memory?
- [ ] Do I understand file purposes (test.out, status.log, diff.bak)?

**Self-test (answer from memory):**
1. What does `remove_diff.pl` do? When does it create `.diff.bak`?
2. What exit status codes exist and what do they mean?
3. Where are gold files located for different platforms?

**If you cannot answer these → Use `read_file` to re-read the docs NOW.**

#### B. Testcase Data Collection ✅

- [ ] Did I run `analyze_testcase.py` (via terminal)?
- [ ] Did I use `cat` to examine `test.out`?
- [ ] Did I use `cat` to check `status.log`?
- [ ] Did I use `ls` to find ALL `*.diff.bak` files?
- [ ] Did I use `cat` to read EACH `*.diff.bak` file?
- [ ] Did I examine `testresults/logs/` content?
- [ ] Did I compare with `golds.linux*/` files?
- [ ] Did I read `Makefile`?
- [ ] Did I read `test.log`?

**If ANY file wasn't examined → Go back and read it NOW.**

#### C. Deep Traversal Verification ✅

- [ ] If Makefile includes other files → Did I read those?
- [ ] If scripts are invoked → Did I read those scripts?
- [ ] If external paths are mentioned in errors → Did I `ls` those paths?
- [ ] If environment variables are used → Did I `echo` them?
- [ ] Did I follow trails to the ROOT CAUSE?

#### C. Root Cause Understanding ✅
- [ ] Do I know EXACTLY which command/target failed?
- [ ] Do I know EXACTLY what the expected output was?
- [ ] Do I know EXACTLY what the actual output was?
- [ ] Do I understand WHY there is a difference (not just WHAT)?
- [ ] Can I trace the failure to a specific execution phase?

**If ANY answer is "I'm not sure" → Investigate more before concluding.**

#### D. Evidence Quality ✅
- [ ] Do I have CONCRETE file contents as proof (not just descriptions)?
- [ ] Are my evidence snippets SPECIFIC (line numbers, exact text)?
- [ ] Is my evidence SUFFICIENT to convince a skeptical reviewer?
- [ ] Have I captured evidence from MULTIPLE sources (not just one file)?

**If evidence feels weak → Find more supporting evidence.**

### Step 2: Category Validation

Before committing to a category, argue AGAINST your chosen category:

#### If you chose REGOLD, try to prove it's NOT REGOLD:
- Is the new output actually WORSE in any way?
- Could this indicate a real bug that happens to look like an improvement?
- Is there any correctness issue hidden in the change?
- Would accepting this as regold mask a real problem?

**If you found valid counter-arguments → Reconsider your categorization.**

#### If you chose SETUP ISSUE, try to prove it's NOT SETUP ISSUE:
- Did the infrastructure actually work correctly?
- Could the "setup issue" be caused by tool behavior?
- Is the environment/configuration actually correct and the tool just mishandled it?
- Could the same inputs work in a different environment?

**If you found valid counter-arguments → Reconsider your categorization.**

#### If you chose REPORT TO RnD, try to prove it's NOT REPORT TO RnD:
- Could this be an environment-specific issue (→ SETUP)?
- Is the "wrong" output actually acceptable (→ REGOLD)?
- Have I verified the setup is truly correct?
- Is this definitely not user error or misconfiguration?

**If you found valid counter-arguments → Reconsider your categorization.**

### Step 3: Confidence Assessment

Rate your confidence level honestly:

| Confidence Level | Action Required |
|------------------|-----------------|
| **HIGH (90%+)** | Proceed to output. Your analysis is solid. |
| **MEDIUM (70-89%)** | Identify what would increase confidence. Gather that data. |
| **LOW (<70%)** | **STOP.** You need more investigation. List what's missing and go get it. |

**Ask yourself these questions:**
1. "If I'm wrong, what would that mean?" 
2. "What evidence would change my conclusion?"
3. "Is there any file I haven't checked that could contradict me?"

### Step 4: Iteration Decision

**If you identified gaps in Steps 1-3, DO NOT PROCEED. Instead:**

1. **List the gaps**: What information is missing?
2. **Plan the iteration**: What specific files/commands will fill those gaps?
3. **Execute**: Go back to the relevant phase and gather the data
4. **Return here**: Re-run this reflection phase after gathering more data

**You may iterate as many times as needed until you reach HIGH confidence.**

### Step 5: Final Validation Statement

Before producing output, mentally complete this statement:

> "I am confident in my categorization of **[CATEGORY]** because:
> 1. I have examined [list key files examined]
> 2. My evidence shows [specific proof]
> 3. I have ruled out other categories because [specific reasons]
> 4. My confidence level is [HIGH/MEDIUM/LOW] because [reason]
> 5. Any remaining uncertainty is [describe or 'none']"

**If you cannot complete this statement confidently → ITERATE.**

### Reflection Checklist Summary

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                         REFLECTION CHECKLIST                                │
├─────────────────────────────────────────────────────────────────────────────┤
│ TOOL USAGE VERIFICATION (Did you actually use tools?)                       │
│ □ Used read_file to read testcase-structure.md (ALL ~750 lines)             │
│ □ Used read_file to read testcase-execution-flow.md (ALL ~1175 lines)       │
│ □ Used run_in_terminal to execute analyze_testcase.py script                │
│ □ Used read_file/cat to read test.out, status.log, Makefile                 │
│ □ Used ls to find all *.diff.bak files                                      │
│ □ Used read_file/cat to read EACH diff.bak file                             │
│ □ Used read_file/cat to read gold files for comparison                      │
├─────────────────────────────────────────────────────────────────────────────┤
│ UNDERSTANDING VERIFICATION                                                  │
│ □ Can list all 10 execution phases from memory                              │
│ □ Can explain what remove_diff.pl does                                      │
│ □ Can explain exit status codes (0, 5, 128+, etc.)                          │
│ □ Understand gold file locations for different platforms                    │
├─────────────────────────────────────────────────────────────────────────────┤
│ ANALYSIS VERIFICATION                                                       │
│ □ Exact failure point identified                                            │
│ □ Root cause understood (not just symptoms)                                 │
│ □ Evidence is concrete and specific (file names, line numbers, content)     │
│ □ Counter-arguments against chosen category considered                      │
│ □ Alternative categories ruled out with evidence                            │
│ □ Confidence level is HIGH (90%+)                                           │
├─────────────────────────────────────────────────────────────────────────────┤
│ ⚠️  If ANY box unchecked → USE TOOLS TO FILL THE GAP, then return here      │
│ ✅ If ALL boxes checked → Proceed to output                                 │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## Output Format (Failed Testcase)

```
## Testcase Analysis Report

**Path**: <testcase_path>
**Status**: ❌ FAILED

---

### Testcase Overview

**Purpose**: <What this testcase validates>
**Commands Executed**: <From TEST_TARGETS>
**Testmodes/Features**: <From readme.tsf or Makefile>

---

### Failure Analysis

**Failure Point**: <Specific command/phase that failed>
**Error Type**: <Core Dump / Exit Status / Diff / etc.>

**Expected Behavior**:
<What should have happened - with evidence from gold files>

**Actual Behavior**:  
<What actually happened - with evidence from testresults>

**Key Evidence**:
```
<Relevant content from diff.bak, logs, status.log>
```

---

### Category: <REGOLD | SETUP ISSUE | REPORT TO RnD>

---

### Proof of Categorization

<Detailed evidence showing WHY this category was chosen>

**Evidence 1**: <file, content, what it proves>
**Evidence 2**: <file, content, what it proves>
...

---

### 🔧 SOLUTION (For SETUP ISSUE - MANDATORY)

**⚠️ If category is SETUP ISSUE, you MUST complete this section:**

#### ❌ What is Incorrect
- **Item**: <Specific configuration/path/variable that is wrong>
- **Location**: <File path and line number>
- **Current value**: <What it currently is>
- **Problem**: <Why this is wrong>

#### ✅ What Needs to be Corrected
```bash
# Exact command or change needed
<specific fix command or code change>
```

#### 📍 Where to Make the Fix
- **File**: <Exact file path to edit>
- **Line**: <Line number if applicable>
- **Change**: <old value> → <new value>

#### 👤 Who Should Fix
- **Owner**: <Testcase owner / Infrastructure team / IT admin>
- **Reason**: <Why they are responsible>

---

### Reasoning Chain

<Your step-by-step reasoning process showing how you arrived at this conclusion>

1. First, I examined... and found...
2. Using my understanding of <execution phase>, I determined...
3. This indicates... because...
4. I ruled out REGOLD because...
5. I ruled out SETUP ISSUE because... (or: This IS a setup issue because...)
6. I ruled out REPORT TO RnD because... (or: This IS an RnD issue because...)
7. Therefore, the final categorization is...

---

### Reflection Summary

**Files Examined**: <List all files you examined>
**Confidence Level**: <HIGH/MEDIUM/LOW>
**Iterations Performed**: <Number of times you went back to gather more data>
**Remaining Uncertainty**: <Any caveats or areas of slight uncertainty, or "None">

---

### Next Steps

**For REGOLD**: 
- Files to regold: `<specific files>`
- Command: `cp testresults/<file> golds.<platform>/<file>`
- See: [regold.md](references/category_next_move/regold.md) for detailed steps

**For SETUP ISSUE**:
- Solution provided in 🔧 SOLUTION section above
- See: [setup-issue.md](references/category_next_move/setup-issue.md) for additional guidance

**For REPORT TO RnD**:
- Summary: `<concise problem statement>`
- Reproduction steps: `<how to reproduce>`
- See: [report-to-rnd.md](references/category_next_move/report-to-rnd.md) for bug report format
```

---

## Phase 10: Generate HTML Report (MANDATORY - DO NOT SKIP)

```
┌─────────────────────────────────────────────────────────────────────────────┐
│              🚨 MANDATORY: YOU MUST EXECUTE THESE STEPS                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  Your analysis is INCOMPLETE until you:                                     │
│  1. Create the JSON file using create_file tool                             │
│  2. Run the Python script using run_in_terminal tool                        │
│  3. Confirm report was generated                                            │
│                                                                             │
│  ⚠️ IF YOU DO NOT DO THIS, THE ANALYSIS IS INVALID                          │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

### Step 1: Create JSON File (USE create_file TOOL)

**You MUST use the `create_file` tool to create this JSON file:**

**File path:** `/lan/fed/etpv/scripts/ayush_scripts/modus_cli_agent/gemini-cli/.gemini/skills/testcase-analysis/assets/analysis_<TESTCASE_NAME>.json`

**Minimal JSON content (fill in your analysis data):**
```json
{
  "testcase_name": "<name from path>",
  "testcase_path": "<full testcase path>",
  "status": "FAILED",
  "category": "<REGOLD | SETUP ISSUE | REPORT TO RnD>",
  "confidence": "HIGH",
  "purpose": "<what the test does>",
  "failure_point": "<where it failed>",
  "error_type": "<type of error>",
  "key_evidence": "<main evidence line>",
  "root_cause": "<one sentence root cause>",
  "reasoning_chain": ["<step1>", "<step2>", "<step3>"],
  "files_examined": ["<file1>", "<file2>"],
  "next_steps": "<what to do next>"
}
```

### Step 2: Run Report Generator (USE run_in_terminal TOOL)

**You MUST use the `run_in_terminal` tool to execute:**

```bash
python3 /lan/fed/etpv/scripts/ayush_scripts/modus_cli_agent/gemini-cli/.gemini/skills/testcase-analysis/scripts/generate_report.py /lan/fed/etpv/scripts/ayush_scripts/modus_cli_agent/gemini-cli/.gemini/skills/testcase-analysis/assets/analysis_<TESTCASE_NAME>.json /lan/fed/etpv/scripts/ayush_scripts/modus_cli_agent/gemini-cli/.gemini/skills/testcase-analysis/assets/
```

**Expected output:** `✅ Report generated: .../assets/report_<name>_<timestamp>.html`

### Step 3: Confirm to User

After the script runs successfully, tell the user:

```
📄 **HTML Report Generated**
- Report: `assets/report_<name>_<datetime>.html`
- Open in browser to view formatted analysis
```

### EXAMPLE - Complete Phase 10 Execution:

**For a testcase named `Broadcast_Scan`:**

1. **Create JSON** (use create_file tool):
   - Path: `.../assets/analysis_Broadcast_Scan.json`
   
2. **Run script** (use run_in_terminal tool):
   ```bash
   python3 .../scripts/generate_report.py .../assets/analysis_Broadcast_Scan.json .../assets/
   ```

3. **Confirm**: Tell user the HTML report location

---

## Global References

For general operations:

- **Rerunning testcases**: [references/global/running-testcase.md](references/global/running-testcase.md)
- **Invoking Modus**: [references/global/running_modus.md](references/global/running_modus.md)

---

## Critical Principles: The Senior Engineer's Code

### Core Principles

1. **Learn first, analyze second** - Read the infrastructure docs before diving deep into failures
2. **Understand the phases** - Know which of the 10 execution phases failed
3. **Investigate thoroughly** - Check multiple files; don't rely on single data points  
4. **Reason, don't pattern-match** - Think about WHY something happened
5. **Prove your conclusions** - Every categorization needs concrete evidence
6. **Eliminate alternatives** - Explain why the failure is NOT the other categories
7. **Be precise** - Cite specific files, line numbers, content
8. **Think like a senior engineer** - Consider context, impact, and edge cases
9. **ALWAYS reflect at checkpoints** - Complete ALL reflection checkpoints; iterate if ANY gaps exist
10. **Never guess** - If confidence is below 90%, go back and gather more evidence before finalizing

### 🔥 Deep Traversal Principles

11. **Follow EVERY trail** - If a path is mentioned, GO CHECK IT. If a script is invoked, READ IT.
12. **Go to the source** - Don't stop at symptoms. Trace to the ROOT CAUSE.
13. **Investigate external paths** - If `/grid/common/pkgs/python` fails, GO INSIDE and check what's there
14. **Read the scripts** - Makefile calls a script? READ THAT SCRIPT. Script sources another? READ THAT TOO.
15. **Verify infrastructure** - Setup issues might be in infrastructure scripts, not the testcase itself
16. **Use your tools** - You have `ls`, `cat`, `read_file`, terminal access. USE THEM to explore.
17. **Take your time** - Complex debugging takes time. Don't rush to a conclusion.

### 🧠 Extended Thinking Principles

18. **⚠️ MANDATORY: Use activate_deep_reasoning at ALL checkpoints** - You MUST call the tool before critical analysis
19. **Use deep thinking at critical points** - Enable high reasoning effort for root cause analysis and categorization
20. **Think out loud** - Explicitly reason through your evidence and conclusions
21. **Connect the dots** - Use extended thinking to find non-obvious connections between evidence
22. **Challenge yourself** - Use deep reasoning to argue against your own conclusions
23. **Don't rush decisions** - At decision points, engage extended thinking to ensure thoroughness

### 🔧 Using the activate_deep_reasoning Tool

**YOU MUST call this tool at the following checkpoints:**

1. **Checkpoint #3: Trail Following Complete** - Before connecting evidence from multiple files
2. **Checkpoint #4: Root Cause Understood** - Before finalizing root cause (if applicable)
3. **DECISION POINT** - Before selecting final category (ALWAYS REQUIRED)
4. **Final Reflection** - Before outputting final analysis (ALWAYS REQUIRED)

**Tool Schema:**
```typescript
activate_deep_reasoning({
  checkpoint: string,  // Which checkpoint you're at
  effort: "low" | "medium" | "high",  // How much reasoning is needed
  reason: string  // WHY you need deep thinking at this point
})
```

**Example usage:**
```
activate_deep_reasoning({
  checkpoint: "Checkpoint #3: Trail Following Complete",
  effort: "high",
  reason: "Need to connect all evidence from multiple files (Makefile, logs, diffs, scripts, paths) to identify root cause"
})
```

**⚠️ FAILURE TO CALL THIS TOOL AT REQUIRED CHECKPOINTS WILL RESULT IN INCOMPLETE ANALYSIS.**

### Reasoning Effort Guidelines

| Phase | Reasoning Effort | Rationale |
|-------|------------------|-----------|
| Phase 1-2: Data gathering | N/A | No tool call needed |
| Phase 3: Status check | N/A | No tool call needed |
| Phase 4: Deep investigation → **Checkpoint #3** | **`high`** | **MANDATORY TOOL CALL** |
| Phase 5: Root cause reasoning → **Checkpoint #4** | **`high`** | **MANDATORY if applicable** |
| Phase 6: Failure-type analysis | N/A | Reference-guided |
| Phase 7: **CATEGORIZATION → DECISION POINT** | **`high`** | **MANDATORY TOOL CALL** |
| Phase 8: Script validation | N/A | Comparison task |
| Phase 9: **FINAL REFLECTION** | **`high`** | **MANDATORY TOOL CALL** |

**🧠 You MUST use the activate_deep_reasoning tool at:**
- ✅ Checkpoint #3 (Trail following complete)
- ✅ Checkpoint #4 (Root cause understood) - if applicable
- ✅ DECISION POINT (Category selection) - ALWAYS
- ✅ Final Reflection - ALWAYS

### The Senior Engineer's Questions

Before finalizing, ask yourself:

- "Would a senior DFT PV engineer with 15 years experience accept this analysis?"
- "Have I followed every trail to its end?"
- "Can I explain the root cause in one clear sentence?"
- "If I'm wrong, what's the impact? Have I done enough to be sure?"
- "Is there any file I haven't checked that could change my conclusion?"
- "Have I used deep thinking at the critical decision points?"

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                    THE SENIOR ENGINEER'S STANDARD                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  A junior engineer reports: "The test failed because file X wasn't found"   │
│                                                                             │
│  A senior engineer reports: "The test failed because file X wasn't found    │
│  at path /a/b/c. This path was set in Makefile line 42, which includes      │
│  common.mk from /infra/makefiles/. The variable TOOL_PATH is set from       │
│  environment variable $MODUS_HOME, which was /old/path instead of           │
│  /new/path because the setup script /scripts/setup.sh line 15 uses a        │
│  hardcoded value that doesn't match this machine's installation.            │
│  ROOT CAUSE: Infrastructure setup script needs update for new install path. │
│  CATEGORY: SETUP ISSUE"                                                     │
│                                                                             │
│  BE THE SENIOR ENGINEER.                                                    │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## Configuration Notes

### Enabling Extended Thinking Mode

Extended thinking/reasoning is now supported via the LiteLLM/JEDAI configuration. To enable it:

**Option 1: Via settings.json (Recommended)**

Add to your settings to enable reasoning for agents:
```json
{
  "agents": {
    "overrides": {
      "codebase_investigator": {
        "modelConfig": {
          "generateContentConfig": {
            "thinkingConfig": {
              "thinkingBudget": 4096,
              "reasoningEffort": "high"
            }
          }
        }
      }
    }
  }
}
```

**Option 2: Environment Variable**

Set the default model to one that supports reasoning:
```bash
export LITELLM_DEFAULT_MODEL=GCP_claude-sonnet-4-5
```

### Supported Models for Extended Thinking

| Model | Reasoning Support | Notes |
|-------|-------------------|-------|
| GCP_claude-sonnet-4-5 | ✅ Yes | Best for complex analysis |
| GCP_claude-opus-4-1 | ✅ Yes | Most capable |
| GCP_claude-sonnet-4 | ✅ Yes | Good balance |
| GCP_gemini-2.5-pro | ✅ Yes | Fast with good reasoning |
| GCP_gemini-2.5-flash | ✅ Yes | Fast |
| GCP_qwen/qwen3-* | ✅ Yes | Good for code |
| on_prem_nvidia/nemotron-* | ✅ Yes | On-premise option |
| on_prem_openai/gpt-oss-120b | ✅ Yes | On-premise option |
| AzureOpenAI_gpt-5 | ✅ Yes | Azure option |
| AzureOpenAI_o4-mini | ✅ Yes | Azure option |
| GCP_claude-3-5-haiku | ❌ No | Does not support reasoning |
| GCP_claude-3-5-sonnet-v2 | ❌ No | Does not support reasoning |

### Recommended Reasoning Budgets

| Use Case | Budget (tokens) | Reasoning Effort |
|----------|-----------------|------------------|
| Simple decisions | 512-1024 | low |
| Root cause analysis | 2048-4096 | medium-high |
| Critical categorization | 4096-8192 | high |
| Complex multi-step debugging | 8192-16384 | high |

### JEDAI API Direct Usage

For direct API calls with reasoning:

**Gemini:**
```python
body = {
    "messages": [...],
    "model": "GEMINI",
    "deployment": "gemini-2.5-pro",
    "reasoning_effort": "high"
}
```

**Claude:**
```python
body = {
    "messages": [...],
    "model": "Claude",
    "deployment": "claude-sonnet-4",
    "reasoning_effort": "high"
}
```

**Response includes:**
```python
response['choices'][0]['message']['content']          # Main response
response['choices'][0]['message']['reasoning_content'] # Reasoning trace
```


---

## FINAL REMINDER: Before You Finish

```
+-----------------------------------------------------------------------------+
|                     CHECKLIST BEFORE COMPLETING                             |
+-----------------------------------------------------------------------------+
|                                                                             |
|  [ ] Did you categorize the testcase? (REGOLD/SETUP ISSUE/REPORT TO RnD)    |
|  [ ] Did you provide root cause with evidence?                              |
|  [ ] Did you create JSON file in assets/ using create_file tool?            |
|  [ ] Did you run generate_report.py using run_in_terminal tool?             |
|  [ ] Did you confirm HTML report was generated?                             |
|                                                                             |
|  If ANY checkbox is unchecked, GO BACK AND COMPLETE IT.                     |
|                                                                             |
+-----------------------------------------------------------------------------+
```

**Your analysis is ONLY complete when the HTML report exists in assets/.**