# fix
> When a test fails, follow this decision tree:
- Author: Nash Gao
- Repository: nashgao/mqtt-client
- Version: 20260210001543
- Stars: 3
- Forks: 1
- Last Updated: 2026-02-09
- Source: https://github.com/nashgao/mqtt-client
- Web: https://mule.run/skillshub/@@nashgao/mqtt-client~fix:20260210001543
---
---
allowed-tools: all
description: **EXECUTE and fix ALL error types in tests** including syntax, semantic, runtime, type, logic errors with comprehensive analysis
intensity: โกโกโกโก
pattern: ๐ง๐ง๐ง๐ง
---
# ๐ง๐ง๐ง๐ง COMPREHENSIVE ERROR RESOLUTION IN TESTS: ALL ERROR TYPES! ๐ง๐ง๐ง๐ง
**THIS FIXES ALL ERROR CATEGORIES - SYNTAX, SEMANTIC, RUNTIME, TYPE, LOGIC, AND TEST FAILURES!**
**๐จ MANDATORY: RUN STATIC ANALYZERS BEFORE TEST EXECUTION! ๐จ**
## โ ๏ธ SPECIFICATION-FIRST PHILOSOPHY (MANDATORY)
**When tests fail, the test is the authority โ the code must conform to the specification.**
- Default assumption: if a test fails, the CODE is wrong (not the test)
- Exception: test itself has a bug (syntax error, wrong assertion logic, stale assertion)
- Never "fix" a test by changing it to match broken implementation behavior
- See `templates/CLAUDE.md` โ "MANDATORY: Specification-First Testing" for full mandate
### Test Failure Decision Framework (MANDATORY)
**DEFAULT ASSUMPTION: The test is correct. The code is wrong.**
When a test fails, follow this decision tree:
```
Test fails
โ Step 1: Is the test itself buggy? (syntax error, stale import, broken setup, wrong fixture)
โ YES: Fix the test bug (this PRESERVES the specification, it doesn't change it)
โ NO: Continue to Step 2
โ Step 2: The test is a valid specification. Fix the PRODUCTION CODE to satisfy it.
โ If fix is clear: apply it
โ If fix is complex/unclear: escalate to user with explanation
โ NEVER: Change a test assertion to match what broken code produces
```
**What counts as a "test bug" (safe to fix):**
- Syntax errors in the test file
- Stale imports or missing use statements
- Broken test setup/teardown (missing fixtures, wrong DB state)
- Wrong assertion method (e.g., `toBe` vs `toEqual` for objects โ only if the matcher is genuinely wrong for the data type)
**What is NOT a "test bug" (fix the production code instead):**
- Assertion expects value X but code returns value Y โ fix the code to return X
- Test expects a method to exist but it doesn't โ implement the method
- Test expects a side effect (email sent, event fired) that doesn't happen โ fix the code
- Test expects an error to be thrown but code silently succeeds โ fix the code
โ **FORBIDDEN**: Changing test assertions to match broken code output
โ **FORBIDDEN**: Weakening test expectations (e.g., loosening type checks, removing assertions)
โ **FORBIDDEN**: Increasing timeouts or adding retries to mask real failures
โ **FORBIDDEN**: Adding `@skip` or commenting out failing tests
## ๐จ ZERO TOLERANCE ENFORCEMENT
**MANDATORY - ALL tests must achieve PERFECT execution:**
### Success Criteria (ALL must be met)
- โ
**0 Failed Tests** - Every single test must pass
- โ
**0 Errors** - No runtime errors allowed
- โ
**0 Warnings** - Warnings are treated as failures
- โ
**0 Deprecations** - Deprecation notices block success
- โ
**0 Incomplete Tests** - Incomplete tests are failures
- โ
**0 Risky Tests** - Risky test detection must pass
- โ
**0 Skipped Tests** - Unless explicitly allowed with justification
### Failure Response Protocol
When ANY issue is detected:
1. **STOP** - Do not proceed to next steps
2. **REPORT** - List all issues with file:line references
3. **FIX** - Resolve ALL issues before continuing
4. **VERIFY** - Re-run tests to confirm 100% clean execution
### Exit Codes
- `0` = Perfect execution (no warnings, no deprecations, no failures)
- `1` = Any failure, warning, deprecation, or incomplete test
- `2` = Configuration or setup error
## ๐ MANDATORY POST-FIX VERIFICATION PROTOCOL
**After EVERY single fix, you MUST verify the ENTIRE test suite passes:**
### Verification Sequence (MANDATORY after each fix)
1. **Fix ONE issue** - Make the minimal change to fix the current failure
2. **Run FULL test suite** - Not just the fixed test, but ALL tests
3. **Verify ALL conditions** - 0 failures, 0 warnings, 0 deprecations, 0 incomplete
4. **Confirm no regressions** - All previously passing tests still pass
5. **Only then proceed** - Move to next fix only after full verification
### โ FORBIDDEN Patterns
- Fixing multiple tests before verification
- Running only the specific test that was fixed
- Assuming a fix works without full suite verification
- Batching fixes to "save time"
### โ
REQUIRED Pattern
```bash
# After EACH fix:
run_full_test_suite
verify_zero_tolerance_conditions
compare_with_baseline # Ensure no new failures
# Only if ALL pass, proceed to next fix
```
### Why This Matters
A fix to Test A may break Test B through:
- Shared fixtures or test infrastructure
- Common production code dependencies
- Side effects in test setup/teardown
- Database state changes
## ๐ COMPREHENSIVE ERROR DETECTION PIPELINE
**โ ๏ธ CRITICAL: DETECT ALL ERROR TYPES FIRST!**
### PHASE 1: Static Analysis (BEFORE running tests)
```bash
# PHP: Use PHPStan FIRST for comprehensive error detection
phpstan analyze --level=9 --no-progress 2>&1 | tee static_errors.log
# Python: Use mypy for type and semantic errors
mypy . --strict 2>&1 | tee static_errors.log
# TypeScript: Use tsc for comprehensive checking
tsc --noEmit --strict 2>&1 | tee static_errors.log
```
**CRITICAL: When analyzing test failures, remember that tests are specifications. If a test fails, first assume the code is wrong, not the test.**
### PHASE 2: Categorize Errors Found
- **SYNTAX ERRORS**: Parse failures, malformed code
- **SEMANTIC ERRORS**: Undefined classes, missing methods, unimplemented abstracts
- **RUNTIME ERRORS**: Null pointer exceptions, array index out of bounds, division by zero, resource exhaustion
- **TYPE ERRORS**: Missing annotations, type mismatches, unsafe access
- **DEPENDENCY ERRORS**: Missing packages requiring `composer require`
- **LOGIC ERRORS**: Unreachable code, missing returns, infinite loops
- **ENUM ERRORS**: Incorrect instantiation (`new Enum()` โ `Enum::CASE`)
### PHASE 3: Test Execution (AFTER fixing static errors)
- **PHP**: Use `composer test` or `./vendor/bin/phpunit`
- **Python**: Use `pytest` with full error output
- **JavaScript**: Use appropriate test runner
- **ALWAYS**: Fix static errors BEFORE running tests
When you run `/test fix`, you are REQUIRED to:
1. **ANALYZE** code with PHPStan/mypy/tsc to find ALL error types
2. **CATEGORIZE** errors: syntax, semantic, runtime, type, logic, dependency
3. **FIX** semantic errors first (missing classes, dependencies)
4. **FIX** type errors second (annotations, type guards)
5. **FIX** runtime errors third (null checks, bounds validation, error handling)
6. **FIX** logic errors fourth (unreachable code, missing returns)
7. **THEN** execute tests to find additional runtime failures
8. **USE MULTIPLE AGENTS** for parallel error resolution:
- Spawn agents for semantic error fixing (class creation, dependencies)
- Spawn agents for type error resolution (annotations, guards)
- Spawn agents for runtime error prevention (null checks, validation)
- Spawn agents for logic error correction
- Say: "I'll spawn multiple agents to fix all error categories in parallel"
9. **VERIFY** all errors resolved with final PHPStan/analyzer run
## ๐ฏ USE MULTIPLE AGENTS
**MANDATORY AGENT SPAWNING FOR TEST DEBUGGING:**
I'll spawn 5 specialized agents using Task tool for comprehensive test fixing:
test-fixer
Analyze test failures and categorize issues
You are the Failure Analysis Agent for comprehensive test debugging.
Your responsibilities:
1. EXECUTE tests first to collect failing test information and error details
2. Categorize failures by type (assertion, timeout, mock, async, environment, flaky)
3. Analyze error patterns and stack traces
4. Prioritize failures by severity and impact
5. Group related failures together
6. Generate comprehensive failure analysis report
7. SAVE results to /tmp/test-fix-analysis-{{TIMESTAMP}}.json for verification
MUST execute actual test commands and verify results. Save detailed analysis with success/failure status.
test-fixer
Implement fixes for identified root causes
You are the Fix Implementation Agent for comprehensive test debugging.
Your responsibilities:
1. READ analysis results from /tmp/test-fix-analysis-*.json
2. Perform deep root cause analysis for each failure category
3. Implement systematic fixes addressing root causes (not symptoms)
4. Handle assertion errors, timeout issues, mock problems, and async errors
5. Apply fixes incrementally with proper rollback capability
6. EXECUTE tests after each fix to verify effectiveness
7. SAVE results to /tmp/test-fix-implementation-{{TIMESTAMP}}.json
Implement comprehensive fixes and VERIFY they work through actual test execution.
test-fixer
Verify fixes work correctly
You are the Validation Agent for comprehensive test debugging.
Your responsibilities:
1. READ implementation results from /tmp/test-fix-implementation-*.json
2. EXECUTE all previously failing tests multiple times to verify stability
3. Check that all previously failing tests now pass consistently
4. Measure performance improvements and execution times
5. RUN full test suite to ensure no regressions introduced
6. SAVE comprehensive results to /tmp/test-fix-validation-{{TIMESTAMP}}.json
7. Generate validation reports with actual test execution results
MUST execute actual tests and verify 100% success rate before reporting completion.
test-fixer
Ensure fixes don't introduce regressions
You are the Regression Prevention Agent for comprehensive test debugging.
Your responsibilities:
1. Identify all tests related to the fixed functionality
2. Execute comprehensive regression test suites
3. Monitor for new test failures introduced by fixes
4. Check integration points and dependency impacts
5. Verify that existing passing tests remain stable
6. Generate regression analysis reports
7. Coordinate rollback if regressions are detected
Ensure all fixes maintain system stability and don't introduce new failures.
test-fixer
Improve test design and prevent future failures
You are the Prevention Enhancement Agent for comprehensive test debugging.
Your responsibilities:
1. Analyze patterns in fixed failures to identify prevention opportunities
2. Implement test reliability improvements (determinism, isolation, timing)
3. Create linting rules and templates to prevent similar issues
4. Add monitoring and alerting for test quality metrics
5. Update documentation with lessons learned and best practices
6. Establish failure prevention measures and quality gates
7. Generate final comprehensive test debugging report
Implement prevention measures to avoid similar test failures in the future.
**AGENT RESULT VERIFICATION:**
```bash
# After spawning agents, MUST verify results:
AGENT_RESULTS_DIR="/tmp"
for result_file in "$AGENT_RESULTS_DIR"/test-fix-*-*.json; do
if [ -f "$result_file" ]; then
AGENT_SUCCESS=$(jq -r '.success // false' "$result_file" 2>/dev/null || echo 'false')
if [ "$AGENT_SUCCESS" != "true" ]; then
echo "โ CRITICAL: Agent failed to execute tests successfully"
echo " Result file: $result_file"
exit 1
fi
else
echo "โ CRITICAL: Agent result file not found: $result_file"
exit 1
fi
done
# Final test execution to verify all fixes
case "$(detect_test_framework)" in
"jest"|"mocha")
npm test || exit 1
;;
"pytest")
python -m pytest || exit 1
;;
"go-test")
go test ./... || exit 1
;;
"phpunit")
./vendor/bin/phpunit || exit 1
;;
*)
echo "โ No test framework detected for final verification"
exit 1
;;
esac
echo "โ
All test fixes verified through actual execution"
```
## ๐จ FORBIDDEN BEHAVIORS
**NEVER:**
- โ "Just comment out the failing test" โ NO! Fix the root cause!
- โ Apply quick fixes without understanding โ NO! Analyze thoroughly!
- โ Ignore test environment issues โ NO! Environment affects reliability!
- โ Skip regression testing โ NO! Verify fixes don't break other tests!
- โ "It's just a flaky test" โ NO! Make tests deterministic!
- โ Fix symptoms instead of root causes โ NO! Address underlying issues!
- โ Modifying test assertions to match broken implementation behavior (the test is the spec)
**MANDATORY WORKFLOW:**
```
1. TEST EXECUTION โ Run tests to identify and reproduce failures
2. Failure analysis โ Categorize and prioritize failing tests
3. IMMEDIATELY spawn agents for parallel debugging
4. Root cause identification โ Deep analysis of underlying issues
5. Fix implementation โ Address root causes comprehensively
6. AGENT RESULT VERIFICATION โ Verify agents completed successfully
7. TEST RE-EXECUTION โ Re-run tests to confirm fixes work
8. FINAL SUCCESS VALIDATION โ Ensure 100% test pass rate
```
**YOU ARE NOT DONE UNTIL:**
- โ
ALL tests have been EXECUTED and failures identified
- โ
ALL failing tests are fixed and passing through ACTUAL EXECUTION
- โ
Root causes are identified and addressed
- โ
Fixes are verified through TEST EXECUTION to not introduce regressions
- โ
Test reliability is improved and verified through execution
- โ
Prevention measures are implemented
- โ
Test debugging knowledge is documented
---
๐ **MANDATORY TEST DEBUGGING CHECK** ๐
1. Re-read ~/.claude/CLAUDE.md RIGHT NOW
2. Check current failing tests and their error patterns
3. Verify you understand the test debugging requirements
Execute comprehensive test debugging and fixing for: $ARGUMENTS
**FORBIDDEN SHORTCUT PATTERNS:**
- "This test is too complex to debug" โ NO, break it down systematically!
- "Random fixes might work" โ NO, use systematic debugging!
- "Environment issues are someone else's problem" โ NO, fix the environment!
- "Flaky tests are normal" โ NO, make them deterministic!
- "Quick fixes are faster" โ NO, fix root causes properly!
Let me ultrathink about the comprehensive test debugging architecture and fixing strategy.
๐จ **REMEMBER: Properly debugged and fixed tests improve code quality and team velocity!** ๐จ
**Comprehensive Test Debugging and Fixing Protocol:**
**Step 0: Test Failure Analysis and Categorization**
- Collect all failing test information and error details
- Categorize failures by type and severity
- Prioritize fixes based on impact and complexity
- Set up debugging environment and tools
- Prepare systematic debugging approach
**Step 1: Comprehensive Failure Analysis**
**Failure Analysis Framework:**
```typescript
interface TestFailure {
test_name: string;
test_file: string;
error_message: string;
stack_trace: string;
failure_type: FailureType;
error_category: ErrorCategory;
severity: 'critical' | 'high' | 'medium' | 'low';
environment_info: EnvironmentInfo;
execution_context: ExecutionContext;
related_failures: string[];
debugging_priority: number;
}
enum FailureType {
ASSERTION_ERROR = 'assertion_error',
TIMEOUT = 'timeout',
SETUP_ERROR = 'setup_error',
TEARDOWN_ERROR = 'teardown_error',
MOCK_ERROR = 'mock_error',
ASYNC_ERROR = 'async_error',
ENVIRONMENT_ERROR = 'environment_error',
DEPENDENCY_ERROR = 'dependency_error',
CONFIGURATION_ERROR = 'configuration_error',
FLAKY_TEST = 'flaky_test'
}
enum ErrorCategory {
LOGIC_ERROR = 'logic_error',
TIMING_ISSUE = 'timing_issue',
RESOURCE_ISSUE = 'resource_issue',
EXTERNAL_DEPENDENCY = 'external_dependency',
TEST_DESIGN = 'test_design',
INFRASTRUCTURE = 'infrastructure'
}
class TestFailureAnalyzer {
async analyzeFailures(testResults: TestResult[]): Promise {
// Analyze test failures
const failures = testResults.filter(result => !result.passed);
const analysis = {
total_failures: failures.length,
failure_categories: new Map(),
failure_types: new Map(),
priority_failures: [],
related_failure_groups: [],
debugging_recommendations: []
};
// Categorize failures
for (const failure of failures) {
const failureInfo = await this.analyzeIndividualFailure(failure);
// Group by category
if (!analysis.failure_categories.has(failureInfo.error_category)) {
analysis.failure_categories.set(failureInfo.error_category, []);
}
analysis.failure_categories.get(failureInfo.error_category).push(failureInfo);
// Group by type
if (!analysis.failure_types.has(failureInfo.failure_type)) {
analysis.failure_types.set(failureInfo.failure_type, []);
}
analysis.failure_types.get(failureInfo.failure_type).push(failureInfo);
}
// Identify priority failures
analysis.priority_failures = failures
.sort((a, b) => b.debugging_priority - a.debugging_priority)
.slice(0, 10);
// Group related failures
analysis.related_failure_groups = this.groupRelatedFailures(failures);
// Generate debugging recommendations
analysis.debugging_recommendations = this.generateDebuggingRecommendations(analysis);
return analysis;
}
private async analyzeIndividualFailure(failure: TestResult): Promise {
// Analyze individual failure
const failureInfo = {
test_name: failure.test_name,
test_file: failure.test_file,
error_message: failure.error_message,
stack_trace: failure.stack_trace,
failure_type: this.determineFailureType(failure),
error_category: this.determineErrorCategory(failure),
severity: this.determineSeverity(failure),
environment_info: await this.collectEnvironmentInfo(),
execution_context: await this.collectExecutionContext(failure),
related_failures: this.findRelatedFailures(failure),
debugging_priority: this.calculateDebuggingPriority(failure)
};
return failureInfo;
}
private determineFailureType(failure: TestResult): FailureType {
const errorMessage = failure.error_message.toLowerCase();
const stackTrace = failure.stack_trace.toLowerCase();
if (errorMessage.includes('timeout') || errorMessage.includes('exceeded')) {
return FailureType.TIMEOUT;
}
if (errorMessage.includes('mock') || errorMessage.includes('stub')) {
return FailureType.MOCK_ERROR;
}
if (errorMessage.includes('async') || errorMessage.includes('promise') ||
errorMessage.includes('await')) {
return FailureType.ASYNC_ERROR;
}
if (errorMessage.includes('setup') || errorMessage.includes('beforeeach')) {
return FailureType.SETUP_ERROR;
}
if (errorMessage.includes('teardown') || errorMessage.includes('aftereach')) {
return FailureType.TEARDOWN_ERROR;
}
if (errorMessage.includes('connection') || errorMessage.includes('network') ||
errorMessage.includes('database')) {
return FailureType.DEPENDENCY_ERROR;
}
if (errorMessage.includes('config') || errorMessage.includes('environment')) {
return FailureType.CONFIGURATION_ERROR;
}
if (this.isFlaky(failure)) {
return FailureType.FLAKY_TEST;
}
return FailureType.ASSERTION_ERROR;
}
private determineErrorCategory(failure: TestResult): ErrorCategory {
const failureType = this.determineFailureType(failure);
switch (failureType) {
case FailureType.ASSERTION_ERROR:
return ErrorCategory.LOGIC_ERROR;
case FailureType.TIMEOUT:
case FailureType.ASYNC_ERROR:
return ErrorCategory.TIMING_ISSUE;
case FailureType.DEPENDENCY_ERROR:
return ErrorCategory.EXTERNAL_DEPENDENCY;
case FailureType.ENVIRONMENT_ERROR:
case FailureType.CONFIGURATION_ERROR:
return ErrorCategory.INFRASTRUCTURE;
case FailureType.FLAKY_TEST:
return ErrorCategory.TEST_DESIGN;
default:
return ErrorCategory.LOGIC_ERROR;
}
}
private isFlaky(failure: TestResult): boolean {
// Check test history for inconsistent results
const testHistory = this.getTestHistory(failure.test_name);
const recentResults = testHistory.slice(-10);
if (recentResults.length < 5) return false;
const passCount = recentResults.filter(r => r.passed).length;
const failCount = recentResults.length - passCount;
// Consider flaky if it has both passes and failures in recent history
return passCount > 0 && failCount > 0;
}
private calculateDebuggingPriority(failure: TestResult): number {
let priority = 0;
// Higher priority for critical functionality
if (this.isCriticalTest(failure.test_name)) {
priority += 10;
}
// Higher priority for blocking other tests
if (this.isBlockingTest(failure.test_name)) {
priority += 8;
}
// Higher priority for frequently failing tests
const failureRate = this.getFailureRate(failure.test_name);
priority += Math.floor(failureRate * 5);
// Higher priority for tests with many dependencies
const dependencyCount = this.countTestDependencies(failure.test_name);
priority += Math.min(dependencyCount, 5);
return priority;
}
}
```
**Step 2: Parallel Agent Deployment for Test Debugging**
**Agent Spawning Strategy:**
After identifying failing tests, I deploy specialized agents through Task tool for targeted fixes:
**Example Agent Deployment:**
"I've identified 23 failing tests across 6 different categories. I'll spawn specialized agents using Task tool invocations above:
1. **Failure Analysis Agent**: Categorizes all 23 failures into assertion errors (8), timeout issues (5), mock errors (4), async errors (3), environment issues (2), and flaky tests (1)
2. **Fix Implementation Agent**: Implements targeted fixes for each failure category using systematic root cause analysis
3. **Validation Agent**: Verifies each fix through multiple test runs to ensure stability and effectiveness
4. **Regression Prevention Agent**: Runs comprehensive regression suites to prevent new failures
5. **Prevention Enhancement Agent**: Analyzes patterns and implements prevention measures
Each agent operates in parallel while coordinating through shared analysis reports and fix validation."
**Step 3: Systematic Test Debugging Implementation**
**Root Cause Analysis Engine:**
```typescript
class RootCauseAnalyzer {
async analyzeRootCause(failure: TestFailure): Promise {
// Analyze root cause
const analysis = {
primary_cause: null,
contributing_factors: [],
evidence: [],
reproduction_steps: [],
fix_recommendations: [],
prevention_measures: []
};
// Analyze based on failure type
switch (failure.failure_type) {
case FailureType.ASSERTION_ERROR:
analysis.primary_cause = await this.analyzeAssertionError(failure);
break;
case FailureType.TIMEOUT:
analysis.primary_cause = await this.analyzeTimeoutError(failure);
break;
case FailureType.MOCK_ERROR:
analysis.primary_cause = await this.analyzeMockError(failure);
break;
case FailureType.ASYNC_ERROR:
analysis.primary_cause = await this.analyzeAsyncError(failure);
break;
case FailureType.FLAKY_TEST:
analysis.primary_cause = await this.analyzeFlakiness(failure);
break;
default:
analysis.primary_cause = await this.analyzeGenericError(failure);
}
// Collect evidence
analysis.evidence = await this.collectEvidence(failure);
// Generate reproduction steps
analysis.reproduction_steps = await this.generateReproductionSteps(failure);
// Generate fix recommendations
analysis.fix_recommendations = await this.generateFixRecommendations(analysis);
// Generate prevention measures
analysis.prevention_measures = await this.generatePreventionMeasures(analysis);
return analysis;
}
private async analyzeAssertionError(failure: TestFailure): Promise {
// Analyze assertion error
const stackTrace = this.parseStackTrace(failure.stack_trace);
const testCode = await this.getTestCode(failure.test_file);
const productionCode = await this.getProductionCode(failure.test_file);
// Analyze assertion details
const assertion = this.extractAssertion(failure.error_message);
const expectedValue = this.extractExpectedValue(assertion);
const actualValue = this.extractActualValue(assertion);
// Analyze why values don't match
const valueAnalysis = await this.analyzeValueMismatch(expectedValue, actualValue);
// Trace back to root cause
const rootCause = await this.traceAssertionRootCause(
assertion,
valueAnalysis,
testCode,
productionCode
);
return {
type: 'assertion_error',
description: `Assertion failed: expected ${expectedValue}, got ${actualValue}`,
root_cause: rootCause,
confidence: this.calculateConfidence(rootCause),
supporting_evidence: valueAnalysis.evidence
};
}
private async analyzeTimeoutError(failure: TestFailure): Promise {
// Analyze timeout error
const timeoutValue = this.extractTimeoutValue(failure.error_message);
const testCode = await this.getTestCode(failure.test_file);
// Analyze what operations might be causing the timeout
const slowOperations = await this.identifySlowOperations(testCode);
// Check for infinite loops or blocking operations
const blockingOperations = await this.identifyBlockingOperations(testCode);
// Analyze async operations
const asyncOperations = await this.analyzeAsyncOperations(testCode);
// Determine most likely cause
const rootCause = await this.determineTimeoutRootCause(
slowOperations,
blockingOperations,
asyncOperations,
timeoutValue
);
return {
type: 'timeout_error',
description: `Test timed out after ${timeoutValue}ms`,
root_cause: rootCause,
confidence: this.calculateConfidence(rootCause),
supporting_evidence: {
slow_operations: slowOperations,
blocking_operations: blockingOperations,
async_operations: asyncOperations
}
};
}
private async analyzeMockError(failure: TestFailure): Promise {
// Analyze mock error
const testCode = await this.getTestCode(failure.test_file);
const mocks = await this.extractMocks(testCode);
// Analyze mock setup
const mockSetup = await this.analyzeMockSetup(mocks);
// Check for mock expectations
const mockExpectations = await this.analyzeMockExpectations(mocks);
// Verify mock cleanup
const mockCleanup = await this.analyzeMockCleanup(mocks);
// Identify mock issues
const mockIssues = await this.identifyMockIssues(
mockSetup,
mockExpectations,
mockCleanup,
failure.error_message
);
return {
type: 'mock_error',
description: `Mock error: ${failure.error_message}`,
root_cause: mockIssues.primary_issue,
confidence: this.calculateConfidence(mockIssues.primary_issue),
supporting_evidence: mockIssues.evidence
};
}
private async analyzeAsyncError(failure: TestFailure): Promise {
// Analyze async error
const testCode = await this.getTestCode(failure.test_file);
const asyncOperations = await this.extractAsyncOperations(testCode);
// Analyze promise handling
const promiseHandling = await this.analyzePromiseHandling(asyncOperations);
// Check for missing await
const missingAwait = await this.checkForMissingAwait(asyncOperations);
// Analyze error propagation
const errorPropagation = await this.analyzeErrorPropagation(asyncOperations);
// Check for race conditions
const raceConditions = await this.checkForRaceConditions(asyncOperations);
const rootCause = await this.determineAsyncRootCause(
promiseHandling,
missingAwait,
errorPropagation,
raceConditions
);
return {
type: 'async_error',
description: `Async error: ${failure.error_message}`,
root_cause: rootCause,
confidence: this.calculateConfidence(rootCause),
supporting_evidence: {
promise_handling: promiseHandling,
missing_await: missingAwait,
error_propagation: errorPropagation,
race_conditions: raceConditions
}
};
}
private async analyzeFlakiness(failure: TestFailure): Promise {
// Analyze test flakiness
const testHistory = this.getTestHistory(failure.test_name);
const testCode = await this.getTestCode(failure.test_file);
// Analyze timing dependencies
const timingDependencies = await this.analyzeTimingDependencies(testCode);
// Check for external dependencies
const externalDependencies = await this.analyzeExternalDependencies(testCode);
// Analyze state dependencies
const stateDependencies = await this.analyzeStateDependencies(testCode);
// Check for race conditions
const raceConditions = await this.analyzeRaceConditions(testCode);
// Analyze environmental factors
const environmentalFactors = await this.analyzeEnvironmentalFactors(testHistory);
const rootCause = await this.determineFlakinessCause(
timingDependencies,
externalDependencies,
stateDependencies,
raceConditions,
environmentalFactors
);
return {
type: 'flaky_test',
description: `Flaky test with ${this.getFailureRate(failure.test_name)}% failure rate`,
root_cause: rootCause,
confidence: this.calculateConfidence(rootCause),
supporting_evidence: {
timing_dependencies: timingDependencies,
external_dependencies: externalDependencies,
state_dependencies: stateDependencies,
race_conditions: raceConditions,
environmental_factors: environmentalFactors
}
};
}
}
```
**Step 4: Automated Test Fix Implementation**
**Test Fix Implementation Engine:**
```typescript
class TestFixImplementer {
async implementFix(rootCause: RootCauseAnalysis): Promise {
// Implement fix for root cause
const implementationResult = {
fix_applied: false,
changes_made: [],
verification_results: null,
rollback_plan: null,
success: false
};
try {
// Create backup of current state
const backup = await this.createBackup(rootCause);
implementationResult.rollback_plan = backup;
// Apply fix based on root cause type
const fixResult = await this.applyFix(rootCause);
implementationResult.changes_made = fixResult.changes;
implementationResult.fix_applied = true;
// Verify fix works
const verificationResult = await this.verifyFix(rootCause);
implementationResult.verification_results = verificationResult;
if (verificationResult.success) {
// Run regression tests
const regressionResult = await this.runRegressionTests(rootCause);
if (regressionResult.success) {
implementationResult.success = true;
// Fix successfully implemented
} else {
// Rollback if regression tests fail
await this.rollbackFix(backup);
implementationResult.success = false;
// Fix caused regressions, rolled back
}
} else {
// Rollback if verification fails
await this.rollbackFix(backup);
implementationResult.success = false;
// Fix verification failed, rolled back
}
} catch (error) {
// Rollback on any error
if (implementationResult.rollback_plan) {
await this.rollbackFix(implementationResult.rollback_plan);
}
implementationResult.success = false;
implementationResult.error = error.message;
// Fix implementation failed
}
return implementationResult;
}
private async applyFix(rootCause: RootCauseAnalysis): Promise {
const changes = [];
switch (rootCause.primary_cause.type) {
case 'assertion_error':
changes.push(...await this.fixAssertionError(rootCause));
break;
case 'timeout_error':
changes.push(...await this.fixTimeoutError(rootCause));
break;
case 'mock_error':
changes.push(...await this.fixMockError(rootCause));
break;
case 'async_error':
changes.push(...await this.fixAsyncError(rootCause));
break;
case 'flaky_test':
changes.push(...await this.fixFlakiness(rootCause));
break;
default:
changes.push(...await this.fixGenericError(rootCause));
}
return { changes };
}
private async fixAssertionError(rootCause: RootCauseAnalysis): Promise {
const changes = [];
// Fix based on specific assertion error type
if (rootCause.primary_cause.description.includes('expected')) {
const expectedValue = this.extractExpectedValue(rootCause.primary_cause.description);
const actualValue = this.extractActualValue(rootCause.primary_cause.description);
// Determine if expectation or implementation is wrong
const correctValue = await this.determineCorrectValue(expectedValue, actualValue, rootCause);
if (correctValue === expectedValue) {
// Fix the implementation
changes.push(await this.fixImplementation(rootCause, correctValue));
} else {
// Fix the test expectation
changes.push(await this.fixTestExpectation(rootCause, correctValue));
}
}
return changes;
}
private async fixTimeoutError(rootCause: RootCauseAnalysis): Promise {
const changes = [];
const evidence = rootCause.supporting_evidence;
// Fix slow operations
if (evidence.slow_operations && evidence.slow_operations.length > 0) {
for (const operation of evidence.slow_operations) {
changes.push(await this.optimizeSlowOperation(operation));
}
}
// Fix blocking operations
if (evidence.blocking_operations && evidence.blocking_operations.length > 0) {
for (const operation of evidence.blocking_operations) {
changes.push(await this.fixBlockingOperation(operation));
}
}
// Fix async operations
if (evidence.async_operations && evidence.async_operations.length > 0) {
for (const operation of evidence.async_operations) {
changes.push(await this.fixAsyncOperation(operation));
}
}
// Increase timeout if necessary
if (this.shouldIncreaseTimeout(rootCause)) {
changes.push(await this.increaseTimeout(rootCause));
}
return changes;
}
private async fixMockError(rootCause: RootCauseAnalysis): Promise {
const changes = [];
const evidence = rootCause.supporting_evidence;
// Fix mock setup issues
if (evidence.mock_setup_issues) {
for (const issue of evidence.mock_setup_issues) {
changes.push(await this.fixMockSetup(issue));
}
}
// Fix mock expectations
if (evidence.mock_expectation_issues) {
for (const issue of evidence.mock_expectation_issues) {
changes.push(await this.fixMockExpectation(issue));
}
}
// Fix mock cleanup
if (evidence.mock_cleanup_issues) {
for (const issue of evidence.mock_cleanup_issues) {
changes.push(await this.fixMockCleanup(issue));
}
}
return changes;
}
private async fixAsyncError(rootCause: RootCauseAnalysis): Promise {
const changes = [];
const evidence = rootCause.supporting_evidence;
// Fix missing await
if (evidence.missing_await) {
for (const missingAwait of evidence.missing_await) {
changes.push(await this.addMissingAwait(missingAwait));
}
}
// Fix promise handling
if (evidence.promise_handling_issues) {
for (const issue of evidence.promise_handling_issues) {
changes.push(await this.fixPromiseHandling(issue));
}
}
// Fix error propagation
if (evidence.error_propagation_issues) {
for (const issue of evidence.error_propagation_issues) {
changes.push(await this.fixErrorPropagation(issue));
}
}
// Fix race conditions
if (evidence.race_conditions) {
for (const raceCondition of evidence.race_conditions) {
changes.push(await this.fixRaceCondition(raceCondition));
}
}
return changes;
}
private async fixFlakiness(rootCause: RootCauseAnalysis): Promise {
const changes = [];
const evidence = rootCause.supporting_evidence;
// Fix timing dependencies
if (evidence.timing_dependencies) {
for (const dependency of evidence.timing_dependencies) {
changes.push(await this.fixTimingDependency(dependency));
}
}
// Fix external dependencies
if (evidence.external_dependencies) {
for (const dependency of evidence.external_dependencies) {
changes.push(await this.fixExternalDependency(dependency));
}
}
// Fix state dependencies
if (evidence.state_dependencies) {
for (const dependency of evidence.state_dependencies) {
changes.push(await this.fixStateDependency(dependency));
}
}
// Fix race conditions
if (evidence.race_conditions) {
for (const raceCondition of evidence.race_conditions) {
changes.push(await this.fixRaceCondition(raceCondition));
}
}
// Make test more deterministic
changes.push(await this.makeDeterministic(rootCause));
return changes;
}
private async verifyFix(rootCause: RootCauseAnalysis): Promise {
// Verify fix
// Run the specific test multiple times
const testResults = await this.runTestMultipleTimes(rootCause.test_name, 5);
// Check if all runs pass
const allPassed = testResults.every(result => result.passed);
// Check execution time improvements
const executionTimes = testResults.map(result => result.execution_time);
const averageTime = executionTimes.reduce((a, b) => a + b, 0) / executionTimes.length;
// Check for stability (consistent results)
const isStable = this.checkStability(testResults);
return {
success: allPassed && isStable,
test_passes: allPassed,
stability_improved: isStable,
average_execution_time: averageTime,
test_results: testResults
};
}
private async runRegressionTests(rootCause: RootCauseAnalysis): Promise {
// Run regression tests
// Find related tests
const relatedTests = await this.findRelatedTests(rootCause.test_name);
// Run related tests
const regressionResults = await this.runTests(relatedTests);
// Check for new failures
const newFailures = regressionResults.filter(result => !result.passed);
return {
success: newFailures.length === 0,
tests_run: relatedTests.length,
new_failures: newFailures,
all_results: regressionResults
};
}
}
```
**Step 5: Test Reliability Improvement**
**Test Reliability Enhancement:**
```typescript
class TestReliabilityEnhancer {
async enhanceTestReliability(fixedTests: string[]): Promise {
// Enhance test reliability
const enhancements = {
deterministic_improvements: [],
isolation_improvements: [],
timing_improvements: [],
error_handling_improvements: [],
monitoring_improvements: []
};
for (const testName of fixedTests) {
const testCode = await this.getTestCode(testName);
// Make test more deterministic
const deterministicImprovements = await this.improveDeterminism(testCode);
enhancements.deterministic_improvements.push(...deterministicImprovements);
// Improve test isolation
const isolationImprovements = await this.improveIsolation(testCode);
enhancements.isolation_improvements.push(...isolationImprovements);
// Improve timing reliability
const timingImprovements = await this.improveTiming(testCode);
enhancements.timing_improvements.push(...timingImprovements);
// Improve error handling
const errorHandlingImprovements = await this.improveErrorHandling(testCode);
enhancements.error_handling_improvements.push(...errorHandlingImprovements);
// Add monitoring
const monitoringImprovements = await this.addMonitoring(testCode);
enhancements.monitoring_improvements.push(...monitoringImprovements);
}
return enhancements;
}
private async improveDeterminism(testCode: string): Promise {
const improvements = [];
// Remove random elements
const randomUsages = this.findRandomUsages(testCode);
for (const usage of randomUsages) {
improvements.push({
type: 'remove_randomness',
description: 'Replace random values with deterministic test data',
change: await this.replaceRandomWithDeterministic(usage)
});
}
// Add data seeding
const seedingNeeded = this.needsDataSeeding(testCode);
if (seedingNeeded) {
improvements.push({
type: 'add_data_seeding',
description: 'Add deterministic data seeding',
change: await this.addDataSeeding(testCode)
});
}
// Fix date/time dependencies
const timeUsages = this.findTimeUsages(testCode);
for (const usage of timeUsages) {
improvements.push({
type: 'fix_time_dependency',
description: 'Use deterministic time values',
change: await this.fixTimeDependency(usage)
});
}
return improvements;
}
private async improveIsolation(testCode: string): Promise {
const improvements = [];
// Add proper setup/teardown
const setupTeardownNeeded = this.needsSetupTeardown(testCode);
if (setupTeardownNeeded) {
improvements.push({
type: 'add_setup_teardown',
description: 'Add proper test setup and teardown',
change: await this.addSetupTeardown(testCode)
});
}
// Fix shared state issues
const sharedStateIssues = this.findSharedStateIssues(testCode);
for (const issue of sharedStateIssues) {
improvements.push({
type: 'fix_shared_state',
description: 'Eliminate shared state dependencies',
change: await this.fixSharedState(issue)
});
}
// Add test data isolation
const dataIsolationNeeded = this.needsDataIsolation(testCode);
if (dataIsolationNeeded) {
improvements.push({
type: 'add_data_isolation',
description: 'Isolate test data between tests',
change: await this.addDataIsolation(testCode)
});
}
return improvements;
}
private async improveTiming(testCode: string): Promise {
const improvements = [];
// Add proper waits
const waitingNeeded = this.needsProperWaits(testCode);
if (waitingNeeded) {
improvements.push({
type: 'add_proper_waits',
description: 'Add deterministic waits instead of sleep',
change: await this.addProperWaits(testCode)
});
}
// Fix race conditions
const raceConditions = this.findRaceConditions(testCode);
for (const condition of raceConditions) {
improvements.push({
type: 'fix_race_condition',
description: 'Fix race condition in test',
change: await this.fixRaceCondition(condition)
});
}
// Add timeouts
const timeoutNeeded = this.needsTimeouts(testCode);
if (timeoutNeeded) {
improvements.push({
type: 'add_timeouts',
description: 'Add appropriate timeouts',
change: await this.addTimeouts(testCode)
});
}
return improvements;
}
}
```
**Step 6: Prevention Measures and Best Practices**
**Test Failure Prevention System:**
```typescript
class TestFailurePreventionSystem {
async implementPreventionMeasures(fixedFailures: TestFailure[]): Promise {
// Implement failure prevention measures
const measures = {
linting_rules: [],
test_templates: [],
documentation_updates: [],
ci_cd_improvements: [],
monitoring_additions: []
};
// Analyze patterns in fixed failures
const patterns = this.analyzeFailurePatterns(fixedFailures);
// Create linting rules to prevent similar issues
measures.linting_rules = await this.createLintingRules(patterns);
// Create test templates for common scenarios
measures.test_templates = await this.createTestTemplates(patterns);
// Update documentation with lessons learned
measures.documentation_updates = await this.updateDocumentation(patterns);
// Improve CI/CD pipeline
measures.ci_cd_improvements = await this.improveCICD(patterns);
// Add monitoring for potential issues
measures.monitoring_additions = await this.addMonitoring(patterns);
return measures;
}
private async createLintingRules(patterns: FailurePattern[]): Promise {
const rules = [];
for (const pattern of patterns) {
switch (pattern.type) {
case 'missing_await':
rules.push({
name: 'require-await-async-test',
description: 'Require await for async operations in tests',
rule: 'Always use await for async operations in tests'
});
break;
case 'improper_mocking':
rules.push({
name: 'proper-mock-cleanup',
description: 'Ensure mocks are properly cleaned up',
rule: 'Always reset mocks in teardown'
});
break;
case 'flaky_timing':
rules.push({
name: 'no-sleep-in-tests',
description: 'Avoid sleep/setTimeout in tests',
rule: 'Use deterministic waiting instead of sleep'
});
break;
}
}
return rules;
}
private async createTestTemplates(patterns: FailurePattern[]): Promise {
const templates = [];
// Create templates for common test scenarios
templates.push({
name: 'async-test-template',
description: 'Template for async test with proper error handling',
template: `
describe('{{MODULE_NAME}}', () => {
beforeEach(async () => {
// Setup test data
await setupTestData();
});
afterEach(async () => {
// Cleanup test data
await cleanupTestData();
});
it('should {{TEST_DESCRIPTION}}', async () => {
// Arrange
const testData = await createTestData();
// Act
const result = await {{FUNCTION_NAME}}(testData);
// Assert
expect(result).toBeDefined();
expect(result).toEqual(expectedResult);
});
});`
});
templates.push({
name: 'mock-test-template',
description: 'Template for tests with proper mocking',
template: `
describe('{{MODULE_NAME}}', () => {
let mockDependency: jest.Mocked;
beforeEach(() => {
mockDependency = {
method: jest.fn()
};
});
afterEach(() => {
jest.clearAllMocks();
});
it('should {{TEST_DESCRIPTION}}', async () => {
// Arrange
mockDependency.method.mockResolvedValue(expectedValue);
// Act
const result = await {{FUNCTION_NAME}}(mockDependency);
// Assert
expect(result).toEqual(expectedResult);
expect(mockDependency.method).toHaveBeenCalledWith(expectedInput);
});
});`
});
return templates;
}
}
```
**Test Debugging Quality Checklist:**
- [ ] All failing tests are analyzed and categorized
- [ ] Root causes are identified through systematic debugging
- [ ] Fixes address root causes, not just symptoms
- [ ] Fixes are verified to work correctly
- [ ] Regression tests confirm no new issues introduced
- [ ] Test reliability is improved through enhancements
- [ ] Prevention measures are implemented
- [ ] Knowledge is documented for future reference
**Agent Coordination for Test Debugging:**
For comprehensive test debugging, I coordinate multiple specialized agents through Task tool:
**Agent Coordination Flow:**
1. **Failure Analysis Agent** โ Provides categorized failure analysis to all other agents
2. **Fix Implementation Agent** โ Receives analysis, implements fixes, reports to validation
3. **Validation Agent** โ Receives fixes, validates effectiveness, reports to regression prevention
4. **Regression Prevention Agent** โ Receives validation results, runs regression tests, reports to prevention
5. **Prevention Enhancement Agent** โ Receives all reports, implements prevention measures, generates final report
**Parallel Processing Architecture:**
- All agents operate simultaneously using Task tool parallelism
- Shared analysis files enable coordination without blocking
- Each agent focuses on their specialized domain
- Real-time progress reporting ensures comprehensive coverage
- Automatic rollback capabilities if any agent detects issues
This Task tool-based approach provides true parallelism and systematic test debugging coverage.
**Anti-Patterns to Avoid:**
- โ Commenting out failing tests (avoiding the problem)
- โ Quick fixes without understanding root causes (symptom fixing)
- โ Ignoring test environment issues (unreliable infrastructure)
- โ Skipping regression testing (introducing new problems)
- โ Accepting flaky tests as normal (poor test quality)
- โ Not implementing prevention measures (repeated failures)
**Final Verification:**
Before completing test debugging:
- Are all failing tests fixed and passing?
- Are root causes identified and addressed?
- Are fixes verified to work correctly?
- Are regression tests passing?
- Are prevention measures implemented?
- Is debugging knowledge documented?
**Final Commitment:**
- **I will**: Perform comprehensive root cause analysis for all failing tests
- **I will**: Use multiple agents for parallel debugging and fixing
- **I will**: Address root causes, not just symptoms
- **I will**: Verify fixes work and don't introduce regressions
- **I will NOT**: Comment out failing tests or apply quick fixes
- **I will NOT**: Ignore test environment or infrastructure issues
- **I will NOT**: Skip prevention measures or documentation
**REMEMBER:**
This is TEST DEBUGGING AND FIXING mode - comprehensive root cause analysis, systematic fixing, and prevention implementation. The goal is to create reliable, maintainable tests that provide accurate feedback about code quality.
Executing comprehensive test debugging and fixing protocol for reliable test suite...