Thea Code Testing Guide

Status: Published Last Updated: 2025-08-10 Category: Developer Guide

Overview

This guide provides comprehensive information about testing in Thea Code, including unit tests, integration tests, end-to-end tests, and benchmarks.

Test Infrastructure
Running Tests
Test Coverage
Unit Testing
Integration Testing
End-to-End Testing
Benchmark Testing
Writing Tests
Common Issues

Test Infrastructure

Test Framework

Thea Code uses the following testing tools:

Jest - Primary test runner for unit and integration tests
Mocha - Used for VSCode extension integration tests
Playwright - Browser automation testing
Custom benchmark harness - For performance testing

Directory Structure

tests/
├── src/__tests__/           # Unit tests (co-located with source)
├── e2e/                     # End-to-end VSCode integration tests
├── test/                    # Test utilities and mock servers
│   ├── generic-provider-mock/
│   ├── mcp-mock-server/
│   ├── openai-mock/
│   └── roo-migration/
└── benchmark/               # Performance benchmarks

Running Tests

All Tests

npm test

Unit Tests Only

npm run test:unit

Integration Tests

npm run test:integration

End-to-End Tests

npm run test:e2e

Coverage Report

npm run test:coverage

Watch Mode

npm run test:watch

Test Coverage

Current Coverage Status

The project maintains a comprehensive test coverage checklist in MASTER_TEST_CHECKLIST.md, generated automatically by:

npm run generate:test-checklist

Coverage Goals

Overall coverage: >80%
Critical paths: >95%
New code: 100%

Viewing Coverage Reports

After running coverage tests, view the HTML report:

open coverage/lcov-report/index.html

Unit Testing

Writing Unit Tests

Unit tests are co-located with source files using the __tests__ directory pattern:

// src/utils/__tests__/path.test.ts
import { arePathsEqual, formatPath } from '../path';

describe('Path Utilities', () => {
  describe('arePathsEqual', () => {
    it('should handle case-insensitive comparison on Windows', () => {
      // Test implementation
    });
  });
});

Mocking

Common mocks are provided in src/__mocks__/:

vscode.js - VSCode API mocks
@modelcontextprotocol/sdk - MCP SDK mocks
fs/promises.ts - File system mocks

Best Practices

Test naming: Use descriptive names that explain what is being tested
Arrange-Act-Assert: Structure tests clearly
One assertion per test: Keep tests focused
Mock external dependencies: Isolate unit tests

Integration Testing

Provider Testing

Test API providers with mock servers:

// src/api/providers/__tests__/ollama.test.ts
describe('Ollama Provider', () => {
  beforeAll(async () => {
    // Start mock server
    await startMockOllamaServer();
  });

  afterAll(async () => {
    // Cleanup
    await stopMockOllamaServer();
  });

  test('should handle streaming responses', async () => {
    // Test implementation
  });
});

MCP Integration Testing

Test MCP (Model Context Protocol) integration:

// src/services/mcp/__tests__/McpIntegration.test.ts
describe('MCP Integration', () => {
  test('should register and execute tools', async () => {
    // Test MCP tool registration and execution
  });
});

End-to-End Testing

VSCode Extension Tests

The e2e/ directory contains full VSCode extension integration tests:

// e2e/src/suite/extension.test.ts
suite('Extension Test Suite', () => {
  test('Extension should activate', async () => {
    const extension = vscode.extensions.getExtension('SolaceHarmony.thea-code');
    assert.ok(extension);
    await extension.activate();
  });
});

Running E2E Tests

# Run VSCode integration tests
npm run test:e2e

# With specific VSCode version
npm run test:e2e -- --vscode-version 1.85.0

Benchmark Testing

Benchmark Harness

The benchmark suite tests performance across multiple languages:

# Build and start Docker environment
npm run docker:start

# Run specific benchmark
npm run docker:benchmark -- -e exercises/javascript/binary

# Run all benchmarks for a language
npm run cli -- run javascript all

# Run all benchmarks
npm run cli -- run all

Supported Languages

C++
Go
Java
JavaScript
Python
Rust

Writing Benchmarks

Create benchmark exercises in benchmark/exercises/[language]/:

// benchmark/exercises/javascript/example.js
export const exercise = {
  name: 'Example Exercise',
  prompt: 'Implement a function that...',
  validate: (result) => {
    // Validation logic
    return result === expectedOutput;
  }
};

Writing Tests

Test Template

import { describe, test, expect, beforeEach, afterEach } from '@jest/globals';

describe('ComponentName', () => {
  let component: ComponentType;

  beforeEach(() => {
    // Setup
    component = new ComponentType();
  });

  afterEach(() => {
    // Cleanup
    jest.clearAllMocks();
  });

  describe('methodName', () => {
    test('should handle normal case', () => {
      // Arrange
      const input = 'test';
      
      // Act
      const result = component.method(input);
      
      // Assert
      expect(result).toBe('expected');
    });

    test('should handle edge case', () => {
      // Test edge cases
    });

    test('should handle error case', () => {
      // Test error handling
      expect(() => component.method(null)).toThrow();
    });
  });
});

Async Testing

test('should handle async operations', async () => {
  const result = await asyncFunction();
  expect(result).toBeDefined();
});

test('should handle streaming', async () => {
  const stream = getStream();
  const chunks = [];
  
  for await (const chunk of stream) {
    chunks.push(chunk);
  }
  
  expect(chunks).toHaveLength(3);
});

Common Issues

Port Conflicts

Tests use dynamic port assignment to avoid conflicts:

import { findAvailablePort } from '../utils/port-utils';

const port = await findAvailablePort(10000);
server.listen(port);

Timeout Issues

Adjust timeouts for slow operations:

test('long running operation', async () => {
  // Test implementation
}, 30000); // 30 second timeout

Mock Server Issues

Ensure mock servers are properly started and stopped:

beforeAll(async () => {
  await waitForPortAvailable(mockPort);
  await startMockServer();
  await waitForPortInUse(mockPort);
});

afterAll(async () => {
  await stopMockServer();
});

Environment Variables

Set required environment variables for tests:

# .env.test
OPENROUTER_API_KEY=test-key
ANTHROPIC_API_KEY=test-key

CI/CD Integration

GitHub Actions

Tests run automatically on:

Pull requests
Push to main branch
Release tags

Pre-commit Hooks

Install pre-commit hooks:

npm run prepare

This runs tests before commits to catch issues early.

Test Improvements

Recent Improvements

Port Management - Dynamic port assignment to prevent conflicts
Timeout Handling - Proper timeout configuration for async operations
Mock Servers - Improved mock server setup and teardown
Coverage Tracking - Automated coverage reporting

Planned Improvements

Increase coverage to >90%
Add performance regression tests
Implement visual regression testing for webview
Add mutation testing

Support

For test-related issues:

Check the Common Issues section
Review existing GitHub Issues
Ask in Discord

Changelog:

2025-08-10: Consolidated testing documentation from multiple sources

Uh oh!

FilesExpand file tree

README.md

Latest commit

History