Skip to content

testing.md

Latest commit

 

History

History
557 lines (401 loc) · 11 KB

testing.md

File metadata and controls

557 lines (401 loc) · 11 KB
Raw

Testing Guide

This document describes the testing philosophy, patterns, and best practices for A.R.C. CLI.

Testing Philosophy

A.R.C. CLI follows a lenient testing approach that balances quality with development velocity:

  • Focus on Functionality: Tests ensure code works correctly, not just coverage numbers
  • Pragmatic Coverage: 60-70% overall coverage (75%+ for critical paths)
  • Clear Intent: Test names describe what's being tested and why
  • Maintainability: Use helpers and fixtures to reduce duplication
  • Fast Execution: Unit tests complete in < 10 seconds

Quick Start

Running Tests

# Run all unit tests
make test

# Run with coverage report
make test-coverage

# Generate HTML coverage report
make test-coverage-html

# Run integration tests (slower)
make test-integration

# Run benchmarks
make test-bench

Writing Your First Test

package mypackage

import (
  "testing"

  "github.com/stretchr/testify/require"
)

func TestMyFunction(t *testing.T) {
  result := MyFunction("input")
  require.Equal(t, "expected", result)
}

Test Organization

Co-Located Tests

Tests live alongside the code they test:

pkg/state/
├── state.go
├── state_test.go          # Unit tests for state.go
├── storage.go
├── storage_test.go        # Unit tests for storage.go
├── integration_test.go    # Integration tests (build tag)
└── testdata/
    └── *.yaml             # Test fixtures

Build Tags for Integration Tests

Integration tests use build tags to separate slow tests from fast unit tests:

//go:build integration

package state

import "testing"

func TestFullStateLifecycle(t *testing.T) {
  // Integration test using real file system
}

Run integration tests separately:

go test -tags=integration ./...

Test Data

Test fixtures belong in testdata/ directories:

pkg/state/testdata/
├── valid_state.yaml
├── corrupted_state.yaml
└── empty_state.yaml

Writing Tests

Table-Driven Tests

Use table-driven tests for multiple scenarios:

func TestStateValidation(t *testing.T) {
tests := []struct {
name    string
state   *State
wantErr bool
}{
{
name:    "valid state",
state:   &State{Version: 1},
wantErr: false,
},
{
name:    "invalid version",
state:   &State{Version: 0},
wantErr: true,
},
}

for _, tt := range tests {
t.Run(tt.name, func (t *testing.T) {
err := tt.state.Validate()
if tt.wantErr {
require.Error(t, err)
} else {
require.NoError(t, err)
}
})
}
}

Test Suites

Use testify suites for shared setup/teardown:

import "github.com/stretchr/testify/suite"

type StorageTestSuite struct {
suite.Suite
tempDir string
storage *Storage
}

func (s *StorageTestSuite) SetupTest() {
s.tempDir = s.T().TempDir()
s.storage = NewStorage(s.tempDir)
}

func (s *StorageTestSuite) TestReadState() {
// Test implementation
}

func TestStorageSuite(t *testing.T) {
suite.Run(t, new(StorageTestSuite))
}

Integration Tests

Integration tests use real dependencies:

//go:build integration

func TestStateStorageIntegration(t *testing.T) {
// Use real file system
dir := t.TempDir()
storage := NewStorage(dir)

// Test full workflow
state := ValidState()
err := storage.WriteState(state)
require.NoError(t, err)

loaded, err := storage.ReadState()
require.NoError(t, err)
require.Equal(t, state.Version, loaded.Version)
}

Test Utilities

A.R.C. provides shared test utilities in internal/testing/:

Helpers

import arctest "github.com/arc-framework/arc-cli/internal/testing"

// Create temporary directory
dir := arctest.TempDir(t)

// Capture output
stdout, stderr := arctest.CaptureOutput(t, func () {
fmt.Println("hello")
})

// Create test files
path := arctest.CreateFile(t, "test.yaml", []byte("content"))

// Create state files
arctest.CreateStateFile(t, dir, validState)

Mocks

// Mock file system
mockFS := arctest.NewMockFS()
mockFS.WriteFile("state.yaml", []byte("data"), 0644)

// Mock clock
mockClock := arctest.NewMockClock(time.Now())
mockClock.Advance(1 * time.Hour)

// Simulate errors
mockFS.SimulateError("/path", os.ErrPermission)

Fixtures

// Valid test data
state := arctest.ValidState()
history := arctest.ValidHistory()

// Test data with specific characteristics
stateWithResources := arctest.StateWithResources(5)
historyWithOps := arctest.HistoryWithOperations(100)

// Invalid/corrupted data for error testing
corruptedYAML := arctest.CorruptedStateYAML()

Custom Assertions

// File assertions
arctest.AssertFileExists(t, "/path/to/file")
arctest.AssertFileContent(t, path, expectedBytes)

// YAML assertions (ignores formatting)
arctest.AssertYAMLEqual(t, expectedState, actualState)

// Directory assertions
arctest.AssertDirExists(t, "/path/to/dir")

Running Tests

Command Line

# Run all tests with verbose output
go test -v ./...

# Run tests with race detector
go test -race ./...

# Run tests in a specific package
go test -v ./pkg/state

# Run a specific test
go test -v -run TestMyFunction ./pkg/mypackage

# Run integration tests
go test -v -tags=integration ./...

# Run with coverage
go test -coverprofile=coverage.txt ./...
go tool cover -html=coverage.txt

# Run benchmarks
go test -bench=. -benchmem ./...

Make Targets

make test                  # Run all unit tests
make test-coverage         # Run with coverage report
make test-coverage-html    # Generate HTML coverage
make test-integration      # Run integration tests
make test-bench            # Run benchmarks

CI/CD

Tests run automatically on:

  • Every commit (via pre-commit hook)
  • Every pull request (GitHub Actions)
  • Before release (release workflow)

Coverage Guidelines

Coverage Targets

Priority Package Target Rationale
P0 pkg/state 75%+ Critical state management
P0 pkg/state/storage 75%+ File I/O correctness
P1 internal/state 70%+ App configuration
P1 pkg/cli 60%+ Command execution
P2 pkg/ui/* 40-50% Visual output, hard to test

What to Prioritize

High Priority:

  • State management (CRUD operations)
  • File I/O operations
  • Data validation and parsing
  • Error handling paths
  • Business logic

Low Priority:

  • UI rendering (visual output)
  • Banner generation
  • Color/theme selection
  • Simple getter/setter methods

Viewing Coverage

# Terminal summary
make test-coverage

# HTML report (detailed view)
make test-coverage-html
open coverage.html

# Package-level coverage
go test -cover ./pkg/state

Best Practices

DO ✅

  • Use descriptive test names: TestStorageReadState_MissingFile_ReturnsDefault
  • Use require for critical checks: Early exit on failure
  • Use assert for non-critical checks: Continue running other assertions
  • Use t.Helper(): Mark helper functions for better error messages
  • Use t.TempDir(): Automatic cleanup of test directories
  • Test error paths: Ensure errors are handled correctly
  • Use table-driven tests: Multiple scenarios in one test
  • Leverage test utilities: Reduce duplication with helpers

DON'T ❌

  • Don't use time.Sleep(): Use MockClock or channels for synchronization
  • Don't create global state: Each test should be independent
  • Don't use real file paths: Use t.TempDir() or mocks
  • Don't test implementation details: Test behavior, not internals
  • Don't duplicate test code: Extract common patterns into helpers
  • Don't skip test cleanup: Always clean up resources
  • Don't write flaky tests: Ensure deterministic outcomes

Assertions

// Use require for critical checks (stops test on failure)
require.NoError(t, err)
require.NotNil(t, result)
require.Equal(t, expected, actual)

// Use assert for non-critical checks (continues on failure)
assert.Equal(t, expected, actual)
assert.True(t, condition)
assert.Contains(t, slice, item)

Troubleshooting

Tests Are Slow

Problem: Test suite takes > 10 seconds

Solutions:

  • Use mocks instead of real dependencies
  • Separate integration tests with build tags
  • Run tests in parallel: t.Parallel()
  • Use testing.Short() for quick smoke tests

Flaky Tests

Problem: Tests pass sometimes, fail other times

Common Causes:

  • Race conditions (use -race to detect)
  • Time-dependent code (use MockClock)
  • Uninitialized state (check setup/teardown)
  • External dependencies (use mocks)

Coverage Too Low

Problem: Coverage below 60% threshold

Solutions:

  • Focus on critical packages first (state management)
  • Test error paths (often missed)
  • Add integration tests for workflows
  • Don't test trivial code (getters/setters)

Import Cycle

Problem: import cycle not allowed

Solution:

  • Move test utilities to internal/testing/
  • Use interfaces to break dependencies
  • Restructure packages if necessary

Examples

Simple Unit Test

func TestResourceValidation(t *testing.T) {
r := Resource{
ID:   "res-123",
Type: "compute",
Name: "My Resource",
}

require.Equal(t, "res-123", r.ID)
require.Equal(t, "compute", r.Type)
}

Table-Driven Test

func TestParseVersion(t *testing.T) {
tests := []struct {
name    string
input   string
want    int
wantErr bool
}{
{"valid", "1", 1, false},
{"invalid", "abc", 0, true},
{"empty", "", 0, true},
}

for _, tt := range tests {
t.Run(tt.name, func (t *testing.T) {
got, err := ParseVersion(tt.input)
if tt.wantErr {
require.Error(t, err)
} else {
require.NoError(t, err)
require.Equal(t, tt.want, got)
}
})
}
}

Using Mocks

func TestStorageWithMockFS(t *testing.T) {
mockFS := arctest.NewMockFS()
mockFS.WriteFile("state.yaml", []byte("version: 1"), 0644)

data, err := mockFS.ReadFile("state.yaml")
require.NoError(t, err)
require.Contains(t, string(data), "version: 1")
}

Integration Test

//go:build integration

func TestEndToEndWorkflow(t *testing.T) {
dir := t.TempDir()

// Create storage
storage := NewStorage(dir)

// Write state
state := arctest.ValidState()
err := storage.WriteState(state)
require.NoError(t, err)

// Read back
loaded, err := storage.ReadState()
require.NoError(t, err)
arctest.AssertYAMLEqual(t, state, loaded)
}

Resources

Contributing

When contributing tests:

  1. Follow existing patterns and conventions
  2. Write clear, descriptive test names
  3. Add tests for all new features
  4. Ensure tests pass: make test
  5. Check coverage: make test-coverage
  6. Run pre-commit checks: make pre-commit

Questions? Check the Development Guide or open an issue.