Roll Your Own Test Framework

Level: Advanced 60–90 min

Concepts: Design PatternsIncremental DesignBoundaries

Solutions: C# | TypeScript | Python

---

Build a minimal unit testing framework and command-line runner. The twist: use TDD to build a tool for TDD.

Inspired by Jason Gorman’s Codemanship kata.

Requirements

Step 1: Test Discovery

Define a convention for marking test methods. Options:

• Naming convention: methods starting with test_ or test
• Annotations/decorators: @Test or [Test]
• Registration: suite.addTest("name", function)

The framework should find all test methods in a given class or module.

Step 2: Test Execution

Run each discovered test and capture the result:

• PASS — test completed without errors
• FAIL — an assertion was not met
• ERROR — an unhandled exception occurred

Each test runs independently — a failure in one test does not prevent others from running.

Step 3: Assertions

Provide at minimum:

• assertTrue(condition) — fails if condition is false
• assertEqual(expected, actual) — fails if values don’t match
• assertThrows(fn) — fails if the function doesn’t throw

Failed assertions should include a message describing what went wrong:

assertEqual failed: expected 5 but got 3

Step 4: Result Reporting

After all tests run, print a summary:

Tests run: 5, Passed: 3, Failed: 1, Errors: 1

FAIL: test_addition - assertEqual failed: expected 5 but got 3
ERROR: test_division - ZeroDivisionError: division by zero

Test Cases

Scenario	Expected
Discover 0 tests	Reports “0 tests run”
Discover 3 test methods	Finds and runs all 3
Passing test	Reported as PASS
Failing assertion	Reported as FAIL with message
Unhandled exception	Reported as ERROR with exception info
Multiple tests, mixed results	Summary counts correct
assertEqual(5, 5)	Passes silently
assertEqual(5, 3)	Fails with “expected 5 but got 3”
assertTrue(false)	Fails with “expected true”
assertThrows with no throw	Fails with “expected exception”

Bonus

• Add setUp() and tearDown() methods that run before/after each test
• Add parameterized tests — run the same test with multiple input sets
• Add test filtering — run only tests matching a pattern
• Add timing — report how long each test took
• Add color output — green for pass, red for fail

Hint

Start with assertEqual. You need it to test everything else. Then build assertTrue on top of it. Once assertions work, build the runner that catches assertion failures vs unhandled exceptions. Test discovery comes last — it’s language-specific plumbing.

Reference Walkthrough

Full C#, TypeScript, and Python implementations live at tddbuddy-reference-katas/roll-your-own-test-framework with twelve scenarios across all three languages, a fluent TestSuiteBuilder, and the domain exception AssertionFailedException that distinguishes FAIL from ERROR. This is a meta-kata — the discovery mechanism diverges sharply per language (C# reflection, TS object keys, Python inspect + test_ prefix).

• C# (.NET 8, xUnit, FluentAssertions) — walkthrough
• TypeScript (Node 20, Vitest, strict types) — walkthrough
• Python (3.11, pytest, dataclasses) — walkthrough

This kata ships in Agent Full-Bake mode (middle gear) — a single commit per language with the full domain design. See the repo’s Gears section for why that’s a deliberate teaching choice.

---

Related reading

• Test Deletion Is a Privileged Operation
  The cheapest way for an agent to make a failing test pass is to delete it. That is logical for the agent and catastrophic for the codebase. Tests are append-only by default. Deletion needs a human author, a separate commit, and a separate review.
• The Instruction File Is Not the Discipline
  Teams that want their agent to do TDD reach for the instruction file. The TDAD paper measured this directly: adding a 'do TDD' instruction raised regressions by nearly two-thirds. Contextual test discovery cut them by seventy percent. The instruction is theatre. The codebase's test surface is the discipline.
• Tamper-Resistant Test Design Is What the Suite Now Owes the Codebase
  Tests were a contract the team made with itself. Agents made the suite an attack surface. Tamper-resistant test design is what the suite now owes the codebase: expensive to spoof, redundantly verified, partially hidden, and impossible to delete without a separate review.