Why You Should Write Your Own Test Runner
Writing a test runner isn't difficult, and doing so gives you full control over how tests are executed. Many developers default to using pre-built testing frameworks like Jest, Mocha, or PyTest, but these tools come with hidden assumptions and unnecessary overhead. By writing your own test runner, you eliminate black-box behaviors and ensure that your tests run exactly the way you intend.
The Basic Structure of a Test Runner
At its core, a test runner only needs to:
- Parse parameters (e.g., which test files to run).
- Loop through a directory to find test files.
- Loop over the tests, execute them, and store results in a hashmap.
- Print out the results in a readable format.
That’s it. Everything beyond this—assertion libraries, pretty output, parallel execution—is just added convenience.
Benefits of Writing Your Own Test Runner
1. Learning Opportunity
Writing a test runner helps you deeply understand the mechanics of testing. You gain direct experience with test isolation, assertions, setup/teardown logic, and how execution order affects test reliability. By implementing these features yourself, you build an intuitive understanding of testing that goes beyond simply using a framework.
2. Avoid Hidden Assumptions
Pre-built test runners often include behaviors that aren’t always obvious, such as:
- Running tests in parallel by default (which can be disastrous if your tests make real API calls and write to a database).
- Automatically stubbing/mocking dependencies in unexpected ways.
- Adding implicit assertions that interfere with debugging.
When you write your own test runner, you control everything. There are no surprises, no unexpected behaviors, and no need to inspect the internals of a third-party library to understand what’s happening under the hood.
3. Natural Extension of Application Growth
As your application evolves, testing often shifts from high-level integration tests to more granular unit tests. A custom test runner allows you to tailor the testing process to match this evolution. Instead of adapting your project to fit an off-the-shelf framework, you can build a test runner that aligns perfectly with your project’s needs.
4. No Overhead
Popular test runners introduce unnecessary learning curves. You have to:
- Learn new assertion functions.
- Understand various patterns for table-driven tests.
- Configure test environments.
When you write your own runner, you skip all of this. You write tests in a format that makes sense to you, using patterns that fit naturally with your project.
5. Flexibility and Easy Extensibility
A custom test runner gives you absolute control over:
- How test results are formatted.
- Whether tests run sequentially or in parallel.
- The level of logging and debugging information available.
- Any additional features you want to include, such as custom assertion messages.
- Easy extensions for debugging and advanced reporting.
For example, in a custom request flow test runner, you could add a stack trace for any failed assertion. This would allow you to see exactly where unexpected output or return values occurred.
Example: “Oh, okay, it was after the second user did a PUT, and the third user tried to do a GET but got nothing, even though they should have had access to the object.”
This level of visibility and control is difficult to achieve with pre-built test runners unless you modify their source code.
When Established Tools Make Sense
While writing your own test runner is often a better approach, there are cases where using a pre-built solution makes sense:
- Team consistency – If you’re working in a large team, established frameworks provide a standard way to write and execute tests.
- CI/CD integration – Many testing frameworks integrate easily with existing CI/CD pipelines.
- Complex formatting & reporting – If you need advanced reporting features, using a pre-built framework can save time.
For unit tests, it's more of a toss-up—pre-written frameworks can be useful if you just need quick, lightweight checks. However, for tests handling larger chunks of code, such as integration or system tests, writing your own test runner is often the better choice. It gives you full control over execution order, dependencies, and prevents unwanted parallel execution that might interfere with real API calls or database writes.
That said, even in a team setting, if you document your test runner well, colleagues can easily contribute and use it effectively. The main reason to use a pre-written tool is convenience—not necessity.
Writing Your Own Test Runner vs. Using a Pre-Written One
| Feature | Writing Your Own | Pre-Written Test Runner |
|---|---|---|
| Control over test execution | ✅ Full control | ❌ Limited to built-in patterns |
| Learning experience | ✅ High | ❌ Low (Black-box abstraction) |
| Setup complexity | ✅ Minimal | ❌ Can require config and dependencies |
| Readability of results | ❌ Requires effort | ✅ Built-in formatting |
| Built-in assertion helpers | ❌ You write your own | ✅ Comes with framework |
| Integration with CI/CD | ❌ Manual setup | ✅ Usually plug-and-play |
Final Thoughts
Most developers default to using Jest, Mocha, or PyTest without questioning whether they actually need them. But if you take the time to write your own test runner, you’ll:
- Gain a deeper understanding of how tests are executed.
- Avoid hidden assumptions that could cause issues.
- Have full control over execution order (ensuring, for example, that database-modifying tests don’t run in parallel unless explicitly allowed).
- Eliminate unnecessary dependencies and complexity in your projects.
- Easily extend and customize your test runner to suit your specific debugging and reporting needs.
For solo projects, internal tools, or highly customized workflows, writing your own test runner is often the simpler, better option.