TLDR: I made a small typescript-compiling front-end for the nodejs test runner: https://www.npmjs.com/package/@robey/gauntlet
I’ve been on a quest to reduce dependencies in my nodejs projects, partly because of things like the left-pad incident and the debug/chalk incident, and partly because my ideal codebase is simple and organized. Deeply nested dependency trees of unknown provenance look to me like a mess to be cleaned up.
You’d be surprised how many important, useful libraries have an indirect dependency on things like “is-arrayish”, a polyfill for Array.isArray (available since ES5 15 years ago). Or “strip-ansi”, a one-line call to string.replace that itself imports a one-line regex from a different package. Most are relics of a past age or land grab of npm’s namespace, and probably considered harmless. But each one adds a layer of needless complexity and another potential attack vector.
As I’ve gradually cut back these dependency weeds, one tool has remained: mocha. It has a lot of dependencies. I’ve been eyeing it for a while, trying to figure out how to replace it, but building my own testing framework is a much bigger project than I want to take on. And in the javascript world, mocha is not a testing framework, it’s the testing framework. Whenever someone recommends a hot new alternate, I find out it’s just a new layer built on top of mocha. Mocha is still there underneath.
There’s also the uncomfortable truth that mocha is not very typescript-friendly. The favored way to run typescript tests is by running mocha with a special config file that sets the loader to “ts-node”, a just-in-time typescript compiler which is increasingly unmaintained and now emits warnings each time you run it. It would be nice to have a test runner that compiled the tests before running them, using the same typescript module I have in my project already.
I’ve noticed that some build systems (like vite) solve this by ignoring the types in tests, and run them as if they were untyped javascript. It’s much faster, no doubt, but I consider a type mismatch between my tests and library to be a bug, so I want that checked explicitly as part of the unit tests. And I want that to happen at build time, not just IDE time.
By the way, this is not a call to bully the maintainers of mocha! I’ve been using it for many years and it’s worked well enough that I’ve rarely thought about it until recently. Mocha clearly strives for long-term backward compatibility in the face of the moving target of nodejs, a commendable and difficult job. They also now have to support all these other test frameworks built on top of them. I think they’re doing great. I just have different goals.
The gauntlet
The idea remained stuck in the back of my head: I could probably replace mocha with a small library, like a minimal “pytest” that just discovered test files, defined describe and it functions, and ran them. But the word “just” is doing a lot of work there, and the more I thought about how to code it, the more load-bearing it became. It seemed like a huge effort for low payoff.
A few years ago, I noticed that nodejs’s builtin “assert” module had quietly added features like regex matching and promises, and could now handily replace my use of “should.js”. It didn’t have every feature I was using, but it had enough that I could make it work. One more dependency down.
And then at some point last year, I saw a new builtin module: “test”. Turns out some nodejs contributors had been thinking along similar lines, and had added most of the features of a standard test runner to the standard library. It can collect tests and suites into a tree, run them, and report the results, either as asynchronous events in javascript, or as one of a few simple text formats. In many cases, you could probably use this module as-is, and remove all your testing dependencies.
I was nerd-sniped. In my spare time across a couple of weeks, I hacked up a front-end to node:test to cover my needs:
- Compile (changed) tests into a cache folder.
- Run them.
- Display a simple summary on success, and details (including stdout & stderr) for any failures.
I call it “gauntlet” after the english phrase “running the gauntlet”, which I recently discovered is actually a corruption/confusion of the swedish word “gatlopp”, and has nothing to do with the glove.
Typescript
The hardest part turned out to be running the typescript compiler as a library.
I’d already done this before. For many years, typescript has been my preferred config file format for JS projects. It’s an easy way to enforce a structured schema, define some custom types (like duration), and let the server fail immediately on startup for errors or typos. I used some sample code to drive the compiler API. It was pretty slow, but that didn’t matter much since a config file only gets recompiled if it changes. I don’t actually remember where I found the sample code. It was probably an earlier version of this “minimal compiler” from the typescript wiki.
Compiling a few dozen test files made speed a real problem, though, since each file took about one second. I want tests to build and run quickly for rapid iteration! Mocha + ts-node are much faster. They’re obviously not using the sample code from the wiki. It was time to get my hands dirty and figure out what they were doing differently.
The sample code in the “minimal compiler” isn’t much more than three steps (and three lines): build a compiler, compile a file, and get the list of errors. The code in ts-node is… a lot. Thousands of lines. Eventually I figured out the core difference, and whittled it down to about 100 lines that could reproduce ts-node’s speed. Their trick is to create a “language service” that can cache file & module contents across multiple typescript source files. The language service requires an environment to handle filesystem access and project metadata – things an IDE might want to control. The ts-node compiler builds a fake environment for each file, directing the language service back at typescript’s own internal implementations. The cache is the secret speed sauce.
It became clear to me that typescript was never really meant to be used as a library like this. Kudos to the author(s) of ts-node for figuring it out and making it look simple, despite the underlying complexity! If the project became unmaintained because they’ve moved to the forest to become a landscape painter, I wish them a happy retirement.
node:test
I wrote code to compile every test file from tests/ into a .gauntlet/ folder, then copy over everything from lib/ too (and any other configured asset folders). That way, tests can use relative imports while remaining in a relatively clean environment, separate from the project workspace.
After that, it was just a matter of hooking up the nodejs test runner, listening for events, assembling those events into some coherent story for the end user… and lots of debugging.
The test module is new enough that the event API documentation is sparse, and it still has a lot of quirks. I assume it’s too late to make significant changes, but if it wasn’t, here are things I would suggest for a future version:
There’s no easy way to know how many test cases are going to run until they finish. The
enqueue/dequeueevents report when a file is processed, but it would help to also report how many test cases they found in each file. This would let a runner estimate progress once the test cases start running… and even show a progress bar.Use different event names for different event types. The
startevent currently means the test runner started reading a file, or possibly started processing a test suite, or possibly started running an individual test case. It’s difficult to infer which one. They should be 3 different events.Some events (like
start) only pass the name of the individual test case, not the full path of enclosing suites. For example, if a suite"queue worker"has a test case"handles shutdown", you’ll get astartevent that only names"handles shutdown". Gauntlet has some convoluted logic to model the tree and figure out the path to each test, but the test runner already has this model of the tree. It built it when reading the test files. It should include the full tree path in each event that refers to a suite or test case.Test case timeouts seem to include the time the test (and its file) are enqueued in the test runner. So if there are many more test files than cores, and some of the files have slow tests, a file can “timeout” with an error, without running any of the tests, before the file is even executed. The runner should start the timer only when a test case is actually started.
Results
The results beat my expectations! Tests are much faster now than they were under mocha.
All of that speed is due to the node:test module, not me. It seems to be launching a thread on each core and running all the test files in parallel. Once the VMs warm up, it can fly through them.
So I’m releasing gauntlet. Zero dependencies – though you’ll need typescript, of course. I gave it an MIT license because it’s not really much code, and it relies entirely on nodejs libraries that use MIT(-ish) licenses themselves, so that seemed only fair.
I admit my requirements are pretty narrowly focused: typescript compilation, ESM, and no dependencies. But if that describes you too, check it out.