Why Your AI-Generated Tests Don't Catch Anything

I asked for tests on a rate limiter last month. I got twelve of them, all green, nice describe blocks, sensible names. I felt good about it for exactly as long as it took someone to send 101 requests in the same millisecond and get a 200 on all of them.

The bug was a fixed-window counter that should have been sliding. And one of the twelve tests — I checked — asserted that exact fixed-window behavior. Not the requirement. The implementation. The test wasn't checking my rate limiter was correct. It was checking that my rate limiter was consistent with itself.

The green checkmark lies

Here's the sequence that produces this every time: you write a function, you ask for tests, the model reads the function you just wrote and generates assertions that match what it does. Not what it's supposed to do — there's no spec in the context, usually, just the code. The model doesn't know your fixed window was meant to be sliding. It only has the fixed window. So it writes a test that says "given these inputs, the fixed window produces this output," runs it, sees green, and hands it back to you looking exactly like verification.

It isn't verification. It's a mirror. A bug baked into the implementation shows up identically in a test that was derived from the implementation — the test and the code agree, perfectly, about the wrong thing.

The three tells

It only tests the happy path. Ask for tests on a function called validateEmail, and you'll get three cases: a valid email, an obviously invalid one ("not an email"), and maybe an empty string. You will not get "a@b" (valid per RFC, rejected by most regex), or a 300-character local part, or a unicode domain, or the email with a + tag that your own signup form generates for every test account. The model tests the cases it can imagine from the function name and one example call site. Real edge cases live in your actual data, which the model has never seen.

It mocks everything, so nothing real gets exercised. Ask for tests on a function that hits a database, and the fast path to green is mocking the database call entirely — db.query = jest.fn().mockResolvedValue(fakeRow). The test now proves your function correctly handles a fake row that always looks exactly the way you told it to. It proves nothing about what happens when the real query returns null, or two rows instead of one, or times out. A mock that always succeeds is indistinguishable, from inside the test, from a mock that's hiding a bug in the query itself.

It changes when the bug fix does. This is the sharpest tell and the easiest to check retroactively: go find a bug you fixed recently, and look at whether the test file changed in the same commit as the fix. If a test had to be edited to make a bug fix pass, that test was never testing the requirement — it was testing whatever the code did before, which was wrong, and it agreed with the wrong answer just as confidently as it now agrees with the right one.

What a real test asserts

A test that would have caught the rate limiter bug doesn't come from reading the rate limiter. It comes from the requirement, written down before the implementation exists: "101 requests in the same second, from the same key, should produce exactly 100 successes and at least 1 rejection — regardless of how they're distributed within that second." That sentence is checkable against a fixed window (which fails it, because a burst can land 100 in the tail of one window and 100 more in the head of the next) and it's checkable against a sliding window (which passes). It doesn't know or care what the implementation is. That's what makes it a test and not a mirror.

This is the same discipline as writing the spec first — the sentence above is a spec, not a test, and it's supposed to exist before the function does. The order matters. Requirement, then implementation, then a test derived from the requirement, is three independent things that have to agree. Implementation, then a test derived from the implementation, is one thing checking itself.

In practice this means writing three to five concrete scenarios by hand before you ask for anything — not full test code, just sentences, the way you'd describe the behavior to a teammate:

• 101 requests in one second from the same key → 100 succeed, rest rejected
• 100 requests spread evenly across two seconds → all succeed
• Two different keys, 100 requests each in the same second → both succeed independently
• Redis unreachable → requests succeed (fail open), and it's logged

Then ask the model to write tests for those scenarios, not "add tests for this function." The difference in the prompt is small. The difference in what gets caught is not — the model can no longer default to reading its own implementation back to you, because you've handed it something external to check against.

Coverage percentage is not the signal you think it is

A rate limiter with a fixed-window bug and twelve tests asserting fixed-window behavior has, technically, respectable line coverage. Coverage tells you which lines executed. It says nothing about whether what those lines did was correct — a suite that mirrors the implementation gets 100% coverage on the wrong behavior just as easily as the right one. Treat a coverage number as evidence the code ran, not evidence it works. Those turn out to be very different claims, and AI-generated tests are unusually good at making the first one look like the second.

None of this means don't ask for generated tests — writing test boilerplate by hand is exactly the kind of work worth delegating. It means the boilerplate should be filling in scenarios you specified, not inventing them from the one artifact — the implementation — that's guaranteed to already agree with itself.

Related: The confidence trap in AI-generated code · The spec is the work now · Testing vibecoded projects