The resume metric we chose not to invent
If we can't defend the last digit, we shouldn't print it
· 5 min read · Building
The most requested feature we've never shipped is a single number.
People want it. "Just tell me my score." It's the obvious thing to build — it demos beautifully, it fits in a card on a landing page, and every competitor has some version of it. We built a working one early on.
Then we looked at it properly, and decided not to ship it. Here's the reasoning, because we think the decision generalizes well beyond us.
The number moved when nothing else did
The first problem showed up immediately. Run the same CV against the same job description twice, and the score comes back different. Not wildly — 84 and then 79 — but different, from identical input.
That's not a bug you fix with a better prompt. It's what happens when you ask a language model to compress a complex, multi-dimensional judgment into a continuous scalar. The model has no internal scale. It isn't measuring anything against a reference; it's producing a plausible-looking number, and plausible-looking numbers wobble.
We could have hidden the wobble — cache the first result, round to the nearest five, seed it deterministically. All of those make the number stable. None of them make it mean anything. We'd have been engineering the appearance of rigor, which is a worse position than having none.
Two digits of precision we hadn't earned
Even stabilized, the format makes a claim we couldn't support.
Writing 73% asserts that 73 is distinguishable from 74. That there exists some real quantity where those differ. For CV-to-role fit there is no such quantity — no agreed unit, no ground truth, nothing to calibrate against. The second digit is decoration.
And decoration isn't harmless here. Precision is a rhetorical device: it signals measurement, and people extend trust to it accordingly. A candidate who sees 73% believes something was measured. They'll make real decisions — whether to apply, whether to keep looking — on the strength of a digit we invented.
It was answering a question we can't answer
The deeper issue is what users reasonably hear.
Nobody reads "87% match" as your document has substantial keyword overlap with this advert. They read it as you have a strong chance here. Those are wildly different claims, and only the first one is in our data.
To honestly output the second, we'd need outcome data — applications, callbacks, offers, linked back to scores, across roles and markets and seniorities, in volume. We don't have that. As far as we can tell, nobody who ships a match score has it either. So the number is computed on text and read as a forecast about someone's career. The gap between those two things is where the dishonesty lives, and it doesn't matter that it's unintentional.
(We wrote separately about what a match score genuinely can't see — competition, unwritten requirements, how you interview. Same problem from the candidate's side.)
Metrics change behaviour, and this one changed it badly
The argument that actually settled it for us wasn't accuracy. It was incentives.
Whatever you put a number on, people optimize. Ship a match percentage and you've told every user that the goal is to make the number go up — and the fastest way to raise keyword overlap is to stuff the CV with the advert's vocabulary. That produces documents that score well and read like they were assembled by a machine, which is exactly the outcome the tool is supposed to prevent.
We'd have built something that made people's CVs worse while telling them they were improving. That's the kind of failure that doesn't show up in your metrics, because your metric is the thing causing it.
What we shipped instead
Countable things.
Rather than a percentage, the analysis reports how many of the stated requirements your CV addresses, which specific ones it doesn't, and which of those are marked required versus preferred. Something closer to "you address 6 of 9 stated requirements; 2 of the 3 you're missing are listed as required" than to a score.
That version is worse in exactly one way: it doesn't fit in a badge. It's better in several. You can verify it — go and count them. It's stable, because counting requirements isn't a judgment call in the way scoring overlap is. It points at the work instead of summarizing it away. And it can't be gamed in a direction that hurts you, because the only way to raise it is to genuinely address a requirement.
Coarse and honest beats smooth and invented. When the underlying judgment is genuinely uncertain, the output should look uncertain.
What it cost
We're not going to pretend this was free.
A percentage is a better demo. It's a better screenshot. Users still ask for it, and some of them find the answer unsatisfying — "just give me a number" is a reasonable thing to want when you're trying to decide whether to spend an evening on an application.
We think it's still right. The test we keep coming back to: can we defend the last digit? If we can't, we shouldn't print it. Most product metrics in this category fail that test, including several we'd find useful if they were real.
There's a version of this industry where every tool reports a confident number, all the numbers disagree, and none of them are anchored to anything. We'd rather ship something smaller that's true.
All posts