How consistent is AI resume scoring? What we measured, what broke, what we changed.

Our latest repeatability check, on 16 October 2026, missed the bar we set for it: 21 of 32 resume and job pairs stayed within 5 points across three runs, against a rule of 80%. We do not claim consistent scores. Earlier, our first benchmark gave worse-looking numbers, and much of that came from a flaw in our own test set, which we found in July and describe below. This article sets out what we measured, what we withdrew and what remains.

May: alarming numbers from a small, mismatched test

On 23 May, we ran 20 job descriptions against a set of resumes, producing 410 scoring rows when repeat runs were included. The benchmark mixed a public labelled resume dataset, O*NET occupation descriptions, and job posts from a public hiring forum. Of the 400 main resume and job-description pairs, 395 deliberately crossed domains. The benchmark mostly compared unrelated pairs.

Of the 309 rows with scores, 280 were below 50, seven were between 70 and 79, and two were at least 80. The median was 25.2, and the maximum was 82.2. These figures describe that mostly mismatched benchmark matrix. They do not describe the scores a recruiter would get from a relevant candidate pool.

At the time, we reported that two of five labelled comparisons fell inside bands we had set ourselves: a “no fit” pair at 0.0 and a “potential fit” pair at 73.5. We called that 2 of 5, or 40%. The other three missed: two scored below the band, and one run returned an error. The 0.0 did not establish that the scorer had identified a poor candidate. The run log suggested it might have been a parsing failure recorded as zero, but that explanation was never confirmed. In July, we withdrew this in-band result because none of the five comparisons used a real job description.

The first run also had an operational problem: 101 of 410 rows, or 24.6%, had no usable score or an error. That is a fact about this benchmark run, not a measured rate in customer workflows.

Two interventions, neither a clean answer

That day, we rewrote the score-band wording and tightened the output instructions, then reran the same set. The reported parse-error count fell from 60 of 400 scorings to 30 of 400. But the share scoring 80 or higher stayed at 0.6%, the mean fell from 28.7 to 22.3, and the reported in-band count remained 2 of 5. Four of the five labelled pairs encountered parse errors after the rewrite, compared with one before it. We later withdrew the in-band count along with the other calibration numbers.

One software engineering pair moved from 75.9 to 83.8. We initially described that as a prompt-driven improvement. The same pair had already scored 75.9, 80.8, and 82.2 in its original three runs. The rewrite’s scored runs, 83.8 and 82.7, sat 1.6 and 0.5 points above the top of that original range, against a 6.3-point spread the same pair already showed before the change. Those reruns are our observation; they do not show that the rewrite caused the change. We reverted the rewrite.

We also tried a lenient scoring mode, a per-job setting that tells the scorer to favour higher scores on transferable skills; balanced is the default. Only 116 of 410 rows returned a usable score. Another 294 returned no score even though the calls finished normally; the runner’s headline count of seven errors captured only exceptions. The 116 scored rows had a mean of 36.2 and a maximum of 69.5. That mean describes survivors only, so it is not comparable with the full-run mean. No row reached 80.

The lenient run did not move the apparent ceiling, but it did not show that lenient scoring inherently fails. Almost all of its pairs were deliberately cross-domain, and the explanations we proposed for its short, empty responses were hypotheses, not tested causes. We kept balanced as the default.

July: we found the test-set bug

On 26 July, we checked the labelled fixtures themselves. All 20 labelled job-description files in our fixtures were byte-identical copies of resumes, including all five we had used in the benchmark. A single wrong path in the corpus builder caused it: the resume file was copied where the job-description file belonged.

In effect, our labelled comparisons asked the scorer to evaluate a resume against criteria extracted from that same resume. The real labelled job descriptions never made it into those comparisons. The “no fit” label belonged to a different real job description that was absent from our set.

We withdrew every calibration number from the May reports, including the 2-of-5 and 0-of-5 tallies. We also withdrew the conclusion that good-fit resumes were being under-scored. Those comparisons were self-matches, not evaluations of fit to a job. We withdrew the domain-drift example as evidence of drift too: the model had been given a resume where we thought it had a job description.

The mistake was ours: every labelled result rested on a resume being scored against itself.

The same review found two complete benchmark generations of 410 rows each where every row was marked done but had no score and empty criteria. We had not noticed. A run could finish successfully in process terms while producing no usable results.

What the accidental self-match showed

The contaminated pairs did produce a useful diagnostic, but only about the benchmark runner’s arithmetic. Across the five self-match examples, stored scores ranged from 76 to 84. In those same examples, practicalities such as location, availability, salary, and work authorization scored from 40 to 65; engagement, including motivations and work-style preferences, scored from 55 to 75. The evidence-based categories scored mostly in the 80s and low 90s. The runner averaged about 11 category scores without weights, so the two weaker categories pulled down the overall mean. Recomputing the same examples without them produced scores from 80.6 to 87.1.

Our first explanation was that the model defaulted to about 50 whenever a resume lacked evidence. We tested that separately across 8,896 sub-criteria notes. The mean was 45.75 for notes that signalled absent evidence and 46.87 for notes that did not. Those are note-level averages, separate from the five self-match results, and they did not support our hypothesis. The clearer difference was spread: the standard deviation was 12.2 for absent-evidence notes and 16.8 for notes with evidence.

This points to the benchmark runner’s aggregation method as the source of the self-match ceiling, rather than prompt tone. It does not establish a ceiling in HireSalt’s production scores. Production weights categories differently, and this analysis did not test that scoring method.

The same problem affects the May repeatability check. All five repeated pairs were later found to be self-matches. Of the five repeated pairs, one was the 0.0-to-73.2 outlier mentioned above. The other four stayed within about 1 to 8 points of each other: 73.5 to 77.0 for finance and accounting, 62.3 to 69.9 for non-tech, 66.5 to 67.2 for product design, and 75.9 to 82.2 for software engineering. The non-tech pair had an error on its third run. These ranges describe repeatability on contaminated pairs, not a valid consistency result for the product. Through August we had no clean consistency figure for resume scoring on verified job and resume pairs. A later October check on a rebuilt corpus, 10 pairs scored twice by one model, gave a mean standard deviation of 1.63 points. That is a small, single-model check and not a general figure.

What remains measured, and what changed

Our extraction review covered 15 real job descriptions. Of 63 extracted categories, 55, or 87%, were judged central to the role; four were domain context, one was unknowable from the description, and three were ambiguous. That result applies to this small set. It does not establish a general rate for every job description.

Some operational counts also remain facts about their runs. The May run had 101 rows without a usable score or with an error out of 410. A later, partial June run used lenient mode and covered only a subset of jobs before it crashed. It scored 99 of 101 rows, with two silent failures. That run used a different model and followed many code changes, so it does not isolate the effect of lenient mode. The lenient failures in May were not shown to be inherent to that mode.

In the August run, the benchmark summary recorded settings such as scoring mode, concurrency, and timeout, and listed errors, silent failures, and timeouts separately. Its scoring phase recorded 177 completed rows, 24 errors, 24 silent failures, and 12 timeouts. The run also recorded 197 variance pairs available and 10 selected, but gave no per-pair results. We cannot report a score distribution or consistency result from that run, and those counts do not give us a clean failure rate.

We reverted the prompt rewrite and kept balanced as the default. By August, the benchmark summary was recording configuration and counting silent failures separately. We proposed excluding or down-weighting the two weaker categories. Practicalities and engagement now carry zero weight by default in the product.

October: a new repeatability check, and what it showed

On 16 October 2026 we reran the check on the production scoring route. We scored 32 resume and job pairs three times each, with no errors. The grade letter held for 26 of the 32 pairs, and five of the six changes were at the line between a C and a D. The score stayed within 5 points for 21 of the 32 pairs, which is 65.6%. We wrote the pass rule before running anything. It had two parts: at least 80% of pairs within 5 points, and a mean gap of 5 points or less. The mean gap was 4.19 points, so the second part held. The first part did not, and the rule needed both, so the test failed.

We also compared PDF and Word versions of 12 resumes. The grade changed for 2 of the 12, one more than the limit we set. The Word files were built from the PDF text, so that test is weak by design. Rescoring the same PDF twice changed the grade for 3 of the 12, which is a larger gap than the format difference.

These results describe one model, one night and one data set. They are not a promise about how any single resume will score. Because the repeatability result missed our own bar, we do not claim consistent scores, and we have taken the earlier 1.63-point figure out of our marketing.

This is one of the tests we ran. The others, including the ones that passed, are collected in our benchmark results. How we now build an evaluation set, including a check that would have caught the fixture bug described above, is in how to build an evaluation set for resume-to-job matching.