Eight LLMs order resume-job pairs similarly, but score them differently
The HireSalt team 6 min read
Three models scored each of 143 resume-job pairs. On 14 pairs—about one in ten—the same pair received scores at least 20 points apart. Across the seven models tested against the baseline, pooled rank correlations ran from 0.80 to 0.96. grok-4.7 had the highest correlation with the baseline (rho 0.96), yet landed in the same grade band on just 35% of its 48 pairs.
The benchmark covered 190 pairs across 40 jobs. The baseline model scored all 190. Each of the other seven models scored one assigned set of 47 or 48 pairs.
The only outside reference was Claude Sonnet, which rated 30 random pairs blind using a recruiter rubric and reading the full files. Sonnet is one LLM; the reference ratings are not from human recruiters. Each of the seven non-baseline models overlapped with Sonnet on only six to nine pairs, too few for a useful comparison.
What the test measured
The sample included 50 pairs labelled good_fit, 50 potential_fit, 50 no_fit, and 40 distractors. The pairs were randomly sampled with a fixed seed. In the source code, a distractor is a pair without a dataset ground-truth flag.
Every model used the criteria the baseline extracted on October 7. The criteria stayed fixed while the scoring model changed. The baseline supplied both the rubric and the comparison point, so agreement describes consistency within a shared scoring setup. Agreement may also reflect the common rubric and scoring pipeline. The benchmark treats the dataset labels as noisy and not a usable accuracy target.
Scores are weighted totals out of 100; the help center explains how the weighted total is calculated. The reported Spearman correlations pool pairs across 40 jobs and four label groups. They describe how models order this mixed sample; they do not show which resume a model would choose for a particular job.
| Model | Pairs | Mean score | Scores 80+ | Rho vs baseline | Same grade band |
|---|---|---|---|---|---|
| Baseline | 190 | 43.0 | 7 | 1.00 | 100% |
| deepseek-v4-pro | 48 | 31.8 | 0 | 0.92 | 60% |
| glm-5.3 | 48 | 34.0 | 0 | 0.94 | 50% |
| gpt-5.4-mini | 47 | 40.1 | 2 | 0.80 | 55% |
| gpt-5.6-luna | 47 | 45.7 | 6 | 0.93 | 70% |
| grok-4.7 | 48 | 30.7 | 0 | 0.96 | 35% |
| muse-spark-1.3 | 47 | 37.0 | 1 | 0.87 | 51% |
| qwen3.8-max | 48 | 37.9 | 0 | 0.92 | 71% |
The seven non-baseline means span 30.7 to 45.7. Each outside model saw its own 47- or 48-pair sample, so that range reflects scorer differences and pair-set differences together. The seven means are not a same-pair comparison. Rho and grade-band agreement against the baseline are the like-for-like measures, each calculated on the pairs that model shared with it. The 80+ column counts score outputs, not validated matches. The baseline had 7 in 190 pairs; gpt-5.6-luna had 6 in 47; gpt-5.4-mini had 2 in 47; muse-spark-1.3 had 1 in 47; the other four models had none. Those counts come from different sample sizes, so they describe what each scorer produced in its assigned set.
Similar order, different scores
grok-4.7 is the clearest example of how those measures diverge. Its pooled rank correlation with the baseline was the highest among the seven models, while its scores fell in the same grade band on 35% of shared pairs. qwen3.8-max had the highest same-band rate, 71%. A similar ordering across a pooled sample can coexist with different score levels and band assignments.
The matrix also reports mean absolute error (MAE) against the baseline. On their respective shared pairs, MAE ranged from 5.2 points for gpt-5.6-luna to 11.1 for deepseek-v4-pro. That figure is the average distance between scores, measured in points.
The 14 large gaps came from sets one to three, where each pair had three scorers: the baseline and two other models. Across those 143 pairs, the mean gap between the highest and lowest score was 12.2 points; the median was 11.7. Set four had only two scorers per pair and is outside that calculation.
A separate repeatability check offers limited context. On 10 pairs rerun twice, the baseline’s mean run-to-run range was 3.3 points and its mean standard deviation was 1.63. The comparison covers one model and ten pairs; it is a small check against which to read the broader cross-model spread.
A second scoring service
Set two allowed a same-pair comparison among five scorers on 48 pairs: the baseline, qwen3.8-max, deepseek-v4-pro, and two modes from a third-party scoring service. Mode A asked about individual criteria before combining the results; mode B asked for one overall fit score. Pairwise rank correlations among the five ranged from 0.89 to 0.96: qwen and mode B were at the low end, while deepseek and mode A, and the two service modes, were at 0.96. It is the only set where five scorers can be compared on the same pairs.
All five gave potential_fit a higher mean than good_fit on this set. Their good_fit / potential_fit means were 45.5 / 58.4 for the baseline, 40.4 / 54.1 for qwen, 35.4 / 48.7 for deepseek, 31.2 / 41.6 for mode A, and 29.7 / 39.9 for mode B. The baseline’s full-run means go the other way—52.3 for good_fit and 49.9 for potential_fit—so the inversion appears confined to some pairs.
The service also scored the 30-pair Sonnet sample in both modes. Those results put the baseline and both service modes beside the same reference ratings:
| Scorer | Pairs | Spearman vs Sonnet | MAE (points) | Mean score |
|---|---|---|---|---|
| Claude Sonnet (reference) | 30 | 1.00 | 0.0 | 20.6 |
| Baseline | 30 | 0.80 | 21.4 | 41.9 |
| Third-party service, mode A | 30 | 0.86 | 7.4 | 24.4 |
| Third-party service, mode B | 30 | 0.92 | 5.0 | 19.5 |
Against Sonnet, the baseline’s 21.4-point MAE is almost entirely a 21.3-point difference between the two mean scores. Its rank correlation on the same pairs was 0.80. Mode B’s lower MAE and higher correlation show closer agreement with Sonnet on this small sample. Both mode B and Sonnet judge fit holistically, so that closeness describes similar ratings.
Reading the corpus
A reasonable working interpretation is that the corpus complicates the comparison as much as the scoring methods do. Sonnet and the two service modes had mean scores of 20.6, 24.4, and 19.5 on the same 30 pairs. One document labelled as a resume was an article, not a candidate document. And in set two, five different scorers put potential_fit above good_fit on average.
The evidence has a clear limit: Sonnet rated 30 pairs, and it is itself an LLM. Sonnet and the third-party service may simply be harsh raters. A separate five-pair probe shows the scoring pipeline can reach the 80s: the baseline averaged 81.6 when good_fit resumes were scored against criteria extracted from those same resumes. That is a ceiling check, because each resume supplied its own criteria.
For HR-tech buyers and ML teams, a score travels with its scorer. The matrix found that broad rank correlation can coexist with low agreement on grade bands. Keep the scorer tied to each score, and do not carry cutoffs across tools or model changes. The benchmark calls the bands model-specific; recalibrate them when the scorer changes. Before setting a shortlist threshold, check whether the score and band still point to the same action in a reviewed sample. Ask what the number means.
Every pass-bar test we ran in October 2026, including the ones we missed, is on the benchmark results page.
Two high scores in the wider baseline run remain uninspected: 91.4 on a no_fit pair and 87.1 on a distractor pair. The benchmark leaves open whether they reflect noisy labels or over-scoring.