How to build an evaluation set for resume-to-job matching

A prior run’s calibration numbers were meaningless because heuristic-labeled pairs and pairs carrying the dataset’s own labels were mixed. The repository’s notes report a 2.6-point separation on the heuristic-labeled pairs and 23.7 points on the dataset-labeled pairs.

At HireSalt, we built a small resume matching evaluation dataset from one public labeled dataset: 40 job descriptions (JDs), 197 pairs carrying the dataset’s own labels, and 40 distractor pairs. The 237 pairs were planned for later scoring. This is a field report on how to build an evaluation set for resume-job matching.

1. Keep each JD with its own resumes

The selection unit is a JD and the resumes matched to it. An earlier selector deduplicated resumes into one shared pool, then paired that same pool with every JD. It ignored which JD each resume had been labeled against. The repository’s notes say this produced more than 95% category-mismatched pairs and no calibration signal.

That failure is easy to miss: a resume’s label belongs to its original JD-resume pair. The per-JD pool preserves that link. A distractor, by contrast, is a resume linked to a different JD and placed in this JD’s pool. A category label cannot restore a link the selection has discarded.

2. Set eligibility rules before selecting JDs

For the full-dataset source, the builder groups rows by the complete JD text after collapsing whitespace. A group must have at least four resumes, all three dataset labels, and at least 200 characters of JD text. The builder’s comment says very short JDs produced degenerate extracted criteria. These checks decide which groups can enter selection; they do not fill in a missing label after the fact.

Three of the final 40 JDs came through an earlier, smaller five-JD pool with a separate set of rules. That selector required at least two Good Fit resumes, one Potential Fit and one No Fit, plus JD text longer than 500 characters. It also limited that pool to one JD per coarse category. The two paths are worth stating because the same final set includes items admitted under different eligibility gates.

Record the eligibility path with each group; the final count alone does not show which gate applied.

3. Spread roles, then acknowledge the length preference

The full-dataset builder assigns each JD to a category with ordered keyword rules: the first matching rule wins, with an “other” fallback. It sorts eligible JDs by text length, takes up to four from each category on the first pass, then relaxes the cap if slots remain. The reason in the builder is practical: longer JDs are expected to yield richer criteria.

The builder defines 27 named categories plus “other.” The selected set contains 18 category values, with three JDs in “other.” The three JDs from the older pool use a different, coarser classifier, so the 18 values do not come from one shared taxonomy. These are categories created by keyword rules, not a separately verified account of job families.

The cap and longest-first order shape which JDs enter the set. They are construction choices, not evidence that the mix represents a broader set of jobs. Record them alongside the count so the length preference stays visible.

4. Build a small pool around every JD

For each JD, the builder takes up to five of its matched resumes. It cycles through Good Fit, Potential Fit and No Fit so each label appears when available. Within a label, it prefers the longest resume text because the builder describes those as less truncated. The final selection has five labeled resumes for 37 JDs and four for the three older-pool JDs.

The selector adds one distractor per JD. It draws that resume from a different JD and prefers a different category, using the same category if needed. Distractor resumes in evaluation remain linked to their original JDs; each is placed in another JD’s pool to create a different target pairing. That distinction matters when interpreting its label later.

The counts make the pool shape concrete: 37 JDs with five matched resumes and one distractor, plus three JDs with four matched resumes and one distractor. That gives 197 matched pairs and 40 distractors, or 237 pairs planned for scoring. Preferring longer resumes may reduce truncation, while also making text length part of selection. It describes how examples were chosen, not the resumes that were left out.

Across the 197 labeled pairs, the counts are 79 Good Fit, 67 Potential Fit and 51 No Fit.

5. Treat label source as a hard filter

Every pair records where its label came from. The selector counts only pairs with the dataset-label source among the 197 labeled examples; pairs labeled by keyword overlap stay outside that count. The heuristic ranks resumes by shared-token count, assigning Good Fit to the top, Potential Fit to the median and No Fit to the bottom. Those labels describe the heuristic’s ranking, not an additional reviewed judgment.

The provenance field matters especially for distractors. Of the 40 selected distractors, 12 carry a dataset label and 28 carry only a keyword-overlap heuristic label. Even the 12 are not counted among the 197 matched pairs. Their dataset label belongs to a resume paired with a different JD; it is not a label for the new target pairing. Keeping every distractor out of the labeled-pair count avoids treating a label from one relationship as if it were assigned to another.

The phrase “ground truth labels for resume screening” needs a qualifier here: these are the dataset’s own labels, and the repository does not audit who assigned them or how consistently. The calibration problem described at the opening is why source is a filter, not a footnote. Keep it attached and decide which source belongs in an analysis before using the labels.

6. Make draws repeatable and JD identity stable

The selector uses base seed 42. Each JD’s distractor draw uses its own random generator, seeded from the base seed, the JD’s position and a fixed salt. That makes the draw repeatable for its seed.

Identity caused a separate failure. Before the fix, derived titles produced 39 distinct titles for 40 JDs. A downstream tool keyed by title silently reused one JD’s extracted criteria for another JD’s resumes. The builder now appends the JD’s stable identifier to fallback titles, so a readable title can remain display text without serving as the only key.

Keep identity separate from presentation: a title can be derived from repeated text, while the stable identifier points to one JD. Here, adding that identifier to fallback titles kept the downstream criteria attached to the intended JD.

7. Add a cheap file-integrity check

The build hashes files in the JD collection and the resume collection, then stops if any JD file is byte-identical to a resume file. The documented purpose is narrow: catch a JD source mistakenly wired to a resume PDF. It does not prove that every source is correctly labeled or that every pair is meaningful. It catches one concrete wiring error at the boundary where the files are assembled.

The failure that check guards against, job description files that were really copies of resumes, and what it did to our early figures, is told in how consistent is AI resume scoring?. Every pass-bar test we ran in October 2026 is on the benchmark results page.

This remains a small set: 40 JDs and 197 pairs carrying the dataset’s own labels, from one public labeled dataset. The repository does not audit who assigned those labels or how consistently, so that is the limit of what the construction details establish.

The August 2 and October 7 selection files are identical in every field except 36 disambiguated title strings. The earlier file has 39 distinct titles across 40 JDs; the later one has 40. The selected pairs stayed the same. The fix changed identity metadata, not selection.