Skip to content
HireSalt
Research & Benchmarks

Do AI-drafted criteria match the role? We rated 117 of them

The HireSalt team 5 min read

When HireSalt reads a job description, it drafts a short list of job-specific criteria to score resumes against. We tested whether those criteria describe the job’s real work. Across 30 job descriptions, HireSalt drafted 117 criteria, and 102 of them (87%) were rated central to the role. The rater was one AI model (Claude) applying a rubric we wrote before the run, not a panel of recruiters. None of the 117 was rated wrong.

The result

Before the run we set a pass rule: at least 80% of criteria central, no more than 5% wrong, and no more than 2 of the 30 drafts failing outright. The test met all three: 87% central, none wrong, no failures.

Stacked horizontal bars of how one AI rater (Claude, written rubric) rated each criterion HireSalt drafted, by source. Public dataset (15 descriptions): 53 of 57 central, 4 peripheral; US O*NET occupations (8 descriptions): 32 of 32 central, 0 peripheral; Hacker News job posts (7 descriptions): 17 of 28 central, 11 peripheral; All sources (30 descriptions): 102 of 117 central, 15 peripheral. No criterion was rated wrong. Dashed line marks the 80% pass bar set before the run, which applies to the all-descriptions total. Central to the role Peripheral (minor) Share of drafted criteria, 0 to 100% Public dataset, 15 descriptions 53 of 57 (93%) US O*NET occupations, 8 descriptions 32 of 32 (100%) Hacker News job posts, 7 descriptions 17 of 28 (61%) All sources, 30 descriptions 102 of 117 (87%) 0% 25% 50% 75% 100% Dashed line on the last bar: 80% pass bar, set before the run
Drafted criteria rated central or peripheral to the role by one AI rater, by source of the job description; counts of criteria, n = 117 from 30 descriptions.

The split by source is where the detail sits. O*NET descriptions, which are written to list an occupation’s duties, got 32 of 32 central. Job descriptions from a public dataset got 53 of 57. Hacker News job posts got 17 of 28, or 61%. Those posts are short and often say more about the company than the job, and the drafts followed them: criteria about handling billions of errors or understanding a healthcare staffing marketplace came from the company’s pitch, not a stated duty, so the rater marked them peripheral.

Two of the 30 descriptions got only one criterion instead of the three to six the drafting instructions ask for. Both replies had to be repaired from broken JSON, and we did not keep the raw text, so we cannot say whether the repair dropped criteria.

How we ran it

On 16 October 2026 we drew 30 job descriptions with a fixed random seed: one from each of 15 role folders in a public dataset, 8 from the US O*NET occupational database and 7 Hacker News “Who is hiring” posts. Each went once through the drafting prompt HireSalt uses when a description is pasted, on our production model. No retries, no reruns.

Each criterion was rated against the description’s text:

  • Central: a main responsibility or requirement the description states, or an unavoidable part of the role.
  • Peripheral: traceable to the description but minor, such as a one-off mention, a nice-to-have, culture fit, or the employer’s sector rather than the job.
  • Wrong: not supported by the text.

When torn between two ratings, the rater had to pick the less favourable one, and every rating carries a one-line written reason. An earlier internal review of criteria drafting used a different sample, an older model and different rating categories; its result is separate from this one.

What happens with a poor job description

The 30 above were real job descriptions. In the same run we wrote six poor ones: two one-liners (“Accountant needed ASAP!!”), two made only of buzzwords, one that contradicts itself (entry level, yet 12 years of experience and “led teams of 20+”) and one that is almost all company boilerplate. We drafted criteria for each and scored the same three resumes against it.

None failed visibly. The one-liners got generic criteria that fit the named job. The contradictory description got four criteria built straight from the contradictions, with nothing flagged. One buzzword description squeezed three very different resumes into an 8-point range. The boilerplate one got three criteria about the company (its twelve countries, its sectors, its charity work) and none about the HR role it named. HireSalt shows a hint only when a description is under 100 characters, and the hint does not block. Under the severity rule we set before the run, 2 of the 6 had a silent problem, which places this as a medium-severity blind spot. If a job description is thin, the criteria will be thin too, and nothing will tell you.

What this test does not show

  • No human rater. One AI model rated each criterion once. No recruiter checked the ratings, and a human panel could disagree.
  • No outcome data. Central to the description is not the same as predictive of a good hire. This says the criteria match the job description; it says nothing about who you should hire.
  • A small, single run. 30 descriptions, one model, one night. Some dataset folders were mislabelled, so the 15 “roles” are 15 folders; ratings used each description’s text, so the figure is unaffected.
  • Criteria, not scores. It rates what HireSalt drafted, not the resume scores built on it.

Related tests: all results are on the benchmark hub. See also whether every score comes with a reason and how consistent the scores are.

Keep reading