Subscribe
Analysis
9 min read

The Algorithm Rejected You First: What the Evidence Says About AI Hiring Tools

Analysis of lab experiments, field data from 3.4 million applicants, and enforcement records reveals that AI hiring tools exhibit significant racial and gender biases, often concentrated per position rather than in aggregate statistics.

Illustration representing algorithmic bias in automated employment decision tools screening resumes
Illustration representing algorithmic bias in automated employment decision tools screening resumes

Evidence from controlled experiments and production data shows that AI hiring tools routinely exhibit racial and gender biases, with systems preferring white-associated names and disadvantaging Black male candidates. Furthermore, human reviewers often mirror these moderate biases, confirming that algorithms are not inherently less biased than humans.

Recruiting is where enterprise AI landed first. SHRM's State of AI in HR report, fielded in December 2025, found that 27% of organizations use AI in talent acquisition, more than in any other HR function, and that adoption reaches 60% at companies with 5,000 or more employees. For job seekers, that means an algorithm, not a person, is now routinely the first reader of an application.

The public debate about that shift has mostly run on anecdotes: one viral horror story after another, followed by vendor reassurance. Over the past two years, though, a real evidence base has accumulated. It comes from randomized lab experiments, field data on millions of actual applications, human-subjects studies of recruiters working alongside AI, and the first enforcement records under the laws that regulate these tools. The findings do not sort neatly into either camp.

What controlled experiments show

The cleanest tests vary one thing about an otherwise identical resume: the name, which signals gender and race to a model the way it signals them to a human reader.

University of Washington researchers fed more than 550 real resumes with 120 name variants into three commercial language-model ranking systems and 500 real job postings across nine occupations, producing over three million comparisons. The rankers preferred white-associated names in 85% of the bias tests and Black-associated names in 9%. Male-associated names were favored 52% of the time, female-associated names 11%. An intersectional analysis by the same research line, summarized by the Brookings Institution, found the steepest penalty for Black men: scored against resumes with white men's names, those with Black men's names were selected 0% of the time.

A larger randomized experiment, summarized by the research portal VoxDev, generated roughly 361,000 fictitious resumes with work history, education, and skills randomly assigned, then had five leading models (GPT-3.5 Turbo, GPT-4o, Gemini 1.5 Flash, Claude 3.5 Sonnet, and Llama 3-70b) score them on a 0-100 scale. With qualifications held constant, the models produced a consistent hierarchy: Black female candidates scored highest (0.38 points above white men), white women next, and Black men lowest, 0.30 points below white men. The direction runs counter to classic human audit studies, where Black-associated names consistently draw fewer callbacks than identical white-associated ones. The models scored these candidates by group separately, then combined the effects, an intersection that behaves differently in machine rankings than in human ones.

Two newer studies point at the mechanism. A Nature paper covered by Science News had ChatGPT write resumes for men and women across 54 occupations: it consistently generated younger, less experienced female candidates, then ranked women's resumes as less qualified. The bias was in the input generator and the output scorer at once. A Princeton and University of Chicago study presented at ICML in July 2026 put models through a simulated hiring game with fictional ethnic groups and success feedback. The models formed their own novel stereotypes, faster and more strongly than human participants in the same game. Ordinary guardrails did little: step-by-step reasoning prompts, added randomness, and shorter decision histories barely reduced stereotyping. An explicit diversity objective did. And a University of Texas at Austin study found that four AI scoring models marked interview answers spoken in accents acoustically distant from American English lower, even when the words were identical.

What production data shows

Lab tests use toy problems at worst. A Stanford Human-Centered AI team got access to something rarer: 4 million real applications from 3.4 million people to 1,700 job postings across 150 employers, every one screened by the same third-party vendor's AI tool.

Applying the EEOC's four-fifths adverse-impact standard position by position, the researchers found that 26% of Black applicants and 15% of Asian applicants submitted to at least one position where the tool disadvantaged their group. Had the system recommended those groups at the rate it recommended the most-favored group, roughly 40,000 more applications would have advanced. Notably, aggregating the vendor's recommendations across all jobs shows no adverse impact at all: the discrimination is visible only job by job, which is exactly how a routine audit could miss it.

The study also tested a worry economists call algorithmic monoculture. When many employers screen with the same vendor, one rejection propagates. Applicants who submitted several applications to positions screened by the same vendor were more likely to be rejected by all of them than if each company had decided independently: 10% of applicants who sent four applications were turned down by every employer they tried. The comparison group, 83,000 applications to 108 Fortune 500 firms where shared AI screening was not the focus, showed no such excess. The coordination of rejection, not rejection itself, was the new phenomenon.

The human-review defense does not hold up

A document sorting machine splitting a paper stream into two trays, mid-century brochure style

The standard answer is that recruiters make the final call. A University of Washington study published in November 2025 tested that directly. It found in a survey that 80% of organizations using AI hiring tools say they never reject an applicant without human review. Then 528 participants picked candidates for 16 jobs alongside simulated AI recommendations that ranged from neutral to severely biased. When the AI's bias was moderate, participants mirrored it: if the model favored white candidates, so did they, and if it favored non-white candidates, so did they. Only blatant bias met real resistance. "Unless bias is obvious," the lead author said, "people were perfectly willing to accept the AI's biases."

There is a counterweight, and it comes with instructions attached. A March 2026 study of a real recruitment platform (Kaya and Bogers) compared three workflows: recruiters searching a CV database by hand, AI-only matching, and a hybrid where recruiters reviewed the AI's recommended slate first and then searched manually. Human-curated lists were fairer on gender than AI-only lists. But the hybrid produced the fairest results of all, better than either component alone. Deliberation, sequencing, and friction mattered more than the model choice.

The rulebook exists. The enforcement is thin.

Five legal facts frame the risk picture:

  • New York City's Local Law 144 has required, since 2023, an annual independent bias audit, a published summary, and 10 business days' notice to candidates before any automated employment decision tool is used, with penalties of $500 to $1,500 per violation. In December 2025 the New York State Comptroller audited DCWP's enforcement from July 2023 through June 2025 and called it "ineffective." Reviewing the same 32 companies' published audits, the city found one potential compliance issue; the state auditors found at least 17. Enforcement has been complaint-driven, and the audit notes that DCWP cannot easily see non-compliance at employers who simply never post an audit.
  • Illinois moved further. Its AI Video Interview Act already requires consent before AI analyzes a video interview, and from January 1, 2026 its HB 3773 makes using AI in ways that discriminate against protected classes a civil rights violation under the Human Rights Act, with notice requirements for any AI used in hiring, promotion, or discipline.
  • Colorado's comprehensive AI Act, which would cover employment as a high-risk domain, was delayed from February 2026 to June 30, 2026 by the legislature and then had its enforcement paused by a federal court in April 2026. It has yet to bind anyone.
  • The EEOC's first AI hiring settlement, iTutorGroup in August 2023, cost $365,000 for software that automatically rejected women over 55 and men over 60. The complaint surfaced because one rejected applicant reapplied with a younger birth date and got an interview.
  • Mobley v. Workday is the case that changed vendor behavior. In July 2024 a federal judge held that an AI screening vendor can be directly liable as an "agent" of the employers using it, and in May 2025 preliminarily certified a nationwide age-discrimination collective. No merits ruling has found the tool discriminatory, but the theory that vendors could hide behind "we just provide software" did not survive.

The bottom line

The evidence does not support the claim that AI hiring tools are simply less biased than humans. Bias in these systems is now documented at laboratory scale, and the field data suggests it concentrates: the same vendor's pattern applied across 150 employers can shut a candidate out systemically, and human reviewers absorb a model's moderate bias rather than correcting it.

What has not survived the evidence is the easy part of the fix. Stripping names does not work, because proxy signals remain; generic instructions to "be fair" do not work; and averaging audit statistics across job titles hides adverse impact that is visible position by position. What did work in the studies so far: explicit diversity objectives, hybrid workflows where humans see the AI slate before searching themselves, and independent audits computed per position.

For candidates, the practical implication is that the first filter is now a scoring system, and the transparency levers that matter are the notice requirements: NYC and Illinois give applicants the right to know they were graded by a machine. For buyers, the compliance floor has moved: after Mobley, "the vendor handles the algorithm" is no longer an answer any employer or any vendor should find reassuring.

Sources

Frequently asked questions

Do AI hiring tools show consistent racial bias?

Yes, controlled experiments showed that rankers preferred white-associated names in 85% of tests and Black-associated names in only 9%, while production data indicated 26% of Black applicants faced disadvantage at least once.

How does algorithmic monoculture affect job applicants?

When many employers use the same AI vendor, rejections can propagate systemically, with 10% of applicants sending four applications to different companies being rejected by every employer if they shared a screening vendor.

Do human recruiters correct AI bias during review?

Research indicates that 80% of organizations say they never reject without human review, but participants in studies mirrored AI biases unless the discrimination was blatant, suggesting humans absorb moderate model biases.

What legal frameworks regulate AI hiring tools in the US?

Regulations include NYC's Local Law 144 requiring annual bias audits, Illinois' HB 3773 making discriminatory AI use a civil rights violation as of Jan 2026, and EEOC enforcement actions like the iTutorGroup settlement.

AnalysishiringPolicy