Your judgment is fine. Your threshold may not be.

Executive summary

• The threshold moves, not the skill. When targets got rare, searchers in a lab task missed more than twice as many (20% → 46%), yet their sensitivity held, if anything rising. What shifted was the evidence they demanded. Experts reading real cases drift the same way: radiologists missed 12% at 50% prevalence and 30% at about 1%.
• You work in a low-prevalence job. Most of the specs, research syntheses, pull requests and dashboards you review are fine, so your threshold drifts, and you cannot feel it drifting.
• Seeing the answer first contaminates the judgment. Showing reviewers an LLM’s answer moved their majority labels from 40% agreement with it to 81–87%; they grew more confident, not faster. Seniority doesn’t protect you: a wrong AI suggestion cut the most experienced mammogram readers’ accuracy from 82.3% to 45.5%. And in a separate study, the bias outlived the AI.
• Simple, consistent models beat experts, modestly. Unit weights predicted graduate performance at .48 against .19 for the admissions committee. For a call you make every quarter, a plain scorecard is the cheapest upgrade there is.
• What doesn’t work: being told to be more careful, slowing down, gathering more information, and worrying about decision fatigue.
• What does: decide the test before you see the answer, salt the deck, map which of your calls ever had feedback, turn repeated calls into unit-weight scorecards, judge independently before discussing, and keep score.

Every fix trades misses for false alarms. In a low-prevalence world that is the right trade, and it is not free.

Picture an ordinary week in product now. Forty AI-drafted artifacts cross your desk: specs, research syntheses, pull-request summaries, launch notes, a dashboard someone built with an agent. Nearly all of them are fine. Somewhere in the pile is one wrong number a roadmap decision will rest on, and you read straight past it.

In a 2007 paper, Jeremy Wolfe reported an experiment that explains why, and it isn’t that you got careless.

He had people search for targets in visual displays. When targets appeared in 50% of trials, they missed about 20% of them. When targets appeared in 2% of trials — same people, same task, same images — they missed 46%.

Miss rate more than doubled. The obvious explanation is that people got sloppy, bored, less attentive.

That’s not what happened. Wolfe decomposed the result using signal detection theory, which separates two things most people treat as one: sensitivity (how well you can tell signal from noise) and criterion (how much evidence you need before you say “that’s one”).

Sensitivity didn’t drop. In this comparison it went up: d′ rose from 2.01 to 2.43, a rise the authors caution may partly be an artifact of the measure. The searchers were, if anything, better at discriminating. The paper’s own summary: prevalence shifts the criterion, not sensitivity.

What moved was the criterion — from −0.02 to 1.07. More than a full unit. They had become far less willing to call something a target, because where almost nothing is a target, not calling it is usually right.

• Targets common (50%)
• Targets rare (2% in the lab; ~1% in real screening)

People searching displays
lab task, same images~20%46%

Board-certified radiologists
real mammograms12%30%

0%10%20%30%40%50%

2.01 → 2.43

sensitivity (d′) in the lab task: no worse at telling target from noise, if anything better

Did not drop

−0.02 → 1.07

criterion: how much evidence they needed before saying “that’s one”

What moved

Sources: Wolfe et al. (2007), Journal of Experimental Psychology: General; Evans et al. (2013): 14 radiologists at ~1%, six of them at 50%.

The numbers

Study	Targets common	Targets rare
Wolfe et al. (2007), miss rate	~20%	46%
Wolfe et al. (2007), sensitivity d′	2.01	2.43
Wolfe et al. (2007), criterion	−0.02	1.07
Evans et al. (2013), miss rate	12%	30%

This isn’t only a lab effect, and expertise doesn’t protect you from it. Evans and colleagues (2013) ran it with fourteen board-certified radiologists reading real mammograms: 30% of cancers missed at roughly 1% prevalence; six of them read the same cases again with cancer in half, and missed 12%. The detail that should stop you: 47% of the cancers missed at low prevalence were caught by all six readers when prevalence was high. Same material, different base rate.

Now look at your own review queue. The base rate of “this is materially wrong” in AI output is low — not zero, but low, and falling. A spec resting on a broken assumption, a research summary that quietly inverted a finding, a metric whose definition changed under you: rare, costly, and the product version of a rare target. You are running a low-prevalence screening program, and nobody told you.

The rest of this essay follows from that. The implications are almost entirely counterintuitive, and they invalidate most of the advice you’ve been given.

What judgment actually is

First, a definition, because the popular version is wrong in a way that matters.

Kahneman and Klein spent years disagreeing about intuition and then published a joint paper, Conditions for Intuitive Expertise: A Failure to Disagree (2009), setting out what they could both accept. Intuitive judgment can be trusted when two conditions hold:

1. A high-validity environment — one with regularities stable enough to be learned.
2. Adequate opportunity to learn them — through “prolonged practice and feedback that is both rapid and unequivocal.”

Nearly every popular summary drops the last four words. They’re the whole thing. Chess has rapid, unequivocal feedback. So does a checkout-button A/B test with enough traffic, or an on-call fix that either stops the errors or doesn’t. Product strategy, hiring, pricing and roadmap bets have slow, ambiguous, confounded feedback — which is why expertise in those calls is so much harder to establish than the confidence of the people making them suggests.

The concept from that paper worth stealing is fractionated expertise. Expertise is domain-partial. A PM may read usability problems in a prototype superbly and be no better than chance at predicting which feature will move retention. The useful question is never “am I an expert here” but “which of my sub-judgments sit in high-validity territory, and which don’t.” Almost nobody does this inventory, and it takes about twenty minutes.

Kahneman and Klein also note something rarely quoted: algorithms outperform humans at both ends of the validity spectrum, not just the low end.

Which brings us to the most replicated finding in judgment research, and the one most product people have never heard.

Simple models beat experts. Consistently. Modestly.

Paul Meehl published Clinical versus Statistical Prediction in 1954 and started an argument that is now settled. Three independent meta-analyses:

• Grove et al. (2000) — 136 studies. Mechanical prediction beat clinical judgment by roughly 10% on average.
• Ægisdóttir et al. (2006) — 67 studies. About 13%.
• Kuncel et al. (2013) — mechanical combination improved job-performance prediction by over 50% (r = .44 versus .28) — and this held even when the human judges were experts who knew the jobs and organizations.

Note the magnitude. “Formulas crush experts” is not what these show. The advantage is real, consistent and modest. What makes it remarkable isn’t the size, it’s the direction: it holds across sixty years, well over a hundred studies, and nearly every domain tested, including ones where the humans were experts in the job.

~10%

better on average, across 136 studies

Grove et al. (2000)

~13%

better, across 67 studies

Ægisdóttir et al. (2006)

.44 vs .28

job-performance prediction, formula against experts who knew the jobs

Kuncel et al. (2013)

Sources: Grove et al. (2000); Ægisdóttir et al. (2006); Kuncel et al. (2013).

The numbers

Meta-analysis	Studies	Result
Grove et al. (2000)	136	Mechanical ~10% better
Ægisdóttir et al. (2006)	67	~13% better
Kuncel et al. (2013)	—	r = .44 vs .28

Robyn Dawes pushed it further in The Robust Beauty of Improper Linear Models (1979). Predicting graduate-school performance from the Oregon data (N=111): the admissions committee’s judgment correlated .19 with outcomes. A cross-validated regression managed .38. And a model with unit weights — no optimization at all, just “pick the variables that matter and add them up” — hit .48.

Dawes had put it plainly five years earlier, with Bernard Corrigan: “The whole trick is to decide what variables to look at and then know how to add.”

In that sample, an unweighted checklist beat both the experts and the statistics. It is the highest-leverage, least-used insight for anyone who makes the same kind of call repeatedly: which bets to fund, which candidates to hire, which accounts are about to churn.

Admissions committeer = .19

Cross-validated regressionr = .38

Unit weights, no optimizationr = .48

Source: Dawes (1979), “The Robust Beauty of Improper Linear Models,” American Psychologist.

The numbers

Method	Correlation
Committee	.19
Regression	.38
Unit weights	.48

Why volume breaks judgment, precisely

Two mechanisms. The first is the one we started with.

Your criterion drifts with the base rate, and you cannot feel it happening. This finding makes “just be more careful” useless advice. The searchers in Wolfe’s study weren’t less careful; they were more discriminating and still missed twice as many. The drift is an adaptation to the environment, and it is rational in the aggregate. It just produces catastrophic individual misses.

A few details from that literature matter for what follows:

• Slowing people down didn’t fix it. Forcing longer search times barely changed the prevalence effect. Time wasn’t the constraint.
• What did work: high-prevalence bursts with feedback. Interleaving sessions where targets were common, with immediate correction, recalibrated the criterion.
• Every fix costs false alarms. You cannot lower your threshold for misses without raising your rate of flagging things that were fine. There is no free lunch, only a choice of which error you’d rather make.

The second mechanism is worse, and it’s specific to AI.

Anchoring to the machine is large, and it persists

The study closest to product work is Schroeder et al. (ACL 2025). 410 crowdworkers reviewed LLM output. Showing them the model’s answer moved their majority labels from 40% agreement with it to 81–87%. Their confidence went up. They were not faster — so this wasn’t people rushing. And the model’s apparent F1 score jumped from 0.47 to 0.79 purely because the humans moved toward it.

Read that last sentence again, because it describes a measurement failure you may be living inside. If you evaluate AI output by having humans review it — an eval where graders see the model’s answer, a design review that starts from the AI mock-up, a code review that starts from the AI’s summary — and the humans have seen the output, your evaluation is substantially measuring the anchoring, not the quality.

Seniority softens the pull; it doesn’t remove it. The cleanest evidence comes from medicine, where AI suggestions can be scored against known answers. Dratsch et al. (2023) gave radiologists AI suggestions on mammograms: the most experienced got 82.3% right when the suggestion was correct and 45.5% when it was wrong, and the least experienced fell from 79.7% to 19.8%. Every tier of the 27 readers was pulled toward the wrong answer. Bond et al. (2018) found the same on ECG interpretation: non-cardiology fellows fell from 86% to 27% with a wrong automated diagnosis, cardiology fellows to about 42%. Put your own seniors in those rows: the staff engineer approving an AI-written change, the head of product skimming an AI-drafted strategy memo.

• No answer shown, or a correct one
• Answer shown, or a wrong one

Reviewers of LLM output
agreement with the model’s answer40%81–87%

Most experienced mammogram readers
accuracy, wrong AI suggestion82.3%45.5%

Non-specialist ECG readers
accuracy, wrong automated diagnosis86%27%

0%25%50%75%100%

Sources: Schroeder et al. (Findings of ACL 2025), n = 410; Dratsch et al. (2023), Radiology, most experienced of 27 readers; Bond et al. (2018), non-cardiology fellows.

The numbers

Study	Before	After
Schroeder et al. (2025)	40% agreement	81–87%
Dratsch et al. (2023), most experienced	82.3%	45.5%
Bond et al. (2018), non-cardiologists	86%	27%

And then Vicente & Matute (2023): the bias persisted after the AI was removed. People who had worked with a biased model kept making the model’s errors when the model was gone. It was a simulated diagnosis task with lay volunteers, not clinicians, but the direction is hard to ignore.

That looks like the deskilling mechanism people keep gesturing at without evidence. It isn’t atrophy. It’s contamination.

What doesn’t work

Before the fixes, four things to stop spending effort on.

“Be more careful.” Mosier and colleagues (2001) tested explicit automation-bias training and on-screen prompts to verify automated information with professional aircrews and found no effect on automation bias. Expertise doesn’t fix it; Parasuraman and Manzey’s review found it in experienced pilots, glass-cockpit pilots and air traffic controllers. Practice doesn’t fix it either.

Slowing down. As above, it didn’t fix the prevalence effect, and the wider evidence on speed is mixed. The best-measured cases are doctors under time pressure. In an experiment with 37 internal-medicine residents, ALQahtani et al. (2016) found time pressure effectively null (η² = 0.002) while case complexity dominated (η² = 0.85). Yet the same group’s other 2016 study did find a cost, larger on difficult cases, and Trueblood et al. (2018) found speed hurt novices’ sensitivity but not experts’. A longer review slot won’t save you. Time is rarely your binding constraint. Structure is.

Gathering more information. The famous Oskamp (1965) finding is usually reported as “more information increases confidence but not accuracy,” and that summary holds up. 32 judges, eight of them clinical psychologists, read a case file in four instalments: accuracy started at 26% and never rose above about 28% (chance was 20%), while confidence climbed from 33% to 53%. Past a point, more information doesn’t make you more right; it makes you more sure. It is the research-synthesis trap: the tenth customer interview makes the deck more convincing, not more correct.

Accuracy, first instalment26%

Accuracy, last instalment28%

Confidence, first instalment33%

Confidence, last instalment53%

Source: Oskamp (1965), 32 judges on the “Joseph Kidd” case; chance accuracy was 20%.

The numbers

Stage	Accuracy	Confidence
First instalment	26%	33%
Last instalment	~28%	53%

Decision fatigue. You will be told your judgment degrades through the day, citing the Israeli parole-judges study. Don’t use it. Glöckner (2016) showed the claimed effect is OR = 35, d = 1.96. Meta-analytic ego-depletion effects run roughly d = −0.10 to 0.25, so this is roughly eight times the top of that range, and more than twice the conventional threshold for a “large” effect. Favorable rulings take longer (7.37 versus 5.21 minutes, in the original authors’ own data), and Glöckner’s simulations show that this alone, with sessions ending on the clock, reproduces much of the curve. And ego depletion itself has largely failed: Hagger’s 23-lab replication (N = 2,141) found d = 0.04; Vohs’ 36-lab attempt (N = 3,531) found d = 0.06 with Bayes factor 4:1 favoring the null.

Parole judges, as claimedd = 1.96

Ego depletion, meta-analytic upper ranged ≤ 0.25

36-lab replication (N = 3,531)d = 0.06

23-lab replication (N = 2,141)d = 0.04

Sources: Glöckner (2016), including the bias-corrected meta-analytic range he cites; Hagger et al. (2016); Vohs et al. (2021).

The numbers

Estimate	d
Parole judges, claimed	1.96
Ego depletion, meta-analytic	−0.10 to 0.25
Vohs et al., 36 labs	0.06
Hagger et al., 23 labs	0.04

What actually works

Six moves, ordered by leverage. All share one property: each changes the structure of the judgment rather than asking you to think harder inside it.

1. Decide the test before you see the answer

Write down what would falsify the artifact — the number that would have to be wrong, the assumption carrying the weight — before you read it.

This is the only defense with a clean mechanism against the anchoring evidence. Once you’ve read a fluent, confident answer, your agreement has already moved 40 points and your confidence has risen. The Schroeder result isn’t about weak-willed reviewers; it’s about the order of operations. The judgment has to exist before the exposure or it isn’t independent of it.

Practically: one line, written first. “This holds only if retention in the enterprise segment is above 85%.” Or, for an AI-drafted spec: “This only works if admins, not end users, send the invites.” Then you check that.

2. Manufacture high-prevalence bursts with feedback

Wolfe’s prevalence experiments tried several interventions that failed. The one that recalibrated the criterion was brief exposure to a high-target-rate environment with immediate correction.

The product version: salt the deck. Periodically review artifacts you know contain errors — plant a wrong number in the metrics-review deck, seed a known bug into a pull request before it goes out for review, keep a folder of real AI outputs where you later found the mistake and re-read them cold, run a session where the team knows three of the ten claims in a research synthesis are wrong and has to find them.

This feels artificial and slightly silly. It is also the best-supported way the literature has found to move the criterion, and it is how airport baggage screening keeps screeners calibrated: the system projects fictional threats onto real bags. It has not been tested on document review, and it is no cure-all; in one 2016 simulation of medical screening, inserted catch trials did no better than a second reader.

Accept the cost: your false-alarm rate will rise. You will challenge things that were fine. That is the price and it is not optional.

3. Fractionate your own expertise

Take twenty minutes. List the recurring judgments you make. For each, ask the Kahneman–Klein questions: is this environment regular enough to have learnable signal, and have I had rapid, unequivocal feedback on it?

You’ll find three groups. A few judgments where you have genuine trained intuition — usually narrow, usually things you’ve done hundreds of times with fast feedback. A middle group where you have experience but the feedback was slow and confounded. And a group where you’ve been running on confidence with no validity behind it at all — typically the highest-stakes, longest-horizon calls.

Trust your gut only in the first group. Use structure for the second. Use a model or defer to base rates for the third.

Trust the gut

Done hundreds of times, with fast, unambiguous feedback: whether a flow confuses people in a usability session, whether a bug report reproduces, whether a line of copy reads clearly.

You have been visibly wrong here, quickly, many times.

Use structure

Real experience, slow or noisy feedback: sizing a feature, reading a research synthesis, a hiring loop, a launch go/no-go.

Decide the test first; score against a short checklist.

Use a model or base rates

Long horizons, confounded outcomes: the annual strategy bet, a pricing change, whether a market will be there in two years.

Start from how often bets like this usually work, then adjust.

Framework: Kahneman & Klein (2009). The examples are illustrations of the three groups, not measurements.

It’s the cheapest high-value thing in this essay.

4. Convert repeated judgments into unit-weight models

If you make the same kind of call more than a few times a year — which features to fund, which candidates to hire, which customers are at risk — write down the three to six factors that predict the outcome, give each equal weight, and score against it. For a feature bet: is there usage evidence for the problem, is the affected segment large, is the change cheap to reverse, does it have a measurable success signal. One point each; add them up.

Not because the model is smart. Because it’s consistent. Dawes got .48 from unit weights against .19 for the expert committee. Grove, Ægisdóttir and Kuncel all found the same direction across 200+ studies.

The evidence from hiring is especially clean now that Schmidt and Hunter’s famous validities have been corrected. Sackett et al. (2022) found the real numbers are much lower than everyone quoted — but the ratio moved sharply toward structure: unstructured interviews fell from .38 to .19, while structured interviews fell only from .51 to .42. The correction weakened selection testing generally and strengthened the case for structure specifically. Structured interviews are now more than twice as predictive as unstructured ones.

• As long quoted
• Corrected

Structured interview.51.42

Unstructured interview.38.19

0.2.4.6

Source: Sackett et al. (2022), Journal of Applied Psychology.

The numbers

Method	Before	Corrected
Structured interview	.51	.42
Unstructured interview	.38	.19

You will resist this because it feels like abdicating judgment. It isn’t. Choosing the variables is the judgment. Dawes’ whole point is that the expertise lives in knowing what to look at, and the human failure is in the adding-up.

5. Independent judgment before discussion

When several people assess the same thing, collect their assessments before anyone speaks. This is the one piece of “decision hygiene” with a clear mechanism: once the first confident voice is in the room, everyone else’s judgment is anchored, exactly as the reviewers were anchored by the model’s answer. In practice: scores in the prioritization sheet before the planning meeting, written comments before the design crit, every interviewer’s hire or no-hire logged before the debrief.

A caveat, since I’ve been strict elsewhere: the insurance noise audit that made this famous — a 55% median gap between two underwriters’ quotes on the same case, where executives guessed 10% — has never been published with data, sample sizes or confidence intervals. It also measures disagreement rather than error; nobody established which underwriter was right. Treat the specific numbers as unverified. The mechanism is still sound, and collecting judgments independently costs roughly nothing.

6. Score yourself, and keep the scores

Judgment can’t improve without feedback, and your job doesn’t supply it. So make some: prediction, date, confidence, then a scoring pass.

One honest note. The usual citation here is Tetlock’s forecasting-training work, and it’s under live challenge. Hauenstein et al. (Psychological Science, 2025) found the training and teaming effects shrank, vanished or reversed when self-selected question choice and response timing were controlled; Year 1 teaming actually inverted to d = −0.285. Mellers and Dougherty have agreed to an adversarial collaboration.

So I’m not claiming calibration training reliably works. The claim is narrower: without a scored record you have no information at all about the quality of your own judgment, and in a profession with no natural feedback that absence is the default state. A two-year record of scored calls is the only evidence that distinguishes good judgment from confident prose — which matters more every month, as the prose gets free.

A protocol

Concretely, for a high-volume artifact review — an AI-drafted PRD, a research synthesis, a pull request, an eval run:

Before opening it. One line: what would make this wrong. One line: what I expect it to say. Thirty seconds total.

While reading. Check the falsifier first, not the argument. Resist reading for overall quality — that’s the mode the evidence says fails. If the artifact is a recommendation, form your own answer before reading theirs.

After. Log any frontier failure in one line. If this is a repeated judgment type, score it against your model rather than your impression.

Weekly. One salted review — something you know contains an error.

Quarterly. Score your open predictions. Re-check your fractionation map: which judgments have earned trust, which haven’t.

Total overhead: maybe ninety minutes a month. That’s the honest cost.

What I’d want to be wrong about

I cut my best argument. I planned to build this around the dilution effect — the finding that adding non-diagnostic information weakens judgments based on diagnostic information. It maps perfectly to AI output: lots of plausible, irrelevant context.

It didn’t survive checking. Highhouse et al. (2022) ran a preregistered direct replication with N = 796 under optimal conditions and found no dilution. Worse for my purposes, Sanborn et al. (2020, five experiments) reinterpret the mechanism in the opposite direction: people were roughly accurate when given diagnostic plus non-diagnostic evidence and overconfident on diagnostic evidence alone. On that reading, the extra information corrected an overconfident baseline rather than degrading a good one. If they’re right, my argument runs backwards. The original (Nisbett, Zukier & Lemley 1981) is internally solid, and McHugh & Igou (2025, N = 2,535) extended it to moral-character judgments, so it isn’t dead. But it cannot carry weight.

The low-prevalence analogy is an analogy. Wolfe and Evans studied visual search for discrete targets. Reviewing a strategy document is not visual search, and I have no direct evidence that criterion shift operates the same way on semantic judgment. The mechanism is plausible and the parallel is strong, but it is transfer, not measurement.

Checklists are not magic. The most famous checklist result comes from surgery, and it is weaker than its fame. Haynes et al. (2009) found surgical mortality falling from 1.5% to 0.8% across eight hospitals — but it was an uncontrolled pre-post study. Urbach et al. (2014), a far larger Ontario study covering over 200,000 procedures, found no significant effect (OR 0.91, CI 0.80–1.03). I’m recommending structure on the strength of the Meehl-Dawes-Grove line, not the checklist literature, which is mixed.

And the cost is real. Every recommendation here trades misses for false alarms. If you implement all of this you will slow down, challenge things that turn out fine, and irritate people. In a low-prevalence environment that is the correct trade — but it is a trade, and anyone selling it as free is selling something.

The short version

Your judgment is not degrading. Your threshold is moving, because almost everything that crosses your desk is fine, and adapting to that is what a well-calibrated system does.

The problem is that the adaptation is invisible from the inside, you can’t fix it by trying harder or going slower, and the one thing that reliably contaminates it — seeing a confident answer before forming your own — is now the default shape of every workflow you have.

So stop trying to be a better reader of artifacts. Decide the test before you see the answer. Salt your own deck. Find out which of your judgments ever had real feedback behind them. Turn the repeated ones into something a checklist could run. And keep score, because in a world where everyone’s output reads well, a record of being right is the only thing left that isn’t free.

Sources

Where the evidence comes from. Each row opens the document itself; the tag says which part of the argument it supports, and ↩︎ goes back to the sentence that uses it.

1. Rare items often missed in visual searchesWolfe, Horowitz & Kenner · Nature 435, 439–440Low prevalenceJournal2005Targets rare, misses common↗︎The prevalence effect
2. Low target prevalence is a stubborn source of errors in visual search tasksWolfe, Horowitz, Van Wert, Kenner, Place & Kibbi · Journal of Experimental Psychology: General 136(4)Low prevalenceJournal2007d′ 2.01 → 2.43; criterion −0.02 → 1.07; misses .20 → .46↗︎Criterion moved, sensitivity held
3. If you don’t find it often, you often don’t find it: why some cancers are missed in breast cancer screeningEvans, Birdwell & Wolfe · PLoS ONE 8(5): e64366RadiologyJournal201314 radiologists in clinic; 6 re-read at 50%; 30% vs 12% missed↗︎The prevalence effect in experts
4. Conditions for intuitive expertise: a failure to disagreeKahneman & Klein · American Psychologist 64(6)ExpertiseJournal2009Two conditions for skilled intuition; fractionated expertise↗︎When intuition can be trusted
5. Clinical versus statistical predictionPaul E. Meehl · University of Minnesota PressMechanical vs clinicalReport1954The original argument↗︎Where the line of research starts
6. Clinical versus mechanical prediction: a meta-analysisGrove et al. · Psychological Assessment 12(1)Mechanical vs clinicalJournal2000136 studies; mechanical ~10% better↗︎Meta-analysis one
7. The meta-analysis of clinical judgment project: fifty-six years of accumulated research on clinical versus statistical predictionÆgisdóttir et al. · The Counseling Psychologist 34(3)Mechanical vs clinicalJournal200667 studies, 92 effect sizes; 13% more accurate (stringent sample)↗︎Meta-analysis two
8. Mechanical versus clinical data combination in selection and admissions decisions: a meta-analysisKuncel et al. · Journal of Applied Psychology 98(6)Mechanical vs clinicalJournal2013r = .44 vs .28↗︎Meta-analysis three
9. The robust beauty of improper linear models in decision makingRobyn M. Dawes · American Psychologist 34(7)Unit weightsJournal1979Oregon, N = 111: committee .19, regression .38, unit weights .48↗︎Unit weights beat the committee
10. Automation bias in mammography: the impact of artificial intelligence BI-RADS suggestions on reader performanceDratsch et al. · RadiologyAI anchoringJournal202327 readers; wrong AI: most experienced 82.3% → 45.5%, least 79.7% → 19.8%↗︎Radiologists anchored by AI
11. Automation bias in medicine: the influence of automated diagnoses on interpreter accuracy and uncertainty when reading electrocardiogramsBond et al. · Journal of Electrocardiology 51(6S)AI anchoringJournal201830 physicians; non-cardiologists 86% → 27% with a wrong diagnosis↗︎The same on ECGs
12. Just put a human in the loop? Investigating LLM-assisted annotation for subjective tasksSchroeder, Roy & Kabbara · Findings of ACL 2025AI anchoringPaper2025410 annotators; overlap 40% → 81–87%; GPT-4 F1 .47 → .79↗︎Reviewers anchored by the model
13. Humans inherit artificial intelligence biasesVicente & Matute · Scientific Reports 13AI anchoringJournal20233 experiments; simulated diagnosis; bias persisted without the AI↗︎Contamination, not atrophy
14. Aircrews and automation bias: the advantages of teamwork?Mosier, Skitka, Dunbar & McDonnell · International Journal of Aviation Psychology 11(1)Automation biasJournal2001Bias training and verify prompts: no effect↗︎Being careful does not fix it
15. Complacency and bias in human use of automation: an attentional integrationParasuraman & Manzey · Human Factors 52(3)Automation biasJournal2010Pilots and controllers; not prevented by training↗︎Expertise does not fix it
16. The influence of time pressure and case complexity on physicians’ diagnostic performanceALQahtani et al. · Health Professions Education 2(2)SpeedJournal201637 residents; time pressure η² = 0.002, complexity η² = 0.85↗︎Time is not the constraint
17. The impact of speed and bias on the cognitive processes of experts and novices in medical image decision-makingTrueblood et al. · Cognitive Research: Principles and Implications 3SpeedJournal2018Speed cut novices’ d′ (p = .03), not pathologists’↗︎Speed and expertise
18. Overconfidence in case-study judgmentsStuart Oskamp · Journal of Consulting Psychology 29(3)InformationJournal1965Accuracy 26% → ~28%; confidence 33% → 53%↗︎More information, more confidence
19. The irrational hungry judge effect revisited: simulations reveal that the magnitude of the effect is overestimatedAndreas Glöckner · Judgment and Decision Making 11(6)Decision fatigueJournal2016Claimed OR = 35, d = 1.96↗︎The parole-judges study, taken apart
20. A multilab preregistered replication of the ego-depletion effectHagger et al. · Perspectives on Psychological Science 11(4)Ego depletionJournal201623 labs, N = 2,141, d = 0.04↗︎Ego depletion, replicated to nothing
21. A multisite preregistered paradigmatic test of the ego-depletion effectVohs et al. · Psychological Science 32(10)Ego depletionJournal202136 labs, N = 3,531, d = 0.06; data 4× likelier under null↗︎And again
22. Revisiting meta-analytic estimates of validity in personnel selection: addressing systematic overcorrection for restriction of rangeSackett, Zhang, Berry & Lievens · Journal of Applied Psychology 107(11)SelectionJournal2022Unstructured .38 → .19; structured .51 → .42; GMA .51 → .31↗︎The correction favoured structure
23. Noise: a flaw in human judgmentKahneman, Sibony & Sunstein · Little, Brown Spark (book)NoiseReport2021Insurance audit: 55% median gap vs 10% expected; no data published↗︎The noise audit, flagged
24. Rethinking the role of teams and training in geopolitical forecasting: the effect of uncontrolled method variance on statistical conclusionsHauenstein, Thomas, Illingworth & Dougherty · Psychological ScienceForecastingJournal2025Effects reduced, eliminated or reversed with controls; Year 1 teaming d = −0.285↗︎Calibration training, challenged
25. The dilution effect: nondiagnostic information weakens the implications of diagnostic informationNisbett, Zukier & Lemley · Cognitive Psychology 13(2)DilutionJournal1981N = 500↗︎The argument I cut
26. Failure to replicate the basic dilution effect in performance predictionHighhouse et al. · International Journal of Selection and Assessment 30(2)DilutionJournal2022Preregistered; N = 796; no dilution in GPA predictions↗︎It did not replicate
27. A dilution effect without dilution: when missing evidence, not non-diagnostic evidence, is judged inaccuratelySanborn, Noguchi, Tripp & Stewart · Cognition 196DilutionJournal2020Five experiments; accurate on mixed evidence, overestimate diagnostic alone↗︎The mechanism, reversed
28. The moral dilution effect: irrelevant information influences judgments of moral characterMcHugh & Igou · Journal of Behavioral Decision Making 38(4)DilutionJournal2025N = 2,535; dilution for good and bad actors↗︎A replication that held
29. A surgical safety checklist to reduce morbidity and mortality in a global populationHaynes et al. · New England Journal of Medicine 360(5)ChecklistsJournal20098 hospitals; mortality 1.5% → 0.8%; uncontrolled↗︎The checklist case for
30. Introduction of surgical safety checklists in Ontario, CanadaUrbach et al. · New England Journal of Medicine 370(11)ChecklistsJournal2014101 hospitals, 215,711 procedures; death OR 0.91 (0.80–1.03)↗︎The checklist case against
31. Does time pressure have a negative effect on diagnostic accuracy?ALQahtani et al. · Academic Medicine 91(5)SpeedJournalMay 201642 residents; time pressure lowered accuracy, more on difficult cases↗︎Time pressure can hurt too
32. Extraneous factors in judicial decisionsDanziger, Levav & Avnaim-Pesso · PNAS 108(17)Decision fatigueJournalApr 2011Favorable rulings 65% → ~0% within sessions; 7.37 vs 5.21 min↗︎The original parole-judges study