Executive summary
- Most popular habits don’t have the evidence. Pre-mortems were only ever tested for confidence, decision journals have no direct study, and implementation intentions shrink to d = .15 once publication bias is corrected.
- The real problem is feedback. Deliberate practice explains 18 to 26% of performance in games, music and sport, and under 1% in professions. A product call made in March is graded in November, if ever.
- AI made the loop worse. Output went up; signal didn’t. Reading AI output doesn’t catch its errors, and past the edge of the model’s competence, people using it were less often right and more convincing.
- The test for any habit: does it manufacture feedback the job doesn’t supply? If not, it’s a ritual, and rituals that feel rigorous grant confidence without earning it.
- Drop eight: reading AI output to judge it, research volume as discovery, synthetic users for segments, options as work, artifacts as a self-measure, dates as the roadmap’s axis, delegating sense-making, and rituals that never get scored.
- Build eight, graded by evidence. If you only take two: decide what would prove an analysis wrong before you read it, and score your predictions.
The habits worth keeping all do one thing: they tell you, soon and plainly, whether you were right.
I set out to write a confident list of habits. Then I checked the evidence behind the ones I was about to recommend, and most of them fell over.
Pre-mortems: one study, 178 participants, measuring only how confident people felt. Nobody has tested whether they improve accuracy. Decision journals: I couldn’t find a single direct study. Implementation intentions, the best-evidenced behavior-change technique in psychology, were revised down by their own authors in 2025, from d = .65 to d = .36, and to d = .15 after correcting for a publication bias they called “extreme.” And what made a plan work wasn’t the paperwork: effects were larger when the plan was a simple if-then, when people were highly motivated, and when they rehearsed it.
n = 178
people in the one pre-mortem study, which measured confidence only
Pre-mortems
0
direct studies of decision journals I could find
Decision journals
Sources: Gollwitzer & Sheeran (2006); Sheeran, Listrom & Gollwitzer (2025), European Review of Social Psychology; Veinott, Klein & Wiggins (2010).
The numbers
| Habit | Evidence |
|---|---|
| Implementation intentions | d = .65 → .36 → .15 bias-corrected |
| Pre-mortems | One study, n = 178, confidence only |
| Decision journals | No direct study found |
So this isn’t a list of rituals. It’s an argument about why most product habits fail, and what the few that survive have in common.
The organizing principle
One finding reorganized the whole essay.
Deliberate practice — structured, effortful repetition with feedback — explains 18 to 26% of performance variance in games, music and sport. In professions, it explains under 1%.
Source: Macnamara, Hambrick & Oswald (2014), Psychological Science, meta-analysis.
The numbers
| Field | Variance explained |
|---|---|
| Games | 26% |
| Music | 21% |
| Sports | 18% |
| Education | 4% |
| Professions | under 1% |
Same technique. Same humans. The difference is feedback. A chess player learns within minutes whether a move was good. A pianist hears the wrong note at once. A product leader makes a call in March and finds out in November, when six other things have changed and the result can’t be traced to anything.
Drawn to one scale, in days: the first two bars are there, just too short to see.
The numbers
| Who | Time to feedback |
|---|---|
| Pianist | Immediate |
| Chess player | Minutes |
| Product leader | Months, confounded |
Product management has always been a low-feedback profession. AI made it worse in a specific way: it produces plausible, well-structured, confident answers with no ground truth attached. The volume of output went up enormously. The volume of signal didn’t move.
That gives us one test that sorts every habit on the list:
Does this habit manufacture feedback the job doesn’t supply on its own?
If yes, it’s worth the effort, even when the direct evidence is thin. If no, it’s a ritual that feels rigorous — and rituals that feel rigorous are dangerous now, because they grant the confidence without earning it.
Habits to drop
1. Reviewing AI output by reading it
This is the big one, and several others rest on it.
People are bad at catching errors in automated output. Skitka and colleagues (1999) found error detection fell from 97% to 59% just because an automated aid was present. Participants also followed the aid’s wrong recommendation about 65% of the time, with contradicting information on their own instruments.
- Without the aid
- With the aid
Source: Skitka, Mosier & Burdick (1999), International Journal of Human-Computer Studies.
The numbers
| Measure | Value |
|---|---|
| Error detection, no aid | 97% |
| Error detection, with aid | 59% |
| Followed wrong advice | ~65% |
What doesn’t fix it: expertise, practice, or being told to check. Mosier and colleagues (2001) found that on-screen prompts to verify, and training about the bias, had no effect.
And it’s worse than random, because of what Dell’Acqua’s team found at HBS. On tasks outside the model’s competence, correctness fell from 84.5% to 60–70%, and the AI users’ answers scored 17.9 to 25.1% higher on coherence than the no-AI group’s, right or wrong. They got more answers wrong and wrote them more convincingly. The output that reads best is not the output most likely to be right.
- Without AI
- With AI
+17.9–25.1%
higher coherence scores for the AI users’ answers, right or wrong
The trap
Source: Dell’Acqua et al. (2026), “Navigating the Jagged Technological Frontier,” Organization Science, n = 758.
The numbers
| Measure | Value |
|---|---|
| Correct, without AI | 84.5% |
| Correct, with AI | 60–70% |
| Coherence, AI users vs no-AI group | 17.9–25.1% higher |
Dropreading AI output end to end and forming an impression of its quality.Domaineverything, but most of all strategy and analysis.
2. Treating research volume as discovery
The sharpest statistic I found. User Interviews surveyed 485 researchers: AI adoption jumped 24 points to 80% in a year. Median research output — two mixed-method, three qualitative and one quantitative study per six months — barely moved.
- A year earlier
- 2025
2
mixed-method studies per six months
Unchanged
3
qualitative studies
Unchanged
1
quantitative study
Unchanged
Source: User Interviews, State of User Research 2025 (485 researchers).
The numbers
| Measure | Before | After |
|---|---|---|
| Researchers using AI | 56% | 80% |
| Median studies per six months | 2 mixed, 3 qual, 1 quant | Same |
Near-universal adoption, next to no gain in throughput.
That suggests AI took over the production of research artifacts — the summaries, the affinity maps, the readouts — when production was never the constraint. Learning was, and learning is limited by how many real conversations you have.
Dropcounting research outputs as evidence that you understand your users.Domaindiscovery.
3. Synthetic users for anything segment-specific
In a separate User Interviews survey, only 8% of researchers use synthetic respondents regularly and 28% reject them outright. The evidence backs the skeptics. Park and colleagues replicated 14 classic studies with LLM respondents: 21% succeeded, 43% produced unusable data. Bisbee and colleagues found the dangerous pattern: aggregate means roughly matched reality while the spread of opinion and the regression coefficients were wrong.
8%
of researchers use synthetic respondents regularly
Adoption
28%
reject them outright
Rejection
Sources: Park, Schoenegger & Zhu (2024); User Interviews, State of Synthetic Users. Bisbee et al. (2024) found averages that matched while the spread of opinion and the regression coefficients did not.
The numbers
| Measure | Value |
|---|---|
| Studies replicated | 21% |
| Unusable data | 43% |
| Researchers using synthetic respondents | 8% |
| Rejecting them | 28% |
That failure is made to fool a product team. The headline number looks plausible, so you ship it, and every segment-level decision under it rests on noise.
Dropsynthetic respondents for anything you’d segment on. Keep them, if at all, for testing question wording before real interviews.Domaindiscovery.
4. Generating options as the work
When options were expensive to produce, having them was valuable, and the PM who could generate them was useful. Now generation costs nothing. Forty feature ideas is one prompt.
The scarce act is elimination, and almost nobody measures it. Teams still count ideas generated, initiatives proposed, roadmap items. Nobody reports a kill count.
Droptreating option generation as a contribution.Domainstrategy and prioritization.
5. Measuring yourself by artifacts produced
Meta’s scaled-back Project OT gave us the cleanest numbers anyone has on this. Across Meta’s internal infrastructure and platforms, year over year: code changes up 220%, user-facing features up 36%, technical incidents up 40%.
Source: Reuters special report, 26 August 2026.
The numbers
| Measure | Change |
|---|---|
| Code changes | +220% |
| User-facing features | +36% |
| Technical incidents | +40% |
Motion tripled. Output rose by a third. Defects rose faster than output.
If your weekly sense of a job well done comes from documents shipped, that instrument is broken. It measured effort, effort has come apart from outcome, and you’ll feel most productive in exactly the weeks you generate the most slop for someone else to review.
Dropartifact count as a measure of yourself.Domainpersonal craft and team leadership.
6. The roadmap as a confidence display
Dates on a roadmap have always been a confidence claim dressed as a schedule. Build time is now the smallest and most variable term in the equation, so a date-based roadmap is less accurate and tells you less than it did.
Meanwhile only 31% of product teams prioritize early iteration, 40% run little or no experimentation, and 84% fear their current products won’t succeed (Atlassian State of Product 2026, 1,000+ respondents). The roadmap is doing reassurance work the evidence doesn’t support.
Source: Atlassian, State of Product 2026, with Wakefield Research.
The numbers
| Measure | Share |
|---|---|
| Prioritize early iteration | 31% |
| Limited or no experimentation | 40% |
| Fear products won’t succeed | 84% |
Dropdates as the primary axis.Domainstakeholders and strategy.
7. Using AI on the tasks that are your judgment
I have to be careful here, because the deskilling literature is much weaker than its reputation, and I nearly wrote something I’d have had to retract.
The “Google effect” — digital amnesia — failed to replicate twice, once in a preregistered study with a Bayes factor of 5.07 favoring the null. Calculators don’t damage arithmetic: in a meta-analysis of 54 studies, the 15 that tested computation without calculators found an effect of .03, statistically indistinguishable from zero. The MIT “Your Brain on ChatGPT” EEG study is still a preprint. Only 18 participants completed the session behind every headline, and a formal commentary flags its small sample, its EEG analysis and its inconsistent reporting.
The 2025 Lancet endoscopy study is the strongest item: 19 experienced endoscopists, whose unassisted detection rates fell from 28.4% to 22.4% after exposure to AI. But it’s observational, the main confounder was never reported, and the authors call it preliminary.
Didn’t hold
The “Google effect” failed to replicate twice. Calculators: an effect of .03 on computation tested without them, in 15 of 54 studies.
A preregistered replication favored the null, Bayes factor 5.07.
Weak
MIT’s EEG study: a preprint, 18 completers. The Lancet endoscopy study: 28.4% → 22.4%, observational.
A formal critique flags its sample, its EEG analysis and its reporting.
Holds
Casner’s pilots kept their stick-and-rudder skills but lost the picture: 4 of 16 lost track of their position and missed the airport.
Automation displaces the model in your head, not the skill in your hands.
Sources: Camerer et al. (2018); Hesselmann (2020), PeerJ; Ellington (2003); Kosmyna et al. (2025), arXiv; Budzyń et al. (2025); Casner et al. (2014).
So the honest version isn’t “AI rots your brain.” It’s Casner’s 2014 aviation finding, which is more interesting and travels better. Pilots kept their stick-and-rudder skills but lost cognitive navigation ability: 4 of the 16 lost track of their position and missed the airport altogether. Automation doesn’t weaken your hands. It displaces the knowledge in your head about where you are and what happens next.
The product equivalent isn’t losing the ability to write a spec. It’s losing the mental model of your own system: why the funnel behaves the way it does, which constraint binds, what happened the last time someone tried this.
Drophanding off the sense-making, as opposed to the drafting.Domainpersonal craft.
8. Rituals that end without a score
Pre-mortems, decision journals, retros, confidence ratings: all of them feel rigorous, and all share one failure. They produce a record that nobody grades.
A pre-mortem you never revisit after launch is theater. A decision journal with no scoring pass is a diary. Martin and Mandel (2025, N = 610 and 871) found no calibration improvement from outcome feedback, or even from structured scoring feedback. Those were short lab sessions on trivia-style questions, not a record kept over years. Still, they’re a warning that scoring alone doesn’t guarantee better judgment.
Dropany ritual whose loop you don’t close.Domainall of them. This one quietly wastes the most time.
Habits to build
Graded honestly. I’ve marked each one [Strong], [Reasoned] or [Bet] so you can spend your effort accordingly — and several things I’d have confidently recommended a week ago come out as bets.
Strong
Falsification-first reading. Own one outcome publicly.
Both rest on what fails: review after the fact, and instructions to be careful.
Reasoned
Scored predictions. A frontier log. A raw-contact quota. A kill count. Teach it.
Argued from the evidence, not tested directly.
Bet
Unassisted reps on the cognitive core.
An hour a week against a plausible mechanism.
1. Score your predictions, don’t just record them Reasoned
Before the decision, write down what you expect: the metric, the number, the date, your confidence. Then go back and grade yourself. That’s the part everyone skips, and it carries all the value.
The evidence, honestly: Tetlock’s forecasting work is the usual citation, but Hauenstein and colleagues reanalyzed the core data in Psychological Science (2025) and found the training and teaming effects “substantially eliminated, reduced, and, in some cases, even reversed” once question selection and submission timing were controlled. Mellers and colleagues have proposed an adversarial collaboration to settle it. What isn’t disputed is that some people are reliably better forecasters than others: the selection effect holds even where the training effect is contested.
So the case for this isn’t “calibration training works.” It’s narrower and sturdier. In a profession where individual judgment is nearly impossible to prove, a two-year record of scored predictions is the only artifact that sets you apart from someone with equally good prose. That argument doesn’t depend on Tetlock at all.
Fixesthe feedback problem, directly.Domainstrategy, and career.
2. Falsification-first reading Strong
Before you read an AI-generated analysis, write down what would make it wrong. Which number would have to be false. Which assumption holds the weight. Then check only that.
This is the one habit on the list with strong support, because it’s built on what fails. Review after the fact doesn’t catch errors, prompts to verify have no effect, and past the frontier AI made people less often right and more convincing. Deciding the test before the persuasion reaches you is the only move not already known to fail.
What the literature says does help, when almost nothing else does: Skitka’s follow-up found that making people accountable for decision accuracy reduced both kinds of error.
Fixesautomation bias.Domainstrategy, analysis, anything where you’re the last check.
3. Keep a frontier log Reasoned
Every time a model fails you in your domain, write down the shape of the failure in one line. Every time it surprises you on the upside, the same.
The jagged-frontier research shows that capability is uneven and the boundary is invisible from outside. It also shows that generic prompt-engineering training made performance outside the frontier worse, because it built confidence in a frontier people still couldn’t see. The boundary doesn’t carry over between domains, so the only way to know yours is to map it yourself.
After six months you have a document nobody else has about your product.
Fixescalibration about the tools.Domaindelivery and personal craft.
4. Unassisted reps on the cognitive core Bet
Once a week, do one piece of thinking with no assistance: read the raw funnel data yourself, work out the constraint, write the argument cold.
I’m marking this a bet on purpose. The deskilling evidence is weak, and I’ve just spent several paragraphs saying so. But Casner’s displaced-cognition finding is a plausible mechanism, the cost is an hour a week, and if I’m wrong, you spent an hour thinking without a tool. That asymmetry is enough to do it anyway.
Fixespossible decay of your mental model.Domainpersonal craft.
5. A raw-contact quota Reasoned
Set a number of unmediated customer conversations per month — not summarized, not synthesized, you in the room. Put it on the calendar as a recurring commitment.
The research-volume finding is the argument: AI took over artifact production while learning stayed flat. If synthesis is now free and exposure is still expensive, exposure is where the input that sets you apart lives. The Nature Human Behaviour finding that AI raises the quality of individual ideas while reducing the diversity of the pool points the same way. If everyone’s synthesis converges, the person with non-consensus raw input has the only real advantage left.
Fixesthe input problem.Domaindiscovery.
6. A kill count Reasoned
Report what you decided not to do, with the reason, as often as you report what shipped.
It makes elimination visible as work, which matters when generation is free. It also turns the most common form of senior judgment — the thing you stopped — from invisible to legible, which is as much a career argument as a practical one.
Fixesmeasuring the wrong thing.Domainstrategy, stakeholders.
7. Own one outcome publicly Strong
Pick a metric, say where you think it’ll be in a quarter, and say it in front of people who’ll remember.
Accountability for outcomes is the one intervention that reduced automation bias in Skitka’s work, where instructions, training and expertise all failed. It’s also the thing AI can’t take from you: a system that can’t be held responsible can’t absorb responsibility.
Fixesautomation bias and the value problem.Domainleadership, career.
8. Teach it Reasoned
Run a session, write the essay, take the cohort.
The compression findings are the reason. AI lifts below-average performers 43% and top performers 17%; the most skilled support agents saw small declines in quality; and in the writing study, the correlation between a person’s performance on task one and task two fell from 0.49 to 0.25. AI is pulling output quality apart from individual ability. Once your documents no longer prove you’re good, explaining the reasoning behind them is one of the few proofs left — and teaching forces the reasoning to be explicit enough to check.
0.49 → 0.25
how well a writer’s first task predicts their second, without AI and with it
Writing study
Sources: Dell’Acqua et al. (2026), Organization Science; Noy & Zhang (2023), Science. The correlation compares the control group with the ChatGPT group.
The numbers
| Measure | Value |
|---|---|
| Below-average performers | +43% |
| Top performers | +17% |
| Task-to-task correlation | 0.49 → 0.25 |
Fixesthe legibility problem.Domaincareer, team.
Which habit fixes what
If you only take two: falsification-first reading and scored predictions. The first is the only defense against the failure that’s growing. The second is the only way to build evidence about your own judgment in a profession that doesn’t generate any.
How to actually install these
Most habit advice is folklore. Here’s what survives scrutiny, including the parts that contradict the popular version.
Expect it to take longer than you’ve been told, and to vary wildly. Lally’s famous “66 days” is a median drawn from 39 of 96 participants (the ones whose data fit the model well), and individuals ranged from 18 to 254 days. A larger review (20 studies, 2,601 people) puts the median at 59–66 days but the mean at 106–154, with a range of 4 to 335 days. Complexity dominates. “Write one prediction per decision” will stick on that timeline. “Restructure how I evaluate evidence” won’t.
Sources: Singh et al. (2024), 20 studies, 2,601 participants. Lally et al. (2010) found 18 to 254 days in the 39 people whose data fit.
The numbers
| Measure | Days |
|---|---|
| Median | 59–66 |
| Mean | 106–154 |
| Range | 4–335 |
Don’t over-invest in writing the plan. The 2025 review of implementation intentions found bigger effects for a simple if-then plan that people rehearsed, not for more elaborate ones. Say when and where — “before I read any AI analysis, I write the falsifier” — then go and do it, rather than building a system around it.
Motivation is a precondition, not an output. Implementation intentions work best when people are already highly motivated, and an earlier study found they helped only when the goal intention was strong. These techniques turn motivation into action; they don’t create it. If you don’t believe your judgment needs evidence, none of this will hold.
If-then
the plan names a cue and a response, not a wish
Format
Motivated
people already wanted the goal; with a weak goal intention, plans didn’t help
Motivation
Rehearsed
the plan was repeated until the cue calls it up
Rehearsal
Sources: Sheeran, Listrom & Gollwitzer (2025), European Review of Social Psychology; Sheeran, Webb & Gollwitzer (2005), Personality and Social Psychology Bulletin.
The numbers
| Component | Finding |
|---|---|
| If-then format | Larger effects |
| High motivation | Larger effects; no benefit with weak goal intentions (2005) |
| Rehearsal | Larger effects |
Change the cue, not the intention. Established habits are triggered by context, not by current goals, even though people believe their habits are driven by goals. So the leverage is in your surroundings: a template that opens with the falsifier field, a recurring calendar block for customer calls, a standing agenda item for the kill count.
Self-chosen beats assigned. This is the uncomfortable one for leaders. Habits people pick themselves tend to become stronger than assigned ones, so mandating these across your team starts them in the weaker category. Run them yourself, visibly, and report what you learn. Adoption by example is slower, and it’s the only version that survives you losing interest.
And ignore “70% of change initiatives fail.” It has no valid empirical basis: Hughes (2011) traced five published sources of it and found evidence behind none.
What I’d want to be wrong about
Most of the habit-change evidence is weaker than the confident version. Implementation intentions at d = .15 after bias correction is a small effect. Pre-mortems have never been tested for accuracy. Decision journals have no direct study at all. I’ve recommended several of these anyway, on cost and benefit rather than evidence, and I’ve marked which.
The deskilling case is mostly unproven. The Google effect failed to replicate twice, the calculator studies found nothing, and the most viral finding is a preprint with 18 completers and a formal critique of its methods. I’ve kept the unassisted-reps habit as an explicit bet.
Tetlock is under live challenge. The forecasting-training effect that half the product world cites is being contested in Psychological Science right now. I’ve rested the prediction habit on the undisputed part instead.
Self-reported time savings are wrong in direction, not just size — METR’s developers believed they were 20% faster while measuring 19% slower. METR’s newer data on early-2026 tools points toward a speedup, though. METR is also redesigning the study, so that figure shouldn’t be swung around as a blanket claim either.
Source: METR (July 2025). Newer METR data on early-2026 tools points toward a speedup; the study is being redesigned.
The numbers
| Measure | Value |
|---|---|
| Believed | 20% faster |
| Measured | 19% slower |
The short version
The job’s feedback loop was always bad. Now it produces confident, articulate, plausible answers at unlimited volume, with no ground truth attached, which makes the loop worse at exactly the moment the volume went up.
Every habit worth building in 2027 does the same thing: it manufactures a feedback signal the job doesn’t produce on its own. Decide the test before you read the answer. Write the prediction before you know. Count what you killed. Log where the model broke. Say the number out loud in front of people.
Everything else is a ritual, and rituals that feel rigorous are now the most expensive thing on your calendar.
Sources
Where the evidence comes from. Each row opens the document itself; the tag says which part of the argument it supports, and ↩︎ goes back to the sentence that uses it.
- Evaluating the Effectiveness of the PreMortem Technique on Plan ConfidenceVeinott, Klein & Wiggins · Proceedings of ISCRAM 2010 (Seattle)Pre-mortem studyJournal2010178 participants, 5 conditions, H1N1 university plan; outcome = confidence ratings onlyPre-mortems tested on confidence, not accuracy
- The when and how of planning: Meta-analysis of the scope and components of implementation intentions in 642 testsSheeran, Listrom & Gollwitzer · European Review of Social Psychology 36(1), 162-194Implementation intentionsJournal2025642 tests; .27 ≤ d ≤ .66 by outcome; d = .36, .15 bias-corrected; larger with if-then format, high motivation, rehearsalImplementation intentions revised down; what makes plans work
- Implementation intentions and goal achievement: A meta-analysis of effects and processesGollwitzer & Sheeran · Advances in Experimental Social Psychology 38, 69-119Implementation intentions 2006Journal200694 tests; d = .65The earlier d = .65 baseline
- Deliberate Practice and Performance in Music, Games, Sports, Education, and Professions: A Meta-AnalysisMacnamara, Hambrick & Oswald · Psychological Science 25(8), 1608-1618Deliberate practiceJournal2014Variance explained: games 26%, music 21%, sports 18%, education 4%, professions <1%Practice explains performance only where feedback is fast
- Does automation bias decision-making?Skitka, Mosier & Burdick · Int. J. Human-Computer Studies 51(5):991–1006Automation biasJournal1999Omission errors 41% with aid vs 3% missed without (detection 97% → 59%); ~65% commission errorsError detection falls with automated aid
- Complacency and Bias in Human Use of Automation: An Attentional IntegrationParasuraman & Manzey · Human Factors 52(3):381–410Complacency reviewJournal2010Review; complacency in naive and expert users, not overcome by practiceExpertise and practice don’t fix automation bias
- Aircrews and Automation Bias: The Advantages of Teamwork?Mosier, Skitka, Dunbar & McDonnell · Int. J. Aviation Psychology 11(1):1–14Verification promptsJournal2001No effects of automation-bias training or display prompts to verifyInstructions to verify had no effect
- Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and QualityDell’Acqua, McFowland, Mollick, Lifshitz, Kellogg et al. · Organization Science 37(2):403–423Jagged frontierJournalMar 2026n=758; outside frontier correct 84.5% (control) vs 70.6% / ~60% with AI; coherence +17.9% / +25.1% vs control meanWrong AI answers read as more coherent
- The State of User Research Report 2025User Interviews (Liz Steelman, Morgan Koufos)UX research surveyReportSep 2025485 researchers, 25 Jul–9 Aug 2025; AI use +24 pts YoY to 80%; median 2 mixed-method, 3 qual, 1 quant studies per 6 months, ‘barely budged’ vs 2024AI adoption up, research output flat
- State of Synthetic UsersUser InterviewsSynthetic users surveyReport2026150 research professionals surveyed May 2026 + 5 interviews; 8% use synthetic-user tools regularly; 28% never used and actively choose not to; 21% tried once or twice8% use synthetic respondents, 28% reject
- Diminished diversity-of-thought in a standard large language modelPeter S. Park, Philipp Schoenegger, Chongyang Zhu · Behavior Research MethodsLLM replicationsJournal202414 Many Labs 2 studies run on GPT-3.5 (text-davinci-003); 6 unanalysable (‘correct answer’ effect, near-zero variance); of 8 analysable, 37.5% replicated the original results21% replicated, 43% unusable
- Synthetic Replacements for Human Survey Data? The Perils of Large Language ModelsJames Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, Jennifer M. Larson · Political Analysis 32(4):401–416Synthetic survey dataJournal2024ChatGPT personas vs 2016–2020 ANES feeling thermometers for 11 groups; averages match closely; less variation than real surveys; regression coefficients often differ significantly; results shift with prompt wording and over 3 monthsaggregate means right, inference wrong
- Mark Zuckerberg had a bold plan to replace Meta staff with AI. Here’s how it imploded (Reuters special report)Reuters (syndicated by CTV News)Meta Project OTNewsAug 2026Code changes +220% YoY across internal infrastructure and platforms (Bosworth, June post); changes reaching users as new/improved features +36%; major technical and security incidents +40%; firefighting time +70%code +220%, features +36%, incidents +40%
- State of Product Report 2026Atlassian with Wakefield ResearchAtlassian State of ProductReportSep 20251,000+ product professionals (US, France, Germany); 31% prioritize rapid experimentation and iterative learning; 40% do little or no experimentation; 84% worry their products won’t succeed in the market; 85% feel empowered31% iterate, 40% no experimentation, 84% fear
- No conclusive evidence that difficult general knowledge questions cause a “Google Stroop effect”. A replication studyHesselmann · PeerJ 8:e10325Google effectJournalNov 2020Preregistered; sequential BF01 = 5.07 favouring nullGoogle effect failed preregistered replication
- A Meta-Analysis of the Effects of Calculators on Students’ Achievement and Attitude Levels in Precollege Mathematics ClassesEllington · J. Research in Mathematics Education 34(5):433–463CalculatorsJournal200354 studies; computation tested without calculators: 15 studies, mean effect .03, not significantCalculators don’t damage arithmetic
- Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing TaskKosmyna et al. (MIT Media Lab) · arXiv 2506.08872Cognitive debtPaperJun 202554 participants sessions 1–3; 18 completed session 4Viral EEG deskilling preprint, 18 completers
- Comment on: Your Brain on ChatGPT: Accumulation of Cognitive Debt When Using an AI Assistant for Essay Writing TasksStanković, Hirche, Kollatzsch & Doetsch · arXiv 2601.00856Cognitive debt critiquePaperJan 2026Critiques sample size, reproducibility, EEG methods, reportingFormal critique of the MIT study
- Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational studyBudzyń, Romańczyk et al. · Lancet Gastroenterology & Hepatology 10:896–903Endoscopist deskillingJournalAug 202519 endoscopists; 1,443 colonoscopies; ADR 28.4% → 22.4%Strongest AI-deskilling evidence, observational
- The Retention of Manual Flying Skills in the Automated CockpitCasner, Geven, Recker & Schooler · Human Factors 56(8):1506–1516Pilot skill retentionJournal201416 airline pilots; manual skills mostly intact; 4 of 16 lost track of position and missed the airportAutomation displaces cognitive, not motor, skill
- Calibration Feedback With the Practical Scoring Rule Does Not Improve Calibration of ConfidenceMartin & Mandel · Futures & Foresight ScienceCalibration feedbackJournal2025Two experiments, N1 = 610, N2 = 871; neither outcome feedback nor Practical-scoring-rule performance feedback improved calibrationOutcome feedback did not improve calibration
- Psychological Strategies for Winning a Geopolitical Forecasting TournamentMellers, Ungar, Baron, ... Tetlock · Psychological Science 25(5), 1106-1115Forecasting tournamentJournal20142-year IARPA tournament; training, teaming and tracking improved accuracyThe original Tetlock training/teaming result
- Rethinking the Role of Teams and Training in Geopolitical Forecasting: The Effect of Uncontrolled Method Variance on Statistical ConclusionsHauenstein, Thomas, Illingworth & Dougherty · Psychological Science 36(1), 3-18Forecasting reanalysisJournal2025IRT reanalysis of Mellers et al. 2014; controls for question selection and timing ‘substantially eliminated, reduced, and, in some cases, even reversed’ the effectsTraining and teaming effects contested
- Adversarial Collaboration on the Forecaster Benefits of Teams and TrainingMellers et al. · Mack Institute for Innovation Management, WhartonAdversarial collaborationArticleJul 2025Proposal: hold questions and timing constant to retest teams and trainingAdversarial collaboration with Hauenstein et al.
- Accountability and automation biasSkitka, Mosier & Burdick · Int. J. Human-Computer Studies 52(4):701–717AccountabilityJournal2000Accountability for performance or accuracy lowered automation biasAccountability reduces both error types
- ChatGPT decreases idea diversity in brainstormingLennart Meincke, Gideon Nave, Christian Terwiesch · Nature Human Behaviour 9:1107–1109Idea diversityJournal2025Five experiments: ChatGPT-assisted brainstorming gave narrower idea sets, significant in 37 of 45 comparisons; framed as a trade-off with gains in individual idea creativityAI shrinks diversity of idea pool
- Generative AI at WorkErik Brynjolfsson, Danielle Li, Lindsey Raymond · Quarterly Journal of Economics 140(2):889–942Customer support AIJournal20255,172 customer-support agents; +14–15% issues resolved/hour on average, +34% for novice/low-skilled; most experienced/skilled agents: small speed gains, small declines in qualitytop agents saw small quality declines
- Experimental evidence on the productivity effects of generative artificial intelligenceShakked Noy, Whitney Zhang · Science 381(6654):187–192Writing studyJournal2023453 college-educated professionals; ChatGPT cut time ~40% and raised quality 18%; task-1/task-2 grade correlation 0.49 in control vs 0.25 in treatmentability-output correlation 0.49 to 0.25
- How are habits formed: Modelling habit formation in the real worldLally, van Jaarsveld, Potts & Wardle · European Journal of Social Psychology 40(6), 998-1009Habit formation curveJournal201096 recruited; 82 usable data; model fitted 62; good fit 39; median 66 days to 95% of asymptote; range 18-254 daysThe 66-day median and its thin base
- Time to Form a Habit: A Systematic Review and Meta-Analysis of Health Behaviour Habit Formation and Its DeterminantsSingh, Murphy, Maher & Smith · Healthcare 12(23), 2488Habit formation reviewJournalDec 202420 studies, 2,601 participants; median 59-66 days, mean 106-154 days (4 studies); range 4-335 daysHabit time is long and variable
- The interplay between goal intentions and implementation intentionsSheeran, Webb & Gollwitzer · Personality and Social Psychology Bulletin 31(1), 87-98Goal intentionsJournal2005Implementation intentions helped goal attainment when goal intentions were strong, not when weakPlans help only with strong motivation
- How do habits guide behavior? Perceived and actual triggers of habits in daily lifeNeal, Wood, Labrecque & Lally · Journal of Experimental Social Psychology 48(2), 492-498Context cuesJournal20122 experiments: performance contexts, not goals, triggered strong habits; people perceive goals as triggersHabits cued by context, not goals
- Do 70 Per Cent of All Organizational Change Initiatives Really Fail?Hughes · Journal of Change Management 11(4), 451-46470% change mythJournal2011Reviewed five published sources of the 70% figure; no valid and reliable empirical evidenceDebunks the 70% change-failure figure
- Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015Camerer et al. · Nature Human Behaviour 2:637–644Replication projectJournal201821 studies; 13 (62%) replicated; Sparrow et al. Google Stroop did notFirst failed Google effect replication
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer ProductivityMETR (Becker, Rush, Barnes, Rein) · METR blog / arXiv 2507.09089METR RCTArticleJul 202516 experienced OSS developers, 246 tasks; forecast 24% speedup, believed 20% faster afterwards; measured 19% slower (CI +2% to +39%)believed 20% faster, measured 19% slower
- We are Changing our Developer Productivity Experiment DesignMETR · METR blogMETR redesignArticleFeb 202657 developers, 143 repos, 800+ tasks; returning developers est. -18% time (speedup, CI -38% to +9%); new recruits -4% (CI -15% to +9%); selection effects make the data only very weak evidence; design being changedlater estimate toward speedup; redesign