Executive summary
- Research went AI-first in two years. 80% of researchers in one 2025 survey use AI in their work, up 24 points in a year, and 91% worry about the accuracy of what it gives them. The companies that sell AI interviews raised large rounds, and one of them is being bought by Salesforce.
- AI interviewers are better than their reputation. In controlled studies they draw out longer, more specific answers than a survey form, and experts rated over 95% of one AI interviewer’s questions as open, neutral and relevant. But talking to one can change what people say next, and almost nobody has tested one against a skilled human.
- AI synthesis is persuasive whether or not it is true. Coding to a fixed codebook works. Open-ended theme-finding misses some themes and invents others, the errors can follow who was speaking, and even the most faithful summarizers on one public benchmark add unsupported content to 1.8% to 23.3% of summaries.
- Synthetic users get the average and lose the person. Simulated samples often match the direction of real results, but in one replication of 154 experiments only 19% of their confidence intervals held the real effect, and simulated answers vary less than people do.
- Your panel may not be people. An AI respondent passed attention checks in 99.8% of 6,000 trials, and at least 4.4% of open-ended answers on one major panel showed AI help.
- The old rules still decide. Five users find 55% to 99% of problems depending on which five, saying predicts doing weakly, and about one idea in three improves the number it was built to move.
Collecting research got cheap. Knowing whether any of it is true did not. Spend what you save on checking.
Product research used to have a clock you could hear. A week to recruit. A week of interviews. A week to synthesize, if you were rigorous. By the time the deck was ready, the decision had usually been made without it.
Every one of those weeks now has an AI version. A tool recruits the participants. An AI interviewer runs the sessions, in any language, at any hour, a thousand at a time. A model reads the transcripts and hands back themes with quotes attached. And if you don’t want to wait for people at all, a synthetic user will answer your questions immediately, in character, forever.
This essay is about what that does to the work. Not whether to use AI in research — that argument is over, and the numbers below say so — but which parts of the new research stack produce evidence, which produce something that only looks like it, and how to tell the difference before a roadmap rests on it.
The short answer is in the title. Research got cheap. Evidence didn’t.
The new research stack, in numbers
Start with adoption, because it moved faster than most people noticed.
User Interviews surveys researchers every year. In its 2025 report (485 respondents, fielded in the summer), 80% said they use AI in their research workflow — a jump of 24 percentage points in a single year. In the same survey, 91% said they were concerned about the accuracy of what AI gives them. That pairing is the whole essay in two numbers: almost everyone uses it, and almost everyone distrusts it.
Maze’s surveys tell the same story from a wider sample that includes designers and product managers. In its 2025 report (800 respondents, fielded around the turn of the year), 58% used AI tools in research, and three in four of those used them to analyze data. By its 2026 report, 69% used AI in at least some projects.
91%
are concerned about the accuracy of AI output
User Interviews 2025
+24 pts
rise in AI use in one year
User Interviews 2025
Sources: User Interviews, State of User Research 2025; Maze, Future of User Research 2025 and 2026. Vendor surveys of self-selected respondents: read the direction, not the decimals.
The numbers
| Survey | Respondents | Use AI in research |
|---|---|---|
| Maze 2025 | 800 | 58% |
| Maze 2026 | about 500 | 69% |
| User Interviews 2025 | 485 | 80% |
Two other things moved at the same time. The first is who does research. In Maze’s 2025 survey, product managers ran research at 42% of the companies represented. In the User Interviews data, 21% of respondents said their company had laid off researchers in the past year, roughly the same as the year before, and in the 2024 edition 14% said their company now had no dedicated researchers at all, against 6–8% in each of the four years before. Fewer specialists, more research, more of it done by people whose main job is something else. If you are a product manager, there is a good chance the research on your desk is research you did.
The second is money. In January 2026, Listen Labs announced a $69M Series B for AI-moderated interviews. In December 2025, Outset raised $30M for the same idea. And at the end of September 2026, Salesforce agreed to buy Listen Labs; the price wasn’t disclosed, and reports put it around $2 billion. Whatever you think of the valuations, the direction is not in doubt. The people who buy software believe interviews are about to become something you run, not something you schedule.
So the stack has four new layers: AI interviewers, AI synthesis, synthetic respondents, and the online panels that feed all of them. Each one deserves its own look, because the evidence for each is very different.
AI interviewers are better than their reputation
The skeptic’s assumption is that a machine asking questions gets shallow answers. The evidence, such as it is, says the opposite — with an important asterisk.
The oldest good study predates large language models. Xiao and colleagues ran a field study with about 600 gamers, half answering a standard online survey and half talking to a chatbot that asked the same open questions conversationally. The chatbot’s answers were significantly more informative, relevant, specific and clear. And that was a chatbot built before today’s language models.
With current models the results hold up. Two economists, Chopra and Haaland, ran 381 thirty-minute AI interviews about why people don’t invest in the stock market. Experts rated over 95% of the AI’s questions as open-ended, neutral and relevant — better than many human interviewers manage — and most participants rated the experience well. A randomized experiment at NORC with 1,800 respondents found that AI follow-up questions produced more detailed and informative answers, at a slight cost to how much respondents enjoyed it.
Here is the asterisk. A large randomized experiment published in Public Opinion Quarterly in August 2026 confirmed that AI interviewing increased the number and specificity of the reasons people gave, with almost no loss in satisfaction. But it also found that after explaining themselves to the AI, respondents gave more polarized answers to the questions that followed. The interview changed the person being interviewed.
That shouldn’t surprise anyone who has run interviews. Asking someone to justify a view firms it up. A human interviewer does it too. But it means an AI interviewer is not a neutral pipe from the user’s head to your spreadsheet. It is an intervention, and its effects travel into whatever you ask next.
And notice what all of these studies compare against. Almost all of them put the AI against a survey form, not against a skilled human interviewer. The only head-to-head I could find, from Wuttke and colleagues, randomized university students to an AI or a human interviewer and found no significant drop in quality — a useful result, from a small pilot. Nobody has yet published a rigorous comparison of AI-moderated and human-moderated interviews in a product setting.
So the honest summary is narrow. AI interviewers beat forms. They can ask good, neutral questions. They let you talk to hundreds of people instead of eight. Whether they match a good researcher in the hour that matters — the one where someone says something you didn’t know to ask about — is not yet shown.
Synthesis is persuasive whether or not it is true
Synthesis is where AI saves the most time, and where it can do the most damage, because a summary with quotes attached reads as evidence even when the quotes are wrong.
The good news first. When the task is deductive — you have a codebook, and the model applies it — language models are competent. In one study, a human coder working with GPT-3.5 reached agreement of κ = 0.87 and 0.81 on two datasets; on the second, two humans coding alone had reached 0.77. If you already know what you are looking for, the model will find it faster than you.
Open-ended synthesis is different. De Paoli re-analyzed published interview studies with GPT-3.5: it found most of the main themes the original researchers reported, missed some, proposed others the researchers hadn’t considered relevant, and gave a different set of themes when asked for a different number of them. That last part matters. The model wasn’t discovering themes; it was producing as many as you asked for.
Then there is the illusion of agreement. A blinded comparison in PLOS Digital Health in 2026 found language models agreed with the reference coding 93.5% of the time and human analysts 92.7% — so the models were as good as the humans. But once you correct for chance, both scored κ = 0.34, which is weak. High percent agreement on a long transcript mostly measures how many lines carry no code at all. If a vendor tells you their synthesis “matches human researchers 93% of the time,” ask for kappa.
The most worrying result is about whose words get lost. Ashwin, Chhabra and Rao coded 2,407 interviews with Rohingya refugees and Bangladeshi hosts against a 19-code scheme built by sociologists. The best of four models reached an F1 of 0.41. Worse, in 10 of the 19 codes the errors lined up with characteristics of the person speaking. That is not noise that averages out. It is bias, and it would tilt any conclusion you drew about one group of users against another.
Summaries have the same tendency. A NAACL 2024 benchmark of nine models found that both model and human summaries of diverse opinions scored low on fairness to the different perspectives. A 2025 study of 18 models aggregating opinions found they consistently under-represented minority viewpoints and were swayed by the order of the input. In product terms: the angry 10% of your users — often the ones telling you something new — is exactly what a summary averages away.
And then there is plain invention. Vectara keeps a public leaderboard that measures how often models add unsupported content when summarizing short documents. In its September 22, 2026 update, the best model did so in 1.8% of summaries, and most frontier models fell between about 5% and 12%. The newest, most capable “reasoning” models were not the most faithful.
Source: Vectara hallucination leaderboard (HHEM-2.3), updated September 22, 2026; summaries of over 7,700 short articles. Interview transcripts are longer and messier, which tends to make this worse, not better.
The numbers
| Model | Hallucination rate |
|---|---|
| finix_s1_32b (lowest) | 1.8% |
| GPT-4.1 | 5.6% |
| Gemini 2.5 Pro | 7.0% |
| Claude Sonnet 4.6 | 10.6% |
| DeepSeek-R1 | 11.3% |
| GPT-5 (high) | 15.1% |
| o3-pro | 23.3% |
Interview transcripts are long, and length is its own problem. The “lost in the middle” study showed that models use information at the start and end of a long context much better than information in the middle. Your fourth interview of eight, or the twentieth minute of an hour-long call, is in the middle.
None of this means you shouldn’t use AI to synthesize. I do, and I built a small tool to do it. But building it taught me what the checks have to be. Every quote verified word for word against its transcript, because a paraphrased quote is the most persuasive kind of error. Every theme shown with how many participants it rests on, because “four of five” and “two of five” are different findings and a fluent summary prints them at the same weight. And contradictions kept as findings rather than rounded into a majority view, because both people and models drift toward consensus, and the disagreement is usually the point.
There is one check no tool can do: whether the quote means what the theme says it means. That is still your job, and it is the part of synthesis that was always the work.
Synthetic users get the average and lose the person
The most ambitious idea in the new stack is to skip people entirely: prompt a model to answer as a 34-year-old operations manager in Ohio, ask it your survey, and read the results. The research on this is now large enough to say something definite, and what it says is more interesting than either the hype or the dismissal.
On averages, synthetic samples can be remarkably good. In the paper that started the field, Argyle and colleagues conditioned GPT-3 on the backstories of real survey respondents and asked how they voted. The simulated votes correlated with the real ones at 0.90, 0.92 and 0.94 across three US elections. Brand, Israeli and Ngwe asked GPT what people would pay for product features; in an early pilot, one estimate of what shoppers would pay for fluoride in toothpaste came in at $3.40, against $3.27 in a study of real consumers.
On everything else, they fail in consistent ways.
They vary less than people. Bisbee and colleagues found ChatGPT’s averages matched a national survey closely, but with less variation than real respondents — and when they ran the same regressions on synthetic and real data, 48% of the coefficients differed significantly, and in 32% of those the sign flipped. The same prompt also gave significantly different answers three months apart. A tool whose answer changes when the vendor updates the model is not measuring your users.
They get the direction and miss the size. Cui, Li and Zhou had GPT-4 act as the participant in 154 published psychology experiments. It reproduced 76% of the main effects — an impressive number — but only 47% of the interactions, and only 19% of its confidence intervals contained the original effect size. Most of its effects were larger than the real ones. And in 72% of the cases where the original study found nothing, the simulation found something significant.
48%
of regression coefficients from synthetic survey data differed from the real ones
Bisbee et al. (2024)
32%
of those had the wrong sign
Bisbee et al. (2024)
Sources: Cui, Li & Zhou (2024), arXiv preprint, not peer reviewed; Bisbee et al. (2024), Political Analysis.
The numbers
| Measure | Value |
|---|---|
| Main effects replicated | 76.0% |
| Interactions replicated | 47.0% |
| Confidence intervals containing the original effect | 19.44% |
| Original null results that came out significant | 71.6% |
| Coefficients differing (Bisbee) | 48% |
| Of those, sign flipped (Bisbee) | 32% |
They are too right and too nice. In one of the first tests of simulated participants, the newer, more aligned models showed what the authors called a “hyper-accuracy distortion”: asked to estimate obscure quantities like a metal’s melting point, most simulated people simply gave the correct answer. Real crowds don’t. Nielsen Norman Group found the product version: synthetic users gave shallow, overly favorable feedback, and in one study of online courses claimed to have finished every course, when real learners often dropped out. A synthetic user will tell you it loves your idea. So will your mother.
They flatten the people at the edges. A 2025 study in Nature Machine Intelligence compared four models with 3,200 real people across 16 identities and found the models portrayed groups the way outsiders describe them rather than the way members describe themselves, and erased the variety within each group. If your product serves anyone who is not the median user, that is the part of the market a synthetic panel can’t see.
And they are weakest exactly where product teams want them. Brand and colleagues found that fine-tuning on past survey data improved the model’s answers for existing products in a known category, but they did not find similar improvements for new product categories or for differences between customer segments — and in their main study with GPT-3.5, the estimate of what people would pay for fluoride came out about three times too high. New categories and segment differences are most of what discovery is for.
There is one genuinely encouraging line of work. Park and colleagues built simulations of 1,052 real people from two-hour interviews with each of them, and found the simulations answered a national social survey 85% as accurately as the people themselves did when re-asked two weeks later. In a revised version of the paper, simulations built from demographics alone reached 74% of that benchmark, against 83–86% for those built from interviews. Nielsen Norman Group’s 2025 review reached the same conclusion: simulations built from real interviews beat persona prompts. Notice what that means. The simulations got good when they were made from a great deal of real research about real individuals. Synthetic users don’t replace research. At their best, they are a way of reusing it.
So use them where being wrong is cheap. To pilot a survey before you send it, and find the confusing question. To generate hypotheses worth testing. To rehearse an interview guide. Not to estimate demand, not to size a segment, not to decide between two concepts, and never for a market you haven’t studied with people.
Your panel may not be people
There is a quieter problem underneath all of this. Even when you do go to real people, through an online panel, you may not get them.
In 2023, researchers at EPFL asked crowd workers on Mechanical Turk to summarize medical abstracts and estimated that 33–46% of the summaries were written with help from a language model. That was a writing task, not a survey, but it showed how quickly the people paid to produce human text started outsourcing it.
By late 2025 it was a survey problem. Sean Westwood built an autonomous AI respondent and set it loose on standard survey quality checks. It passed attention checks in 99.8% of 6,000 trials, kept a consistent persona, and remembered its own earlier answers. In seven major national polls before the 2024 US election, adding between 10 and 52 such fake responses would have flipped the predicted result. In a follow-up audit of 2,898 open-ended answers on Prolific, at least 4.4% showed signs of AI assistance; the authors call that a lower bound.
99.8%
of attention checks passed, across 6,000 trials
Westwood (2025), PNAS
10–52
fake responses would have flipped each of seven 2024 national polls
Westwood (2025), PNAS
≥ 4.4%
of open-ended answers on one panel showed AI help
Westwood & Frederick (2026), n = 2,898
Sources: Westwood (2025), PNAS 122(47); Westwood & Frederick (2026), PNAS. Panel providers report lower rates from their own audits.
Panel providers dispute the size, and their own audits report lower rates. But the direction matters more than the decimal. The open-ended answer — the one field where a real person says something in their own words — is the easiest one for a model to fake and the hardest one to check. A synthesis of contaminated answers is a model summarizing a model, and it will read beautifully.
The practical response is provenance. Know where every respondent came from. Prefer your own customers, recruited from your own product, over anonymous panels. Watch for the tells: copy-pasted answers, inhuman typing speeds, answers that are fluent and generic in the same way. And weight what people did — in your product, in your logs — above what an anonymous panelist typed.
What didn’t change
Everything above is new. What follows is old, and AI changes none of it. If anything, cheap research makes these rules more important, because it makes them easier to ignore at scale.
Five users is an average, not a guarantee
The most quoted rule in usability is that five users find about 85% of the problems. It comes from Jakob Nielsen’s 2000 article, built on a model he published with Thomas Landauer, and it assumes each user finds, on average, 31% of the problems. Average is the important word.
Laura Faulkner tested it with 60 users on one application, then drew random sets of five from them. Some sets of five found 99% of the problems. Others found only 55%. With ten users, the worst set still found 80%; with twenty, 95%.
Source: Faulkner (2003), Behavior Research Methods, Instruments, & Computers 35(3); 60 users of a web-based timesheet application.
The numbers
| Users per set | Worst set found |
|---|---|
| 5 | 55% (best: 99%) |
| 10 | 80% |
| 20 | 95% |
This is the first place cheap research should change your habits. If an AI interviewer makes the twenty-first session nearly free, run more rounds and more people. But don’t mistake volume for coverage of the people who matter: twenty sessions with the same kind of user find the same problems twenty times.
Saturation depends on the question
How many interviews are enough? In the best-known experiment, with 60 interviews on a narrow topic, nearly all the themes appeared within the first twelve interviews, and the basic shape within six. A 2022 systematic review of 23 empirical tests found saturation within 9–17 interviews — for studies with relatively homogeneous participants and narrowly defined questions.
Those conditions are doing a lot of work. Ask a broad question of a varied market and you need more. And hearing a theme is not the same as understanding it. AI can make a hundred interviews cheap. It can’t make one well-defined question.
Saying is not doing
Research mostly collects what people say. Products succeed on what they do. The gap between the two is one of the best-measured findings in social science.
Paschal Sheeran’s review of 422 studies (N = 82,107) found intentions correlated with behavior at r = .53 — which means intentions explain about 28% of what people actually do. Experiments that deliberately changed people’s intentions found that a medium-to-large change in intention (d = 0.66) produced only a small-to-medium change in behavior (d = 0.36). And for purchases specifically, stated intentions predict sales worse for new products than for existing ones, and worse over long horizons than short ones.
New products, long horizons: that is product discovery. The questions we most want research to answer are the ones where asking is weakest. An AI that summarizes what thousands of users said inherits this gap at scale and adds its own.
Most ideas fail when measured
The last old rule is the most humbling. Ronny Kohavi and colleagues reported that at Microsoft only about one-third of the ideas tested in controlled experiments improved the metric they were designed to improve. They quote the figure for Google as about 10% of experiments leading to business changes, and in well-optimized areas like search, failure rates of 80–90%.
1 in 3
ideas tested at Microsoft improved the metric they were built to move
Kohavi et al. (2013)
~10%
of Google’s experiments led to a business change
Quoted in Kohavi et al. (2013)
80–90%
failure in well-optimized areas such as search
Kohavi et al. (2013)
Source: Kohavi, Deng, Frasca, Walker, Xu & Pohlmann (2013), “Online controlled experiments at large scale,” KDD.
These were ideas that survived research, design review and prioritization at some of the most data-rich companies in the world. Most still didn’t work. Research can tell you which ideas are worth testing. It has never been good at telling you which ones will work.
And here is the part of the new age that is genuinely good news. The same AI that made research cheap made building cheap. When a working prototype takes an afternoon, the cheapest way to learn whether people will use something is often to let them try to. The best research question now is frequently not “would you use this?” but “here it is — do you?”
Research after AI: a protocol
Put the evidence together and a way of working falls out of it. Seven rules.
Use it
Transcription. Coding to a codebook you wrote. Piloting a survey or interview guide on synthetic users. AI interviews for breadth, when compared with a survey form.
Agreement with human coders of κ 0.81–0.87 when a person stays in the loop.
Use it, then check it
Open-ended synthesis. Summaries of many voices. AI-moderated interviews you will act on.
Verify every quote, show coverage on every theme, keep the contradictions, and read some transcripts yourself.
Don’t use it as evidence
Synthetic users for demand, pricing, segment differences or new categories. Anonymous panel answers without provenance.
Only 19% of simulated confidence intervals held the real effect; an AI respondent passes 99.8% of attention checks.
Framework: the author’s, built from the studies cited in this essay. The examples summarize those studies; they are not new measurements.
1. Write down the decision first
Before any research, write the decision it informs and what result would change it. Cheap research tempts you to collect first and decide what it means later, and a model will always find themes in whatever you collected. If no result would change the decision, you don’t need research; you need to make the decision.
2. AI for breadth, people for depth
Use AI interviews to cast the wide net: two hundred conversations instead of eight, in every market and language you serve. Then read what came back for the surprises, and talk to five of those people yourself. The machine finds where to dig. You do the digging, because that is the part where nobody has shown the machine is as good.
3. Read two transcripts before any summary
Before you look at any synthesis, read two transcripts in full. It costs an hour and it inoculates you: you will know what these people sound like, and a summary that doesn’t sound like them will stand out.
4. Make the synthesis show its work
Every quote checked word for word against its source. Every theme labeled with how many participants it rests on. Every contradiction kept as a finding. Ask for chance-corrected agreement, not percent match, from anyone selling you automated coding. If a tool can’t do these things, it is producing impressions, not findings.
5. Synthetic users for rehearsal only
Pilot your questions on them. Generate hypotheses with them. Never let a synthetic panel estimate demand, set a price, compare segments or tell you about a market you haven’t studied with real people. If you want simulations worth trusting, build them from your own interviews with real customers — which means doing the research first.
6. Know where every respondent came from
Prefer your own users, recruited inside your own product. Treat anonymous open-ended answers as suspect until checked. Log provenance with the data, so that a finding can be traced to the people behind it.
7. Close the loop with behavior
Research decides what is worth testing. Behavior decides what is true. Now that building is cheap, put a prototype, a fake door or a small experiment behind every finding that matters, and expect most of them to fail. That isn’t research failing. It is research doing its job, which was always to make your bets better, not to make them safe.
What I’d want to be wrong about
The AI-interviewer evidence comes from the wrong field. Almost every study I cited comes from survey methodology and economics, and almost all of them compare the AI with a survey form, not a skilled researcher. Product interviews are different — looser, more exploratory, more dependent on following a surprise. My guess is that the gap to a good human is real in exactly that hour. But it is a guess, and the first good head-to-head study in a product setting could prove me wrong.
Models are moving faster than papers. Many of the synthesis and simulation studies tested GPT-3.5 or GPT-4. Newer models are better at many things. On the one benchmark here that is current — the hallucination leaderboard — newer did not mean more faithful, which is why I’m not assuming the problems will simply disappear. But some of these failure rates will fall.
Several key sources are preprints or vendor surveys. The replication of 154 experiments, the crowd-worker study and the 1,000-person simulation are not peer reviewed. The adoption numbers come from companies that sell research tools, surveying people who chose to answer. I’ve leaned on them for direction, and said so.
Rich simulations may get good. If simulations built from real interviews keep improving, the line between “synthetic user” and “well-organized research archive” will blur, and some of rule five will need rewriting. I’d welcome that. It would mean the research got reused instead of skipped.
The short version
Every step of product research now has a machine that does it faster. Recruiting, interviewing, synthesizing, even the users. Most of those machines are useful, and some are better than their reputation.
But none of them made it cheaper to know whether what they produced is true. AI interviewers change the people they talk to. AI synthesis invents some things and drops others, and its errors can follow who is speaking. Synthetic users get the average and lose the person. The panels feeding all of it are taking on bots. And underneath, the old facts stand: small samples vary, saying isn’t doing, and most ideas fail when measured.
So take the time AI saves you and spend it on the part it didn’t make cheaper. Check the quotes. Count the participants behind each theme. Keep the disagreements. Talk to some people yourself. And put the findings that matter in front of real behavior, now that building the thing is cheaper than asking about it.
Sources
Where the evidence comes from. Each row opens the document itself; the tag says which part of the argument it supports, and ↩︎ goes back to the sentence that uses it.
- The State of User Research Report 2025User InterviewsPracticeReport2025n = 485 researchers; 80% use AI (+24 pts); 91% concerned about accuracy; 21% saw researcher layoffsAI is now the default in research
- The Future of User Research Report 2025MazePracticeReportJan 2025n = 800; 58% use AI; 74% of them to analyze data; PMs do research at 42% of companiesWho does research now
- The Future of User Research Report 2026MazePracticeReportMar 2026Nearly 500 respondents; 69% use AI; humans essential for nuance (82%) and framing questions (76%)Adoption keeps climbing
- The State of User Research 2024 ReportUser InterviewsPracticeReport2024n = 759; 22% saw researcher layoffs; 14% of companies with no dedicated researchersFewer researchers, more research
- Announcing Listen’s Series BListen Labs (company announcement)MarketNewsJan 2026$69M led by Ribbit CapitalMoney behind AI interviews
- Outset Series BOutset (company announcement)MarketNewsDec 2025$30M led by Radical Ventures; $51M raised in totalMoney behind AI interviews
- Salesforce to acquire AI customer research startup Listen Labs in reported $2B dealSiliconANGLEMarketNewsOct 2026Definitive agreement; price not disclosed (about $2B reported)The category gets bought
- Tell me about yourself: using an AI-powered chatbot to conduct conversational surveys with open-ended questionsXiao, Zhou, Liao, Mark, Chi, Chen & Yang · ACM TOCHI 27(3)AI interviewsJournal2020About 600 gamers; chatbot answers more informative, relevant, specific and clearA chatbot draws out more
- Conducting qualitative interviews with AIChopra & Haaland · CESifo Working Paper 10666AI interviewsWorking paper2023n = 381 AI interviews; over 95% of questions rated open, neutral and relevantAn AI interviewer that experts approve
- AI-assisted conversational interviewing: effects on data quality and respondent experienceBarari et al. (NORC at the University of Chicago) · arXivAI interviewsPreprintApr 2025n = 1,800, randomized; more detailed answers, slight cost to experienceRicher answers, at a small cost
- An experimental comparison of AI-enabled semi-structured interviews and fixed surveysAustin, Hohe, Kennedy, Litman, Minozzi & Moses · Public Opinion QuarterlyAI interviewsJournalAug 2026Randomized; more and more specific reasons; later answers more polarizedThe interview changes the answers
- AI conversational interviewing: transforming surveys with LLMs as adaptive interviewersWuttke, Aßenmacher, Klamm, Lang, Würschinger & Kreuter · LaTeCH-CLfL 2025AI interviewsPaper2025Students randomized to AI or human interviewer; no significant drop in qualityThe only head-to-head, and small
- LLM-in-the-loop: leveraging large language model for thematic analysisDai, Xiong & Ku · Findings of EMNLP 2023SynthesisPaper2023Human plus GPT-3.5 coder: κ = 0.87 and 0.81Coding to a codebook works
- Performing an inductive thematic analysis of semi-structured interviews with a large language modelStefano De Paoli · Social Science Computer Review 42(4)SynthesisJournal2024Most main themes found; some missed, some added; output shifts with the promptOpen-ended synthesis drifts
- Large language models for thematic analysis in healthcare research: a blinded mixed-methods comparison with human analystsPLOS Digital HealthSynthesisJournalApr 2026LLMs 93.5% agreement, κ = 0.34; humans 92.7%, κ = 0.34Agreement that hides weak reliability
- Using large language models for qualitative analysis can introduce serious biasAshwin, Chhabra & Rao · Sociological Methods & ResearchSynthesisJournal20252,407 interviews; best model F1 0.41; errors tied to who answered in 10 of 19 codesErrors that follow the person
- Fair abstractive summarization of diverse perspectivesZhang et al. · NAACL 2024SynthesisPaper2024Six datasets, nine LLMs; model and human summaries both low on fairnessSome voices get dropped
- Can AI truly represent your voice in deliberations? A comprehensive study of large-scale opinion aggregation with LLMsZhu, Yang, Bakker, Pentland & Pei · arXivSynthesisPreprintOct 202518 LLM summarizers; minority views under-represented; order effectsMinority views fade
- Hallucination leaderboardVectara (HHEM-2.3)SynthesisDataSep 22, 2026Summaries of 7,700+ articles; 1.8% to 23.3% hallucinatedEven good summarizers invent
- Lost in the middle: how language models use long contextsLiu et al. · TACL 12SynthesisJournal2024Accuracy highest at the start and end of the context, worst in the middleThe middle of a transcript
- Synthesizer: interview themes with every quote checkedBets & Ships case studySynthesisArticle2026Five transcripts; coverage on every theme; every quote verifiedWhat a checked synthesis looks like
- Out of one, many: using language models to simulate human samplesArgyle, Busby, Fulda, Gubler, Rytting & Wingate · Political Analysis 31(3)Synthetic usersJournal2023Vote choice vs ANES: tetrachoric r 0.90, 0.92, 0.94Right on the average
- Using LLMs for market researchBrand, Israeli & Ngwe · Harvard Business School Working Paper 23-062Synthetic usersWorking paper2023–2025Pilot estimates close; main GPT-3.5 study about 3× too high for fluoride; no gain for new categoriesFine for the known, not the new
- Can AI predict whether shoppers would pick Crest or Colgate?Harvard Business School Working KnowledgeSynthetic usersArticle2023Fluoride toothpaste: $3.27 human study vs $3.40 GPTA number that matched
- Synthetic replacements for human survey data? The perils of large language modelsBisbee, Clinton, Dorff, Kenkel & Larson · Political Analysis 32(4)Synthetic usersJournal202448% of coefficients differ; 32% of those flip sign; less variance; drift over 3 monthsWrong on the spread
- Can AI replace human subjects? A large-scale replication of psychological experiments with LLMsCui, Li & Zhou · arXivSynthetic usersPreprintAug 2024154 experiments; 76.0% main effects, 47.0% interactions; 19.44% of CIs hold the original; 71.6% of nulls significantDirection yes, size no
- Using large language models to simulate multiple humans and replicate human subject studiesAher, Arriaga & Kalai · ICML 2023Synthetic usersPaper2023Three classic studies replicated; “hyper-accuracy distortion” in wisdom of crowdsToo right to be human
- Synthetic users: if, when, and how to use AI-generated “research”Rosala & Moran · Nielsen Norman GroupSynthetic usersArticleJun 2024Shallow, overly favorable feedback; claimed to finish every courseToo agreeable to be useful
- Large language models that replace human participants can harmfully misportray and flatten identity groupsWang, Morgenstern & Dickerson · Nature Machine Intelligence 7Synthetic usersJournalFeb 2025Four LLMs vs 3,200 people, 16 identities; out-group portrayals, flattened groupsGroups flattened
- Generative agent simulations of 1,000 peoplePark et al. · arXivSynthetic usersPreprintNov 20241,052 people, 2-hour interviews; 85% of own test–retest on the GSS; 74% from demographics alone (revised)Rich data closes the gap
- Evaluating AI-simulated behavior: insights from three studies on digital twins and synthetic usersRaluca Budiu · Nielsen Norman GroupSynthetic usersArticleAug 2025Twins built from interviews beat persona-only synthetic usersWhere simulation helps
- Artificial artificial artificial intelligence: crowd workers widely use large language models for text production tasksVeselovsky, Ribeiro & West · arXivPanelsPreprintJun 2023MTurk summaries: 33–46% produced with LLM helpCrowd work went synthetic
- The potential existential threat of large language models to online survey researchSean Westwood · PNAS 122(47)PanelsJournalNov 2025Passed attention checks in 99.8% of 6,000 trials; 10–52 fakes would flip 7 pollsA respondent that isn’t
- Reply to Van der Stigchel et al.: empirical evidence that AI survey contamination is real and substantialWestwood & Frederick · PNASPanelsJournal2026Prolific, n = 2,898 open-ended answers; 4.4% flagged, a lower boundIt is already in the data
- Why you only need to test with 5 usersJakob Nielsen · Nielsen Norman GroupSample sizeArticleMar 2000L = 31% per user; five users find about 85% of problemsThe five-user rule
- A mathematical model of the finding of usability problemsNielsen & Landauer · INTERCHI ’93Sample sizePaper1993Problems found = N(1 − (1 − L)n), 11 studiesThe model behind it
- Beyond the five-user assumption: benefits of increased sample sizes in usability testingLaura Faulkner · Behavior Research Methods, Instruments, & Computers 35(3)Sample sizeJournal200360 users; sets of 5 found 55% to 99%; worst set of 10 found 80%, of 20 found 95%An average, not a guarantee
- How many interviews are enough? An experiment with data saturation and variabilityGuest, Bunce & Johnson · Field Methods 18(1)SaturationJournal200660 interviews; saturation within the first 12, basic metathemes by 6When new interviews stop adding
- Sample sizes for saturation in qualitative research: a systematic review of empirical testsHennink & Kaiser · Social Science & Medicine 292SaturationJournal202223 studies; saturation at 9–17 interviews for narrow, homogeneous studiesSaturation, reviewed
- Intention–behavior relations: a conceptual and empirical reviewPaschal Sheeran · European Review of Social Psychology 12Say–do gapJournal2002422 studies, N = 82,107; r = .53, about 28% of varianceSaying is not doing
- Does changing behavioral intentions engender behavior change? A meta-analysis of the experimental evidenceWebb & Sheeran · Psychological Bulletin 132(2)Say–do gapJournal2006Intention change d = 0.66 → behavior change d = 0.36Even moved intentions move half
- When do purchase intentions predict sales?Morwitz, Steckel & Gupta · International Journal of Forecasting 23(3)Say–do gapJournal2007Intentions predict worse for new products and long horizonsWeakest where discovery asks
- Online controlled experiments at large scaleKohavi, Deng, Frasca, Walker, Xu & Pohlmann · KDD ’13ExperimentsPaper2013One-third of ideas at Microsoft improved their metric; Google about 10%; 80–90% fail in searchMost ideas fail when measured