Can You Trust This AI Summary? — The UX Research Library

The UX Research Library

Can You Trust This AI Summary?

Five real-looking summaries. Something's wrong with each one — can you catch it before reading the answer?

Case 1 of 5

Original data

7 of 10 participants said they were "somewhat satisfied." 3 said "very satisfied." Nobody said they were dissatisfied.

AI summary

"Participants were satisfied with the product overall."

What did this summary get wrong?

It dropped the qualifier "somewhat" — most people were satisfied with reservations, not fully satisfied Nothing — this is a fair summary of the data It invented a number that wasn't in the original data

Correct. "Somewhat satisfied" and "satisfied" are not the same finding. This is the most common failure — a real, true-sounding sentence that has quietly softened or dropped a qualifier along the way.

Not quite. Read it again next to the original — 7 of 10 people qualified their satisfaction. "Satisfied" alone quietly erases that hesitation.

Not quite. No numbers were invented here — the problem is subtler than a fabricated statistic. Look at the words themselves.

Case 2 of 5

Original data

Participant A: "I gave up after the third error message." Participant B: "Honestly, the error messages were fine for me."

AI summary

"As Participant A noted, the error messages were generally fine, though occasionally frustrating."

What did this summary get wrong?

This is an accurate paraphrase of what was said It added a participant who wasn't in the original data The sentiment got attributed to the wrong participant

Not quite. Check the names — Participant A is the one who gave up in frustration, not the one who found things fine.

Not quite. No new participant was introduced — the real issue is that an existing one got misquoted and misattributed.

Correct. Participant B said the messages were fine; Participant A is the one who gave up. The summary swapped them and blended both into one falsely moderate opinion — attributed to the wrong person, with a tone neither of them actually expressed.

Case 3 of 5

Original data

Participant: "I stopped using it after the last update." (No reason given, and none was asked.)

AI summary

"One participant stopped using the product after an update, likely due to the confusing new interface."

What did this summary get wrong?

Nothing — "likely due to" is clearly framed as a guess, not a fact It invented a plausible-sounding reason the participant never actually gave It understated how strongly the participant felt

Not quite. "Likely due to" reads as hedged, but it's still presenting a reason as analysis when it's actually a fabrication — the participant never mentioned the interface at all.

Correct. The participant gave no reason. "Likely due to the confusing new interface" sounds like a reasonable inference, but it's invented — and once it's in a report, it reads as a finding, not a guess.

Not quite. The intensity isn't the issue here — the problem is a reason was added that was never actually stated.

Case 4 of 5

Original data

2 of 10 participants mentioned confusion with the checkout button's placement. The other 8 didn't bring it up at all.

AI summary

"Users found the checkout button placement confusing."

What did this summary get wrong?

It turned a minority view (2 of 10) into a statement that reads like a majority finding Nothing — if even a few users struggled, it's worth flagging as "users" It understated how confusing the button actually was

Correct. "Users found it confusing" implies most people, when actually 8 of 10 never mentioned it. The finding might still matter, but the scope has been quietly inflated.

Not quite. Worth flagging, sure — but "users found X confusing" reads as a general finding, not a minority signal from 2 of 10 people.

Not quite. If anything the summary overstates it, not understates it — look at how many people this was actually true for.

Case 5 of 5

Original data

8 participants preferred the new design. 2 explicitly said they preferred the old one and felt strongly about it.

AI summary

"Participants preferred the new design."

What did this summary get wrong?

Nothing — 8 out of 10 is a clear majority It invented a percentage that wasn't in the original data It dropped the 2 participants who disagreed, erasing a real and possibly important dissenting signal

Not quite. Majority, yes — but "participants preferred" with no caveat erases the 2 who felt strongly the other way, which might matter a lot depending on who they were.

Not quite. No percentage was fabricated here — the issue is an omission, not an invented number.

Correct. A clean majority statement quietly deleted a real, strongly-held dissenting opinion. Whether that dissent matters depends on context you can only get by keeping it visible, not folding it into an average.

The pattern across all five

None of these summaries look wrong on a skim. They read as clean, confident prose — which is exactly what makes them dangerous. The only real check is comparing specific claims back to the raw quotes, not re-reading the summary and deciding it sounds right.

One habit that makes this faster: ask your AI to map every finding to the specific quote(s) it came from, right in the output. Checking stops being "does this sound right" and becomes "does this quote actually say that" — seconds per claim instead of re-reading everything from scratch.

Get the full toolkit — Beyond Screens

Part of The UX Research Library — leiamanin.com

Can You Trust This AI Summary?

Paramètres avancés