A randomized test of Google AI Mode found 18.8 points fewer clicks. What CPG brands should take from it

Almost every number in the AI-search-and-clicks debate comes from comparing searches that had an AI summary with searches that didn't. That's useful, but it leaves a nagging question: maybe the searches that get summaries are just the kind people were never going to click on anyway. The fix for that is an experiment, and one has now been run on real Google users.

The paper, posted to arXiv on August 18 by researchers at the University of Pennsylvania and Northeastern, is titled "AI in Search Reduces Publisher Referrals Without Improving User Experience." It hasn't been through peer review yet. We think it's worth a CPG brand's time anyway, partly because of a small test Google ran last week.

How the experiment worked

The team recruited US adults who use Chrome and Google, mostly through the research platform Prolific, in March 2026. Each installed a browser extension. For three days everyone searched normally so the researchers could measure a baseline. Then each person was randomly assigned to one of three conditions for seven days:

• No AI Search: the extension hid AI Overviews and AI Mode.
• Current Search: nothing changed.
• AI Mode Search: searches went to AI Mode, Google's chat-style answer.

In total 1,100 people searched at least once during the treatment week, and the analysis was preregistered, meaning the team wrote down what it would measure and how before it saw the results. During the baseline, 36% of participants' searches triggered an AI Overview, and people chose AI Mode on their own for just 0.6% of searches.

AI Mode cut clicks sharply

The headline: people pushed into AI Mode clicked through to an outside site on 18.8 percentage points fewer searches than people using normal Google, with a 95% confidence interval of 15.3 to 22.2 points. They also searched less, by 0.92 sessions a day, while each session ran 0.43 minutes longer. They were reading the answer instead of the web.

The drop wasn't spread evenly. The researchers looked at what share of people clicked through to particular kinds of sites at all.

Show the data

Clicked through to	Change (percentage points)
Any outside site (click-through rate)	−18.8 pts
Reddit	−21.2 pts
News sites	−12.5 pts
Wikipedia	−9.9 pts
Ads	−42.7 pts

Source: Wang, Gleason, Bart, Wilson and Metaxa, AI in Search Reduces Publisher Referrals Without Improving User Experience, arXiv, August 2026. The first row is the change in click-through rate on searches; the others are changes in the share of participants who clicked that type of site. The ads figure mostly reflects that AI Mode showed no ads during the study.

Reddit is the line we'd circle. It took the biggest hit among the sources tracked, which fits with what we see in AI answers: they summarize the forum consensus so you don't have to go read the thread. For a CPG brand that has worked hard on Reddit, as our community tip suggests, it's a reminder that the payoff is increasingly that the discussion shapes the answer, not that shoppers click into the thread.

The ads line looks dramatic but has a simple explanation the authors give themselves: AI Mode didn't show ads at the time. Google has since added Shopping ads with promotions to AI Mode, so that number is already out of date.

Hiding AI helped clicks, with an asterisk

The other arm is harder to read. Partway through the study Google changed the HTML of its AI Overviews, which broke the extension's ability to find and hide them. Only 51.1% of AI Overviews were actually hidden in the No AI group. The simple comparison (everyone assigned to No AI versus everyone on normal Google) didn't show a statistically clear change in click-through.

The researchers also estimated the effect among people for whom the hiding worked, a standard method in experiments where not everyone receives the treatment. On that basis, removing AI raised click-through by 8.8 percentage points, with a confidence interval of 2.3 to 15.3. It points the same way as earlier observational work, such as Pew's browsing data showing people clicked a result on 8% of visits with an AI summary versus 15% without, and Ahrefs' estimate of a 58% lower click-through rate for the top result when an Overview is present.

People didn't love it

The title says AI search didn't improve the experience, and the survey data backs that up.

Measure	Change
Trust in information found on Google	-0.34 points on a 7-point scale
Used Bing, DuckDuckGo or Yahoo during the week	+11.2 percentage points
Intention to switch to Bing	+1.25 points on a 7-point scale
Rated usefulness	-0.59 standard deviations
Satisfaction	-0.73 standard deviations
Sense of control over the search	-0.66 standard deviations

Source: Wang et al., arXiv 2608.18352, survey responses from the post-experiment questionnaire.

In open-ended answers, 33.6% of responses were negative about the experience and 29.3% positive, with the rest mixed or neutral. The complaints were specific: losing control, finding it hard to reach websites, and seeing a narrow set of sources. The people who liked it mostly liked the speed.

A fair reading is that being forced into something new for a week isn't the same as choosing it. Some of the dislike may be about change itself. The sample also skewed younger, more educated and more left-leaning than the US population, which the authors note.

Why the AI Mode tab test matters here

The experiment forced people into AI Mode. Google may be heading toward nudging them there. On October 23, Search Engine Watch reported a test of a new header in which only two tabs show up front: AI Mode and All. Shopping and the other tabs move into a dropdown under All.

A test isn't a launch, and Google tests many things that never ship. But it's consistent with a direction we've been tracking: AI Overviews on 88% of commercial searches, on 83% of branded searches, and now the Shopping tab possibly stepping back. If more searching happens in AI Mode, this experiment is our best estimate of what happens to clicks: far fewer of them, with the answer doing the work the click used to do.

If AI Mode becomes the front door, the answer is the shelf. A product that isn't named in the answer has very few other ways to be seen.

What a CPG brand should take from it

We'd draw three practical conclusions, none of them panicky.

Traffic is the wrong scoreboard for this channel. If clicks fall by a fifth when people use AI Mode, a brand measuring AI search by referral visits will always conclude it doesn't matter. The question is whether your product is named, correctly, in the answer. That's the point we made in our case against tracking AI rank, and this experiment strengthens it.

The facts have to be in the answer, not behind a click. In AI Mode, the shopper reads Google's summary of what your product is, what it costs, and how it compares. Our tips on content that gets cited and on checking what assistants say are where we'd start. A regular check of the shopper questions that matter for your category, across Google and the chat assistants, tells you more than a traffic chart. Some brands use outside tools for this, such as SearchPanel from XENA Intelligence, which asks the assistants shopper-style questions and shows the answers; others keep a monthly spreadsheet.

Don't opt out on the strength of this. It's tempting to read "AI cuts clicks" as a reason to block Google's AI features. We've looked at that in tip 34. Blocking snippets also limits how your pages appear in normal results, and a product absent from the answer isn't going to win the click either.

What we'd still like to know

The study measured all kinds of searches, not shopping ones. We'd expect product searches to behave differently: people who want to buy something eventually have to click somewhere, whether it's a retailer, a Shopping ad or Universal Cart. The experiment also ran before AI Mode had ads and before Universal Cart launched. A version of this study on commercial queries in the holiday season would be the one to read. Until then, it's the best causal evidence we have, and it says the same thing the correlations did, only more firmly.