SEO teams routinely declare tests a success when the evidence supports no such verdict, according to a new Search Engine Journal analysis that identifies six methodological mistakes undermining common experiments. The author, a researcher who moved into SEO, trained teams to apply scientific controls to title changes, template tests and internal-linking experiments — and reports that most of the errors stem from skipping four basic research steps.

The piece arrives as testing has expanded across the discipline. Practitioners began with titles, moved to internal linking, then content formats and templates, and now increasingly test AI-generated changes. The author's core argument: running a test and learning something from it are not the same thing, and many teams cannot tell the difference.

What makes a hypothesis falsifiable?

The first mistake is testing without a solid, falsifiable hypothesis. Good research starts with a question driven by a real commercial uncertainty, not with a list of things that can be tested. Examples from the article include: why did traffic drop on product pages since last quarter, or did AI mentions fall in the past month?

"Improving page titles will improve SEO performance" fails as a hypothesis, the author writes, because nothing in it defines "improving" or "performance," and no result could prove it wrong. A useful version makes a specific prediction the experiment can contradict. Her example: "Updating the titles to include a country of reference on 700 pages will generate an additional 10% clicks over a four-week period, as compared to the four weeks preceding the change." Under that design, a 12% uplift proves the hypothesis; 9% falsifies it.

Why do concurrent changes break SEO tests?

The second mistake is failing to isolate variables or establish a valid comparison. Websites are not sterile labs. While a team tests titles or a new template placement, a product team may deploy a sitewide feature, marketing may launch a campaign, search demand may shift, or Google may roll out an update. Researchers call these alternative explanations confounding variables.

The author cites her own case. In a recent study on whether people respond differently to AI-generated versus human-created content, she realized the materials differed in messaging and participant familiarity. She scrapped the design and rebuilt the stimuli, which took significantly longer — but, she writes, was less painful than confidently reaching the wrong conclusion. Her practical advice: keep a log of major changes across SEO, engineering, product and marketing during every test window.

What happens when success criteria move after the fact?

The third mistake is deciding what success looks like after seeing the results. The author points to confirmation bias and cites the old industry joke: if traffic goes up, it's your work; if it goes down, it was a Google update. Research counters this with preregistration — documenting what will be tested, and how, before results arrive. She suggests borrowing the principle rather than literally preregistering a title test on the Open Science Framework.

Without pre-committed metrics, teams rationalize post hoc: "traffic didn't move, but impressions did," or "average position is down, but look at these six new keywords we gained." Give yourself enough metrics and enough freedom after the fact, she warns, and something will eventually look positive.

Can a test honestly return 'we don't know'?

The fourth mistake is drawing conclusions the test cannot support. Three outcomes exist — evidence for an effect, evidence against it, or insufficient evidence — and they are not the same. Short windows compound the problem, because SEO experimental time differs from calendar time: changes need crawling, indexing and search-engine response, and user behavior varies by day, week and season. Seven days sound tidy, she notes, but do not guarantee seven meaningful days of exposure.

Her key principle: "Absence of evidence isn't necessarily evidence of absence." A non-finding may mean the experiment lacked the power to detect a real effect. More data will not rescue a badly designed experiment, she adds, but it strengthens a well-crafted one.

Is one test ever enough to act on?

The fifth mistake is treating every test as an isolated event. A single study shows what happened with one sample, one methodology, under one condition — it does not mean "Google definitely prefers this," and a winning change should not become SEO law across every market and page type. The author recommends experiment logs documenting, at minimum: question, hypothesis, treatment, sample, dates, context, result, interpretation, limitations and next action — for losers and inconclusive tests included. One test gives a result; 20 documented tests start telling you something about your site, audience and markets.

When should you skip the test entirely?

The sixth mistake is testing for testing's sake. Before running an experiment, complete the sentence: "Depending on what we find, we will…" — scale, revert, change direction, or do nothing. The last option matters: a statistically convincing result is not necessarily commercially meaningful. A change that improves organic clicks by 0.3% but consumes six months of engineering resources may have "worked" while still justifying rejection. The author has worked where experiment volume was treated as a maturity marker; value, she argues, lies in learning and resulting actions, not counts.

Her closing framing: the output of an experiment is not the chart, but a decision. Teams that pre-register criteria, log every outcome — winner, loser or inconclusive — and accept unexpected answers will, within six months, hold a body of evidence worth more than any single result. Watch whether experimentation platforms and in-house teams adopt research-style preregistration and shared repositories as AI-driven content changes make attribution harder still.