One-tailed vs two-tailed: the setting that fakes wins
Day 6, my test hit 'significant.' Then I noticed the tool was set to one-tailed.
Quick translation: one-tailed only asks 'is B better than A?' Two-tailed asks 'is B different, better OR worse?' One-tailed needs less data to call a win, so vendors love defaulting to it. Feels great. It's also more likely to crown a false winner.
Flipped to two-tailed. My 'significant' result evaporated. Saved me from shipping junk.
The tradeoff: two-tailed is stricter and slower, but it's honest. One-tailed is faster and flatters you.
Use two-tailed by default. Only go one-tailed if you genuinely don't care about a 'worse' outcome (rare).
Go check this exact setting in your tool right now. Report back what you find.

ЕЮ Иванов продолжает кошмарить АффПапу, конторку, которая накинула говна на вентилятор этим летом. Тогда в AffPapa не знали, с каким говном идут бодаться, поэтому заслуженно проиграли. 😏

На этот раз ЕЮ зарегал товарный знак AffPapa — совсем скоро имя компании будет официально принадлежать ему. Чтобы убедиться в трушности мува, переходим по ссыл-Очке и вводим серийный номер: 2026793242. Там видим, что заявка на регистрацию подана лично Евгением Юрьичем.

Всё это выглядит забавно, но давайте не забывать, в какой сфере мы работаем и что реально может произойти с жирным троллем за воровство нейминга. Впрочем, толстому не привыкать отхватывать пиздов за проделки в интернете, поэтому ждем очередную фотку разбитого ебала и длинный пост с извинениями. 😏😏😏

В арбитраже денег нет 💵

Edge testing vs origin testing: the speed play
My A/B snippet was adding 200ms. On a landing page, 200ms is a tax every variant pays.
Option A, origin/server-side test: clean, but the page render waits on your app deciding the variant.
Option B, edge testing (Cloudflare Workers, Vercel Edge): the variant decision happens at the CDN edge, milliseconds from the user. No origin round-trip, no flicker, near-zero added latency.
I moved bucketing logic to a Worker. Page stayed fast, variant assignment was instant, and the test was finally a fair fight.
Tradeoff: edge needs you comfortable writing a little Worker code. Not a marketer's solo job.
If your tests keep losing to latency, the fix isn't a better variant. It's moving the test to the edge. Go time your assignment logic. Report back.

On-site surveys vs just A/B testing it
Had 5 headline ideas, didn't know which 2 to test. Testing all 5 = months of traffic.
Instead I ran a one-question on-site survey (Hotjar/Typeform popup): 'What almost stopped you from buying?' 200 replies in 3 days. The answers wrote my best 2 headlines for me.
The tradeoff: surveys tell you what people SAY, not what they DO. Stated preference lies. So surveys don't replace the test, they narrow it.
My flow now: survey to generate + prioritize hypotheses, A/B test to validate. Qualitative narrows, quantitative decides.
Stop brainstorming variants in a meeting. Go ask your actual users one question this week, then test the top answer. Report back.

Convert vs VWO: the privacy/cookie angle nobody tests for
EU traffic. Cookie banner. Half my visitors decline tracking. My test sample got quietly mangled.
Dug into vendors:
— VWO: powerful, but heavier cookie footprint, more to disclose in your consent flow.
— Convert: built leaner on privacy, has a cookie-less / consent-aware mode that still buckets users for the test without the full tracking cookie load.
The tradeoff: a privacy-light tool keeps more of your EU sample in the test (less consent-decline leakage) but you trade some deep-analytics richness.
For GDPR-heavy traffic, the 'best' tool is the one that doesn't lose 40% of your sample to a declined banner.
Go check what % of your traffic declines tracking. If it's high, your tool choice just changed. Report back.

Holdout group vs pre/post comparison
Launched a sitewide change I couldn't cleanly A/B (a new nav). Two ways to measure.
Pre/post: compare conversion before launch vs after. Easy, but garbage when seasonality, a sale, or a traffic-source shift muddies it. I've fooled myself this way twice.
Holdout: keep 5-10% of users on the OLD experience permanently for a few weeks. Compare against the 90% who got the new thing. Same time window, same season, same traffic. Clean read.
The tradeoff: a holdout 'wastes' a little traffic on the old version. That's the price of a real causal answer.
For anything you can't slot into a normal test, a holdout beats pre/post every single time.
Got a big launch coming? Don't go all-in. Hold back 5%. Report back.

Multi-armed bandit vs classic 50/50 split
Flash sale, 6 days, testing 3 banner variants. A classic A/B/C split wastes traffic on losers the whole time.
Multi-armed bandit: the algorithm shifts traffic toward the winning variant AS it learns. More conversions DURING the test. Perfect for short, high-stakes windows.
But here's where I got burned: I used a bandit for a long-term landing page test and it adapted so fast it never gave me a clean, reportable lift number. Bandits optimize earnings, not learning.
My rule:
— Short campaign, want max conversions now → bandit.
— You need a defensible 'B beat A by X%' for the team/roadmap → classic 50/50 split.
Don't reach for the fancy bandit by default. Pick by your goal: earn or learn. Report back.