Case studies: logs during a site migration

What the access log caught that nobody else did.

→ Builtvisible replatform audit — watched Googlebot's 301 hits post-launch; discovered the bot still requesting 88k old URLs after 3 weeks, all chaining through 4 redirects. Flattening the chains recovered crawl and rankings stabilized.
Takeaway: migrations succeed or fail in the redirect logs.

⭐ Pick of the week — Distilled launch-day monitor — streamed live logs on cutover and caught a misconfigured rule serving 302s instead of 301s within 40 minutes, before Google recrawled at scale.
Takeaway: watch logs live on launch day, not the morning after.

→ Screaming Frog redirect-chain guide — pairs with the log view to map the chains.
Takeaway: chains are invisible until you trace each hop.

Claude Cowork, Claude Design объединили в один Claude

➡️ Читайте на сайте: https://aff.top/blog/claude-cowork-claude-design-obedinili-v-odin-claude

🧠 Ещё больше инсайтов → в канале AFF.top

Microsoft планирует вставлять рекламу в игры

➡️ Читайте на сайте: https://aff.top/blog/microsoft-planiruet-vstavliat-reklamu-v-igry

🧠 Ещё больше инсайтов → в канале AFF.top

Case studies: image and video crawl, logged

Googlebot-Image is a separate line in your log.

→ Retailer image-crawl audit (Lumar) — found Googlebot-Image accounted for 41% of all bot requests, hammering 1.5MB uncompressed product shots. Compressing them cut total bot bandwidth 35% and freed HTML crawl.
Takeaway: image crawl can quietly dominate your budget.

⭐ Pick of the week — Merj media-crawl study — separated Googlebot, Googlebot-Image and Googlebot-Video in the log and showed video discovery lagged 3 weeks behind page discovery, explaining missing video-rich results.
Takeaway: each media bot has its own schedule, visible per user-agent.

→ Google image-SEO best practices — what the image bot rewards.
Takeaway: pairs with the audit above.

Playbook: verify a Googlebot hit is really Googlebot
Four sources that turn "is this real?" into a repeatable check.
→ Google Search Central docs — the canonical reverse-then-forward DNS method. Takeaway: host 66.249.66.1 must end in .googlebot.com or .google.com, then forward-resolve back to the same IP.
→ Google's published IP ranges — the googlebot.json and special-crawlers.json files. Takeaway: match against the ranges instead of DNS when you need speed at scale.
★ Pick of the week — Stack Overflow thread on awk-based rDNS — a one-pass script piping unique bot IPs through host. Takeaway: dedupe IPs first, you'll cut lookups by 90%.
→ Cloudflare bot docs — note the GoogleOther and Google-InspectionTool agents share infra but serve different products. Takeaway: don't block them assuming they're Googlebot.
Never trust the user-agent string alone — it's trivially spoofed.

Mistake: logrotate eats the data you were about to analyze

Default rotation discards old logs before you've sampled a full crawl cycle. Sources:

→ logrotate man page — rotate 4 + weekly = only ~4 weeks retained, and create vs copytruncate changes whether in-flight writes are lost.
Takeaway: bump retention, or ship to cold storage before rotation.

→ Nginx / Apache reopen docs — Why copytruncate can drop lines under load if the app doesn't reopen the handle.
Takeaway: prefer signal-based reopen (USR1) over copytruncate for accuracy.

★ Pick of the week — a tiny ship-on-rotate hook — postrotate script that gzips and pushes to object storage.
Takeaway: never analyze from the live file you're still rotating.

Checklist: find crawl-budget waste in 20 minutes
A tight sequence pulled from the people who do this for a living.
→ OnCrawl's log methodology — bucket Googlebot hits by URL pattern, then sort by hit count. Takeaway: the top 20 patterns usually hold 80% of the waste.
→ Screaming Frog log analyzer guide — cross-join logs with your crawl to flag URLs crawled but not in the sitemap. Takeaway: those orphans are pure budget drain.
★ Pick of the week — Gus Pelogia's parameter-URL teardown — counts Googlebot hits on ?sort=, ?filter=, faceted junk. Takeaway: one regex over a week of logs exposes the worst offenders instantly.
→ Google's "managing crawl budget" doc — confirms duplicate and soft-404 URLs eat budget. Takeaway: pair the doc with your own 404/302 counts from logs.
Work the patterns, not individual URLs — that's where the leverage is.

Playbook: mine status codes from raw access logs
The grep/awk recipes worth keeping in a snippets file, credited to their authors.
→ Julia Evans' (b0rk) log one-liners — awk '{print $9}' access.log | sort | uniq -c | sort -rn for a full status histogram. Takeaway: field 9 is the status in Combined format.
→ Server Fault canonical answer on isolating 5xx spikes by hour with awk date slicing. Takeaway: bucket by the bracketed timestamp to catch incident windows.
★ Pick of the week — Elijah Manor's grep-for-Googlebot-errors snippet — filters to bot UA then to 4xx/5xx in one pipe. Takeaway: grep Googlebot | awk '$9 ~ /^[45]/' surfaces what Google actually sees broken.
→ GoAccess docs — when grep gets tedious, it renders the same counts live. Takeaway: good for a second pair of eyes on the numbers.
Always confirm your field positions before trusting any one-liner.

Mistake: trusting X-Forwarded-For for bot verification

Client-supplied forwarding headers are spoofable, so verifying Googlebot off them lets fakes in. Sources:

→ OWASP / MDN on X-Forwarded-For — Anyone can set the header; only the IP your trusted proxy appended is reliable.
Takeaway: verify against the connection IP or your CDN's authenticated header, never the raw XFF.

→ Cloudflare docs (CF-Connecting-IP) — Use the provider's trusted header and lock origin to the CDN.
Takeaway: drop or distrust XFF chains from untrusted hops.

★ Pick of the week — a trusted-proxy parsing note — How to take the right-most untrusted IP from the chain.
Takeaway: pin your trusted-proxy list, then verify Googlebot on the real source.