Status-code mining with one-liners: the awk greatest hits
Four snippets worth pinning above your terminal, all credited to where I first saw them.
🔗 Julia Evans (wizardzines) — her awk explainers make
🔗 nixCraft — classic tutorial for isolating 5xx by hour:
★ Pick of the week — Brendan Gregg's "awk and the log" notes — treats logs as columnar data and shows running-tally tricks (associative arrays) that beat piping to sort.
🔗 Greg's Wiki (BashFAQ) — the quoting and field-number gotchas that save you from off-by-one column bugs.
Takeaway: 90% of log triage is field 9 (status) grouped by something. Master that, skip the heavy tools.
Four snippets worth pinning above your terminal, all credited to where I first saw them.
🔗 Julia Evans (wizardzines) — her awk explainers make
awk '{print $9}' | sort | uniq -c | sort -rn finally click. Best mental model for field-splitting logs.🔗 nixCraft — classic tutorial for isolating 5xx by hour:
awk '$9~/^5/{print substr($4,14,2)}'. Find the spike, then the cause.★ Pick of the week — Brendan Gregg's "awk and the log" notes — treats logs as columnar data and shows running-tally tricks (associative arrays) that beat piping to sort.
🔗 Greg's Wiki (BashFAQ) — the quoting and field-number gotchas that save you from off-by-one column bugs.
Takeaway: 90% of log triage is field 9 (status) grouped by something. Master that, skip the heavy tools.
Crawl budget, measured from logs (not guessed)
Everyone talks crawl budget; these sources actually quantify it from server data.
→ Google's Gary Illyes (crawl budget post) — the source definition: crawl rate limit + crawl demand. Read it before any tool's dashboard.
→ OnCrawl's log studies — ties Googlebot hit frequency to indexation lag with real charts. Their "crawl-to-index ratio" framing is genuinely useful.
★ Pick of the week — JetOctopus crawl-frequency breakdown — shows how to bucket URLs by hits/30 days and spot the long tail Google ignores. Concrete percentile thresholds, not vibes.
→ Screaming Frog log analyser docs — solid for matching crawled URLs against your sitemap to find orphan hits.
Takeaway: crawl budget waste = bot hits on parameter/faceted URLs. Logs are the only place you see it before it costs you.
Everyone talks crawl budget; these sources actually quantify it from server data.
→ Google's Gary Illyes (crawl budget post) — the source definition: crawl rate limit + crawl demand. Read it before any tool's dashboard.
→ OnCrawl's log studies — ties Googlebot hit frequency to indexation lag with real charts. Their "crawl-to-index ratio" framing is genuinely useful.
★ Pick of the week — JetOctopus crawl-frequency breakdown — shows how to bucket URLs by hits/30 days and spot the long tail Google ignores. Concrete percentile thresholds, not vibes.
→ Screaming Frog log analyser docs — solid for matching crawled URLs against your sitemap to find orphan hits.
Takeaway: crawl budget waste = bot hits on parameter/faceted URLs. Logs are the only place you see it before it costs you.
Pairs well with this channel
@LinkEquityHeat — Strong, unfiltered opinions on internal linking — why your 'related posts' widget is… Quietly one of the better feeds in the space.
@LinkEquityHeat — Strong, unfiltered opinions on internal linking — why your 'related posts' widget is… Quietly one of the better feeds in the space.
Log sampling done right: when a slice lies to you
Sampling huge logs is fine — until it quietly drops your rare-but-important events.
→ Honeycomb's dynamic sampling docs — the clearest explanation of why uniform sampling buries 5xx and bot edge cases, and how head/tail sampling fixes it.
★ Pick of the week — Liz Fong-Jones on observability sampling — argues for keeping 100% of errors and sampling the boring 200s. The principle transfers straight to crawl logs: never sample away Googlebot's 4xx hits.
→ Elastic's sampling guide — practical knobs if you're in the ELK stack.
→ VividCortex's "approximate" series — good stats intuition on confidence intervals for sampled counts.
Takeaway: sample your 200-OK noise, keep every error and every bot 3xx/4xx/5xx. Those are the rows you analyse logs for.
Sampling huge logs is fine — until it quietly drops your rare-but-important events.
→ Honeycomb's dynamic sampling docs — the clearest explanation of why uniform sampling buries 5xx and bot edge cases, and how head/tail sampling fixes it.
★ Pick of the week — Liz Fong-Jones on observability sampling — argues for keeping 100% of errors and sampling the boring 200s. The principle transfers straight to crawl logs: never sample away Googlebot's 4xx hits.
→ Elastic's sampling guide — practical knobs if you're in the ELK stack.
→ VividCortex's "approximate" series — good stats intuition on confidence intervals for sampled counts.
Takeaway: sample your 200-OK noise, keep every error and every bot 3xx/4xx/5xx. Those are the rows you analyse logs for.
GoAccess and friends: real-time log dashboards without a stack
For when you want a live view, not a data warehouse.
🔗 GoAccess official docs — single binary, terminal or HTML dashboard, parses combined format out of the box. The benchmark for zero-setup.
🔗 Allan Barizo's GoAccess + geoip tutorial — adds country breakdown so you can spot scraper farms by region.
★ Pick of the week — Gerardo Orellana's custom log-format guide (GoAccess author) — how to write a
🔗 lnav (Log File Navigator) — underrated TUI that auto-detects formats and does SQL over logs. Great for ad-hoc digging.
Takeaway: GoAccess for the live wall-display, lnav for interactive spelunking. Neither needs a database.
For when you want a live view, not a data warehouse.
🔗 GoAccess official docs — single binary, terminal or HTML dashboard, parses combined format out of the box. The benchmark for zero-setup.
🔗 Allan Barizo's GoAccess + geoip tutorial — adds country breakdown so you can spot scraper farms by region.
★ Pick of the week — Gerardo Orellana's custom log-format guide (GoAccess author) — how to write a
--log-format string for any weird Nginx or CDN log. Stops the "token not found" frustration cold.🔗 lnav (Log File Navigator) — underrated TUI that auto-detects formats and does SQL over logs. Great for ad-hoc digging.
Takeaway: GoAccess for the live wall-display, lnav for interactive spelunking. Neither needs a database.
Log the right fields: Nginx/Apache format tweaks for SEO
Default log formats omit data you'll wish you had. These sources fix that.
→ Nginx log_format docs — add
→ Apache mod_log_config manual —
★ Pick of the week — Barry Adams on "log everything Googlebot sees" — argues for capturing full UA, response time, and bytes, because that's how you prove slow pages throttle crawl rate.
→ Cloudflare Logpush field reference — if a CDN fronts you, your origin logs lie; pull edge logs instead.
Takeaway: add response time and Host now. You can't analyse fields you never logged.
Default log formats omit data you'll wish you had. These sources fix that.
→ Nginx log_format docs — add
$request_time and $upstream_response_time so you can see which URLs make Googlebot wait.→ Apache mod_log_config manual —
%D (microseconds) and %{Host}i for multi-site servers. Without Host you can't split bot traffic per domain.★ Pick of the week — Barry Adams on "log everything Googlebot sees" — argues for capturing full UA, response time, and bytes, because that's how you prove slow pages throttle crawl rate.
→ Cloudflare Logpush field reference — if a CDN fronts you, your origin logs lie; pull edge logs instead.
Takeaway: add response time and Host now. You can't analyse fields you never logged.
grep for logs: the patterns I keep reaching for
Not a beginner intro — these are the references that sharpened my filtering.
🔗 Julia Evans' grep zine —
🔗 ripgrep (BurntSushi) docs — rg over gzipped logs with
★ Pick of the week — "Use The Index, Luke"-style log thinking by Jon Henshaw — combining grep with
🔗 Greg's Wiki on grep+zcat pipelines — handling rotated
Takeaway: learn ripgrep's
Not a beginner intro — these are the references that sharpened my filtering.
🔗 Julia Evans' grep zine —
grep -P for Perl regex and why -F (fixed strings) is faster when you just want a literal IP.🔗 ripgrep (BurntSushi) docs — rg over gzipped logs with
-z is dramatically faster than zcat | grep on big archives.★ Pick of the week — "Use The Index, Luke"-style log thinking by Jon Henshaw — combining grep with
grep -v chains to strip your own monitoring/uptime bots before counting real crawl. The cleanup nobody documents.🔗 Greg's Wiki on grep+zcat pipelines — handling rotated
.gz logs without decompressing to disk.Takeaway: learn ripgrep's
-z and the grep -v exclusion chain. Faster searches, cleaner counts.Screaming Frog Log File Analyser: the under-read docs
Most people own it for the crawler; the log tool is quietly excellent.
→ Screaming Frog's official log analyser guide — import, then the "URLs not in crawl" view instantly surfaces orphan pages bots find but your site doesn't link.
★ Pick of the week — Dan Sharp's tutorial on matching logs to crawl data — the killer move: import a crawl AND logs, then filter for URLs Googlebot hits that return non-200. Bot-discovered errors, ranked.
→ Aleyda Solis on log-driven prioritization — using crawl frequency to decide which fixes Google will actually re-crawl soon.
→ Glenn Gabe's case notes — real examples of log analysis catching a spike of bot hits on a redirect loop.
Takeaway: combine crawl export + logs. The intersection (bot-hit + broken) is your fix list.
Most people own it for the crawler; the log tool is quietly excellent.
→ Screaming Frog's official log analyser guide — import, then the "URLs not in crawl" view instantly surfaces orphan pages bots find but your site doesn't link.
★ Pick of the week — Dan Sharp's tutorial on matching logs to crawl data — the killer move: import a crawl AND logs, then filter for URLs Googlebot hits that return non-200. Bot-discovered errors, ranked.
→ Aleyda Solis on log-driven prioritization — using crawl frequency to decide which fixes Google will actually re-crawl soon.
→ Glenn Gabe's case notes — real examples of log analysis catching a spike of bot hits on a redirect loop.
Takeaway: combine crawl export + logs. The intersection (bot-hit + broken) is your fix list.
Spotting fake crawlers in your logs: 5 solid reads
Scrapers love wearing a Googlebot costume. Here's how the pros unmask them.
→ Google's "verifying Googlebot" page — rDNS is step one, always.
★ Pick of the week — Cloudflare's bot-management writeup — breaks down the tells: UA claims Googlebot but the IP is a residential proxy or a datacenter ASN Google doesn't own. ASN lookup beats DNS for speed.
→ Detectify / ipinfo ASN guides — map an IP to its owning network; "Googlebot" from Hetzner or DigitalOcean is fake, period.
→ DataDome's bot-traffic reports — useful baselines for what fraction of "Googlebot" hits are typically spoofed.
→ AbuseIPDB — cross-check noisy IPs you keep seeing.
Takeaway: real Googlebot lives in Google's ASNs (e.g. AS15169). Wrong ASN = impostor, no matter the UA.
Scrapers love wearing a Googlebot costume. Here's how the pros unmask them.
→ Google's "verifying Googlebot" page — rDNS is step one, always.
★ Pick of the week — Cloudflare's bot-management writeup — breaks down the tells: UA claims Googlebot but the IP is a residential proxy or a datacenter ASN Google doesn't own. ASN lookup beats DNS for speed.
→ Detectify / ipinfo ASN guides — map an IP to its owning network; "Googlebot" from Hetzner or DigitalOcean is fake, period.
→ DataDome's bot-traffic reports — useful baselines for what fraction of "Googlebot" hits are typically spoofed.
→ AbuseIPDB — cross-check noisy IPs you keep seeing.
Takeaway: real Googlebot lives in Google's ASNs (e.g. AS15169). Wrong ASN = impostor, no matter the UA.
Don't lose the evidence: log rotation and retention sources
The best log analysis fails if logrotate ate last month's data first.
🔗 logrotate man page —
🔗 DigitalOcean's logrotate tutorial — the cleanest walkthrough of a custom config and testing with
★ Pick of the week — Will Critchlow on keeping 90+ days of logs for SEO — argues crawl patterns only reveal themselves over months; default 7-day retention blinds you to seasonal bot behavior.
🔗 GoAccess persistence docs — incremental processing so you keep aggregates even after raw logs rotate away.
Takeaway: enable
The best log analysis fails if logrotate ate last month's data first.
🔗 logrotate man page —
rotate, compress, dateext. Set dateext so files are access.log-20240115 not .1, .2 — far easier to script against.🔗 DigitalOcean's logrotate tutorial — the cleanest walkthrough of a custom config and testing with
logrotate -d (debug, no changes).★ Pick of the week — Will Critchlow on keeping 90+ days of logs for SEO — argues crawl patterns only reveal themselves over months; default 7-day retention blinds you to seasonal bot behavior.
🔗 GoAccess persistence docs — incremental processing so you keep aggregates even after raw logs rotate away.
Takeaway: enable
dateext, keep 90 days compressed, and aggregate before deletion.ELK for access logs: a curated starting path
Elasticsearch + Kibana is overkill for some, perfect for others. Five honest references.
→ Elastic's Filebeat nginx module docs — pre-built parsing and dashboards; you're querying bots in an hour, not a week.
→ grok debugger (Elastic) — test your log-line pattern before it silently drops malformed lines.
★ Pick of the week — Daniel Berman's "parsing access logs with Logstash" — the cleanest grok pattern for combined format plus a geoip + user-agent filter chain that classifies bots on ingest.
→ Kibana Lens tutorials — building a "Googlebot hits per URL path" viz without writing query DSL.
→ Elastic's data-stream + ILM guide — auto-roll indices so storage doesn't explode.
Takeaway: use the Filebeat module's defaults first. Only hand-roll grok when your format is non-standard.
Elasticsearch + Kibana is overkill for some, perfect for others. Five honest references.
→ Elastic's Filebeat nginx module docs — pre-built parsing and dashboards; you're querying bots in an hour, not a week.
→ grok debugger (Elastic) — test your log-line pattern before it silently drops malformed lines.
★ Pick of the week — Daniel Berman's "parsing access logs with Logstash" — the cleanest grok pattern for combined format plus a geoip + user-agent filter chain that classifies bots on ingest.
→ Kibana Lens tutorials — building a "Googlebot hits per URL path" viz without writing query DSL.
→ Elastic's data-stream + ILM guide — auto-roll indices so storage doesn't explode.
Takeaway: use the Filebeat module's defaults first. Only hand-roll grok when your format is non-standard.