Redirect chains are technical debt with interest
Every hop is a request, a wait, and one more place something can break. Chains form quietly: a site move, then a category rename, then a protocol switch, and now one old address walks through three hops before it lands anywhere.
How to clean them:
— Crawl your own site and export every URL returning a 3xx.
— For each one, follow the chain to the end and record only the final destination.
— Rewrite the rule so the original points straight at that destination. One hop, always.
— Delete rules for URLs that no longer receive traffic or links. Rule files rot, and a bloated one taxes every request you serve.
— Watch for loops. Two rules pointing at each other will happily serve errors forever.
Then check the type. A temporary redirect that has been in place for years is telling crawlers to keep the old address indexed. That is rarely what anyone meant.
One hop or no hop. There is no third option worth defending.
Every hop is a request, a wait, and one more place something can break. Chains form quietly: a site move, then a category rename, then a protocol switch, and now one old address walks through three hops before it lands anywhere.
How to clean them:
— Crawl your own site and export every URL returning a 3xx.
— For each one, follow the chain to the end and record only the final destination.
— Rewrite the rule so the original points straight at that destination. One hop, always.
— Delete rules for URLs that no longer receive traffic or links. Rule files rot, and a bloated one taxes every request you serve.
— Watch for loops. Two rules pointing at each other will happily serve errors forever.
Then check the type. A temporary redirect that has been in place for years is telling crawlers to keep the old address indexed. That is rarely what anyone meant.
One hop or no hop. There is no third option worth defending.
Decision rule: noindex or disallow, never both
This is the most common technical mistake I see, and it is a logic error rather than a config error.
Disallow stops the crawl. Noindex asks for removal from the index. If you disallow a page, the crawler cannot fetch it, which means it can never see the noindex tag. The page then sits in the index as a bare address with no description, indefinitely.
The rule:
— Want it out of the index? Let it be crawled, serve noindex, wait until it drops, then block it if you want to save the crawl.
— Want to save crawl on pages that were never indexed anyway? Disallow alone is fine.
— Page has links pointing at it and you want that value to go somewhere? Neither. Redirect it.
— Sensitive content? Neither tag is a security control. Put it behind authentication.
Order matters more than the tags. Removal first, blocking second.
Robots controls access. Meta robots controls indexing. They are not synonyms.
This is the most common technical mistake I see, and it is a logic error rather than a config error.
Disallow stops the crawl. Noindex asks for removal from the index. If you disallow a page, the crawler cannot fetch it, which means it can never see the noindex tag. The page then sits in the index as a bare address with no description, indefinitely.
The rule:
— Want it out of the index? Let it be crawled, serve noindex, wait until it drops, then block it if you want to save the crawl.
— Want to save crawl on pages that were never indexed anyway? Disallow alone is fine.
— Page has links pointing at it and you want that value to go somewhere? Neither. Redirect it.
— Sensitive content? Neither tag is a security control. Put it behind authentication.
Order matters more than the tags. Removal first, blocking second.
Robots controls access. Meta robots controls indexing. They are not synonyms.
The sitemap checklist most teams quietly fail
A sitemap is a statement of intent: these are my canonical, indexable, live pages. Most sitemaps say something else entirely.
Run through this:
— Every URL returns 200. No redirects, no errors, no soft not-founds.
— Every URL is self-canonical. If a page points its canonical elsewhere, it does not belong here.
— No noindexed URLs. You are asking for a fetch of a page you asked to have dropped.
— No URLs blocked by robots. Same contradiction, opposite direction.
— One protocol, one host, consistent trailing slashes. Pick and enforce.
— Last-modified values that are real. If everything updates overnight because a template changed, the field is now noise and gets ignored.
— Split by section once the file grows, with an index file. Then you can see which section stopped being crawled.
Treat the sitemap as a report you can be graded on. Clean it and resubmit, rather than appending forever.
A sitemap is a statement of intent: these are my canonical, indexable, live pages. Most sitemaps say something else entirely.
Run through this:
— Every URL returns 200. No redirects, no errors, no soft not-founds.
— Every URL is self-canonical. If a page points its canonical elsewhere, it does not belong here.
— No noindexed URLs. You are asking for a fetch of a page you asked to have dropped.
— No URLs blocked by robots. Same contradiction, opposite direction.
— One protocol, one host, consistent trailing slashes. Pick and enforce.
— Last-modified values that are real. If everything updates overnight because a template changed, the field is now noise and gets ignored.
— Split by section once the file grows, with an index file. Then you can see which section stopped being crawled.
Treat the sitemap as a report you can be graded on. Clean it and resubmit, rather than appending forever.
Crawl budget is a big-site problem you probably do not have
Unpopular position: on a site of a few thousand pages, crawl budget is not your bottleneck. Your pages are not indexed because they are not worth indexing, not because the crawler ran out of patience.
Budget starts to matter when you hold hundreds of thousands of addresses, when URLs are generated from parameters, or when the server is slow enough that the crawler throttles itself.
What to do instead on a small site:
— Delete the pages nobody would miss. Thin, duplicated, near-identical template output.
— Consolidate. Three weak pages on one topic beat each other up. One good page does not.
— Make sure internal links actually reach everything you care about. Orphans go uncrawled because nothing points at them.
If you are small and deep in log files, you are doing the fun work instead of the useful work. I have been guilty of exactly this.
Fix worth before you optimise access.
Unpopular position: on a site of a few thousand pages, crawl budget is not your bottleneck. Your pages are not indexed because they are not worth indexing, not because the crawler ran out of patience.
Budget starts to matter when you hold hundreds of thousands of addresses, when URLs are generated from parameters, or when the server is slow enough that the crawler throttles itself.
What to do instead on a small site:
— Delete the pages nobody would miss. Thin, duplicated, near-identical template output.
— Consolidate. Three weak pages on one topic beat each other up. One good page does not.
— Make sure internal links actually reach everything you care about. Orphans go uncrawled because nothing points at them.
If you are small and deep in log files, you are doing the fun work instead of the useful work. I have been guilty of exactly this.
Fix worth before you optimise access.
How faceted navigation quietly ate an entire crawl
A catalogue site with a modest number of real products. Logs showed the crawler hammering the server constantly, and new products still took forever to appear in search.
The cause was filters. Colour, size, brand, price, sort order, page number — each combination generated its own address, and each one was linked from the page. The real page count was small. The crawlable address count was effectively unbounded.
What fixed it:
— Decide which filter combinations deserve to be indexable pages. Usually very few, and only the ones people actually search for.
— Everything else stays usable for humans but stops generating crawlable addresses, with variants canonicalised to the clean category page.
— Block the parameters that add nothing. Sort order and view mode first.
— Only then look at the logs again.
Crawling of real product pages recovered. Nothing was added. Things were removed.
Infinite address space is a bug, not a feature.
A catalogue site with a modest number of real products. Logs showed the crawler hammering the server constantly, and new products still took forever to appear in search.
The cause was filters. Colour, size, brand, price, sort order, page number — each combination generated its own address, and each one was linked from the page. The real page count was small. The crawlable address count was effectively unbounded.
What fixed it:
— Decide which filter combinations deserve to be indexable pages. Usually very few, and only the ones people actually search for.
— Everything else stays usable for humans but stops generating crawlable addresses, with variants canonicalised to the clean category page.
— Block the parameters that add nothing. Sort order and view mode first.
— Only then look at the logs again.
Crawling of real product pages recovered. Nothing was added. Things were removed.
Infinite address space is a bug, not a feature.
Build the redirect map before the migration, not after
Migrations fail in one specific way: somebody owns the design and somebody owns the platform, and nobody owns the addresses until launch week.
Do this first, while there is still time to argue:
— Export every URL the old site has. From a crawl, from the sitemap, from the logs, from your analytics tool. Four sources, because each one misses something different.
— Deduplicate, then mark each address: keep, merge, or retire.
— Map every kept and merged address to exactly one destination. One to one wherever possible.
— Never map everything to the homepage. That is a graceful way of saying you gave up.
— For retired addresses with no equivalent, decide deliberately whether they simply go missing or are declared gone.
— Test the map against staging before launch, not after.
— Keep the old logs. Afterwards they are the only way to find what you forgot.
The map is the migration. The rest is decoration.
Migrations fail in one specific way: somebody owns the design and somebody owns the platform, and nobody owns the addresses until launch week.
Do this first, while there is still time to argue:
— Export every URL the old site has. From a crawl, from the sitemap, from the logs, from your analytics tool. Four sources, because each one misses something different.
— Deduplicate, then mark each address: keep, merge, or retire.
— Map every kept and merged address to exactly one destination. One to one wherever possible.
— Never map everything to the homepage. That is a graceful way of saying you gave up.
— For retired addresses with no equivalent, decide deliberately whether they simply go missing or are declared gone.
— Test the map against staging before launch, not after.
— Keep the old logs. Afterwards they are the only way to find what you forgot.
The map is the migration. The rest is decoration.
Your canonical tag is a suggestion, and it loses arguments
People treat canonical as a command. It is one signal among several, and when the other signals disagree, it loses.
Signals that fight your canonical:
— Internal links. If the whole site links to the variant instead of the canonical, you have voted against yourself.
— Sitemaps listing the non-canonical version.
— Redirects pointing the other way.
— Pages declaring each other canonical in a loop.
— A canonical target that redirects, errors, or is blocked from crawling.
— Content that is not actually equivalent. Canonical is for duplicates, not for "sort of related".
How to check: inspect the address and see which version is genuinely treated as canonical. If it is not yours, stop adjusting the tag and start fixing the links.
Consistency across every signal beats a perfect tag standing alone.
—
Если тема зашла, посмотри @ParcelMap
People treat canonical as a command. It is one signal among several, and when the other signals disagree, it loses.
Signals that fight your canonical:
— Internal links. If the whole site links to the variant instead of the canonical, you have voted against yourself.
— Sitemaps listing the non-canonical version.
— Redirects pointing the other way.
— Pages declaring each other canonical in a loop.
— A canonical target that redirects, errors, or is blocked from crawling.
— Content that is not actually equivalent. Canonical is for duplicates, not for "sort of related".
How to check: inspect the address and see which version is genuinely treated as canonical. If it is not yours, stop adjusting the tag and start fixing the links.
Consistency across every signal beats a perfect tag standing alone.
—
Если тема зашла, посмотри @ParcelMap
Playbook: pages that get found and never indexed
"Found, not indexed" means the address is known and the crawler decided it was not worth fetching yet. That is a priority signal, not a bug.
Work through it in this order:
— Check the server. Slow or unstable responses make the crawler back off on its own. Cheapest fix, most ignored.
— Check internal links to the page. Zero or one link from somewhere deep tells the crawler precisely how much you value it.
— Check for near-duplicates. If you have many pages built from one template with a word swapped, the pattern has already been seen.
— Confirm the sitemap includes it and that it returns 200.
— Then improve the page. Something on it should exist nowhere else.
— Request indexing once. Requesting repeatedly does nothing except make you feel busy.
If it still does not get indexed after all that, take the hint. Not every page deserves to exist.
"Found, not indexed" means the address is known and the crawler decided it was not worth fetching yet. That is a priority signal, not a bug.
Work through it in this order:
— Check the server. Slow or unstable responses make the crawler back off on its own. Cheapest fix, most ignored.
— Check internal links to the page. Zero or one link from somewhere deep tells the crawler precisely how much you value it.
— Check for near-duplicates. If you have many pages built from one template with a word swapped, the pattern has already been seen.
— Confirm the sitemap includes it and that it returns 200.
— Then improve the page. Something on it should exist nowhere else.
— Request indexing once. Requesting repeatedly does nothing except make you feel busy.
If it still does not get indexed after all that, take the hint. Not every page deserves to exist.
Decision rule: missing, gone, or moved
Retiring addresses gets sloppy because it feels harmless. It is not. You are deciding what happens to every link and every crawl of that address from now on.
The rule I use:
— Direct equivalent exists? Permanent redirect to it. One hop.
— No equivalent, but a genuinely relevant parent page exists and a visitor would be served by it? Redirect there, sparingly. If you cannot say the destination answers the same need, do not.
— Content is gone and nothing replaces it? Serve a not-found. It is a normal status code, not a personal failure.
— Gone permanently and you want it dropped faster? Serve gone. Same outcome, stated more firmly.
— Never redirect a mass of dead addresses to the homepage. It gets treated as a soft not-found anyway, and you lose the ability to see what died.
Then confirm your error page returns the status it displays. A page saying "not found" while returning 200 is the worst of both.
Dead is fine. Lying about dead is not.
Retiring addresses gets sloppy because it feels harmless. It is not. You are deciding what happens to every link and every crawl of that address from now on.
The rule I use:
— Direct equivalent exists? Permanent redirect to it. One hop.
— No equivalent, but a genuinely relevant parent page exists and a visitor would be served by it? Redirect there, sparingly. If you cannot say the destination answers the same need, do not.
— Content is gone and nothing replaces it? Serve a not-found. It is a normal status code, not a personal failure.
— Gone permanently and you want it dropped faster? Serve gone. Same outcome, stated more firmly.
— Never redirect a mass of dead addresses to the homepage. It gets treated as a soft not-found anyway, and you lose the ability to see what died.
Then confirm your error page returns the status it displays. A page saying "not found" while returning 200 is the worst of both.
Dead is fine. Lying about dead is not.
Playbook: read raw access logs before you touch anything else
Crawl problems hide in logs, not in dashboards. Dashboards are sampled and smoothed. Logs are what actually happened.
The pass I run:
— Pull at least four weeks. Less and you're reading noise.
— Split bot from human by user agent, then verify the bot hits by reverse lookup. Fake crawlers are common and they skew everything.
— Group requests by path pattern, not by URL. /product/* tells you more than ten thousand rows ever will.
— Sort by hit count descending. Look at the top twenty patterns and ask one question: do I want the crawler spending its time here?
— Now sort by status code. Anything that isn't a 200 or a 304 inside those top patterns is waste.
— Cross the list against your sitemap. Pages crawled but absent from the sitemap, and pages in the sitemap never crawled — both are findings.
Most sites discover the crawler is spending the bulk of its budget on parameter URLs, ancient redirects and internal search results.
You cannot optimise crawl until you can see it.
Crawl problems hide in logs, not in dashboards. Dashboards are sampled and smoothed. Logs are what actually happened.
The pass I run:
— Pull at least four weeks. Less and you're reading noise.
— Split bot from human by user agent, then verify the bot hits by reverse lookup. Fake crawlers are common and they skew everything.
— Group requests by path pattern, not by URL. /product/* tells you more than ten thousand rows ever will.
— Sort by hit count descending. Look at the top twenty patterns and ask one question: do I want the crawler spending its time here?
— Now sort by status code. Anything that isn't a 200 or a 304 inside those top patterns is waste.
— Cross the list against your sitemap. Pages crawled but absent from the sitemap, and pages in the sitemap never crawled — both are findings.
Most sites discover the crawler is spending the bulk of its budget on parameter URLs, ancient redirects and internal search results.
You cannot optimise crawl until you can see it.
Why relying on a single traffic source is the fastest way to kill your ROI
Relying on one platform creates a single point of failure. When algorithms shift, the funnel collapses. A sustainable strategy requires distributing risk across multiple channels. This diversification protects your margins from sudden platform-specific volatility.
Prioritize the balance between acquisition cost and lifetime value. High-performing campaigns focus on retention to offset rising entry costs. Shifting to a "customer journey" mindset ensures profitability even when traffic costs fluctuate.
Creatives are volatile; even the best hook loses its edge. Use a testing loop with small budgets for experimental angles and larger allocations for proven winners. This prevents sudden performance drops that ruin scaling efforts.
Before increasing spend, ensure the backend is optimized. Build your strategy around data ownership to ensure no external change can zero out your business. 🛡
Relying on one platform creates a single point of failure. When algorithms shift, the funnel collapses. A sustainable strategy requires distributing risk across multiple channels. This diversification protects your margins from sudden platform-specific volatility.
Prioritize the balance between acquisition cost and lifetime value. High-performing campaigns focus on retention to offset rising entry costs. Shifting to a "customer journey" mindset ensures profitability even when traffic costs fluctuate.
Creatives are volatile; even the best hook loses its edge. Use a testing loop with small budgets for experimental angles and larger allocations for proven winners. This prevents sudden performance drops that ruin scaling efforts.
Before increasing spend, ensure the backend is optimized. Build your strategy around data ownership to ensure no external change can zero out your business. 🛡
Forwarded from В арбитраже денег нет?
Тем временем подстилка коричневых Кардиналов и лично Кустова главред Максим Огненный завёл собственный канал, где, наверное, опять будет писать стихи и анонсировать вьюхи с ме**дроновыми наркоманами.🤡🤡
Чем вообще известен этот персонаж? Собсна, только порцией отборного кринжа, например, не так давно он выебывался на Иванова в "Письмах кардинала", но недожал и тема осталась нераскрытой. ЕЮ заслужил даже высокоинтеллектуальные выпады, которые тот не понял в силу врождённого аутизма:
Нихуя не разбираясь в аффилке, он на серьёзных щах брал интервью у Мефмедова и пытался построить серьёзный диалог с объёбанной ракетой. Из остальных достижений только нытье в чатиках: персонаж настолько невзрачный, что даже хуесосить его не так интересно.¯\_(ツ)_/¯
А теперь Огненный завёл собственный канал, где собрался выебать всех, но, походу, ебать будет лишь подписоту своей графоманией. Похожей хуйнёй занимался и оунер ФБ Киллы, который так любит рассуждать про экономику и политику. Оно и понятно: что у коричневых, что у киллы, что у партнёркина — один инвестор и одна крыша, поэтому подопечные синхронно занимаются бестолковой хуйнёй. Кому не похуй, подписывайтесь, будет интересно (нет): https://t.me/+dr240TeLtPc2Nzdk
В арбитраже деньги есть, но только у тех, кто с LuckyCards 💵
Чем вообще известен этот персонаж? Собсна, только порцией отборного кринжа, например, не так давно он выебывался на Иванова в "Письмах кардинала", но недожал и тема осталась нераскрытой. ЕЮ заслужил даже высокоинтеллектуальные выпады, которые тот не понял в силу врождённого аутизма:
Высокохудожественная лексика героя, демонстрируемая им не только на своем канале, но и на любой публичной площадке, настолько пленит любого слушателя, что мало кто может стать собеседником жертвы во второй раз.
Нихуя не разбираясь в аффилке, он на серьёзных щах брал интервью у Мефмедова и пытался построить серьёзный диалог с объёбанной ракетой. Из остальных достижений только нытье в чатиках: персонаж настолько невзрачный, что даже хуесосить его не так интересно.¯\_(ツ)_/¯
А теперь Огненный завёл собственный канал, где собрался выебать всех, но, походу, ебать будет лишь подписоту своей графоманией. Похожей хуйнёй занимался и оунер ФБ Киллы, который так любит рассуждать про экономику и политику. Оно и понятно: что у коричневых, что у киллы, что у партнёркина — один инвестор и одна крыша, поэтому подопечные синхронно занимаются бестолковой хуйнёй. Кому не похуй, подписывайтесь, будет интересно (нет): https://t.me/+dr240TeLtPc2Nzdk
В арбитраже деньги есть, но только у тех, кто с LuckyCards 💵