SO CSharp
The second pillar has fallen 😂 #cloudflare #suggestion_box #cloud
Cloudflare has published the outage post-mortem.
Here is a very brief summary:
- Cloudflare's Bot Management system relies on a periodically generated "feature file", created by a query running on a ClickHouse distributed cluster.
- To query data from all shards they use so-called distributed tables (powered by the table engine Distributed) in a database called
- Somewhere along the pipeline the following metadata query was used:
Which as you can see does not filter by the database name.
Cloudflare was rolling out a new permissions model which resulted in the aforementioned query to return tables from not only the
P.S. Thanks to Valery for another entry in the #suggestion_box!
#clickhouse #random_PoK #cloudflare
Here is a very brief summary:
- Cloudflare's Bot Management system relies on a periodically generated "feature file", created by a query running on a ClickHouse distributed cluster.
- To query data from all shards they use so-called distributed tables (powered by the table engine Distributed) in a database called
default. The Distributed engine queries underlying tables in a database r0. - Somewhere along the pipeline the following metadata query was used:
SELECT
name,
type
FROM system.columns
WHERE
table = 'http_requests_features'
order by name;
Which as you can see does not filter by the database name.
Cloudflare was rolling out a new permissions model which resulted in the aforementioned query to return tables from not only the
default database but the underlying ones too. This effectively doubled the result set, which caused the downstream validation logic (written in #rust! 😁) to fail with kernel panic due to an unexpected number of results.P.S. Thanks to Valery for another entry in the #suggestion_box!
#clickhouse #random_PoK #cloudflare
👍3👏2🤝1