[discord: uvatbc]
The RabbitMQ service had gone mad and all other services were suffering as a result.
I have shot the rabid rabbit in the head. A new rabbit has taken its place.
Things are back to working state.

[discord: uvatbc]
You know how killing one rabbit is never enough when there's zombies around.
now I have to burn the land and rebuild it.
explanation:
the root cause as always is running out of storage space.
run out of disk space -> rabbit goes mad -> everyone goes mad
I'm clearing out disk space (again) and restarting from scratch.
Your builds will remain in queue and will not need to be restarted.
this shouldn't be more than 3 hours max. (edited)

[discord: uvatbc]
- Storage has been brought under control again.
- Build queue is moving.
- I will need to clean up the devspace machine as well, but I intend to delay that for a day at least

[discord: uvatbc]
System down: Its yet another disk space problem.
Working on it.
ETA: Unknown (edited)

[discord: uvatbc]
Sunday funday. Recovery completed, earlier than expected.
Stop touching grass. Start building! NAAAAOOOO!

[discord: uvatbc]
If you want a clean build, please ask an admin permission for it.
If I see a clean build without reason, I will hand out the cancel at the worst possible time: Once you have waited out your queue time, once your build starts.... Canceled.
That should be annoyance enough for you to not start clean builds (edited)

[discord: uvatbc]
Time to cleanup your devspace storage. We're reaching 86% usage now, so do it yourself for the next 24 hours.
During the weekend, I will run scripts to mass cleanup everything from your devspaces.
Here's the command you should run to clean out your own crave clone-s:

crave clone list --json | jq -r '.clones[]."Cloned At"' | while read line ; do crave clone destroy -y $line ; done ; exit

[discord: sounddrill]
We're seeing problems with the Google Cloud API right now. Please stay tuned for further updates
https://cdn.discordapp.com/attachments/1249126184135168011/1382797538880520242/image.png?ex=684c7610&is=684b2490&hm=932665b9e2045f4bb2472a089691752b342e39a8bf4a7fd56d2f32842073e40f&

By [uvatbc] on [discord]:
We're seeing problems with the Google Cloud API right now. Please stay tuned for further updates
https://cdn.discordapp.com/attachments/1249126184135168011/1382839669947105310/image.png?ex=684c9d4d&is=684b4bcd&hm=acecccdc6c3e84e8b5054b399fd4bcee4ed33b95bc21077a56b3d6b6dd557913&

By [sounddrill] on [discord]:
We're seeing problems with the Google Cloud API right now. Please stay tuned for further updates
https://cdn.discordapp.com/attachments/1249126184135168011/1383049329375449128/Screenshot_20250613-171354.Discord.png?ex=684d608f&is=684c0f0f&hm=d98e0c4886b62d1ad7b85689897505ec6d698aa6e488223dba3c14facaf75190&

By [uvatbc] on [discord]:
Hurray for storage problems.... At least this time I checked before it all imploded. Storage has been cleaned up.
All your builds are still running.
Your devspace sessions should now connect again

By [uvatbc] on [discord]:
I have terminated a whole bunch of devspace sessions that were eating up a lot of CPU.
You should now be able to connect to your own devspace sessions

By [uvatbc] on [discord]:
I have also had to kill a bunch of queued builds as a result.
Please reconnect to your sessions and restart you builds.
Sorry for the inconvenience.

By [uvatbc] on [discord]:
Build cache is going to be purged today to reclaim space.
Some initial builds will feel the impact: Slower builds until the cache is refilled. (edited)

By [uvatbc] on [discord]:
- Terminating devspace sessions because its being extra slow. Please restart your sessions.
- NO CLEAN BUILDS WITHOUT PERMISSION. Next person with an unapproved clean build: Ban
- NO CCACHE BUILDS AT ALL. Next person with ccache build: Ban

By [uvatbc] on [discord]:
I will be upgrading the system over the weekend. This will involve clearing out the cache, vacuuming the DB, and reclaiming storage space.
It is likely that this will take several hours.
I will send out another notification when this upgrade process actually begins.
Thank you for your attention to this matter.

By [uvatbc] on [discord]:
Maintenance mode enabled.
This means all queued builds will finish, but no new builds will start.
Maintenance window ETA: 8-10 hours

By [uvatbc] on [discord]:
Maintenance over.
1. DB vacuumed.
1. Older build outputs and patches cleaned up
1. Cache cleared out.
1. Devspace node recreated
Start your builds!

By [imrank1618] on [discord]:
@everyone system going for emergency upgrade

By [imrank1618] on [discord]:
will be unavailable for sometime