CatOps
5.68K subscribers
94 photos
4 videos
19 files
2.27K links
DevOps and other issues by Yurii Rochniak (@grem1in) - SRE @ Preply && Maksym Vlasov (@MaxymVlasov) - Engineer @ Star. Opinions on our own.

We do not post ads including event announcements. Please, do not bother us with such requests!
Download Telegram
Some time ago I had a task to split the helm template output into separate files per object.

So, I found this issue in the Helm’s repository. People were suggesting using AWK for that, but that didn’t work well for me at the time, so I opted out for YQ.

A couple of days ago someone left a comment to that issue that apparently there is a tool called Kubesplit that can do exactly that. So, feel free to use it if you need to achieve something similar to what I did.

#kubernetes
​​For today’s Donations Monday I would like to remind you about Pavlo and Naya, who raise money on drones and telecommunication equipment.

P.S. Here’s a photo of a drone you helped to buy last week and it’s already on its way to the place.
Resend had a 12 hour outage on the 21st of February.

tl;dr:
> The database migration accidentally deleted data from production servers…
> … we performed a database migration command locally, but it incorrectly pointed to the production environment instead…

You can read it in more details is the article, but here are some of the action items from this postmortem:

- No accessible user role should have write privileges on the production database.
- Improve local development to reduce risks related to database migrations.
- Create redundancy to preserve sending function even during a database outage.
- Increase cadence for disaster recovery tests.
- Implement incident banner on Resend dashboard to inform users quickly.

So, I dunno, check your database. Maybe, you have such a risk as well.

Also, it’s kinda strange that people rarely talk about network isolation not only between their production and non-production environments, but also between their local environment and production. Make production access conscious. Put it on a separate role/VPN. Add some friction accessing it.

Moreover, for the love of god, validate your DB backups.

#postmortem #databases
S3 cache for GitHub Actions - a drop-in replacement for the native GHA cache functionality that is technically unlimited because of S3.

I haven’t used it personally and I don’t have an AWS account to test it. This thing comes as a part of RunsOn - a solution to setup your self-hosted runners for GHA in AWS.

Might be worth checking if you want to bring your own worker nodes to GHA but don’t want the hustle configuring them.

#github #gha #cicd
Kubernetes: tracing requests with AWS X-Ray, and Grafana data source is a step-by-step guide on how to setup tracing in your EKS cluster using AWS X-Ray by Arseniy Zinchenko - a member of the Ukrainian DevOps community.

Also, make sure to subscribe to his Substack! He posts new things quite often and I have no idea where does he find time and willpower to do so 😅

#aws #kubernetes #observability
​​I got a bit distracted in the recent days, so I make posts with delays.

Today we have a Donations Monday with a twist.

We are raising funds for two foundations at the same time:

- For NayTak for camouflage nets.
- For UA Responders an IVL and a defibrillator for medics from Kraken.

The twist is that you can win a remnant of an S-300 rocket (on the picture).

Every donation for >50 UAH is a chance to win!

You can donate on:

- a Monobank jar: https://send.monobank.ua/jar/5SizeGGzBM
- top up the card directly: 5375 4112 1191 0851

Please, add your contact details if you don’t use MonoBank for donations, so they know how to find you in case you win.

P.S. Tomorrow I will send a new newsletter issue, that I should’ve sent yesterday.

#Donations #Ukraine
I know that many folks have a mix of corporate and private repositories on their laptops. By private, I mean their small projects, dotfiles repo, forks of public repositories, etc.

Here's an interesting guide on one of the ways of how to keep separate users (email, signing key, name) for different repositories.

This would also work if you have repositories backed by different provides. For example, if you need to push to both GitHub and GitLab and use different SSH keys for that.

Here's a condensed version of this article on StackOverflow.

#git
​​On behalf of the Architecture Stage organizational committee I want to invite you to the DOU Day Conference!

It’ll take place offline in Kyiv on the 18th of May.

The thing is that if you buy a ticket now, you will get -50% off for the second one. So, a great opportunity to grab some tickets together with your teammates.

#event
The core idea of this article is pretty simple: you need to protect your Terraform states. I don't think this is a debatable topic, and anyone has a different opinion on this matter.

To quote the article itself:

 an attacker can modify the Terraform state file it’s game over and bad times ahead.


However, in the very end, this article provides some suggestions that I never saw implemented IRL:

- Store the state lock in a separately permissioned location
- Use a read-only role for terraform plan executions

#terraform
​​For todays Donations Monday I want to share with you a fundraiser for 101 Starlink terminals by Dzyga Paw foundation:

https://dzygaspaw.com/starlinks-101

#Donations #Ukraine