🚨 The Police of AWS

When I joined AWS, my very first team was AWS CloudTrail – the service that records API activity across your account, giving you visibility into who did what, when, and how.

As a customer back in 2020, I often asked myself questions like:

Who created this IAM role?

When was this S3 bucket policy changed?

Why do I have EC2 instances running when I don’t remember starting them?


At that time, I discovered CloudTrail and realized it’s basically the “security camera system” of AWS. It tracks every API call across services, whether from the console, CLI, SDKs, or even from services acting on your behalf.

When I joined CloudTrail as an engineer as my first team at AWS, I got to work in the Integrations Monitoring team. As soon as I joined, I started designing a new proactive integration health monitoring system between AWS services and CloudTrail and led its design, implementation and global deployment across 30+ AWS regions worldwide.

Today, I’m proud that my AWS journey started with CloudTrail, building the foundation for visibility and accountability in the cloud. Without observability, there is no security (which is "job zero" at AWS) – and CloudTrail gives customers that critical lens into their AWS activity.

Root cause analysis of the Amazon DynamoDB Service Disruption in Northern Virginia (US-EAST-1) Region on Oct 20, 2025:

https://aws.amazon.com/message/101925/

How AWS raises the bar for Code Reviews (CRs)

In my first team at AWS CloudTrail, we had the CR Bar Raiser mechanism.

CR Bar Raisers are a group of engineers handpicked by senior and principal engineers within the org. The selection is based on your past code reviews: 1) the technical depth of your comments, 2) evidence that you are steering code authors to produce high quality, well tested and the right changes for our customers.

Every CR/PR submitted is automatically required to have at least 1 approval from the CRBR group before merging.

I was fortunate to be also elected as one of the CloudTrail Code Review Bar Raisers (CRBR).

In the upcoming few posts, I will be sharing some of my own review practices as a CRBR at AWS, so that you can also bar raise your team's codebase.

#aws #cr #pr #codereview #crbr

Code Review Bar Raiser Series, Part 1
Separate Behavioral and Refactoring Changes

Similar to everything else in Software Engineering, the code review (CR) process is all about trade-offs.
As a CR author, you probably care about 2 things:
1) ensuring your reviewers catch any bug you might have missed
2) helping them do it fast, so that you ship faster for your customers.

But in reality, you submit a two-page CR and either get an instant "LGTM!" (Looks Good To Me) in two minutes, or if you’re lucky, some thoughtful comments two days later. So you realize you’re only getting half of what you wanted, not both!

As a CRBR, it was my responsibility to help CR authors achieve both. So whenever I got a review request, the first thing I checked was: are these changes focused purely on business logic, or are they mixed with refactoring and cleanups? If it was a mix, I'd immediately ping the author and ask them to split it into two CRs: one for behavior changes, one for refactoring.

As an example, look at the following diff. At a glance, it looks like a refactoring change: a big chunk of reformatting, introduced User class, renamed parameters. But looking deeper the author is also modifying the behavior: added a duplicate check and a new exception "User already exists".

--- a/src/user_service.py
+++ b/src/user_service.py
@@ -1,22 +1,31 @@
-class UserService:
-    def __init__(self, db):
-        self.db = db
-
-    def create_user(self, username, email):
-        if not username or not email:
-            raise ValueError("Invalid input")
-        user = {"username": username, "email": email}
-        self.db.insert(user)
-        return user
-
-    def get_user(self, username):
-        return self.db.find_one({"username": username})


+import logging
+
+class User:
+    def __init__(self, name, email):
+        self.name = name
+        self.email = email
+
+class UserService:
+    def __init__(self, database):
+        self.database = database
+
+    def create_user(self, name, email):
+        if not name:
+            raise ValueError("Username cannot be empty")
+
+        # new: check for duplicates
+        if self.database.find_one({"name": name}):
+            raise ValueError("User already exists")
+
+        user = User(name, email)
+        self.database.insert({"name": user.name, "email": user.email})
+        logging.info("User created: %s", name)
+        return user

A reviewer now has to mentally separate what logic actually changed from what was just cleanup or renaming, which increases review time and risk of missing subtle behavior bugs.

And that’s exactly why I started insisting on separating behavior changes from refactors. It keeps reviews focused, and everyone (including future you and your customers) ends up happier.

#aws #crbr #cr #pr #codereview

Code Review Bar Raiser Series, Part 2
Atomic changes

Let's say you are working on a new feature, and you find a bug (at least you think is a bug) in a part of the codebase completely unrelated to feature you're working on. Then you decide to fix it in one go, and create a single CR (and potentially a single commit) with the following description:

1. Feature: Implement event payload compression for `Orders` DynamoDB table.
2. Fix: Mark Class ABC as `Singleton` to avoid instance recreation

If you would send me this CR, I would probably not approve it as a CRBR even though both changes look good.

Why? For rollbacks and operational safety.

When things go wrong in production (and they will), you want the ability to revert a single feature or bug fix without side effects. Atomic commits make this straightforward. For the example above, assume your 2nd change (marking a class as Singleton) broke production because the class was stateful. You would then need to rollback the whole revision, including the 1st change with DynamoDB compression, and potentially breaking data consumers already assuming compression on the table, causing cascading impact.

That’s why as a CRBR, I always look for atomicity in changes. Each CR should have a single, clear intent that can be safely reviewed, tested, deployed, and rolled back independently.

Mixing unrelated changes, even if both are "good", increases operational risk and reduces clarity during incidents.
Engineering excellence isn't just about writing correct code, it's also about writing operationally safe, reversible code.

#aws #crbr #cr #pr #codereview

#AI

From the Ising model to large language models, Gibbs Measures in Machine Learning offers a complete journey through one of the most powerful concepts connecting statistical physics and modern AI.

By (author): Laziz U Abdullaev (National University of Singapore, Singapore) and Utkir A Rozikov (V I Romanovskiy Institute of Mathematics, Uzbekistan)

https://www.worldscientific.com/worldscibooks/10.1142/14350#t=aboutBook

Highly recommend this AWS re:Invent session on the Oct 20, 2025 outage, presented by a Senior Principal Engineer from DynamoDB:

https://www.youtube.com/watch?v=YZUNNzLDWb8

Also, proud to have been part of the working group that launched this at AWS re:Invent this year:

Amazon CloudWatch application map un-instrumented services discovery

TL;DR
Now, CloudWatch Application map is capable of detecting and visualizing service topologies not instrumented with Application Signals, providing out-of-the-box observability coverage in your distributed environment.

re:Invent session: https://youtu.be/MQmNzKZsx44?si=oPQzxsMj1d7VVWns&t=666s

What's new post: https://aws.amazon.com/about-aws/whats-new/2025/11/amazon-cloudwatch-application-map-un-instrumented-discovery/

Bugun 19:00 da Stream qilaylik bir.

Reja quyidagicha:

1. Rust dasturlash tilida "Snake Game" yozamiz. Live coding.
2. AMA qilamiz.
3. Geoguessr da 1-2ta round o'ynaymiz.

Ko'rishguncha.

https://www.youtube.com/live/5QRFbw9BAJE

Came across a nice, interactive article by an engineer at Cloudflare on structured logging and wide events to improve observability:

https://loggingsucks.com/

My 2 cents:

Structured logging + wide events is a powerful observability pattern you can implement in your service. But the key aspect I would like to emphasize is to have that discipline in the codebase via a mechanism to follow consistency.

It is best to invest on a reusable library that provide the standardized utility methods so that developers don't need to spend their cognitive energy to maintain the consistency, and let the system enforce it in a mechanized, and repeatable way.

Why nobody is a 'Backend Engineer' at Amazon

I have talked about Amazon Leadership Principles (LPs) before in this blog. One of my favorites is Ownership:

Leaders are owners. They think long term and don’t sacrifice long-term value for short-term results. They act on behalf of the entire company, beyond just their own team. They never say “that’s not my job.”

This is highly visible in how uniquely engineering is done at Amazon. In traditional companies (at least from my experience before I joined Amazon), software engineers are usually seen as "code factory" that takes input and tasks from PMs, execute them, and throw to an army of QA engineers. Depending on the role, whether Frontend Engineer or Backend Engineer, you might also throw a follow-up task to DevOps/infra team: to create some AWS resources for your feature to work. Most teams also have a dedicated Scrum master to run Scrum meetings, manage sprints, etc.

At Amazon things work differently.

Engineers are not seen as Frontend, Backend, or Java engineers, but as "Product Engineers". Knowing the technology (Frontend, Backend, Java, Python, etc.) has always been the baseline at Amazon, which is now getting even more widespread in the AI era.

In my 3.5 years at Amazon so far, I have
- Written code in 7+ programming languages
- Created APIs, event driven async systems
- Done DevOps: infra creation, monitoring, observability
- Handled oncall: handled incidents and tickets; written COEs
- Written design docs
- Done cross-team collaboration
- Run sprints: planning, grooming, retro meetings
- Created UIs in React
- Created designs in Figma
...
- Done whatever it takes to make the product I own successful.

Fun fact: Starting recently, AWS engineers are also responsible for updating their public service documentation, a task previously done by creating a ticket for the tech writers team. Everyone on my team loved this change. Now we move faster, as this avoids back-and-forth with an external team. Also, executing usually takes less energy than coordinating with someone else to do it.

Especially as we enter 2026 in the AI era, ask yourself: "Am I a Product Engineer or still a Backend/Frontend Engineer?"

If the latter is true, now is the best time to plan your transition to becoming a Product Engineer. This will give you better understanding of user needs and business impact, and make you less vulnerable to being replaced by AI.