My 3 years at Amazon (so far): 3 key lessons learned

I joined Amazon on September 1, 2022 — and today celebrating my 3 years of Day 1’s 🎉.

These 3 years have been AWSome — full of growth and valuable experiences, some shaped by lessons learned from mistakes.

Today I wrote an article on the most important lessons I learned along this journey so far.

👉🏻 Read the article: https://abdullaev.dev/my-3-years-at-amazon-3-key-lessons-learned/

A conversation with a Senior engineer about a broken code library

Me: Hi, do you know who owns this library? I need some help with an issue.

Senior engineer: "Remember: success has many fathers, but failure has none"

Me: 🫠

🔎 The Search Engine of AWS

When I first started using AWS (as a customer) back in early 2020, I was super eager to try out many of the services in different AWS regions. Honestly, it's hard not to - just look at the gallery of shiny service icons!

I was cautious with my resource usage to not to breach my free tier limits. Luckily, almost each AWS service has its own AWS Console UI that I used to search for my created resources in that service and delete them when not needed. But over time, I experimented with multiple services and regions and I forgot which specific service console and in which region I needed to go to delete the unnecessary resources.

I had noticed the search bar on the top, but it had only worked for searching AWS service names and the documentation.
So I couldn't get an easy answer to a question like "What are all my EC2 instances created across all regions in my account?"... back in 2020.

In 2022, AWS launched a service called AWS Resource Explorer, which gave customers the Google-like search experience for the actual AWS resources. It supports search across all regions, and across all accounts in your AWS Organization in a single console, and also via the search bar on the top 🎉.

Today, I'm proud to say that I currently work (as my 2nd team) in one of the teams part of AWS Resource Explorer powering core functionalities like resource discovery, helping AWS customers (like myself from 5 years ago 🙂), to search and discover their AWS resources.

React with 👌 if you'd like to read a similar post about my first team at AWS.

🚨 The Police of AWS

When I joined AWS, my very first team was AWS CloudTrail – the service that records API activity across your account, giving you visibility into who did what, when, and how.

As a customer back in 2020, I often asked myself questions like:

Who created this IAM role?

When was this S3 bucket policy changed?

Why do I have EC2 instances running when I don’t remember starting them?


At that time, I discovered CloudTrail and realized it’s basically the “security camera system” of AWS. It tracks every API call across services, whether from the console, CLI, SDKs, or even from services acting on your behalf.

When I joined CloudTrail as an engineer as my first team at AWS, I got to work in the Integrations Monitoring team. As soon as I joined, I started designing a new proactive integration health monitoring system between AWS services and CloudTrail and led its design, implementation and global deployment across 30+ AWS regions worldwide.

Today, I’m proud that my AWS journey started with CloudTrail, building the foundation for visibility and accountability in the cloud. Without observability, there is no security (which is "job zero" at AWS) – and CloudTrail gives customers that critical lens into their AWS activity.

Root cause analysis of the Amazon DynamoDB Service Disruption in Northern Virginia (US-EAST-1) Region on Oct 20, 2025:

https://aws.amazon.com/message/101925/

How AWS raises the bar for Code Reviews (CRs)

In my first team at AWS CloudTrail, we had the CR Bar Raiser mechanism.

CR Bar Raisers are a group of engineers handpicked by senior and principal engineers within the org. The selection is based on your past code reviews: 1) the technical depth of your comments, 2) evidence that you are steering code authors to produce high quality, well tested and the right changes for our customers.

Every CR/PR submitted is automatically required to have at least 1 approval from the CRBR group before merging.

I was fortunate to be also elected as one of the CloudTrail Code Review Bar Raisers (CRBR).

In the upcoming few posts, I will be sharing some of my own review practices as a CRBR at AWS, so that you can also bar raise your team's codebase.

#aws #cr #pr #codereview #crbr

Code Review Bar Raiser Series, Part 1
Separate Behavioral and Refactoring Changes

Similar to everything else in Software Engineering, the code review (CR) process is all about trade-offs.
As a CR author, you probably care about 2 things:
1) ensuring your reviewers catch any bug you might have missed
2) helping them do it fast, so that you ship faster for your customers.

But in reality, you submit a two-page CR and either get an instant "LGTM!" (Looks Good To Me) in two minutes, or if you’re lucky, some thoughtful comments two days later. So you realize you’re only getting half of what you wanted, not both!

As a CRBR, it was my responsibility to help CR authors achieve both. So whenever I got a review request, the first thing I checked was: are these changes focused purely on business logic, or are they mixed with refactoring and cleanups? If it was a mix, I'd immediately ping the author and ask them to split it into two CRs: one for behavior changes, one for refactoring.

As an example, look at the following diff. At a glance, it looks like a refactoring change: a big chunk of reformatting, introduced User class, renamed parameters. But looking deeper the author is also modifying the behavior: added a duplicate check and a new exception "User already exists".

--- a/src/user_service.py
+++ b/src/user_service.py
@@ -1,22 +1,31 @@
-class UserService:
-    def __init__(self, db):
-        self.db = db
-
-    def create_user(self, username, email):
-        if not username or not email:
-            raise ValueError("Invalid input")
-        user = {"username": username, "email": email}
-        self.db.insert(user)
-        return user
-
-    def get_user(self, username):
-        return self.db.find_one({"username": username})


+import logging
+
+class User:
+    def __init__(self, name, email):
+        self.name = name
+        self.email = email
+
+class UserService:
+    def __init__(self, database):
+        self.database = database
+
+    def create_user(self, name, email):
+        if not name:
+            raise ValueError("Username cannot be empty")
+
+        # new: check for duplicates
+        if self.database.find_one({"name": name}):
+            raise ValueError("User already exists")
+
+        user = User(name, email)
+        self.database.insert({"name": user.name, "email": user.email})
+        logging.info("User created: %s", name)
+        return user

A reviewer now has to mentally separate what logic actually changed from what was just cleanup or renaming, which increases review time and risk of missing subtle behavior bugs.

And that’s exactly why I started insisting on separating behavior changes from refactors. It keeps reviews focused, and everyone (including future you and your customers) ends up happier.

#aws #crbr #cr #pr #codereview

Code Review Bar Raiser Series, Part 2
Atomic changes

Let's say you are working on a new feature, and you find a bug (at least you think is a bug) in a part of the codebase completely unrelated to feature you're working on. Then you decide to fix it in one go, and create a single CR (and potentially a single commit) with the following description:

1. Feature: Implement event payload compression for `Orders` DynamoDB table.
2. Fix: Mark Class ABC as `Singleton` to avoid instance recreation

If you would send me this CR, I would probably not approve it as a CRBR even though both changes look good.

Why? For rollbacks and operational safety.

When things go wrong in production (and they will), you want the ability to revert a single feature or bug fix without side effects. Atomic commits make this straightforward. For the example above, assume your 2nd change (marking a class as Singleton) broke production because the class was stateful. You would then need to rollback the whole revision, including the 1st change with DynamoDB compression, and potentially breaking data consumers already assuming compression on the table, causing cascading impact.

That’s why as a CRBR, I always look for atomicity in changes. Each CR should have a single, clear intent that can be safely reviewed, tested, deployed, and rolled back independently.

Mixing unrelated changes, even if both are "good", increases operational risk and reduces clarity during incidents.
Engineering excellence isn't just about writing correct code, it's also about writing operationally safe, reversible code.

#aws #crbr #cr #pr #codereview

#AI

From the Ising model to large language models, Gibbs Measures in Machine Learning offers a complete journey through one of the most powerful concepts connecting statistical physics and modern AI.

By (author): Laziz U Abdullaev (National University of Singapore, Singapore) and Utkir A Rozikov (V I Romanovskiy Institute of Mathematics, Uzbekistan)

https://www.worldscientific.com/worldscibooks/10.1142/14350#t=aboutBook

Highly recommend this AWS re:Invent session on the Oct 20, 2025 outage, presented by a Senior Principal Engineer from DynamoDB:

https://www.youtube.com/watch?v=YZUNNzLDWb8

Also, proud to have been part of the working group that launched this at AWS re:Invent this year:

Amazon CloudWatch application map un-instrumented services discovery

TL;DR
Now, CloudWatch Application map is capable of detecting and visualizing service topologies not instrumented with Application Signals, providing out-of-the-box observability coverage in your distributed environment.

re:Invent session: https://youtu.be/MQmNzKZsx44?si=oPQzxsMj1d7VVWns&t=666s

What's new post: https://aws.amazon.com/about-aws/whats-new/2025/11/amazon-cloudwatch-application-map-un-instrumented-discovery/

Bugun 19:00 da Stream qilaylik bir.

Reja quyidagicha:

1. Rust dasturlash tilida "Snake Game" yozamiz. Live coding.
2. AMA qilamiz.
3. Geoguessr da 1-2ta round o'ynaymiz.

Ko'rishguncha.

https://www.youtube.com/live/5QRFbw9BAJE