Highly recommend this AWS re:Invent session on the Oct 20, 2025 outage, presented by a Senior Principal Engineer from DynamoDB:

https://www.youtube.com/watch?v=YZUNNzLDWb8

Also, proud to have been part of the working group that launched this at AWS re:Invent this year:

Amazon CloudWatch application map un-instrumented services discovery

TL;DR
Now, CloudWatch Application map is capable of detecting and visualizing service topologies not instrumented with Application Signals, providing out-of-the-box observability coverage in your distributed environment.

re:Invent session: https://youtu.be/MQmNzKZsx44?si=oPQzxsMj1d7VVWns&t=666s

What's new post: https://aws.amazon.com/about-aws/whats-new/2025/11/amazon-cloudwatch-application-map-un-instrumented-discovery/

Bugun 19:00 da Stream qilaylik bir.

Reja quyidagicha:

1. Rust dasturlash tilida "Snake Game" yozamiz. Live coding.
2. AMA qilamiz.
3. Geoguessr da 1-2ta round o'ynaymiz.

Ko'rishguncha.

https://www.youtube.com/live/5QRFbw9BAJE

Came across a nice, interactive article by an engineer at Cloudflare on structured logging and wide events to improve observability:

https://loggingsucks.com/

My 2 cents:

Structured logging + wide events is a powerful observability pattern you can implement in your service. But the key aspect I would like to emphasize is to have that discipline in the codebase via a mechanism to follow consistency.

It is best to invest on a reusable library that provide the standardized utility methods so that developers don't need to spend their cognitive energy to maintain the consistency, and let the system enforce it in a mechanized, and repeatable way.

Why nobody is a 'Backend Engineer' at Amazon

I have talked about Amazon Leadership Principles (LPs) before in this blog. One of my favorites is Ownership:

Leaders are owners. They think long term and don’t sacrifice long-term value for short-term results. They act on behalf of the entire company, beyond just their own team. They never say “that’s not my job.”

This is highly visible in how uniquely engineering is done at Amazon. In traditional companies (at least from my experience before I joined Amazon), software engineers are usually seen as "code factory" that takes input and tasks from PMs, execute them, and throw to an army of QA engineers. Depending on the role, whether Frontend Engineer or Backend Engineer, you might also throw a follow-up task to DevOps/infra team: to create some AWS resources for your feature to work. Most teams also have a dedicated Scrum master to run Scrum meetings, manage sprints, etc.

At Amazon things work differently.

Engineers are not seen as Frontend, Backend, or Java engineers, but as "Product Engineers". Knowing the technology (Frontend, Backend, Java, Python, etc.) has always been the baseline at Amazon, which is now getting even more widespread in the AI era.

In my 3.5 years at Amazon so far, I have
- Written code in 7+ programming languages
- Created APIs, event driven async systems
- Done DevOps: infra creation, monitoring, observability
- Handled oncall: handled incidents and tickets; written COEs
- Written design docs
- Done cross-team collaboration
- Run sprints: planning, grooming, retro meetings
- Created UIs in React
- Created designs in Figma
...
- Done whatever it takes to make the product I own successful.

Fun fact: Starting recently, AWS engineers are also responsible for updating their public service documentation, a task previously done by creating a ticket for the tech writers team. Everyone on my team loved this change. Now we move faster, as this avoids back-and-forth with an external team. Also, executing usually takes less energy than coordinating with someone else to do it.

Especially as we enter 2026 in the AI era, ask yourself: "Am I a Product Engineer or still a Backend/Frontend Engineer?"

If the latter is true, now is the best time to plan your transition to becoming a Product Engineer. This will give you better understanding of user needs and business impact, and make you less vulnerable to being replaced by AI.

'Greedy' Career Path
Part 1
🇺🇸

Greedy is a class of algorithms, making a locally optimal choice at each step, hoping it leads to a globally optimal solution.

Growing with passion in mathematics and computers, I had this question in early 2018 when starting my studies at university: what's next? Well, I was going to become a software engineer. But I specifically wanted to go into a subfield that blends with mathematics. I believe that the closer a subject is to mathematics, the closer it is to life. Web Development, which was popular at the time, did not align with this view.

There was already a hype around AI/ML, so I learned that's probably where I wanted to go, both engineering and mathematics combined. The problem was that at the time, there were close to 0 job opportunities, at least for starters. In late 2018, somewhat "luckily", a company had a guest talk at the university. They said they were creating an AI division in Uzbekistan, headed by a Swiss AI expert, and would start accepting applications for an internship program. That felt like a perfect opportunity. I passed the interviews and was accepted to their "internship preparation" program, which lasted for 6 months. They just told us to study some AI/ML courses on Coursera. At the end, they conducted a final exam with questions on mathematics and ML foundations. I waited for 1 month and was told I scored high. I was among the 3 out of 10 who passed. That felt like a dream coming true, and I was waiting for the next steps...silence. Another month passed...

However, and maybe not unexpectedly, they sent an email, saying they no longer needed AI interns, as "there is not enough AI work to do" in the project. With that, I had spent my 9 months, and this email was the dead end of my dreams for a career in AI.

To be continued...

#career #growth #ai

'Greedy' Karyera Yo‘li
1-qism
🇺🇿

Greedy — bu algoritmlar bir turi bo'lib, unda har bir qadamda mahalliy eng yaxshi tanlov qilinadi, va bu tanlov oxir-oqibat global eng yaxshi yechimga olib borishiga umid qilinadi.

Matematika va kompyuterlarga bo‘lgan qiziqish bilan ulg‘ayganim sababli, 2018 yil boshida menda bir savol paydo bo‘lgan edi: keyingi qadam nima? Javob oddiydek tuyulardi — men dasturiy ta’minot muhandisi bo‘laman. Ammo aniqrog‘i, matematika bilan chambarchas bog‘liq bo‘lgan yo‘nalishni tanlashni xohlardim. Menimcha, fan qanchalik matematikaga yaqin bo‘lsa, shunchalik hayotga ham yaqin bo‘ladi. O‘sha paytda juda ommabop bo‘lgan Web Development esa bu qarashimga unchalik mos kelmasdi.

Sun'iy intellekt sohasi (AI) atrofida allaqachon katta shov-shuv bor edi, shuning uchun muhandislik va matematikani birlashtirgan aynan shu soha men uchun to‘g‘ri yo‘l bo‘lsa kerak, degan xulosaga keldim. Muammo shundaki, o‘sha vaqtda, ayniqsa yangi boshlovchilar uchun, deyarli hech qanday ish imkoniyatlari yo‘q edi.
2018 yil oxirida esa biroz “omadli” tarzda, bir kompaniya universitetimizda mehmon ma’ruza o‘tkazdi. Ular O‘zbekistonda sun’iy intellekt bo‘limini ochayotganini, unga shveysariyalik AI mutaxassisi rahbarlik qilishini va tez orada amaliyot (internship) dasturiga qabul boshlanishini aytishdi. Bu men uchun ideal imkoniyatdek tuyuldi.
Suhbatlardan muvaffaqiyatli o‘tdim va ularning 6 oy davom etadigan “internshipga tayyorlov” dasturiga qabul qilindim. Asosan bizga Coursera’dagi AI/ML kurslarini mustaqil o‘rganish topshirildi. Dastur oxirida matematika va mashinaviy o‘rganish asoslariga oid yakuniy imtihon bo‘ldi. Bir oy kutdim va natijam yuqori ekanligi aytildi. 10 kishidan atigi 3 nafari muvaffaqiyatli o‘tgan edi — men ham ular orasida edim. Bu orzularim ushalayotgandek tuyuldi. Keyingi qadamlarni intiqlik bilan kutdim… ammo sukut. Yana bir oy o‘tdi…

Va kutilganidek (yoki kutilmaganda), email keldi: ular endi AI amaliyotchilariga ehtiyoj yo‘qligini, loyiha doirasida “yetarlicha AI ishi yo‘qligi”ni aytishdi. Shu bilan, 9 oyim sarflandi va bu xat mening AI sohasidagi karyera orzularim uchun berk ko‘cha bo‘lib tuyuldi.

Davomi bor…

#career #growth #ai

'Greedy' Career Path
Part 2
🇺🇸

That closed door was a good learning to open the right next door for my career. I had 2 paths that I had considered next:

Path 1: continue with AI, thinking long term with the hope that AI will take off one day, where I could be part of the ship, with the caveat of having 0 real opportunities at that moment.

Path 2: pivot to Web Development, hundreds of immediate opportunities, with peers (as students) already earning 2-3x more than country's average salaries, with the caveat that this work seemed automatable in the quite near future and eventually a saturated market.

I applied what we call in Computer Science a greedy algorithm, optimizing for the immediate best option at that moment: Web Development. Fortunately, a close friend from university, who had already been successful as a Web Developer, referred me for my first Web Development job at the company he was working at. From there, things started moving fast.

Looking back, that locally optimal choice opened doors and a career path I couldn't have imagined, that eventually brought me to Amazon as a Software Engineer. You can read more about my path to Amazon here.

But did the greedy algorithm actually work? Or did I just get lucky?

I will answer that in Part 3.

#career #growth #ai

'Greedy' Karyera Yo'li
2-qism
🇺🇿

O'sha yopilgan eshik karyeramda keyingi to'g'ri eshikni ochish uchun yaxshi saboq bo'ldi. Buyog'iga oldimda 2 ta yo'l bor edi:

1-yo'l: Uzoqni ko'zlagan holda AI bilan davom etish: "bir kun kelib AI rivojlanib ketganda, men uning bir qismi bo'lishim mumkin" degan fikr bilan. Lekin, o'sha paytlarda ish bozorida hech qanday real imkoniyatlar yo'q edi.

2-yo'l: Web Development ga o'tish: ish bozorida talab katta, yuzlab vakansiyalar mavjud, tengdoshlarim (talabalik davrida) allaqachon o'rtacha oyliklardan 2-3 baravar ko'proq topishayotgan edi. Lekin bu ish yaqin kelajakda avtomatlashtirilib, oxir-oqibat to'yingan bozorda talab yo'qolib ketish xavfi bor edi.

Men informatika (Computer Science) da greedy algoritm deb ataluvchi algoritmni qo'lladim, ya'ni o'sha paytdagi eng yaxshi variantni tanladim: Web Development. Yaxshiyamki, universitetdagi yaqin do'stim, allaqachon Web Developer sifatida muvaffaqiyatli ishlab kelayotgan edi va u meni o'zi ishlayotgan kompaniyaga ishga tavsiya qildi. Bu mening birinchi Web Development'dagi ishim edi. O'sha ondan boshlab, ish faoliyatim tez sur'atlarda rivojlana boshladi...

Orqaga qarasam, o'sha lokal optimal tanlov men tasavvur qila olmagan eshiklar va karyera yo'lini ochdi, natijada meni Amazon'ga dasturchi (Software Engineer) sifatida olib kelgandi. Amazon'ga yo'limni bu yerda batafsil o'qishingiz mumkin.

Lekin greedy algoritm haqiqatan ham ish berdimi? Yoki shunchaki omadim keldimi?

Bunga 3-qismda javob beraman.

#career #growth #ai

Friday deployments

Today is Friday and engineers finish their day hoping to have incident-less weekend ahead.

Engineers all know it is not a good idea to deploy software on Fridays (for obvious reasons). In most places, it is a good intention: "hey, we don't deploy on Fridays, please", and someone still does.

At Amazon, we don't rely on good intentions, instead we rely on guardrails (more on that here). All CI/CD pipelines that deploy to production are automatically blocked across all teams on Fridays and outside business hours.

I'm the oncall for my team this week, and thanks to this guardrail, I know I won't get a weekend incident caused by a teammate deploying a bug without my awareness.

Have a nice weekend!

#amazon #aws #operations #oncall #monitoring #cicd

'Greedy' Career Path
Part 3
🇺🇸

Coming back to the question from Part 2: "Did the greedy algorithm actually work? Or did I just get lucky?"
Answer: it wasn't luck; the greedy algorithm worked, but in ways I never expected.

If you still remember, I was rejected from an AI internship in 2018, because "there was no AI work to do". Fast forward to 2026, I'm at AWS, and we are being told "if you are not using AI for your job, you literally don't belong here". If we are not using AI, we are under-performing. Not only that, over the last year me and my team built an agentic AI pipeline that collects data, generates artifacts, and evaluates their quality, completely automating a big piece of manual work. Basically, we built a team of agents working for us overnight, and lets us focus on novel problems.

The path I walked away from a few years ago suddenly became my daily work. I couldn't have gotten here by waiting for AI in 2018. In 2018, I optimized for what was visible: job opportunities, immediate growth, tangible skills. Web Development taught me to deliver business value. It taught me how systems work at scale. It brought me to Amazon. And Amazon put me right in the center of AI opportunities at scale. The 2018 version of me couldn't have planned this path. He could only take the best step available at that moment.

If you're facing a "passion vs. pragmatic" decision, just know that you are not abandoning your goals by applying a greedy algorithm, and it can eventually lead to the global optimum.

The algorithm worked once. Maybe it's time to run it again: switch to AI research?

#career #growth #ai

'Greedy' Karyera Yo'li
3-qism
🇺🇿

2-qismdagi savolga qaytamiz: "Greedy algoritm haqiqatan ham ishladimi? Yoki shunchaki omadim keldimi?"
Javob: bu omad emas edi; greedy algoritm ishladi, lekin men kutmagan yo'llar bilan.

Agar hali esingizda bo'lsa, 2018-yilda AI internship'dan rad etishgan edi, chunki "AI da qilishga ish yo'q edi". 2026-yilga kelib, men AWS'daman va bizga "agar siz ishingizda AI'dan foydalanmayotgan bo'lsangiz, bu yerga tegishli emassiz" deyishmoqda. Agar biz AI'dan foydalanmasak, past natija ko'rsatyapmiz demakdir. Bundan tashqari, o'tgan yil davomida men va jamoam ma'lumot to'playdigan, artefaktlar yaratadigan va ularning sifatini baholaydigan agentic AI pipeline qurdik, bu katta hajmdagi qo'lda bajariladigan ishlarni to'liq avtomatlashtirdi. Oddiy qilib aytganda, o'zimiz uchun tun-u kun ishlaydigan agentlar jamoasini yaratdik va bu bizga yangi muammolarga e'tibor qaratish imkonini berdi.

Bir necha yil oldin tark etgan yo'lim to'satdan kundalik ishimga aylandi. 2018-yilda AI'ni kutib o'tirib bu yerga yetib kelolmasdim. 2018-yilda men ko'rinadigan narsalarni optimallashtirdim: ish imkoniyatlari, darhol o'sish, aniq ko'nikmalar. Web Development menga biznesga foydali bo'lishni o'rgatdi. U menga tizimlar qanday katta miqyosda ishlashini o'rgatdi. U meni Amazon'ga olib keldi. Va Amazon meni katta miqyosdagi AI imkoniyatlarining markaziga qo'ydi. 2018-yildagi "men" bu yo'lni rejalashtira olmas edim. "U" faqat o'sha paytdagi eng yaxshi qadamni qo'ya olardi.

Agar siz "ishtiyoq yoki pragmatik" qaror oldida turgan bo'lsangiz, shuni bilingki, greedy algoritmni qo'llash orqali maqsadlaringizdan voz kechayotganingiz yo'q, va u oxir-oqibat global optimumga olib kelishi mumkin.

Algoritm bir marta ishladi. Ehtimol uni yana ishga tushirish vaqti kelgandir: AI research'ga o'tishim kerakmi?

#career #growth #ai

States of Mind

"Bir aylanib kelsam yaxshiroq g'oyalar kela boshlaydi", - kabi gaplarni ko'pchilikdan eshitgansiz.

Aslida bu tasodif emas. Miyamizda Default Mode Network (DMN) deb ataluvchi neural tarmoq e'tiboringizni tor topshiriqqa qaratmay erkin qo'yganingizda miyangizdagi o'y-xayollarga mas'ul bo'ladi.

Faraz qiling miyangizda juda ko'p g'oyalar mavjud (states). Ammo, ikki g'oya orasidagi bog'lanish aniq emas. Berilgan masalani ishlash uchun esa o'sha state'lar ichidan yechimga olib boruvchi yo'l (path) topish kerak.

Ongli ravishda unday katta grafdan kerakli boshlang'ich nuqtani topish ham, keyingi to'g'ri state'ni tanlash ham mushkul (yechim qotib qoldi).

"Bir aylanib kelganda" esa DMN miya state'lari orasida tasodifiy yurish (random walk) amalga oshiradi. Bu esa o'z aqlingiz bilan tanlashingiz ehtimoli kam bo'lgan state'larning tanlanish ehtimolini orttiradi.

Natijada, tasodifan boshi berk ko'chadan chiqish yo'li ko'rina boshlaydi.

___

Large Reasoning Models (LRM) ham shunga tabiatan o'xshash usulda boshqarilishi mumkin. Bunda keyingi so'z bashorat qilinayotgan (next token prediction) taqsimot sun'iy ravishda o'tkir (peaky) taqsimotdan yoyiqroq (uniform) taqsimotga temperature annealing usulida o'tkaziladi. Bu esa tanlanishi mumkin bo'lgan keyingi so'zlar to'plamini ancha kengaytirishga xizmat qiladi.

@lazizabdullaev

Got a Google offer

...and I rejected it.

I interviewed with Google around a year ago, passed all their 4 interview rounds, and received an offer for their Munich office.

Total compensation was about 20% higher than what I was making in Berlin. But Munich is at least 20% more expensive.

Moving for essentially the same compensation didn't make sense. I'm happy with my work at Amazon, and the Google offer just wasn't compelling enough to justify the move.

Google'dan offer oldim

...va rad etdim.

Taxminan bir yil oldin Google bilan intervyu o'tkazdim, ularning 4 ta intervyu bosqichini muvaffaqiyatli o'tdim va Myunxen ofisi uchun ish taklifi (offer) oldim.

Umumiy taklif qilingan maosh Berlindagi maoshimdan taxminan 20% yuqori edi. Lekin Myunxen yashash xarajatlari Berlindan kamida 20% qimmat.

Deyarli bir xil maosh uchun Myunxenga ko'chish mantiqiy emas edi. Men Amazondagi ishimdan mamnunman va Google taklifi ko'chishni oqlaydigan darajada jozibador ko'rinmadi.

Designing systems for 'unknown unknowns'
Part 1
🇺🇸

In large scale distributed systems, anything can fail at some point. So as part of the system design, engineers are expected to list the anticipated failure modes and how their design mitigate and recover from them. So we cover the failure modes that we know with automated tests.

However, that is limited by only what is known. When the system is deployed to the wild, there will anyways be stuff that fails in unexpected ways, meaning failure is inevitable because of unknown unknowns.

What do we do with them?

We acknowledge we cannot prevent all failures with tests and now what matters for these failures is how fast we recover from them. There are standard metrics that measure this more formally:

- MTTD (mean time to detection): how quickly do we find out something failed?
- MTTA (mean time to acknowledgement): how quickly do we start responding?
- MTTR (mean time to recovery): how quickly we recover.

For example,

MTTD = sum(detection_time_i - incident_start_time_i) / number of incidents. 

The smaller these numbers are, the more resilient your system is to failures.

How do we make these numbers smaller?

I will answer that in Part 2.

#distributedsystems #operations #aws #operationalexcellence #testing