General Advice When working with long running jobs

When you are working with long running jobs specially that involve a lot of data, it is recommended to have the following:

- Make sure to have Exception handling in place in every step of the job.
- Make sure to use logging in every step of the job and log that to a file.
- Make sure to use checkpoints in your job. This will help you to recover from failures.
- In case of failures, make sure the job is restarted from the last successful checkpoint. This will help you to avoid reprocessing of data.

- If the processed data have their processed status stored in a database, make sure to check the status of the data before processing it again.

- If time permits, try to sleep for a few seconds before retrying the job. This will help you to avoid overloading the system. Use exponential backoff for retrying the job.

- If you have 3rd party API calls in your job and if there might be similar calls for the same data, make sure to cache the data in memory and use it for subsequent calls. This will help you to avoid making the same API calls again and again.

- Make sure to document your work and comment your code. This will help you and others to understand the code in future.

- Setup a monitoring system for your job. This will help you to monitor the job and also help you to debug the job in case of failures.

You can use tools like Prometheus and Grafana for monitoring your job. You can also use some kind of notification system like Telegram Bot, Email, SNS, Slack to notify you incase of failures.


Conclusion

One of the most important things to remember when working with long running jobs is to make sure that the job is fault tolerant. This will help you to avoid reprocessing of data and also help you to avoid overloading the system.

Add your own tips in the comments section below.

Do you know Telegram bot can only upload 50MB limit ?

But what if you want to upload more than 50 MB using your bot. Well am here To tell you that you can increase 50MB size to 2000MB.

Docs available [HERE]

Helper script to host it.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install make git zlib1g-dev libssl-dev gperf cmake g++
git clone --recursive https://github.com/tdlib/telegram-bot-api.git
cd telegram-bot-api
rm -rf build
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=.. ..
cmake --build . --target install
cd ../..
ls -l telegram-bot-api/bin/telegram-bot-api*

After you do this you need to logout of the cloud telegram server using the below 👇

curl https://api.telegram.org/bot{TOKEN}/logOut

Now instead of calling to the telegram API server you make the call to your Local server using

https://LOCAL SERVER/bot{TOKEN}/method

Now you can lift a lot of limitations and you can enjoy the super power of telegram bot.

Please read more about it

DOCS
GITHUB
TOOL

Do you know in Telegram Two Bots can't talk to each other at all?

Not even see other bot messages.

Which means if your have 2 bots in the same group one bot can't see the other bots message!

The reason is if the bot start talking to eachother they might keep talking forever 😆 (nonstop 🦜)

So for those reasons Telegram made this restriction.

Shadow Testing

Shadow testing is one approach to reduce risks before going to production.

Shadow Testing reduces risks when you consider replacing the current environment (V-Current) with candidate environment with new feature (V-Next). This approach is monitoring and capturing differences between two environments then compare and reduces all risks before you introduce a new feature/release.

https://youtu.be/ckYBLT1VG-M?si=x5TSmYEhtBxZy60l

Amazing concept from Prime.

Btw make sure to give him a Follow. He is so amazing

Amazing Spotify clone app.

Check this GitHub

They have app for Android, Apple, Windows and Linux.

🌃 Features

🚫 No ads, thanks to the use of public & free Spotify and YT Music APIs
⬇️ Downloadable tracks
🖥️ 📱 Cross-platform support
🪶 Small size & less data usage
🕵️ Anonymous/guest login
🕒 Time synced lyrics
✋ No telemetry, diagnostics or user data collection
🚀 Native performance
📖 Open source/libre software
🔉 Playback control is done locally, not on the server

https://github.com/KRTirtho/spotube

Can you live without Docker ?


I think recently the hardest thing for me might be moving out of docker. The amount of work i do to use or test something is like instant due to docker.

Specially all my Database are on docker Postgres, MySQL and you name it, it is on docker.

Docker is just a huge part of my life now.

How about you ?

I have just posted first session of Backend mentorship video to your in RAW. NO EDIT.

Check it out if you want: https://youtu.be/-GZVJJzBzA0

Good morning my people 🌞,


I think reading book is so underrated now a
days.

People think books are outdated so they advice other to not read books.

I think differently in this matter. I think books are still the best way to learn stuff. Even course doesn't come close to books when it comes to learning (Good books) for me.

To the contrary sometimes Books might be too outdated they might teach you something that you won't use at all.

Regardless, I still root for Programming books than Blog, course, or YouTube Video. But that doesn't mean you don't need those material. I think we need courses and others too. They are just something that comes after book's tho.

I use Book as a primary and the other material to interact more to the other people experience.

Anyway Read Book.

You have a chance to win a prize if you fill out the Python Developers Survey 2023! Fill it out today to get your name in the hat 🪄🎩🎁

p.s. if you think snakes are cute: image search "snakes wearing hats" 🥹 you're welcome! #python

https://survey.alchemer.com/s3/7554174/python-developers-survey-2023

Source: https://twitter.com/ThePSF/status/1749707292814061651?t=8Fmw9yJ_o8Bg3Epea0TeWA&s=19

Somebody please finish the rest of the Quote.


Enjoy this music my people https://youtu.be/IfCuarZqJzM?si=LekT-cnCscTaqd3p

A knife in my heart couldn’t slow me down
‘Cause power is power, my fire never goes out
I rise from my scars, nothing hurts me now
‘Cause power is power, now watch me burn it down

Coding Skill Doesn't make your company successful.

Offical recognition for 3k members.


There might be some giveaways 😜

Respond to Baki For My Post on Programming Books.

FYI: This is just conversation. NO HATE only educated people talking on the matter of the above title.

I think books are for people who want to learn and apply those things to their workflow to improve their skills. In life there are many things we forget to remember but apply them unconsciously. For example i don't think you can name all error syntax yet you know one when you see it. Reading makes you smarter by injecting this kind of mindset to the reader.

I don't think people are expected to name all of the stuff they read about. They rather expected to explain what they got from the book and how they applied it to their workflow. I think this is the most important part of reading books.

In fact with his argument below

"I'm not saying you shouldn't read, but if you don't apply what you learn, then you've wasted your time."

👆 I don't think you can remember the code you wrote a year ago for your company so what is the point of writing code if you are not going to remember it, YOU SHOULD QUIT PROGRAMMING. right?

So people should read whether it is blog, books, and other educational materials.

Another thing he said was "

For me, action is more important, and I learn best through trial and error"

How can a person expected to walk in a dark room without a light? Regardless you will always fail unless you just know your stuff haha. You must have a starting point and that is where these kind of materials come in handy. Books are just one part of the learning process. Blogs can be a really good source of information too Yet Reading is a part of the learning process.

May be not many people achieve financial freedom by reading books "Rich Dad Poor Dad" but it is a good starting point for people who want to learn about financial freedom. I am sure you might have read, watch or listen about How can one create programming language, Have you created one? or another argument might be am sure you know how to avoid syntax error, yet you still make them. Does that mean knowing how to avoid syntax error is useless?

Anyways, I think whether or not people in programming like reading books(or others), I suggest you fight yourself to read stuff. At some point of your life this skill come in handy where you can read and understand things faster.

Have a beloved day.

Yes am coding 😜

This might be one of the best sitcom ever.

I just love this sooo mush

history is written tonight

https://t.me/samsonendale

I don't know who needs to hear this but as a student you have more time than after graduation so use your time properly. Improve yourself as much as possible.