There is a common misconception that GIL was invented to protect developers from problems with concurrent access to data. But this is not true.

GIL, of course, will prevent you from parallelizing an application using threads (but not processes). Simply put, GIL is a lock that must be taken before any access to Python (not that important if Python code is executing or calls using Python C API). Therefore, GIL will protect internal structures from non-consistent states, but you will have to use synchronization primitives like in any other language.

#python

The popular method to declare an abstract method in Python is to use NotImplentedError exception:

def func(self):
    raise NotImplementedError

Though it's pretty popular and even has IDE support (Pycharm considers such a method to be abstract) this approach has a downside. You get the error only upon method call, not upon class instantiation.

Use abc module to avoid this problem:

from abc import ABCMeta, abstractmethod
class Service(metaclass=ABCMeta):
    @abstractmethod
    def func(self):
        pass

#python

In Python if there are no references to the object, it is destroyed immediately, instead of waiting for garbage collection. GC is needed for complicated cases when we have cyclic references.

#python

The list may contain itself. Python detects this and does not loop in the output.

>>> a = []
>>> a.append(a)
>>> a
[[...]]

#python

Python doesn't have tail recursion optimization, not because Guido couldn't handle it, but because he doesn't want to overcomplicate things.

If you really want to, you can implement a Y-combinator with optimization and use it.

#python

Just like tail recursion in Python, Guido wanted to avoid getting lambda functions in the language. The lambda function is just sort of syntactic sugar. It creates an unnamed function - that's literally it. There is no magic to it.

Those two are equivalent:

foo = lambda x: x * 2

def foo(x): 
   return x * 2 
foo.__qualname__ = '<lambda>' 

We all know the lambda functions were introduced in the language, but there was a long fight behind it, which Guido did not win. Lambda still exists because it is extremely convenient and nothing else.

#python

A function can have two returns called. For example:

def foo():
    try:
        return 1
    finally:
        return 2

2 will be returned here.

#python

There are compilers for Python code. And not only JIT ones like Numba, but also ordinary ones. For example, Cython, Nuitka that compiles your Python code into true machine instructions rather than interpreted.

Why?

Basically for the sake of both performance gains and a more portable runtime. So basically it's for the guys who shouldn't use Python in the first place.

#python

Remember I told you about numbers from -5 to 255 interned. That is, the numbers are preliminary placed and cached in memory. So, there is more to it.

You can access this memory if you want and change value. Say, change literal 4 to value 5. I'll probably be damned by many people for this post, but here is a sample code:

>>> import ctypes
>>> ctypes.memmove(id(4) + 24, id(5) + 24, 8)
>>> print(2 * 2) # 5

Have fun debugging!

P.S. Try to replace 0 with 1

#python

Let's talk about a couple of core concepts in data space and how they relate to each other and how they started.

⚡️ACID vs BASE: Comparison of two Design Philosophies

⚡️CAP and PACELC theorems in plain English

Google has made its AutoML algorithm public🤯

https://github.com/google/model_search

#ml

Recent Advances in Language Model Fine-tuning

By Sebastian Ruder:

https://ruder.io/recent-advances-lm-fine-tuning/

Apache Software Foundation (ASF) has announced the retirement to its "Attic" of at least 19 open source projects, 13 of which are big data-related and ten of which are part of the Hadoop ecosystem.

https://www.zdnet.com/article/apache-software-foundation-retires-slew-of-hadoop-related-projects/

I've really never used any of these projects, but could this be a signal that the Hadoop era is coming to an end?

It's another proof that one shouldn't build one's career around a particular tool, but rely on concepts and understanding of the industry. Technology is a very volatile thing, especially in fast-paced industries like Big Data and Data Science. Especially when industry consolidation is still in process.

#big_data

Kafka Visualization

https://softwaremill.com/kafka-visualisation/

#big_data

To build this feature store, DoorDash benchmarked several key-value stores—Redis, Cassandra, CockroachDB, ScyllaDB, and YugabyteDB—before sticking with Redis. More details on their benchmarking process and optimizing Redis in their excellent write-up.

#ml

Learn concepts not frameworks

In the world of engineering and data analytics, enthusiastic people often have strong opinions about using a particular platform. We've all come across such a person — someone who zealously promotes Apache Spark or pushes for all data work to be done using HQL. There's a strong emphasis on the tools and not so much on the problem-solution pair.

Sometimes this behavior is driven by a desire for standardization, aligning team skills, and simplification of the hiring process. Those are certainly valid considerations. But more often than not, I've seen people zealously defend a tool simply because they are passionate about it or, even worse, have built their professional careers around it.

I wish concepts were emphasized a bit more. In the real world, knowing Java doesn't make you a programmer.

Knowing C++ syntax doesn't make you a programmer.

Knowing concepts is what makes you a great programmer. Problem solving skills make you a great programmer.

Simply put, you should never build your professional career around a tool. Tools and applications come and go, and what's hot today may go bad tomorrow. That alone should give you a reason to pause before advocating for technology too quickly. Ask anyone who has worked with JavaScript.

The Silver Bullet Syndrome - Hadi Hariri

#whining

Caching... There is so much in that word - the pain of invalidation and the joy of reusing computation. In Spark, this is known as an optimization technique...

https://luminousmen.com/post/explaining-the-mechanics-of-spark-caching

A good conversation is like a miniskirt, short enough to retain interest but long enough to cover the subject — Celeste Headlee

When your job hinges on how well you talk to people, you learn a lot about how to have conversations -- and that most of us don't converse very well. I came across this Ted talk on how to have a better conversation.

Some of this may seem too obvious, but you have to admit there are some things here that you knew about, but you forgot to apply.

1. Don't do several things at once. Be in the moment, put your smartphone away.
2. You don't have to be important or clever. If you are not very interested in the opinion of others — blog as I do 😇
3. Ask open-ended questions (that don't involve a "yes" or "no" answer).
4. Swim with the flow. If a thought comes up during a conversation, let it go and listen to the person you're talking to.
5. If you don't know something, admit it. This will save you from embarrassment and preserve your authority.
6. There is no need to draw parallels with the experience of your interlocutors. You don't need to tell others what an asshole your boss is in response to your companion's complaints about his boss.
7. Don't repeat what you've said.
8. Avoid unnecessary details. Names, dates, and other details will quickly slip his mind.
9. Listen. You can't learn anything with your mouth open.
10. Be brief. Brevity is the sister of you-know-what.

#soft_skills

Great PySpark style guide from Palantir team:

https://github.com/palantir/pyspark-style-guide

#spark

Coming into the interview, engineers have the idea that HDFS and S3 are fundamentally the same system. Even if S3 clients make it seem like they are, they are not.

https://luminousmen.com/post/hdfs-vs-cloud-based-object-storage-s3