If you have a CPU-heavy task and want to utilize all the cores you have, then
multiprocessing.Pool
is for you. It spawns multiple processes and delegates tasks to them automatically. Simply create a pool with Pool(number_of_processes)
and run p.map
with the list of inputs.In : import math
In : from multiprocessing import Pool
In : inputs = [i ** 2 for i in range(100, 130)]
In : def f(x):
...: return len(str(math.factorial(x)))
...:
In : %timeit [f(x) for x in inputs]
1.44 s ± 19.2 ms per loop (...)
In : p = Pool(4)
In : %timeit p.map(f, inputs)
451 ms ± 34 ms per loop (...)
Python
That means that threads are only useful when your program spends time outside of Python interpreter, usually waiting for IO. For example, downloading three different Wikipedia articles at once with threads will be as efficient as with processes (and thrice as efficient as downloading using only one process):
On the other hand, it doesn't make much sense to solve CPU-heavy tasks with threads:
multiprocessing
module allows you to spawn not only processes but threads as well. Mind, however, than CPython is notorious for its GIL (global interpreter lock), the interpreter feature that doesn't allow different threads run Python bytecode simultaneously.That means that threads are only useful when your program spends time outside of Python interpreter, usually waiting for IO. For example, downloading three different Wikipedia articles at once with threads will be as efficient as with processes (and thrice as efficient as downloading using only one process):
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool
def download_wiki_article(article):
url = 'http://de.wikipedia.org/wiki/'
return requests.get(url + article)
process_pool = Pool(3)
thread_pool = ThreadPool(3)
thread_pool.map(download_wiki_article, ['a', 'b', 'c'])
# 376 ms ± 11 ms
process_pool.map(download_wiki_article, ['a', 'b', 'c'])
# 373 ms ± 3.17 ms
[download_wiki_article(a) for a in ['a', 'b', 'c']]
# 1.09 s ± 27.9 ms
On the other hand, it doesn't make much sense to solve CPU-heavy tasks with threads:
import math
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool
def f(x):
return len(str(math.factorial(x)))
process_pool = Pool(4)
thread_pool = ThreadPool(4)
inputs = [i ** 2 for i in range(100, 130)]
[f(x) for x in inputs]
# 1.48 s ± 7.61 ms
thread_pool.map(f, inputs)
# 1.48 s ± 7.78 ms
process_pool.map(f, inputs)
# 478 ms ± 7.55 ms
When Python executes a method call, say
Python supports only single-dispatch polymorphism because a single object alone (
However, multiple dispatch can be emulated via single-dispatch. The visitor design pattern is created exactly for this. What visitor do is essentially calling single-dispatch twice to imitate double-dispatch.
Mind, that the ability to overload methods (like in Java and C++) is not the same as multiple dispatch. Dynamic dispatch works in runtime while overloading solely affects compile time.
a.f(b, c, d)
, it should first select the right f
function. Due to polymorphism, it depends on the type of a
. The process of choosing the method is usually called dynamic dispatch.Python supports only single-dispatch polymorphism because a single object alone (
a
in the example) affects the method selection. Some other languages, however, may also consider a type of b
, c
and d
. This mechanism is called multiple disaptch. C# is a notable example of languages that support that technique.However, multiple dispatch can be emulated via single-dispatch. The visitor design pattern is created exactly for this. What visitor do is essentially calling single-dispatch twice to imitate double-dispatch.
Mind, that the ability to overload methods (like in Java and C++) is not the same as multiple dispatch. Dynamic dispatch works in runtime while overloading solely affects compile time.
Though decorators and context managers are quite similar and often interchangeable, context managers are severely more limited. You can't skip a block or execute it twice; it always runs exactly one time.
However, you can control whether the exception that is raised inside a context should be propagated to the caller or not. It's done by the slightly obscure way: the exception is suppressed if
The output is:
However, you can control whether the exception that is raised inside a context should be propagated to the caller or not. It's done by the slightly obscure way: the exception is suppressed if
__exit__
return a true value:class Atomic:
def __enter__(self):
print('BEGIN')
def __exit__(self, exc_type, exc_value, traceback):
if exc_type:
print(
'ROLLBACK due to {}({})'.format(
exc_type, exc_value
)
)
else:
print('COMMIT')
return True
with Atomic():
print('A')
with Atomic():
print('B')
raise RuntimeError('C')
The output is:
BEGIN
A
COMMIT
BEGIN
B
ROLLBACK due to <type 'exceptions.RuntimeError'>(C)
If you need to search through a sorted collection, binary search is what you need. This simple algorithm compares the target value to the middle of the array; the result determines which half should be searched next.
Python standard library provides a way to use binary search without directly implementing it.
Python standard library provides a way to use binary search without directly implementing it.
bisect_left
function returns the leftmost position in a sorted list for the element, while bisect_right
return the rightmost one.In : from random import randrange
In : from bisect import bisect_left
In : n = 1000000
In : look_for = 555555
In : lst = sorted(randrange(0, n) for _ in range(n))
In : %timeit look_for in lst
69.7 ms ± 449 µs per loop
In : %timeit look_for == lst[bisect_left(lst, look_for)]
927 ns ± 2.28 ns per loop
To store any information in memory or on a storage device, you should represent it in bytes. Python usually provides the level of abstraction where you can think about data itself, not its byte form.
Still, when you write, say, a string to a file, you deal with a physical structure of data. To put characters into a file you should transform them into bytes; that is called encoding. When you get bytes from a file, you probably want to convert them into meaningful characters; that is call decoding.
There are hundreds of encoding methods out there. The most popular one is probably Unicode, but you can't transform anything to bytes with it. In the sense of byte representation, Unicode is not even an encoding. Unicode defines a mapping between characters and their integer codes. 🐍 is 128 013, for example.
But to put integers into a file, you need a real encoding. Unicode is usually used with
Still, when you write, say, a string to a file, you deal with a physical structure of data. To put characters into a file you should transform them into bytes; that is called encoding. When you get bytes from a file, you probably want to convert them into meaningful characters; that is call decoding.
There are hundreds of encoding methods out there. The most popular one is probably Unicode, but you can't transform anything to bytes with it. In the sense of byte representation, Unicode is not even an encoding. Unicode defines a mapping between characters and their integer codes. 🐍 is 128 013, for example.
But to put integers into a file, you need a real encoding. Unicode is usually used with
utf-8
, which is (usually) a default in Python. When you read form a file, Python automatically decodes utf-8
. You can choose any other encoding with encoding=
parameter of the open
function, or you can read plane bytes by appending b
to its mode.Is Python interpreted or compiled? The simple answer here is interpreted; the right one is — it's both.
Python compiles your source code to bytecode (
Despite that, Python is usually called interpreted language while Java is called compiled language, which is, strictly speaking, not entirely correct.
Here is an article on the subject with more details and explanations.
Python compiles your source code to bytecode (
.pyc
files). It does that implicitly, but it's still an essential phase of Python code execution. Java, for example, does the same but explicitly: you compile with javac
and run with java
.Despite that, Python is usually called interpreted language while Java is called compiled language, which is, strictly speaking, not entirely correct.
Here is an article on the subject with more details and explanations.
You can use any object as a dictionary key in Python as long as it implements the
You also should avoid using mutable objects as keys, because once the object becomes not equal to the old self, it can't be found in a dictionary anymore.
There is also one bizarre thing that might surprise you during debugging or unit testing:
In CPython
__hash__
method. This method can return any integer as long as the only requirement is met: equal objects should have equal hashes (not vice versa).You also should avoid using mutable objects as keys, because once the object becomes not equal to the old self, it can't be found in a dictionary anymore.
There is also one bizarre thing that might surprise you during debugging or unit testing:
In : class A:
...: def __init__(self, x):
...: self.x = x
...:
...: def __hash__(self):
...: return self.x
...:
In : hash(A(2))
Out: 2
In : hash(A(1))
Out: 1
In : hash(A(0))
Out: 0
In : hash(A(-1)) # sic!
Out: -2
In : hash(A(-2))
Out: -2
In CPython
-1
is internally reserved for error states, so it's implicitly converted to -2
.Creating an external process in Python is an easy task, you can do it with
Let's suppose we ask
Now we have to read from them. The problem is, you can't just do
The primary process creates the child and waits for
In this case
To solve this problem, you should read from both
subprocess
module. However, reading both stdout
and stderr
of the spawned process may be more challenging.Let's suppose we ask
Popen
to create two pipes, one for stdout
and one for stderr
:p = subprocess.Popen(
["python", "-c", "..."],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
Now we have to read from them. The problem is, you can't just do
readline()
for any of those pipes since it can cause deadlocks. Consider the more concrete example:import subprocess
SUBPROCESS_CODE = """
import sys
sys.stderr.write('err')
print('out')
"""
p = subprocess.Popen(
["python", "-c", SUBPROCESS_CODE],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
print(p.stdout.readline())
The primary process creates the child and waits for
stdout
. The child process first writes to stderr
pipe and then to stdout
. The main process successfully receives 'out'
from the pipe. But what about 'err'
? It's stored in the pipe buffer until the main process does p.stderr.readline()
. But what if the buffer is full?import subprocess
SUBPROCESS_CODE = """
import sys
for _ in range(100000):
sys.stderr.write('err')
print('out')
"""
p = subprocess.Popen(
["python", "-c", SUBPROCESS_CODE],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
print(p.stdout.readline())
In this case
sys.stderr.write('err')
will be blocked at some point until someone reads from the buffer. But no one ever will: the main process waits for data in stdout
. This is the deadlock we are talking about.To solve this problem, you should read from both
stdout
and stderr
at once. You can do it with select
module or simply use p.communicate()
. The second approach is much more straightforward but doesn't let you read data line by line.Sometimes you need to create a function from a more universal one.
For example,
The
It can be helpful when you need to pass a function as an argument to another higher order function, but some arguments should be locked:
Without
For example,
int()
has a base
parameter which we would like to freeze to have new base2
function:>>> int("10")
10
>>> int("10", 2)
2
>>> def base2(x):
... return int(x, 2)
...
>>> base2("10")
2
The
functools.partial
allows you to do the same more accurate and semantically clear:base2 = partial(int, base=2)
It can be helpful when you need to pass a function as an argument to another higher order function, but some arguments should be locked:
>>> map(partial(int, base=2), ["1", "10", "100"])
[1, 2, 4]
Without
partial
you do something like this:>>> map(lambda x: int(x, base=2), ["1", "10", "100"])
[1, 2, 4]
Liskov substitution principle tells us that if
Whenever you work with
Strictly speaking, it doesn't. The
It may seem that
S
is a subtype of T
, then all occurrences of T
may be replaced with S
without breaking any code. That means that S
should satisfy all the guarantees that T
introduces.Whenever you work with
dict
you usually assume that as long as x in d
returns False, d[x]
raises KeyError
. But if your dict
is a defaultdict
, that it's simply not true. Does defaultdict
violate the LSP then?Strictly speaking, it doesn't. The
dict
documentation explicitly says that d[x]
may return something even though x
is not present in d
(if the __missing__
method defined). So you still potentially get something from the dictionary even if the key is not in it. That means that you shouldn't ever do x in d
to check the existence of element as long as you want to support all dict
subclasses.It may seem that
defaultdict
violates the LSP, but it “breaks” the guarantees that were never there.Sometimes you want to know whether something is a function or not. The obvious solution is to check the object class with
Now you can do checking:
Luckily all above code is already written for you:
Note, that you usually don't care whether something is a function, but rather if it's callable or not. It can be done with
isinstance
. The class of functions is function
, but you can't access directly. You can instead get type
of any existing function:FunctionType = type(lambda: None)
Now you can do checking:
def isfunction(object):
return isinstance(object, FunctionType)
Luckily all above code is already written for you:
FunctionType
is an existing member of types
and isfunction
already exists in the inspect
module.Note, that you usually don't care whether something is a function, but rather if it's callable or not. It can be done with
callable
:>>> callable(int)
True
>>> callable(42)
False
>>> callable(callable)
True
The
The field name is used to specify which argument exactly should be used as a replacement:
The conversion let you ask
Finally, the format specification is a way to define how values are presented:
This specification may be applied to a single object with
The
format
method of Python string is a mighty tool that supports a lot of things that you are probably not even aware of. Each replacement placeholder ({...}
) may contain three parts: field name, conversion and format specification.The field name is used to specify which argument exactly should be used as a replacement:
>>> '{}'.format(42)
'42'
>>> '{1}'.format(1, 2)
'2'
>>> '{y}'.format(x=1, y=2)
'2'
The conversion let you ask
format
to use repr()
(or ascii()
) instead of str()
while converting objects to strings:>>> '{!r}'.format(datetime.now())
'datetime.datetime(2018, 5, 3, 23, 48, 49, 157037)'
>>> '{}'.format(datetime.now())
'2018-05-03 23:49:01.060852'
Finally, the format specification is a way to define how values are presented:
>>> '{:+,}'.format(1234567)
'+1,234,567'
>>> '{:>19}'.format(1234567)
' 1234567'
This specification may be applied to a single object with
format
function (not the str
method):format(5000000, '+,')
'+5,000,000'
The
format
function calls __format__
method of the object internally so you can alter its behavior for your types.Let's suppose you have some datetime object and want to know how much time has passed since the start of the day. How do you do that?
First of all, to know what day we are talking about, we need to have the timezone; having the datetime is not enough. As long you have the timezone you need to convert the datetime and strip time with the
Having the date, we can get its midnight time. To do this, we glue
And now we put things together:
First of all, to know what day we are talking about, we need to have the timezone; having the datetime is not enough. As long you have the timezone you need to convert the datetime and strip time with the
date()
method:def date_of_time(datetime_object, tz):
return datetime_object.astimezone(tz).date()
Having the date, we can get its midnight time. To do this, we glue
00:00:00
back to the datetime object and assign the original timezone:def midnight_of_date(date_in_given_timezone, tz):
midnight = datetime.datetime.combine(
date_in_given_timezone, datetime.time()
)
return tz.localize(midnight)
And now we put things together:
def midnight(datetime_object, tz):
return midnight_of_date(
date_of_time(datetime_object, tz), tz
)
PEP 424 allows generators and other iterable objects that don't have the exact predefined size to expose a length hint. For example, the following generator will likely return ~50 elements:
If you write an iterable and want to add the hint, define the
If you use an iterable and want to know its expected length, use
(x for x in range(100) if random() > 0.5)
If you write an iterable and want to add the hint, define the
__length_hint__
method. If the length is known for sure, use __len__
instead.If you use an iterable and want to know its expected length, use
operator.length_hint
.In Python,
If you do care about indexes you can iterate over
But perhaps the more semantically correct and expressive way to do the same it to use
for
lets you withdraw elements from a collection without thinking about their indexes:def find_odd(lst):
for x in lst:
if x % 2 == 1:
return x
return None
If you do care about indexes you can iterate over
range(len(lst))
:def find_odd(lst):
for i in range(len(lst)):
x = lst[i]
if x % 2 == 1:
return i, x
return None, None
But perhaps the more semantically correct and expressive way to do the same it to use
enumerate
:def find_odd(lst):
for i, x in enumerate(lst):
if x % 2 == 1:
return i, x
return None, None
The
Sometimes you want to know whether a generator is empty (rather say, exhausted). To do this, you have to try getting the next element from the generator. If it works, you would like to put element back in the generator, which of course is not possible. You can glue it back with
Usage example:
itertools.chain
function is a way to iterate over many iterables as though they are glued together:In : list(chain(['a', 'b'], range(3), set('xyz')))
Out: ['a', 'b', 0, 1, 2, 'x', 'z', 'y']
Sometimes you want to know whether a generator is empty (rather say, exhausted). To do this, you have to try getting the next element from the generator. If it works, you would like to put element back in the generator, which of course is not possible. You can glue it back with
chain
instead:def sum_of_odd(gen):
try:
first = next(gen)
except StopIteration:
raise ValueError('Empty generator')
return sum(
x for x in chain([first], gen)
if x % 2 == 1
)
Usage example:
In : sum_of_odd(x for x in range(1, 6))
Out: 9
In : sum_of_odd(x for x in range(2, 3))
Out: 0
In : sum_of_odd(x for x in range(2, 2))
...
ValueError: Empty generator
In Python, an
The common usage for this is to search something in a loop and use
else
block could be presented not only after if
, but after for
and while
as well. The code inside else
is executed unless the loop was interrupted by break
.The common usage for this is to search something in a loop and use
break
when found:In : first_odd = None
In : for x in [2,3,4,5]:
...: if x % 2 == 1:
...: first_odd = x
...: break
...: else:
...: raise ValueError('No odd elements in list')
...:
In : first_odd
Out: 3
In : for x in [2,4,6]:
...: if x % 2 == 1:
...: first_odd = x
...: break
...: else:
...: raise ValueError('No odd elements in list')
...:
...
ValueError: No odd elements in list
Since loops in Python don't create scopes, you usually need an extra function to create closures. The straightforward way doesn't work:
Let's add the extra function:
It works this way, but the code can be clumsy, especially if you need
To make it slightly more readable, you can write universal function and get partials of it:
You can always emulate
Fun fact: thanks to the
multipliers = []
for i in range(10):
multipliers.append(lambda x: x * i)
[multipliers[i](2) for i in range(5)]
# [18, 18, 18, 18, 18]
Let's add the extra function:
multiplier_creator = lambda i: lambda x: x * i
for i in range(10):
multipliers.append(multiplier_creator(i))
It works this way, but the code can be clumsy, especially if you need
def
, not lambda
:def multiplier_creator(i):
def multiplier(x):
return x * i
for i in range(10):
return multiplier
multipliers.append(multiplier_creator(i))
To make it slightly more readable, you can write universal function and get partials of it:
multiplier = lambda x, i: x * i
for i in range(10):
multipliers.append(partial(multiplier, i=i))
You can always emulate
partial
with custom lambda
, but repr
of partials are generally more readable:In : partial(int, base=2)
Out: functools.partial(<class 'int'>, base=2)
In : lambda x: int(x, base=2)
Out: <function __main__.<lambda>>
Fun fact: thanks to the
operator
module this particular example can be expressed even more appealing:for i in range(10):
multipliers.append(partial(operator.mul, i))