#python python... PYTHON 🔛 🚀
#cool #python #books #from t.me/Tylorn #Tylorn - #DMIA #teacher
‘’’
Item 24: Use @classmethod Polymorphism to Construct
Objects Generically
In Python, not only do the objects support polymorphism, but the classes do as well. What does that mean, and what is it good for?
Polymorphism is a way for multiple classes in a hierarchy to implement their own unique versions of a method. This allows many classes to fulfill the same interface or abstract base class while providing different functionality (see Item 28: “Inherit from collections.abc for Custom Container Types” for an example).
For example, say you’re writing a MapReduce implementation and you want a common class to represent the input data. Here, I define such a class with a read method that must be defined by subclasses:
Click here to view code image
class InputData(object):
def read(self):
raise NotImplementedError
Here, I have a concrete subclass of InputData that reads data from a file on disk:
Click here to view code image
class PathInputData(InputData): def __init__(self, path):
super().__init__() self.path = path
def read(self):
return open(self.path).read()
You could have any number of InputData subclasses like PathInputData and each of them could implement the standard interface for read to return the bytes of data to process. Other InputData subclasses could read from the network, decompress data transparently, etc.
You’d want a similar abstract interface for the MapReduce worker that consumes the input data in a standard way.
Click here to view code image
class Worker(object):
def __init__(self, input_data):
self.input_data = input_data self.result = None
def map(self):
raise NotImplementedError
def reduce(self, other):
raise NotImplementedError
Here, I define a concrete subclass of Worker to implement the specific MapReduce function I want to apply: a simple newline counter:
Click here to view code image
class LineCountWorker(Worker):
def map(self):
data = self.input_data.read() self.result = data.count(‘\n’)
def reduce(self, other): self.result += other.result
It may look like this implementation is going great, but I’ve reached the biggest hurdle in all of this. What connects all of these pieces? I have a nice set of classes with reasonable interfaces and abstractions—but that’s only useful once the objects are constructed. What’s responsible for building the objects and orchestrating the MapReduce?
The simplest approach is to manually build and connect the objects with some helper functions. Here, I list the contents of a directory and construct a PathInputData instance for each file it contains:
Click here to view code image
def generate_inputs(data_dir):
for name in os.listdir(data_dir):
yield PathInputData(os.path.join(data_dir, name))
Next, I create the LineCountWorker instances using the InputData instances
returned by generate_inputs. Click here to view code image
def create_workers(input_list): workers = []
for input_data in input_list: workers.append(LineCountWorker(input_data))
return workers
I execute these Worker instances by fanning out the map step to multiple threads (see Item 37: “Use Threads for Blocking I/O, Avoid for Parallelism”). Then, I call reduce repeatedly to combine the results into one final value.
Click here to view code image
def execute(workers):
threads = [Thread(target=w.map) for w in workers] for thread in threads: thread.start()
for thread in threads: thread.join()
first, rest = workers[0], workers[1:]
for worker in rest:
first.reduce(worker) return first.result
Finally, I connect all of the pieces together in a function to run each step.
Click here to view code image
def mapreduce(data_dir):
inputs = generate_inputs(data_dir) workers = create_workers(inputs) return execute(workers)
Running this function on a set of test input files works great.
Click here to view code image
from tempfile import TemporaryDirectory def write_test_files(tmpdir):
#...
Item 24: Use @classmethod Polymorphism to Construct
Objects Generically
In Python, not only do the objects support polymorphism, but the classes do as well. What does that mean, and what is it good for?
Polymorphism is a way for multiple classes in a hierarchy to implement their own unique versions of a method. This allows many classes to fulfill the same interface or abstract base class while providing different functionality (see Item 28: “Inherit from collections.abc for Custom Container Types” for an example).
For example, say you’re writing a MapReduce implementation and you want a common class to represent the input data. Here, I define such a class with a read method that must be defined by subclasses:
Click here to view code image
class InputData(object):
def read(self):
raise NotImplementedError
Here, I have a concrete subclass of InputData that reads data from a file on disk:
Click here to view code image
class PathInputData(InputData): def __init__(self, path):
super().__init__() self.path = path
def read(self):
return open(self.path).read()
You could have any number of InputData subclasses like PathInputData and each of them could implement the standard interface for read to return the bytes of data to process. Other InputData subclasses could read from the network, decompress data transparently, etc.
You’d want a similar abstract interface for the MapReduce worker that consumes the input data in a standard way.
Click here to view code image
class Worker(object):
def __init__(self, input_data):
self.input_data = input_data self.result = None
def map(self):
raise NotImplementedError
def reduce(self, other):
raise NotImplementedError
Here, I define a concrete subclass of Worker to implement the specific MapReduce function I want to apply: a simple newline counter:
Click here to view code image
class LineCountWorker(Worker):
def map(self):
data = self.input_data.read() self.result = data.count(‘\n’)
def reduce(self, other): self.result += other.result
It may look like this implementation is going great, but I’ve reached the biggest hurdle in all of this. What connects all of these pieces? I have a nice set of classes with reasonable interfaces and abstractions—but that’s only useful once the objects are constructed. What’s responsible for building the objects and orchestrating the MapReduce?
The simplest approach is to manually build and connect the objects with some helper functions. Here, I list the contents of a directory and construct a PathInputData instance for each file it contains:
Click here to view code image
def generate_inputs(data_dir):
for name in os.listdir(data_dir):
yield PathInputData(os.path.join(data_dir, name))
Next, I create the LineCountWorker instances using the InputData instances
returned by generate_inputs. Click here to view code image
def create_workers(input_list): workers = []
for input_data in input_list: workers.append(LineCountWorker(input_data))
return workers
I execute these Worker instances by fanning out the map step to multiple threads (see Item 37: “Use Threads for Blocking I/O, Avoid for Parallelism”). Then, I call reduce repeatedly to combine the results into one final value.
Click here to view code image
def execute(workers):
threads = [Thread(target=w.map) for w in workers] for thread in threads: thread.start()
for thread in threads: thread.join()
first, rest = workers[0], workers[1:]
for worker in rest:
first.reduce(worker) return first.result
Finally, I connect all of the pieces together in a function to run each step.
Click here to view code image
def mapreduce(data_dir):
inputs = generate_inputs(data_dir) workers = create_workers(inputs) return execute(workers)
Running this function on a set of test input files works great.
Click here to view code image
from tempfile import TemporaryDirectory def write_test_files(tmpdir):
#...
#python python... PYTHON 🔛 🚀
#cool #python #books #from t.me/Tylorn #Tylorn - #DMIA #teacher
with TemporaryDirectory() as tmpdir: write_test_files(tmpdir)
result = mapreduce(tmpdir)
print(‘There are’, result, ‘lines’)
>>>
There are 4360 lines
What’s the problem? The huge issue is the mapreduce function is not generic at all. If you want to write another InputData or Worker subclass, you would also have to rewrite the generate_inputs, create_wor
‘’’
result = mapreduce(tmpdir)
print(‘There are’, result, ‘lines’)
>>>
There are 4360 lines
What’s the problem? The huge issue is the mapreduce function is not generic at all. If you want to write another InputData or Worker subclass, you would also have to rewrite the generate_inputs, create_wor
‘’’
Forwarded from Python Books. Книги по питону
☝🏻Сегодня, 27 ноября в 20:00 мск приглашаем на День открытых дверей курса «Инфраструктурная платформа на основе Kubernetes»: https://otus.pw/I9Dj/
📌На бесплатном вебинаре:
- вы познакомитесь с преподавателем-практиком Юрием Игнатовым, ведущим инженером Express42;
- сможете задать любые вопросы по Kubernetes;
- узнаете подробнее о хардкорной программе курса, формате обучения и выпускном проекте;
- расскажем о карьерных перспективах.
🔥Чёрная пятница уже началась - чтобы попасть на курс со скидкой 30%, пройдите вступительный тест (честно - дешевле уже не будет): https://otus.pw/vh6f/
📌На бесплатном вебинаре:
- вы познакомитесь с преподавателем-практиком Юрием Игнатовым, ведущим инженером Express42;
- сможете задать любые вопросы по Kubernetes;
- узнаете подробнее о хардкорной программе курса, формате обучения и выпускном проекте;
- расскажем о карьерных перспективах.
🔥Чёрная пятница уже началась - чтобы попасть на курс со скидкой 30%, пройдите вступительный тест (честно - дешевле уже не будет): https://otus.pw/vh6f/
Forwarded from Python Books. Книги по питону
Python Pocket Reference, Fifth Edition
Автор: Mark Lutz
Год издания: 2014
Скачать книгу
#Python #english
Автор: Mark Lutz
Год издания: 2014
Скачать книгу
#Python #english
Forwarded from Python Books. Книги по питону
Forwarded from UDEMY FREE ️| Coursevania
[100%OFF]Python Data Science basics with Numpy, Pandas and Matplotlib -
In this course, we will learn the basics of Python Data Structures and the most important Data Science libraries like NumPy and Pandas
https://tricksinfo.net/python-data-science-basics-with-numpy-pandas-and-matplotlib/
In this course, we will learn the basics of Python Data Structures and the most important Data Science libraries like NumPy and Pandas
https://tricksinfo.net/python-data-science-basics-with-numpy-pandas-and-matplotlib/
Forwarded from Инжиниринг Данных
20 Python libraries.pdf
4.1 MB
ebook - 20 python libraries that you don't use but should
Forwarded from TelePy | Бесплатно научим питону
Medium
Как начать?
Установка python
Forwarded from Moscow Python (Valentin Dombrovsky)
Будем благодарны за вашу поддержку Python Software Foundation:
For the 1st the PSF is participating in Giving Tuesday, December 3rd. By giving, you support sprints, meetups, community events, Python documentation, fiscal sponsorships, software development, and community projects. https://www.python.org/psf/donations/2019-giving-tuesday-drive/
For the 1st the PSF is participating in Giving Tuesday, December 3rd. By giving, you support sprints, meetups, community events, Python documentation, fiscal sponsorships, software development, and community projects. https://www.python.org/psf/donations/2019-giving-tuesday-drive/
Python.org
Giving Tuesday Fundraiser
The official home of the Python Programming Language
Forwarded from Moscow Python (Valentin Dombrovsky)
Будем благодарны за вашу поддержку Python Software Foundation:
For the 1st the PSF is participating in Giving Tuesday, December 3rd. By giving, you support sprints, meetups, community events, Python documentation, fiscal sponsorships, software development, and community projects. https://www.python.org/psf/donations/2019-giving-tuesday-drive/
For the 1st the PSF is participating in Giving Tuesday, December 3rd. By giving, you support sprints, meetups, community events, Python documentation, fiscal sponsorships, software development, and community projects. https://www.python.org/psf/donations/2019-giving-tuesday-drive/
Python.org
Giving Tuesday Fundraiser
The official home of the Python Programming Language
Forwarded from Oleg Churkin
Тот факт, что аннотацию на такой тип данных написать сложно, говорит скорее всего о том тип данных представлен не оптимально, я могу лишь догадываться, но в данном случае можно попробовать NamedTuple:
from typing import NamedTuple, Tuple
from datetime import datetime
class PaymentData(NamedTuple):
payment_date: datetime
merchant_id: int
payments: Tuple[float, ...]
def my_func() -> PaymentData:
return PaymentData(datetime.now(), 1, (10.22, 17.90, 20.66))