https://help.ubuntu.com/community/CronHowto
#Cron is a system daemon used to #execute desired #tasks (in the background) at designated times.
A crontab file is a simple text file containing a list of commands meant to be run at specified times. It is edited using the crontab command. The commands in the crontab file (and their run times) are checked by the cron daemon, which executes them in the system background.
#Cron is a system daemon used to #execute desired #tasks (in the background) at designated times.
A crontab file is a simple text file containing a list of commands meant to be run at specified times. It is edited using the crontab command. The commands in the crontab file (and their run times) are checked by the cron daemon, which executes them in the system background.
http://selenium-python.readthedocs.io/
Selenium Python bindings provides a simple #API to write functional/acceptance tests using Selenium WebDriver. Through Selenium Python API you can access all functionalities of Selenium #WebDriver in an intuitive way.
Selenium Python bindings provide a convenient API to access Selenium WebDrivers like Firefox, Ie, Chrome, Remote etc. The current supported Python versions are 2.7, 3.2 and above.
Selenium Python bindings provides a simple #API to write functional/acceptance tests using Selenium WebDriver. Through Selenium Python API you can access all functionalities of Selenium #WebDriver in an intuitive way.
Selenium Python bindings provide a convenient API to access Selenium WebDrivers like Firefox, Ie, Chrome, Remote etc. The current supported Python versions are 2.7, 3.2 and above.
http://sahandsaba.com/understanding-asyncio-node-js-python-3-4.html
I spent this summer working on a #web_platform running on #Node.js. This was the first time I worked full-time with Node.js and one thing that became quite apparent after a few weeks of working with it was that many developers, including myself at the time, lack clarify on exactly how the #asynchronous features of Node.js work, and how they are implemented at a lower level. Since I believe the only way to use a platform efficiently is to have a clear understanding of how it works, I decided to dig deeper. This curiosity also made me start playing around with implementing similar asynchronous features in other languages, in particular Python, it being my go-to language for experimenting and learning. This led me to Python 3.4's asynchronous IO library asyncio in particular, which intersected with my already existing interest in coroutines (see my post on combinatorial generation using coroutines in Python.) This post is about exploring the questions and answers that came up while I was learning more about this subject, which I hope can help clarify and answer some questions for others as well.
I spent this summer working on a #web_platform running on #Node.js. This was the first time I worked full-time with Node.js and one thing that became quite apparent after a few weeks of working with it was that many developers, including myself at the time, lack clarify on exactly how the #asynchronous features of Node.js work, and how they are implemented at a lower level. Since I believe the only way to use a platform efficiently is to have a clear understanding of how it works, I decided to dig deeper. This curiosity also made me start playing around with implementing similar asynchronous features in other languages, in particular Python, it being my go-to language for experimenting and learning. This led me to Python 3.4's asynchronous IO library asyncio in particular, which intersected with my already existing interest in coroutines (see my post on combinatorial generation using coroutines in Python.) This post is about exploring the questions and answers that came up while I was learning more about this subject, which I hope can help clarify and answer some questions for others as well.
Sahandsaba
Understanding Asynchronous IO With Python's Asyncio And Node.js
An exploration of asynchronous IO, event loops, threads, and coroutines through code written for Node.js and Python 3.4. Coding, mathematics, and problem solving by Sahand Saba.
http://www.giantflyingsaucer.com/blog/?p=5557
In spring 2014 Python 3.4 shipped a provisional package (#asyncio) which according to the docs “provides infrastructure for writing single-threaded #concurrent code using #coroutines, #multiplexing I/O access over #sockets and other resources, running network clients and servers, and other related primitives“. I can’t possibly cover everything in this article but I can introduce some of the things you can do with it. As per my New’s Years resolution I’ll be building these #examples using Python 3.4.2 (Asyncio has been ported back to Python 3.3 now as well).
In spring 2014 Python 3.4 shipped a provisional package (#asyncio) which according to the docs “provides infrastructure for writing single-threaded #concurrent code using #coroutines, #multiplexing I/O access over #sockets and other resources, running network clients and servers, and other related primitives“. I can’t possibly cover everything in this article but I can introduce some of the things you can do with it. As per my New’s Years resolution I’ll be building these #examples using Python 3.4.2 (Asyncio has been ported back to Python 3.3 now as well).
http://masnun.com/2015/11/20/python-asyncio-future-task-and-the-event-loop.html
On any platform, when we want to do something #asynchronously, it usually involves an #event loop. An event loop is a loop that can register #tasks to be executed, execute them, delay or even cancel them and handle different events related to these operations. Generally, we #schedule multiple async functions to the event loop. The loop runs one function, while that function waits for #IO, it pauses it and runs another. When the first function completes IO, it is resumed. Thus two or more functions can #co_operatively run together. This the main goal of an event loop.
On any platform, when we want to do something #asynchronously, it usually involves an #event loop. An event loop is a loop that can register #tasks to be executed, execute them, delay or even cancel them and handle different events related to these operations. Generally, we #schedule multiple async functions to the event loop. The loop runs one function, while that function waits for #IO, it pauses it and runs another. When the first function completes IO, it is resumed. Thus two or more functions can #co_operatively run together. This the main goal of an event loop.
Abu Ashraf Masnun
Python asyncio: Future, Task and the Event Loop - Abu Ashraf Masnun
Event Loop On any platform, when we want to do something asynchronously, it usually involves an event loop. An event loop is a loop that can register tasks to be executed, execute them, delay or even cancel them and handle different events related to these…
http://masnun.com/2015/11/20/python-asyncio-future-task-and-the-event-loop.html
Python #asyncio: #Future, #Task and the #Event Loop
Python #asyncio: #Future, #Task and the #Event Loop
Abu Ashraf Masnun
Python asyncio: Future, Task and the Event Loop - Abu Ashraf Masnun
Event Loop On any platform, when we want to do something asynchronously, it usually involves an event loop. An event loop is a loop that can register tasks to be executed, execute them, delay or even cancel them and handle different events related to these…
https://blog.miguelgrinberg.com/post/designing-a-restful-api-with-python-and-flask
In recent years #REST (REpresentational State Transfer) has emerged as the standard architectural design for #web services and web #APIs.
In this article I'm going to show you how easy it is to create a RESTful web service using Python and the Flask microframework.
What is REST?
The characteristics of a REST system are defined by six design rules:
Client-Server: There should be a separation between the #server that offers a service, and the #client that consumes it.
Stateless: Each request from a client must contain all the information required by the server to carry out the #request. In other words, the server cannot store information provided by the client in one request and use it in another request.
Cacheable: The server must indicate to the client if requests can be cached or not.
Layered System: Communication between a client and a server should be standardized in such a way that allows intermediaries to respond to requests instead of the end server, without the client having to do anything different.
Uniform Interface: The method of communication between a client and a server must be uniform.
Code on demand: Servers can provide executable code or scripts for clients to execute in their context. This constraint is the only one that is optional.
In recent years #REST (REpresentational State Transfer) has emerged as the standard architectural design for #web services and web #APIs.
In this article I'm going to show you how easy it is to create a RESTful web service using Python and the Flask microframework.
What is REST?
The characteristics of a REST system are defined by six design rules:
Client-Server: There should be a separation between the #server that offers a service, and the #client that consumes it.
Stateless: Each request from a client must contain all the information required by the server to carry out the #request. In other words, the server cannot store information provided by the client in one request and use it in another request.
Cacheable: The server must indicate to the client if requests can be cached or not.
Layered System: Communication between a client and a server should be standardized in such a way that allows intermediaries to respond to requests instead of the end server, without the client having to do anything different.
Uniform Interface: The method of communication between a client and a server must be uniform.
Code on demand: Servers can provide executable code or scripts for clients to execute in their context. This constraint is the only one that is optional.
http://lucumr.pocoo.org/2016/12/29/careful-with-str-format/
#security
This should have been obvious to me for a longer time, but until earlier today I did not really realize the severity of the issues caused by str.format on untrusted user input. It came up as a way to bypass the Jinja2 Sandbox in a way that would permit retrieving information that you should not have access to which is why I just pushed out a security release for it.
However I think the general issue is quite severe and needs to be a discussed because most people are most likely not aware of how easy it is to exploit.
#security
This should have been obvious to me for a longer time, but until earlier today I did not really realize the severity of the issues caused by str.format on untrusted user input. It came up as a way to bypass the Jinja2 Sandbox in a way that would permit retrieving information that you should not have access to which is why I just pushed out a security release for it.
However I think the general issue is quite severe and needs to be a discussed because most people are most likely not aware of how easy it is to exploit.
http://stackoverflow.com/questions/6434482/python-function-overloading
I know that Python does not support method #overloading, but I've run into a problem that I can't seem to solve in a nice Pythonic way.
I am making a #game where a character needs to shoot a variety of bullets, but how do I write different functions for creating these bullets? For example suppose I have a function that creates a bullet travelling from point A to B with a given speed.
I know that Python does not support method #overloading, but I've run into a problem that I can't seem to solve in a nice Pythonic way.
I am making a #game where a character needs to shoot a variety of bullets, but how do I write different functions for creating these bullets? For example suppose I have a function that creates a bullet travelling from point A to B with a given speed.
Stack Overflow
Python function overloading
I know that Python does not support method overloading, but I've run into a problem that I can't seem to solve in a nice Pythonic way.
I am making a game where a character needs to shoot a variety of
I am making a game where a character needs to shoot a variety of
https://www.python.org/dev/peps/pep-0443/
This PEP proposes a new mechanism in the #functools standard library module that provides a simple form of generic programming known as #single_dispatch #generic functions.
A generic function is composed of multiple functions implementing the same operation for different types. Which implementation should be used during a call is determined by the #dispatch algorithm. When the implementation is chosen based on the type of a single argument, this is known as #single_dispatch .
#overloading
This PEP proposes a new mechanism in the #functools standard library module that provides a simple form of generic programming known as #single_dispatch #generic functions.
A generic function is composed of multiple functions implementing the same operation for different types. Which implementation should be used during a call is determined by the #dispatch algorithm. When the implementation is chosen based on the type of a single argument, this is known as #single_dispatch .
#overloading
Python Enhancement Proposals (PEPs)
PEP 443 – Single-dispatch generic functions | peps.python.org
This PEP proposes a new mechanism in the functools standard library module that provides a simple form of generic programming known as single-dispatch generic functions.
https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html
👌Making 1 million requests with python -#aiohttp
Apr 22, 2016 - by Paweł Miech - about: #asyncio, aiohttp, #python
In this post I’d like to test limits of python aiohttp and check its performance in terms of requests per minute. Everyone knows that asynchronous code performs better when applied to network operations, but it’s still interesting to check this assumption and understand how exactly it is better and why it’s is better. I’m going to check it by trying to make 1 million #requests with aiohttp client. How many requests per minute will aiohttp make? What kind of exceptions and crashes can you expect when you try to make such volume of requests with very primitive scripts? What are main gotchas that you need to think about when trying to make such volume of requests?
👌Making 1 million requests with python -#aiohttp
Apr 22, 2016 - by Paweł Miech - about: #asyncio, aiohttp, #python
In this post I’d like to test limits of python aiohttp and check its performance in terms of requests per minute. Everyone knows that asynchronous code performs better when applied to network operations, but it’s still interesting to check this assumption and understand how exactly it is better and why it’s is better. I’m going to check it by trying to make 1 million #requests with aiohttp client. How many requests per minute will aiohttp make? What kind of exceptions and crashes can you expect when you try to make such volume of requests with very primitive scripts? What are main gotchas that you need to think about when trying to make such volume of requests?
pawelmhm.github.io
Making 1 million requests with python-aiohttp
In this post I’d like to test limits of python aiohttp and check its performance in terms of requests per minute. Everyone knows that asynchronous code perf...
https://pymotw.com/3/asyncio/subprocesses.html
Working with Subprocesses
It is frequently necessary to work with other programs and processes, to take advantage of existing code without rewriting it or to access libraries or features not available from within Python. As with network I/O, asyncio includes two abstractions for starting another program and then interacting with it.
Using the Protocol Abstraction with #Subprocesses
This example uses a coroutine to launch a process to run the Unix command df to find the free space on local disks. It uses subprocess_exec() to launch the process and tie it to a protocol class that knows how to read the df command output and parse it. The methods of the protocol class are called automatically based on I/O events for the subprocess. Because both the stdin and stderr arguments are set to None, those communication channels are not connected to the new process.
Working with Subprocesses
It is frequently necessary to work with other programs and processes, to take advantage of existing code without rewriting it or to access libraries or features not available from within Python. As with network I/O, asyncio includes two abstractions for starting another program and then interacting with it.
Using the Protocol Abstraction with #Subprocesses
This example uses a coroutine to launch a process to run the Unix command df to find the free space on local disks. It uses subprocess_exec() to launch the process and tie it to a protocol class that knows how to read the df command output and parse it. The methods of the protocol class are called automatically based on I/O events for the subprocess. Because both the stdin and stderr arguments are set to None, those communication channels are not connected to the new process.
https://www.obeythetestinggoat.com/testing-async-asyncio-and-performance.html
#Testing, #async, #asyncio, and #performance
Sun 27 December 2015 By Harry
I recently did some experimenting with asyncio, and wanted to report back on how I got on with writing tests for it. While I was at it I was also able to compare its performance with a couple of other approaches to #mutlitasking in Python, namely #threads and #gevent, so I'll report on that here too. (tl;dr: it's much of a muchness).
#Testing, #async, #asyncio, and #performance
Sun 27 December 2015 By Harry
I recently did some experimenting with asyncio, and wanted to report back on how I got on with writing tests for it. While I was at it I was also able to compare its performance with a couple of other approaches to #mutlitasking in Python, namely #threads and #gevent, so I'll report on that here too. (tl;dr: it's much of a muchness).
https://docs.python.org/3/whatsnew/3.6.html
This article explains the #new features in Python 3.6, compared to 3.5. Python 3.6 was released on December 23, 2016. See the changelog for a full list of changes.
PEP 498, formatted string literals.
PEP 515, underscores in numeric literals.
PEP 526, syntax for variable annotations.
PEP 525, asynchronous generators.
PEP 530: asynchronous comprehensions.
This article explains the #new features in Python 3.6, compared to 3.5. Python 3.6 was released on December 23, 2016. See the changelog for a full list of changes.
PEP 498, formatted string literals.
PEP 515, underscores in numeric literals.
PEP 526, syntax for variable annotations.
PEP 525, asynchronous generators.
PEP 530: asynchronous comprehensions.
http://www.aparat.com/v/TkR0X
Lars has had a diverse career spanning 40 years. His early adoption of C++ in the late 80s gained him a formidable reputation as an emergency services programmer. Lars adopted Python in 2002. Lars currently works for #Mozilla Corporation where he has been the primary engineer behind the #Firefox crash reporting system, Socorro.
#mind #AI
Lars has had a diverse career spanning 40 years. His early adoption of C++ in the late 80s gained him a formidable reputation as an emergency services programmer. Lars adopted Python in 2002. Lars currently works for #Mozilla Corporation where he has been the primary engineer behind the #Firefox crash reporting system, Socorro.
#mind #AI
آپارات
K Lars Lohn_Keynote_PyCon2016_bSfe5M_zG2s
Lars has had a diverse career spanning 40 years. His early adoption of C++ in the late 80s gained him a formidable reputation as an emergency services programmer. Lars adopted Python in 2002. Lars currently works for Mozilla Corporation where he has been…
https://github.com/pywren/pywren
A simple package with a powerful premise, #PyWren lets you run Python-based scientific computing workloads as multiple instances of AWS Lambda functions. A profile of the project at The New Stack describes PyWren using AWS Lambda as a giant #parallel_processing_system, tackling projects that can be sliced and diced into little tasks that don't need a lot of memory or storage to run.
One downside is that #lambda_functions can't run for more than 300 seconds max. But if you need a job that takes only a few minutes to complete and need to run it thousands of times across a data set, PyWren may be a good option to #parallelize that work in the cloud at a scale unavailable on user hardware.
#Machine_learning
A simple package with a powerful premise, #PyWren lets you run Python-based scientific computing workloads as multiple instances of AWS Lambda functions. A profile of the project at The New Stack describes PyWren using AWS Lambda as a giant #parallel_processing_system, tackling projects that can be sliced and diced into little tasks that don't need a lot of memory or storage to run.
One downside is that #lambda_functions can't run for more than 300 seconds max. But if you need a job that takes only a few minutes to complete and need to run it thousands of times across a data set, PyWren may be a good option to #parallelize that work in the cloud at a scale unavailable on user hardware.
#Machine_learning
GitHub
GitHub - pywren/pywren: Teraflops and microservices
Teraflops and microservices. Contribute to pywren/pywren development by creating an account on GitHub.
https://github.com/riga/tfdeploy
Google's TensorFlow framework is taking off big-time now that it's at a full 1.0 release. One common question about it: How can I make use of the models I train in TensorFlow without using TensorFlow itself?
#Tfdeploy is a partial answer to that question. It exports a trained TensorFlow model to "a simple #NumPy-based callable," meaning the model can be used in Python with Tfdeploy and the the NumPy math-and-stats library as the only dependencies. Most of the operations you can perform in TensorFlow can also be performed in Tfdeploy, and you can extend the behaviors of the library by way of standard Python metaphors (such as overloading a class).
Now the bad news: Tfdeploy doesn't support GPU acceleration, if only because NumPy doesn't do that. Tfdeploy's creator suggests using the gNumPy project as a possible replacement.
#Machine_learning
Google's TensorFlow framework is taking off big-time now that it's at a full 1.0 release. One common question about it: How can I make use of the models I train in TensorFlow without using TensorFlow itself?
#Tfdeploy is a partial answer to that question. It exports a trained TensorFlow model to "a simple #NumPy-based callable," meaning the model can be used in Python with Tfdeploy and the the NumPy math-and-stats library as the only dependencies. Most of the operations you can perform in TensorFlow can also be performed in Tfdeploy, and you can extend the behaviors of the library by way of standard Python metaphors (such as overloading a class).
Now the bad news: Tfdeploy doesn't support GPU acceleration, if only because NumPy doesn't do that. Tfdeploy's creator suggests using the gNumPy project as a possible replacement.
#Machine_learning
GitHub
GitHub - riga/tfdeploy: Deploy tensorflow graphs for fast evaluation and export to tensorflow-less environments running numpy.
Deploy tensorflow graphs for fast evaluation and export to tensorflow-less environments running numpy. - GitHub - riga/tfdeploy: Deploy tensorflow graphs for fast evaluation and export to tensorflo...
https://github.com/spotify/luigi
Writing batch jobs is generally only one part of processing heaps of data; you also have to string all the jobs together into something resembling a #workflow or a #pipeline. #Luigi, created by Spotify and named for the other plucky plumber made famous by Nintendo, was built to "address all the plumbing typically associated with long-running batch processes."
With Luigi, a developer can take several different unrelated data processing tasks — "a Hive query, a Hadoop job in Java, a Spark job in Scala, dumping a table from a database" — and create a workflow that runs them, end to end. The entire description of a job and its dependencies are created as Python modules, not as XML config files or another data format, so it can be integrated into other Python-centric projects.
#Machine_learning
Writing batch jobs is generally only one part of processing heaps of data; you also have to string all the jobs together into something resembling a #workflow or a #pipeline. #Luigi, created by Spotify and named for the other plucky plumber made famous by Nintendo, was built to "address all the plumbing typically associated with long-running batch processes."
With Luigi, a developer can take several different unrelated data processing tasks — "a Hive query, a Hadoop job in Java, a Spark job in Scala, dumping a table from a database" — and create a workflow that runs them, end to end. The entire description of a job and its dependencies are created as Python modules, not as XML config files or another data format, so it can be integrated into other Python-centric projects.
#Machine_learning
GitHub
GitHub - spotify/luigi: Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution…
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in. ...
https://github.com/safarijv/kubelib
If you're adopting Kubernetes as an orchestration system for #machine_learning jobs, the last thing you want is for the mere act of using Kubernetes to create more problems than it solves. Kubelib provides a set of Pythonic interfaces to #Kubernetes, originally to aid with Jenkins scripting. But it can be used without Jenkins as well, and it can do everything exposed through the kubectl #CLI or the Kubernetes #API.
If you're adopting Kubernetes as an orchestration system for #machine_learning jobs, the last thing you want is for the mere act of using Kubernetes to create more problems than it solves. Kubelib provides a set of Pythonic interfaces to #Kubernetes, originally to aid with Jenkins scripting. But it can be used without Jenkins as well, and it can do everything exposed through the kubectl #CLI or the Kubernetes #API.
GitHub
safarijv/kubelib
kubelib - Python library to simplify kubernetes scripting
https://github.com/4Catalyzer/pykubectl
A python bridge to kubectl providing additional functionalities useful for CD and #automation.
#machine_learning
A python bridge to kubectl providing additional functionalities useful for CD and #automation.
#machine_learning
GitHub
4Catalyzer/pykubectl
pykubectl - A python bridge to kubectl