Download a static file from a remote URL by using requests in python:

# author -> Roman Podlinov
 def download_file(url):
     local_filename = url.split('/')[-1]
     r = requests.get(url, stream=True)
     with open(local_filename, 'wb') as f:
         for chunk in r.iter_content(chunk_size=1024):
             if chunk: # filter out keep-alive new chunks
                 f.write(chunk)
     return local_filename

NOTE: stream=True parameter is used for returning large files in chunk and save it into a file

#python #requests #stream #large_file #download

Hanlde basic authentication (username, password in header) via requests in python:

>>> from requests.auth import HTTPBasicAuth
 >>> requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))
 <Response [200]>

As you can see above you just need to provide username & password in auth parameter in order to have basic authentication.

#python #requests #authentication #basic #basic_authentication

In python you can use requests to connect to an endpoint an fetch result or create a new endpoint and so on. When you GET result from an endpoint which has an invalid https, the library will give an error. In order to solve this problem you can pass verify=False to requests.get().

A sample would be like:

payload = requests.get(url, auth=HTTPBasicAuth(username, password), verify=False)

Now another issue happens and a WARNING message appears in your log InsecureRequestWarning. You can surpass this message as well by the command below:

from requests.packages.urllib3.exceptions import InsecureRequestWarning

 requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

NOTE: I have done this on a local server behind a firewall, be extremely cautious in case you want to do this on your production server.

#python #requests #InsecureRequestWarning #verify #urllib3

https://realpython.com/python-web-scraping-practical-introduction/

#python #requests #web_scraping