UNDERCODE COMMUNITY

▁ ▂ ▄ U𝕟𝔻Ⓔ𝐫Ć𝔬𝓓ⓔ ▄ ▂ ▁

🦑Python crawler uses dynamic switching ip to prevent blocking ?

1) The biggest difference between a forward proxy and a reverse proxy is that the domain name of the reverse proxy is often fixed, while the forward proxy is accessed at will through an http proxy port, but the http protocol will be modified on the proxy side to help you access

2) If it is python, in fact, it is enough to simply call socket bind to bind a certain ip, but what is the concept of title rotation? It is to maintain different socket bind objects, and then you can take turns! I have talked to some people who specialize in crawlers in the industry, and they basically use this technology.

# -*- coding=utf-8 -*-
import socket
import urllib2
import re
true_socket = socket.socket

ipbind='xx.xx.xxx.xx'

def bound_socket(*a, **k):
sock = true_socket(* a, **k)
sock.bind((ipbind, 0))
return sock

socket.socket = bound_socket

response = urllib2.urlopen('http://www. Testsite .com')
html = response.read()
ip= re.search(r'code.(.*?)..code',html)
print ip.group(1)

🦑I found some ideas for solutions given by foreigners, It is the export ip address constructed with the help of urllib2's HTTPHandler.

import functools
import httplib
import urllib2

class BoundHTTPHandler(urllib2.HTTPHandler):

def init(self, source_address=None, debuglevel=0):
urllib2.HTTPHandler.init(self, debuglevel)
self.http_class = functools.partial(httplib.HTTPConnection,
source_address=source_address)

def http_open(self, req):
return self.do_open(self.http_class, req)

handler = BoundHTTPHandler(source_address=("192.168.1.10", 0))
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)

import functools
import httplib
import urllib2

class BoundHTTPHandler(urllib2.HTTPHandler):

def init(self, source_address=None, debuglevel=0):
urllib2.HTTPHandler.init(self, debuglevel)
self.http_class = functools.partial(httplib.HTTPConnection,
source_address=source_address)

def http_open(self, req):
return self.do_open(self.http_class, req)

handler = BoundHTTPHandler(source_address=("192.168.1.10", 0))
opener = urllib2.build_opener(handler)
urllib2.install_opener (opener)
Then there is a ready-made module netifaces. In fact, the netifaces module is the function package of the socket binding ip just above.
Address: https://github.com/raphdg/netifaces

import netifaces
netifaces.interfaces()
netifaces.ifaddresses('lo0')
netifaces.AF_LINK
addrs = netifaces.ifaddresses('lo0')
addrs[netifaces.AF_INET]
[{'peer': '127.0.0.1','netmask': '255.0.0.0','addr': '127.0.0.1'}]

import netifaces
netifaces.interfaces()
netifaces.ifaddresses('lo0')
netifaces.AF_LINK
addrs = netifaces.ifaddresses('lo0')
addrs[netifaces.AF_INET]
[{'peer': '127.0.0.1','netmask': '255.0.0.0','addr': '127.0.0.1'}]
Thanks for reading, I hope it can help everyone, thank you for your support to this site!

@UndercodeTesting
▁ ▂ ▄ U𝕟𝔻Ⓔ𝐫Ć𝔬𝓓ⓔ ▄ ▂ ▁

GitHub

GitHub - raphdg/netifaces: I'm not the author of netifaces, I just added the project on github and I added a spec file to build…

I'm not the author of netifaces, I just added the project on github and I added a spec file to build the RPM package for the basic Amazon Linux AMI 2012.03 - GitHub - raphdg/netifaces: I&am...

323 viewsedited 20:20