Hi Scrapers! 🕷

If you're diving into the world of data extraction, using a proxy is essential to keep your machine hidden and secure.

But let's be real—navigating the pricing structures of different proxy providers can be a real headache when you're trying to budget! 🤕

Each provider has its own unique billing methods and credit systems, which can make comparisons tricky. 💸

Thanks to Pierluigi from TheWebScrapingClub!

He’s made it easier for us by simplifying the comparison process across various proxy providers, giving you a clearer picture of their pricing structures.

Check out the worksheet below! 📊

✨ Google Sheets: Proxy Provider Pricebook

📝 Complete Pierluigi's explanation on TheWebScrapingClub's blog

This is how your scraping frameworks and libraries was detected while the real browsers are not 😏

A video from John Watson Rooney

🐙 GitHub repo: OxyMouse

Mouse Movement Algorithms

https://github.com/oxylabs/OxyMouse

🎉 Mitmproxy new release now fully support HTTP/3 🔥

https://mitmproxy.org/posts/releases/mitmproxy-11/

This is insane! 😱

🐙 GitHub repo: TikTokLive

Python library to receive live stream events (comments, gifts, etc.) in realtime from TikTok LIVE.

https://github.com/isaackogan/TikTokLive

🗓 Update on the Scraping Universe

Did you know that TikTok's parent company scrapes all around the web massively? One of the most aggressive scraping on the internet! 🤯

ByteDance, the company behind TikTok, has been scraping data from websites at an insane rate.

According to Sam Crowther, the CEO of Kasada, the bot called Bytespider is blowing away the competition, hoovering up data 25 times faster than GPTbot, which scrapes data for ChatGPT. And 3,000 times quicker than ClaudeBot, the scraper bot used by Anthropic. 🤖

What do you think guys? 😀

In case you miss Extract Summit 2024 event by Zyte, you can access full days talks on Youtube

💼 Job market requirement insight for Scrapers

Zyte is opening a position as Principal Reverse Engineer and these are skills they required for the candidate:

• Hacker mindset
• Understand techniques and tools for crawling, extracting, and processing data
• Proficiency in programming languages: JavaScript/Node.js, Python, Java
• Reverse engineering skills: static, dynamic, and concolic analysis
• Understand operating systems and computer networking concepts
• Can use tools like Wireshark, Burp Suite, etc to intercept and debug network traffic
• Understand browser engines, browser fingerprinting, and ad-blocker mechanisms

And will be liked if:
• Experience with Decompilers, IDA Pro, Ghidra or Frida, Jadx, and Babel
• Experience with C/C++
• Core contributions to Mozilla or Chromium projects

🐙 GitHub repo: google-maps-scraper

scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

https://github.com/gosom/google-maps-scraper

⚙ Tech Stack at Apify

Frontend: React.js, styled-components, Storybook, Cypress

Backend: TypeScript/Node.js, Next.js, Nest.js, Docusaurus, Jest

Infra: AWS, Kubernetes, Helm, MongoDB, Redis, DynamoDB, S3, GitHub Actions

Monitoring: New Relic, LogDNA, Sentry, PagerDuty

Tools: GitHub, ZenHub, Notion, GSuite

AI Tools: Langchain, LlamaIndex, Pinecone, OpenAI API, Web agents

🐙 GitHub repo: google-play-scraper

Google play scraper for Python

https://github.com/JoMingyu/google-play-scraper

🐙 GitHub repo: Scrapegraph-ai

Python scraper based on AI

https://github.com/ScrapeGraphAI/Scrapegraph-ai

🐙 GitHub repo: Google-Maps-Scraper

Google maps scraper with gui

https://github.com/Zubdata/Google-Maps-Scraper

🐙 GitHub repo: scrape

CLI utility to scrape emails from websites

https://github.com/lawzava/scrape