coder-hxl/x-crawl
x-crawl is a flexible Node.js AI-assisted crawler library. Making crawler work more efficient, intelligent and convenient. ------ x-crawl 是一个灵活的 Node.js AI 辅助爬虫库。使爬虫工作变得更加高效、智能和便捷。
Language:TypeScript
Total stars: 939
Stars trend:
#typescript
#ai, #aicrawl, #chromium, #crawl, #crawler, #fingerprint, #flexible, #javascript, #multifunction, #nodejs, #puppeteer, #spider, #typescript
x-crawl is a flexible Node.js AI-assisted crawler library. Making crawler work more efficient, intelligent and convenient. ------ x-crawl 是一个灵活的 Node.js AI 辅助爬虫库。使爬虫工作变得更加高效、智能和便捷。
Language:TypeScript
Total stars: 939
Stars trend:
19 Apr 2024
12am ▍ +3
1am █▉ +15
2am ██ +16
3am █▌ +12
4am ▋ +5
5am ▍ +3
6am █ +8
7am █▏ +9
8am ▍ +3
9am ▏ +1#typescript
#ai, #aicrawl, #chromium, #crawl, #crawler, #fingerprint, #flexible, #javascript, #multifunction, #nodejs, #puppeteer, #spider, #typescript
ngc660sec/NGCBot
A WeChat robot based on the ✨HOOK mechanism, supporting 🌱security news regular push [FreeBuf, Prophet, Security Guest, Qi Anxin Attack and Defense Community], 👯Kfc copywriting, ⚡️Registration query, ⚡️Mobile phone number location query, ⚡️WHOIS information query, 🎉 Horoscope query, ⚡️Weather query, 🌱Fishing calendar, ⚡️Microstep threat intelligence query, 🐛Beauty video, ⚡️Beauty pictures, 👯Help menu. 📫 Supports points function, ⚡️supports automatic pulling of people, ⚡️detects ads, 🌱automatic group sending, 👯Ai reply, 😄rich in customization, even beginners can easily get started!
Language:Python
Total stars: 691
Stars trend:
#python
#bot, #crawler, #security, #weixin, #weixin, #wxbot
A WeChat robot based on the ✨HOOK mechanism, supporting 🌱security news regular push [FreeBuf, Prophet, Security Guest, Qi Anxin Attack and Defense Community], 👯Kfc copywriting, ⚡️Registration query, ⚡️Mobile phone number location query, ⚡️WHOIS information query, 🎉 Horoscope query, ⚡️Weather query, 🌱Fishing calendar, ⚡️Microstep threat intelligence query, 🐛Beauty video, ⚡️Beauty pictures, 👯Help menu. 📫 Supports points function, ⚡️supports automatic pulling of people, ⚡️detects ads, 🌱automatic group sending, 👯Ai reply, 😄rich in customization, even beginners can easily get started!
Language:Python
Total stars: 691
Stars trend:
16 May 2024
3am ▍ +3
4am +0
5am ▏ +1
6am +0
7am ▎ +2
8am ▎ +2
9am ██▋ +21
10am █▍ +11
11am █▊ +14
12pm █▊ +14
1pm ▉ +7#python
#bot, #crawler, #security, #weixin, #weixin, #wxbot
projectdiscovery/katana
A next-generation crawling and spidering framework.
Language:Go
Total stars: 9079
Stars trend:
#go
#cli, #crawler, #gocrawler, #headless, #spiderframework, #webspider
A next-generation crawling and spidering framework.
Language:Go
Total stars: 9079
Stars trend:
18 Jun 2024
7am ▎ +2
8am ▏ +1
9am ▍ +3
10am ▌ +4
11am █▏ +9
12pm █ +8
1pm █▏ +9
2pm ██ +16
3pm █▊ +14
4pm █▏ +9#go
#cli, #crawler, #gocrawler, #headless, #spiderframework, #webspider
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Language:Python
Total stars: 208
Stars trend:
#python
#apify, #automation, #beautifulsoup, #crawler, #crawling, #headless, #headlesschrome, #pip, #playwright, #python, #scraper, #scraping, #webcrawler, #webcrawling, #webscraping
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Language:Python
Total stars: 208
Stars trend:
9 Jul 2024
7am ██▏ +17
8am █▍ +11
9am █▎ +10
10am ███▏ +25
11am ██▊ +22
12pm ███ +24
1pm ███▍ +27#python
#apify, #automation, #beautifulsoup, #crawler, #crawling, #headless, #headlesschrome, #pip, #playwright, #python, #scraper, #scraping, #webcrawler, #webcrawling, #webscraping
D4Vinci/Scrapling
Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 268
Stars trend:
#python
#automation, #crawler, #crawling, #crawlingpython, #css, #dommanipulation, #hacktoberfest, #lxml, #playwright, #python, #python3, #scraping, #selectors, #selenium, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 268
Stars trend:
11 Nov 2024
6am ▌ +4
7am ▌ +4
8am ▏ +1
9am ▏ +1
10am ▏ +1
11am ▋ +5
12pm ▎ +2
1pm ▌ +4
2pm ▉ +7
3pm ██▋ +21
4pm █▊ +14
5pm ███▏ +25#python
#automation, #crawler, #crawling, #crawlingpython, #css, #dommanipulation, #hacktoberfest, #lxml, #playwright, #python, #python3, #scraping, #selectors, #selenium, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
xiaoxiunique/x-kit
一个用于抓取和分析 X (Twitter) 用户数据和推文的工具。
Language:TypeScript
Total stars: 167
Stars trend:
#typescript
#crawler, #kols, #twitter, #x
一个用于抓取和分析 X (Twitter) 用户数据和推文的工具。
Language:TypeScript
Total stars: 167
Stars trend:
26 Dec 2024
6pm ▏ +1
7pm +0
8pm ▋ +5
9pm ▏ +1
10pm ▍ +3
11pm █ +8
27 Dec 2024
12am █▏ +9
1am █▎ +10
2am █▉ +15
3am █▎ +10
4am ▋ +5
5am █▏ +9#typescript
#crawler, #kols, #twitter, #x
mendableai/firecrawl
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Language:TypeScript
Total stars: 21906
Stars trend:
#typescript
#ai, #aiscraping, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler, #webscraping
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Language:TypeScript
Total stars: 21906
Stars trend:
20 Jan 2025
5pm ▏ +1
6pm ▎ +2
7pm ▉ +7
8pm ▊ +6
9pm ▉ +7
10pm █ +8
11pm ▋ +5
21 Jan 2025
12am ▉ +7
1am ▌ +4
2am ▍ +3
3am ██ +16
4am █▍ +11#typescript
#ai, #aiscraping, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler, #webscraping
Evil0ctal/Douyin_TikTok_Download_API
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
Language:Python
Total stars: 10524
Stars trend:
#python
#api, #async, #crawler, #douyin, #douyinapi, #douyinscraper, #douyintiktokapi, #douyintiktokdownload, #fastapi, #nowatermark, #onlineparsing, #python, #pywebio, #scraper, #spider, #tiktok, #tiktokapi, #tiktokscraper, #tiktoksignature, #webscraping
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
Language:Python
Total stars: 10524
Stars trend:
25 Jan 2025
11pm ▍ +3
26 Jan 2025
12am ▌ +4
1am █▎ +10
2am ▌ +4
3am ▊ +6
4am ▌ +4
5am █▏ +9
6am ██▏ +17
7am █ +8
8am ▌ +4
9am ▊ +6#python
#api, #async, #crawler, #douyin, #douyinapi, #douyinscraper, #douyintiktokapi, #douyintiktokdownload, #fastapi, #nowatermark, #onlineparsing, #python, #pywebio, #scraper, #spider, #tiktok, #tiktokapi, #tiktokscraper, #tiktoksignature, #webscraping
iawia002/lux
👾 Fast and simple video download library and CLI tool written in Go
Language:Go
Total stars: 28284
Stars trend:
#go
#bilibili, #crawler, #download, #downloader, #go, #golang, #iqiyi, #qq, #scraper, #tumblr, #video, #youku, #youtube
👾 Fast and simple video download library and CLI tool written in Go
Language:Go
Total stars: 28284
Stars trend:
28 Jan 2025
1pm █▌ +12
2pm █▎ +10
3pm █▍ +11
4pm █▉ +15
5pm █ +8
6pm ▌ +4
7pm ▉ +7
8pm ▎ +2
9pm ▏ +1
10pm ▎ +2
11pm █▋ +13#go
#bilibili, #crawler, #download, #downloader, #go, #golang, #iqiyi, #qq, #scraper, #tumblr, #video, #youku, #youtube
D4Vinci/Scrapling
🕷️ Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 2301
Stars trend:
#python
#ai, #aiscraping, #automation, #crawler, #crawling, #crawlingpython, #data, #dataextraction, #hacktoberfest, #playwright, #python, #python3, #scraping, #selectors, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
🕷️ Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 2301
Stars trend:
12 Feb 2025
6am ▍ +3
7am ▎ +2
8am ▋ +5
9am ▍ +3
10am ▊ +6
11am ▉ +7
12pm █▉ +15
1pm █▌ +12
2pm █▉ +15
3pm █▌ +12
4pm ▊ +6
5pm █▋ +13#python
#ai, #aiscraping, #automation, #crawler, #crawling, #crawlingpython, #data, #dataextraction, #hacktoberfest, #playwright, #python, #python3, #scraping, #selectors, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Language:Python
Total stars: 55667
Stars trend:
#python
#crawler, #crawling, #framework, #hacktoberfest, #python, #scraping, #webscraping, #webscrapingpython
Scrapy, a fast high-level web crawling & scraping framework for Python.
Language:Python
Total stars: 55667
Stars trend:
4 Jun 2025
10am ▋ +5
11am █▎ +10
12pm ▉ +7
1pm █ +8
2pm █▏ +9
3pm █▌ +12
4pm █▊ +14
5pm ▉ +7
6pm █▎ +10
7pm █▌ +12#python
#crawler, #crawling, #framework, #hacktoberfest, #python, #scraping, #webscraping, #webscrapingpython