coder-hxl/x-crawl
x-crawl is a flexible Node.js AI-assisted crawler library. Making crawler work more efficient, intelligent and convenient. ------ x-crawl 是一个灵活的 Node.js AI 辅助爬虫库。使爬虫工作变得更加高效、智能和便捷。
Language:TypeScript
Total stars: 939
Stars trend:
#typescript
#ai, #aicrawl, #chromium, #crawl, #crawler, #fingerprint, #flexible, #javascript, #multifunction, #nodejs, #puppeteer, #spider, #typescript
x-crawl is a flexible Node.js AI-assisted crawler library. Making crawler work more efficient, intelligent and convenient. ------ x-crawl 是一个灵活的 Node.js AI 辅助爬虫库。使爬虫工作变得更加高效、智能和便捷。
Language:TypeScript
Total stars: 939
Stars trend:
19 Apr 2024
12am ▍ +3
1am █▉ +15
2am ██ +16
3am █▌ +12
4am ▋ +5
5am ▍ +3
6am █ +8
7am █▏ +9
8am ▍ +3
9am ▏ +1#typescript
#ai, #aicrawl, #chromium, #crawl, #crawler, #fingerprint, #flexible, #javascript, #multifunction, #nodejs, #puppeteer, #spider, #typescript
ngc660sec/NGCBot
A WeChat robot based on the ✨HOOK mechanism, supporting 🌱security news regular push [FreeBuf, Prophet, Security Guest, Qi Anxin Attack and Defense Community], 👯Kfc copywriting, ⚡️Registration query, ⚡️Mobile phone number location query, ⚡️WHOIS information query, 🎉 Horoscope query, ⚡️Weather query, 🌱Fishing calendar, ⚡️Microstep threat intelligence query, 🐛Beauty video, ⚡️Beauty pictures, 👯Help menu. 📫 Supports points function, ⚡️supports automatic pulling of people, ⚡️detects ads, 🌱automatic group sending, 👯Ai reply, 😄rich in customization, even beginners can easily get started!
Language:Python
Total stars: 691
Stars trend:
#python
#bot, #crawler, #security, #weixin, #weixin, #wxbot
A WeChat robot based on the ✨HOOK mechanism, supporting 🌱security news regular push [FreeBuf, Prophet, Security Guest, Qi Anxin Attack and Defense Community], 👯Kfc copywriting, ⚡️Registration query, ⚡️Mobile phone number location query, ⚡️WHOIS information query, 🎉 Horoscope query, ⚡️Weather query, 🌱Fishing calendar, ⚡️Microstep threat intelligence query, 🐛Beauty video, ⚡️Beauty pictures, 👯Help menu. 📫 Supports points function, ⚡️supports automatic pulling of people, ⚡️detects ads, 🌱automatic group sending, 👯Ai reply, 😄rich in customization, even beginners can easily get started!
Language:Python
Total stars: 691
Stars trend:
16 May 2024
3am ▍ +3
4am +0
5am ▏ +1
6am +0
7am ▎ +2
8am ▎ +2
9am ██▋ +21
10am █▍ +11
11am █▊ +14
12pm █▊ +14
1pm ▉ +7#python
#bot, #crawler, #security, #weixin, #weixin, #wxbot
projectdiscovery/katana
A next-generation crawling and spidering framework.
Language:Go
Total stars: 9079
Stars trend:
#go
#cli, #crawler, #gocrawler, #headless, #spiderframework, #webspider
A next-generation crawling and spidering framework.
Language:Go
Total stars: 9079
Stars trend:
18 Jun 2024
7am ▎ +2
8am ▏ +1
9am ▍ +3
10am ▌ +4
11am █▏ +9
12pm █ +8
1pm █▏ +9
2pm ██ +16
3pm █▊ +14
4pm █▏ +9#go
#cli, #crawler, #gocrawler, #headless, #spiderframework, #webspider
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Language:Python
Total stars: 208
Stars trend:
#python
#apify, #automation, #beautifulsoup, #crawler, #crawling, #headless, #headlesschrome, #pip, #playwright, #python, #scraper, #scraping, #webcrawler, #webcrawling, #webscraping
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Language:Python
Total stars: 208
Stars trend:
9 Jul 2024
7am ██▏ +17
8am █▍ +11
9am █▎ +10
10am ███▏ +25
11am ██▊ +22
12pm ███ +24
1pm ███▍ +27#python
#apify, #automation, #beautifulsoup, #crawler, #crawling, #headless, #headlesschrome, #pip, #playwright, #python, #scraper, #scraping, #webcrawler, #webcrawling, #webscraping
D4Vinci/Scrapling
Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 268
Stars trend:
#python
#automation, #crawler, #crawling, #crawlingpython, #css, #dommanipulation, #hacktoberfest, #lxml, #playwright, #python, #python3, #scraping, #selectors, #selenium, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 268
Stars trend:
11 Nov 2024
6am ▌ +4
7am ▌ +4
8am ▏ +1
9am ▏ +1
10am ▏ +1
11am ▋ +5
12pm ▎ +2
1pm ▌ +4
2pm ▉ +7
3pm ██▋ +21
4pm █▊ +14
5pm ███▏ +25#python
#automation, #crawler, #crawling, #crawlingpython, #css, #dommanipulation, #hacktoberfest, #lxml, #playwright, #python, #python3, #scraping, #selectors, #selenium, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
👍2
xiaoxiunique/x-kit
一个用于抓取和分析 X (Twitter) 用户数据和推文的工具。
Language:TypeScript
Total stars: 167
Stars trend:
#typescript
#crawler, #kols, #twitter, #x
一个用于抓取和分析 X (Twitter) 用户数据和推文的工具。
Language:TypeScript
Total stars: 167
Stars trend:
26 Dec 2024
6pm ▏ +1
7pm +0
8pm ▋ +5
9pm ▏ +1
10pm ▍ +3
11pm █ +8
27 Dec 2024
12am █▏ +9
1am █▎ +10
2am █▉ +15
3am █▎ +10
4am ▋ +5
5am █▏ +9#typescript
#crawler, #kols, #twitter, #x
mendableai/firecrawl
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Language:TypeScript
Total stars: 21906
Stars trend:
#typescript
#ai, #aiscraping, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler, #webscraping
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Language:TypeScript
Total stars: 21906
Stars trend:
20 Jan 2025
5pm ▏ +1
6pm ▎ +2
7pm ▉ +7
8pm ▊ +6
9pm ▉ +7
10pm █ +8
11pm ▋ +5
21 Jan 2025
12am ▉ +7
1am ▌ +4
2am ▍ +3
3am ██ +16
4am █▍ +11#typescript
#ai, #aiscraping, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler, #webscraping
Evil0ctal/Douyin_TikTok_Download_API
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
Language:Python
Total stars: 10524
Stars trend:
#python
#api, #async, #crawler, #douyin, #douyinapi, #douyinscraper, #douyintiktokapi, #douyintiktokdownload, #fastapi, #nowatermark, #onlineparsing, #python, #pywebio, #scraper, #spider, #tiktok, #tiktokapi, #tiktokscraper, #tiktoksignature, #webscraping
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
Language:Python
Total stars: 10524
Stars trend:
25 Jan 2025
11pm ▍ +3
26 Jan 2025
12am ▌ +4
1am █▎ +10
2am ▌ +4
3am ▊ +6
4am ▌ +4
5am █▏ +9
6am ██▏ +17
7am █ +8
8am ▌ +4
9am ▊ +6#python
#api, #async, #crawler, #douyin, #douyinapi, #douyinscraper, #douyintiktokapi, #douyintiktokdownload, #fastapi, #nowatermark, #onlineparsing, #python, #pywebio, #scraper, #spider, #tiktok, #tiktokapi, #tiktokscraper, #tiktoksignature, #webscraping
iawia002/lux
👾 Fast and simple video download library and CLI tool written in Go
Language:Go
Total stars: 28284
Stars trend:
#go
#bilibili, #crawler, #download, #downloader, #go, #golang, #iqiyi, #qq, #scraper, #tumblr, #video, #youku, #youtube
👾 Fast and simple video download library and CLI tool written in Go
Language:Go
Total stars: 28284
Stars trend:
28 Jan 2025
1pm █▌ +12
2pm █▎ +10
3pm █▍ +11
4pm █▉ +15
5pm █ +8
6pm ▌ +4
7pm ▉ +7
8pm ▎ +2
9pm ▏ +1
10pm ▎ +2
11pm █▋ +13#go
#bilibili, #crawler, #download, #downloader, #go, #golang, #iqiyi, #qq, #scraper, #tumblr, #video, #youku, #youtube
D4Vinci/Scrapling
🕷️ Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 2301
Stars trend:
#python
#ai, #aiscraping, #automation, #crawler, #crawling, #crawlingpython, #data, #dataextraction, #hacktoberfest, #playwright, #python, #python3, #scraping, #selectors, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
🕷️ Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 2301
Stars trend:
12 Feb 2025
6am ▍ +3
7am ▎ +2
8am ▋ +5
9am ▍ +3
10am ▊ +6
11am ▉ +7
12pm █▉ +15
1pm █▌ +12
2pm █▉ +15
3pm █▌ +12
4pm ▊ +6
5pm █▋ +13#python
#ai, #aiscraping, #automation, #crawler, #crawling, #crawlingpython, #data, #dataextraction, #hacktoberfest, #playwright, #python, #python3, #scraping, #selectors, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Language:Python
Total stars: 55667
Stars trend:
#python
#crawler, #crawling, #framework, #hacktoberfest, #python, #scraping, #webscraping, #webscrapingpython
Scrapy, a fast high-level web crawling & scraping framework for Python.
Language:Python
Total stars: 55667
Stars trend:
4 Jun 2025
10am ▋ +5
11am █▎ +10
12pm ▉ +7
1pm █ +8
2pm █▏ +9
3pm █▌ +12
4pm █▊ +14
5pm ▉ +7
6pm █▎ +10
7pm █▌ +12#python
#crawler, #crawling, #framework, #hacktoberfest, #python, #scraping, #webscraping, #webscrapingpython
NaiboWang/EasySpider
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
Language:JavaScript
Total stars: 39505
Stars trend:
#javascript
#batchprocessing, #batchscript, #codefree, #crawler, #datacollection, #frontend, #gui, #html, #inputparameters, #layman, #parameters, #robotics, #rpa, #scraper, #spider, #visual, #visualization, #visualprogramming, #web, #www
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
Language:JavaScript
Total stars: 39505
Stars trend:
3 Jul 2025
11pm █▌ +12
4 Jul 2025
12am ██▋ +21
1am ███▎ +26
2am ██▌ +20
3am ██▏ +17
4am █▏ +9
5am █▍ +11
6am █▋ +13
7am █▏ +9
8am ▋ +5
9am ▎ +2
10am ▎ +2#javascript
#batchprocessing, #batchscript, #codefree, #crawler, #datacollection, #frontend, #gui, #html, #inputparameters, #layman, #parameters, #robotics, #rpa, #scraper, #spider, #visual, #visualization, #visualprogramming, #web, #www
D4Vinci/Scrapling
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
Language:Python
Total stars: 5620
Stars trend:
#python
#ai, #aiscraping, #automation, #crawler, #crawling, #crawlingpython, #data, #dataextraction, #hacktoberfest, #playwright, #python, #python3, #scraping, #selectors, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
Language:Python
Total stars: 5620
Stars trend:
7 Jul 2025
8pm ██▏ +17
9pm █▎ +10
10pm █▋ +13
11pm █▋ +13
8 Jul 2025
12am █▎ +10
1am █▌ +12#python
#ai, #aiscraping, #automation, #crawler, #crawling, #crawlingpython, #data, #dataextraction, #hacktoberfest, #playwright, #python, #python3, #scraping, #selectors, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
hect0x7/JMComic-Crawler-Python
Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀
Language:Python
Total stars: 3415
Stars trend:
#python
#18comic, #crawler, #downloader, #githubactions, #jmcomic, #pypi, #python, #readthedocs
Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀
Language:Python
Total stars: 3415
Stars trend:
28 Jul 2025
1am ▎ +2
2am ▍ +3
3am ▊ +6
4am ▌ +4
5am ▊ +6
6am ▍ +3
7am █▎ +10
8am █▍ +11
9am █▍ +11
10am █ +8
11am █▊ +14
12pm █ +8#python
#18comic, #crawler, #downloader, #githubactions, #jmcomic, #pypi, #python, #readthedocs