Elegant Scraper and Crawler Framework for Golang
#
spider
Repositories 1,588
爬虫集合
Updated Dec 6, 2018
Python爬虫代理IP池(proxy pool)
[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.
crawler
spider
multi-interface
golang
distributed-crawler
high-concurrency-crawler
fastest-crawler
cross-platform-crawler
Go
Updated Mar 5, 2019
越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）
HTML
Updated Jan 11, 2019
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
JavaScript
Updated Mar 12, 2019
Incredibly fast crawler designed for OSINT.
Python
Updated Mar 21, 2019
A collection of awesome web crawler,spider in different languages
Updated Feb 11, 2019
Python
Updated Nov 11, 2018
Every web site provides APIs.
Python
Updated Dec 6, 2018
BitTorrent DHT Protocol && DHT Spider.
Go
Updated Mar 20, 2019
Web crawling framework based on asyncio.
Python
Updated Mar 19, 2018
admin ui for scrapy/open source scrapinghub
Python
Updated Mar 21, 2019
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be ex…
Go
Updated Nov 16, 2017
owllook-在线网络小说阅读网站&小说搜索引擎&小说推荐系统[搜索、追书、收藏、追更、小说API]
Python
Updated Mar 14, 2019
简单易用的Python爬虫框架，QQ交流群：597510560
Python
Updated Feb 26, 2019
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js
JavaScript
Updated Mar 3, 2019
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
PHP
Updated Aug 30, 2018
PHP
Updated Mar 20, 2019
Creating Scrapy scrapers via the Django admin interface
Python
Updated Feb 15, 2019
A configurable web spider with a easy-to-use web console
Java
Updated Aug 21, 2018
Nodejs实现的一个磁力链接爬虫 http://findit.keenwon.com (原域名http://findit.so )
JavaScript
Updated Dec 28, 2018
Async Python 3.6+ web scraping micro-framework based on asyncio.
Python
Updated Mar 13, 2019
链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁
Python
Updated Feb 16, 2019
zhihu-crawler是一个基于Java的爬虫实战项目
Java
Updated Mar 6, 2019
Your web font utility belt. It can subset web fonts. It can find unicode-ranges for you automatically. It makes julie…
JavaScript
Updated Jan 3, 2019
A high performance web crawler in Elixir.
Free Web Scraping Tool with Java
JavaScript
Updated Feb 3, 2019