Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

A collection of awesome web crawler,spider in different languages

Apache Nutch is an extensible and scalable web crawler

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

简单易用的Python爬虫框架，QQ交流群：597510560

A scalable, mature and versatile web crawler based on Apache Storm

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Internet search engine for text-oriented websites. Indexing the small, old and weird web.

扫描“微信读书”已购图书并下载本地PDF的爬虫

Run a high-fidelity browser-based crawler in a single Docker container

Opensource Korean chatbot framework

ACHE is a web crawler for domain-specific search.

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

The simple, easy to use command line web crawler.

GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Library for Rapid (Web) Crawler and Scraper Development