Scrapy project

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling

Python 8,375 36,115 426 (39 issues need help) 286 Updated Feb 20, 2020
scrapyd

A service daemon to run Scrapy spiders

Python 477 1,987 85 24 Updated Feb 20, 2020
queuelib

Collection of persistent (disk-based) queues

Python 45 188 2 6 Updated Feb 19, 2020
w3lib

Python library of web-related functions

python

Python 81 301 10 14 Updated Feb 18, 2020
scrapy.org

The scrapy.org website

html

HTML 93 38 3 5 Updated Feb 14, 2020
parsel

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

css python xml scraping selectors xpath lxml

Python 87 505 16 14 Updated Jan 3, 2020
protego

A pure-Python robots.txt parser with support for modern conventions.

python robots-txt robots-parser

DIGITAL Command Language BSD-3-Clause 8 9 0 1 Updated Jan 3, 2020
scrapyd-client

Command line client for Scrapyd server

Python 93 507 15 9 Updated Dec 6, 2019
scrapely

A pure-python HTML screen-scraping library

HTML 242 1,647 24 5 Updated Nov 28, 2019
cssselect

CSS Selectors for Python

css python selectors

Python 43 209 15 11 Updated Nov 14, 2019
scrapy-bench

A CLI for benchmarking Scrapy.

python web-crawler scrapy command-line-tool benchmark-suite scrapy-bench

Python MIT 12 23 6 0 Updated Oct 28, 2019
quotesbot

This is a sample Scrapy project for educational purposes

Python MIT 568 839 0 5 Updated Oct 21, 2019
loginform

Fill HTML login forms automatically

Python 64 227 9 2 Updated Oct 18, 2019
booksbot
Forked from stummjr/books_crawler
A crawler for http://books.toscrape.com

Python 516 22 0 1 Updated Mar 12, 2019
scurl

Performance-focused replacement for Python urllib

python cython chromium gurl urlparse

Python Apache-2.0 5 14 10 (1 issue needs help) 1 Updated Oct 2, 2018
url-chromium

url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url

chromium gurl

C++ 1 0 0 0 Updated Aug 8, 2018
base-chromium

base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/

C++ 1 0 0 0 Updated Aug 1, 2018
dirbot

Scrapy project to scrape public web directories (educational) [DEPRECATED]

Python 1,137 1,585 0 0 Updated Oct 27, 2017
scrapy-bench-speedcenter
Forked from Parth-Vader/scrapy-bench-speedcenter
Codespeed for scrapy-bench

Python 1 2 0 0 Updated Aug 28, 2017
pypydispatcher

A fork of http://pydispatcher.sourceforge.net/ with PyPy support

Python 2 10 1 0 Updated Jul 3, 2017
scrapy-itemloader

Library to populate Scrapy items using XPath and CSS with a convenient API

Python BSD-3-Clause 6 5 2 0 Updated Jul 26, 2016
slybot

57 221 5 0 Updated Apr 27, 2015
gsoc2014-integration-tests

GSoC2014 - Scrapy Integration tests project

Shell 2 3 0 0 Updated Mar 18, 2014

Top languages

Python C++ HTML Shell DIGITAL Command Language

Most used topics

Loading…

Grow your team on GitHub

Top languages

Most used topics

People