Scrapy Plugins

Repositories

• scrapy-crawlera-fetch

  Scrapy Downloader Middleware for Crawlera Fetch API
  Python BSD-3-Clause 6 3 1 0 Updated Feb 16, 2021

• scrapy-jsonschema

  Scrapy schema validation pipeline and Item builder using JSON Schema
  Python BSD-3-Clause 10 35 1 1 Updated Feb 2, 2021

• scrapy-magicfields

  Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.
  Python BSD-3-Clause 7 46 0 2 Updated Dec 16, 2020

• scrapy-splash

  Scrapy+Splash for JavaScript integration

  scrapy hacktoberfest headless-browsers
  Python 392 2,484 44 16 Updated Dec 5, 2020

• scrapy-crawlera

  Crawlera middleware for Scrapy

  plugin crawler proxy scraping scrapy crawler-detection
  Python 77 280 4 (1 issue needs help) 1 Updated Dec 1, 2020

• scrapy-deltafetch

  Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls
  Python 41 212 13 5 Updated Sep 17, 2020

• scrapy-headless
  Python BSD-3-Clause 5 15 2 1 Updated Jul 26, 2020

• scrapy-dotpersistence

  A scrapy extension to sync `.scrapy` folder to an S3 bucket
  Python BSD-3-Clause 7 9 1 0 Updated May 21, 2020

• scrapy-feedexporter-sftp
  Python BSD-3-Clause 3 7 1 1 Updated May 14, 2020

• scrapy-statsd
  Python BSD-3-Clause 3 3 1 2 Updated Mar 8, 2020

• scrapy-pagestorage

  A scrapy extension to store requests and responses information in storage service
  Python BSD-3-Clause 6 23 2 0 Updated Oct 30, 2019

• scrapy-jsonrpc

  Scrapy extension to control spiders using JSON-RPC
  Python 66 262 7 (1 issue needs help) 2 Updated Aug 26, 2019

• scrapy-hcf

  Scrapy spider middleware to use Scrapinghub's Hub Crawl Frontier as a backend for URLs
  Python BSD-3-Clause 6 4 1 1 Updated Aug 28, 2018

• scrapy-djangoitem

  Scrapy extension to write scraped items using Django models
  Python BSD-3-Clause 81 469 9 2 Updated Dec 9, 2017

• scrapy-monkeylearn

  A Scrapy pipeline to categorize items using MonkeyLearn
  Python 13 39 0 0 Updated Apr 28, 2017

• scrapy-splitvariants

  Scrapy spider middleware to split an item into multiple items using a multi-valued key
  Python BSD-3-Clause 1 18 0 0 Updated Feb 8, 2017

• scrapy-streaming
  Python 6 17 2 6 Updated Oct 12, 2016

• scrapy-querycleaner

  Scrapy spider middleware to clean up query parameters in request URLs
  Python BSD-3-Clause 1 20 0 0 Updated Jun 30, 2016

• scrapy-bigml

  Scrapy pipeline for writing items to BigML datasets
  Python 3 4 0 0 Updated Nov 16, 2015

• scrapy-streamitem

  Scrapy support for working with streamcorpus Stream Items.
  Python 3 9 0 0 Updated Jan 7, 2015