Topic: crawling

scrapy / scrapy

32.1k

Scrapy, a fast high-level web crawling & scraping framework for Python.

python scraping crawling framework crawler

Python Updated Mar 29, 2019 46 issues need help

codelucas / newspaper

7.8k

News, full-text, and article metadata extraction in Python 3. Advanced docs:

python news crawler crawling scraper news-aggregator

Python Updated Mar 17, 2019

gocolly / colly

7.2k

Elegant Scraper and Crawler Framework for Golang

golang scraper framework crawler scraping crawling spider go

Go Updated Mar 28, 2019 5 issues need help

yujiosaka / headless-chrome-crawler

3.9k

Distributed crawler powered by Headless Chrome

headless-chrome puppeteer jquery crawler crawling scraper scraping chrome chromium promise

JavaScript Updated Mar 30, 2019

MontFerret / ferret

3.5k

Declarative web scraping

golang query-language data-mining scraping scraping-websites dsl cdp crawling scraper crawler go chrome

Go Updated Mar 29, 2019 9 issues need help

iawia002 / Lulu

637

[Unmaintained] A simple and clean video/music/image downloader 👾

downloader video python python3 crawler scraper crawling scraping

Python Updated Apr 2, 2018 2 issues need help

transitive-bullshit / awesome-puppeteer

619

A curated list of awesome puppeteer resources.

puppeteer headless-chrome awesome awesome-list scraping crawling automation

Updated Mar 21, 2019

essandess / isp-data-pollution

375

ISP Data Pollution to Protect Private Browsing History with Obfuscation

web crawling data obfuscation privacy-enhancing-technologies data-analytics privacy

Python Updated Dec 16, 2018

zhuyingda / webster

245

a reliable high-level web crawling & scraping framework for Node.js.

scraping-framework crawler crawling headless-chrome chromium spider automation-ui automation-test nodejs nodejs-framework javascript javascript-framework

JavaScript Updated Jul 5, 2018 1 issue needs help

slotix / dataflowkit

242

Extract structured data from web sites. Web sites scraping.

golang golang-library extract-data chrome-fetcher scraping-websites crawling scraper scraping cdp go headless

Go Updated Mar 29, 2019

MorvanZhou / easy-scraping-tutorial

239

Simple but useful Python web scraping tutorial code.

beautifulsoup scrapy regex urllib scraping crawler crawling requests asyncio distributed-scraper

Jupyter Notebook Updated Jul 25, 2018

DarkSand / Sasila

236

一个灵活、友好的爬虫框架

python scraping crawling framework crawler http requests

Python Updated Dec 13, 2017

webfolderio / cdp4j

232

cdp4j - Chrome DevTools Protocol for Java

java chromium chrome test-automation chrome-debugger-protocol chrome-devtools chrome-developer-protocol automation cdp web web-automation chrome-devtools-protocol selenium selenium-webdriver webdriver crawling crawling-framework chrome-headless

Java Updated Mar 20, 2019

infinitbyte / gopa

230

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

spider crawler lightweight elasticsearch web-crawler crawling web-spider web-scraping scraping

Go Updated Feb 23, 2019

rivermont / spidy

192

The simple, easy to use command line web crawler.

web-crawler web-spider python python3 crawling crawler

Python Updated Oct 25, 2018 1 issue needs help

stopstalk / stopstalk-deployment

156

Stop stalking and start StopStalking 😉

web2py competitive-programming python materializecss crawling stalking codechef codeforces spoj hackerearth hackerrank uva programming-contests online-judges aws timus stopstalk

Python Updated Mar 25, 2019 39 issues need help

alephdata / memorious

136

Distributed crawling framework for documents and structured data.

crawling scraping scraping-framework

Python Updated Mar 26, 2019

clemfromspace / scrapy-selenium

129

Scrapy middleware to handle javascript pages using selenium

scrapy selenium crawling

Python Updated Feb 25, 2019

antchfx / antch

127

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

golang web-crawler crawling scraping framework crawler web-spider

Go Updated Jul 28, 2018

forkonlp / N2H4

124

네이버 뉴스 수집을 위한 도구

naver news crawler crawling korean getcomments sort

R Updated Mar 6, 2019

dimkouv / massivedl

108

Download a large list of files in parallel

golang crawling downloader download-manager

Go Updated Feb 19, 2019

N0taN3rd / Squidwarc

73

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

webarchiving webarchives crawler high-fidelity-preservation chrome-headless chrome puppeteer headless-chrome crawling browser-automation

JavaScript Updated Mar 12, 2019

estin / pomp

59

Screen scraping and web crawling framework

python scraping crawling asyncio framework crawler

Python Updated Apr 25, 2017

googlei18n / corpuscrawler

57

Crawler for linguistic corpora

corpus-linguistics corpus-builder crawling linguistics minority-language

Python Updated Aug 29, 2018

howie6879 / talospider

53

talospider - A simple,lightweight scraping micro-framework

spider web-spider python crawling crawler

Python Updated Feb 22, 2019

TransparencyToolkit / Harvester

50

Web crawling and document processing through a usable interface.

web crawling osint document ocr api interface

JavaScript Updated Jul 22, 2017

Swader / diffbot-php-client

47

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

diffbot php crawling crawl scrape scraping scraper scraped-data machine-learning nlp ai artificial-intelligence bot

PHP Updated Jul 4, 2018 2 issues need help

scrapinghub / learn.scrapinghub.com

44

Scrapinghub Learning Center. Report issues in Jira: Report issues in Jira: https://scrapinghub.atlassian.net/projects…

scrapy python scraping crawling learning tutorial

CSS Updated Feb 20, 2019

afunTW / Python-Crawling-Tutorial

43

Python crawling tutorial

python ipynb-jupyter-notebook crawling

Jupyter Notebook Updated Oct 20, 2018

guptachetan1997 / crawling-projects

38

Web scraping and automation using python

crawling scraping-websites automation

Python Updated Oct 3, 2017

crawling

Repositories 310