Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.

Updated Jun 11, 2020
Java

Potelo / laravel-block-bots

Star

Block crawlers and high traffic users on your site by IP using Redis

laravel bots crawlers scrapper

Updated May 14, 2020
PHP

macabeus / spyck

Star

⛏ An extensible framework for data mining

crawlers

Updated Jun 9, 2016
Python

solidusio-contrib / solidus_sitemap

Star

Provide a sitemap of your Solidus store.

sitemap ecommerce google product solidus crawlers

Updated Jun 26, 2020
Ruby

versioneye / crawl_r

Star

VersionEye crawlers implemented in Ruby.

ruby crawlers versioneye

Updated Feb 5, 2020
Roff

octoparse / Scrape-Amazon

Star

Scrape production information on Amazon

ecommerce scraper amazon crawlers scraping-websites

Updated Oct 31, 2018

BaseMax / StackoverflowCrawler

Star

A web crawler which crawls the stackoverflow website.

crawler web-crawler stackoverflow crawling text-processing crawlers stackoverflow-questions stackoverflow-api python-crawler web-crawler-python stackoverflow-answer stackoverflow-crawler crawler-testing stackoverflow-tags crawler-detector text-processor crawler-python test-crawler stackoverflow-get stackoverflow-analyse

Updated Aug 4, 2019
Python

michaelradu / web-crawler

Star

A Web Crawler developed in Python.

python crawler web script scripting web-crawler scripts python-script scripting-language python3 python-3 crawlers webcrawler web-crawling web-crawler-python webcrawling webcrawl crawler-python web-crawlers

Updated Aug 17, 2020
Python

BaseMax / FirstSelenium

Star

Some sample codes for using selenium in Python just for fun.

python crawler crawling selenium python3 crawl crawlers selenium-tests selenium-python python-selenium selenium-sample selenium-example selenium-py selenium-website

Updated Oct 31, 2019
Python

stamaimer / MPWeChatRSS

Star

Article of WeChat to RSS

wechat crawlers rss-feed-scraper

Updated Mar 25, 2018
Python

BaseMax / StockExchangeCrawler

Star

A crawler program to extract all of the data and the price for symbols in the global stock exchange.

crawler crawling stock stock-market stock-price-prediction crawlers stock-data stock-prices stock-exchanges crawl-pages stock-exchange-simulator stock-prediction crawling-framework crawlspider stock-analysis stock-trading stock-exchange crawling-sites stock-exchange-platform stock-exchange-crawler

Updated Jul 2, 2019
PHP

BaseMax / TwitterBotCrawler

Star

A bot to login in Twitter and process page with selenium using Python.

crawler twitter-bot twitter crawlers twitter-crawler selenium-crawler twitter-python selenium-sample selenium-example twitter-py twitter-selenium selenium-twitter crawler-twitter

Updated Nov 7, 2019
Python

narkhedesam / Proxy-List-Scrapper

Star

Proxy List Scrapper

crawler data-mining proxy proxies rotating-proxy proxy-list crawlers scrapper web-scrapper web-scrapping freeproxy free-proxies freeproxylist proxyscrape

Updated Sep 1, 2020
Python

beyondcoin-project / beyondcoin-seeder

Star

Dnsseeder for the Beyondcoin protocol

dns-server crawlers dnsseed beyondcoin

Updated Aug 4, 2020
C++

dlion / IsGoogle

Star

Performs a reverse lookup for a certain ip to check if it belongs to Google crawlers

go dns golang ip check crawlers google-crawler

Updated May 7, 2018
Go

herrbischoff / robots.txt

Star

A sane, minimal robots.txt file (for the western world)

robots-txt crawlers robotstxt

Updated Jul 31, 2020

FEZIRO / wechat-miniprogram-spider-demo

Star

微信小程序云开发网络爬虫教程

cheerio axios crawlers cloudfunction

Updated May 8, 2020
JavaScript

IvRRimum / ivro_check

Star

Ussfull tool for Hackers

ruby crawler simple crawlers

Updated Nov 16, 2013

marinakiseleva / wiki-scraper

Star

This web crawler uses Scrapy py to crawl Wikipedia. It prints the page title, total word count, and page category (using openpyxl) to an Excel workbook, in order to analyze the verbosity of articles by category.