A module that integrates selenium and requests session, encapsulates common page operations, can achieve seamless switching between the two modes.
-
Updated
Apr 16, 2021 - Python
#
crawlers
Here are 99 public repositories matching this topic...
An R web crawler and scraper
-
Updated
May 22, 2020 - R
TelegramBot that finds flats on immobilienscout24.de, wg-gesucht.de, ebay-kleinanzeigen.de, immowelt.de, idealista.it, subito.it and immobiliare.it
-
Updated
Jun 20, 2021 - Python
Powerful dork searcher and vulnerability scanner for windows platform
crawler
automation
sql
csharp
exploit
hacking
xss
smartthings
vulnerability
crawlers
scrapper
xss-detection
scanners
dork
lfi
lfi-detection
-
Updated
Mar 5, 2021 - C#
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
search-engine
whitelist
user-agent
seo
crawling
twitterbot
robots-txt
googlebot
crawlers
web-crawling
bingbot
robots-exclusion-standard
blocking-bots
web-robots
search-engine-optimization
baiduspider
-
Updated
Dec 17, 2020
hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)
-
Updated
Jun 11, 2021 - Python
Tổng hợp script crawl dữ liệu từ các mạng xã hội & website tiếng Việt
instagram
crawler
scraper
youtube
requests
crawlers
scraping-websites
crawling-framework
selenium-python
-
Updated
Jan 12, 2021 - Python
Proxy List Scrapper
http
crawler
data-mining
https
proxy
socks
socks5
proxies
rotating-proxy
proxypool
proxy-list
crawlers
scrapper
socks4
web-scrapper
web-scrapping
freeproxy
free-proxies
freeproxylist
proxyscrape
-
Updated
Apr 3, 2021 - Python
Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.
java
crawler
information-retrieval
data-mining
scraper
automation
framework
webdriver
scraping
crawling
selenium
information-extraction
crawl
crawlers
extract-data
dynamic-webpages
webspider
crawling-framework
selenium-crawler
scraping-framework
dynamic-website
-
Updated
Jun 23, 2021 - Java
An open source web crawling platform
-
Updated
May 6, 2018 - Go
htcap is a web application scanner able to crawl single page application (SPA) in a recursive manner by intercepting ajax calls and DOM changes.
crawler
security-audit
single-page-app
scanner
crawling
single-page-applications
crawlers
scanning
vulnerability-detection
vulnerability-scanners
javascript-crawler
-
Updated
Mar 21, 2018 - Python
Licitações de Feira de Santana de fácil acesso aos cidadãos 🏦
-
Updated
Mar 2, 2020 - Python
Tiny script to crawl information of a specific application in the Google play/store base on PHP.
api
php
crawler
google-play
google-play-services
crawlers
hacktoberfest
php-crawler
google-play-store
google-play-games
google-play-service
google-playstore
hacktoberfest2020
google-play-api
crawler-php
-
Updated
Jan 1, 2021 - PHP
Scrape production information on Amazon
-
Updated
Oct 31, 2018
User agent database in JSON format of bots, crawlers, certain malware, automated software, scripts and uncommon ones.
-
Updated
Nov 22, 2020 - Shell
A web crawler which crawls the stackoverflow website.
crawler
web-crawler
stackoverflow
crawling
text-processing
crawlers
stackoverflow-questions
stackoverflow-api
python-crawler
web-crawler-python
stackoverflow-answer
stackoverflow-crawler
crawler-testing
stackoverflow-tags
crawler-detector
text-processor
crawler-python
test-crawler
stackoverflow-get
stackoverflow-analyse
-
Updated
Aug 4, 2019 - Python
A crawler program to extract all of the data and the price for symbols in the global stock exchange.
crawler
crawling
stock
stock-market
stock-price-prediction
crawlers
stock-data
stock-prices
stock-exchanges
crawl-pages
stock-exchange-simulator
stock-prediction
crawling-framework
crawlspider
stock-analysis
stock-trading
stock-exchange
crawling-sites
stock-exchange-platform
stock-exchange-crawler
-
Updated
Jul 2, 2019 - PHP
A bot to login in Twitter and process page with selenium using Python.
crawler
twitter-bot
twitter
crawlers
twitter-crawler
selenium-crawler
twitter-python
selenium-sample
selenium-example
twitter-py
twitter-selenium
selenium-twitter
crawler-twitter
-
Updated
Nov 7, 2019 - Python
Data and Feature Catalogue in Go
metadata
machine-learning
catalogue
crawlers
feature-engineering
metadata-extraction
feature-catalogue
-
Updated
Jun 10, 2021 - Go
A Web Crawler developed in Python.
python
crawler
web
script
scripting
web-crawler
scripts
python-script
scripting-language
python3
python-3
crawlers
webcrawler
web-crawling
web-crawler-python
webcrawling
webcrawl
crawler-python
web-crawlers
-
Updated
Aug 17, 2020 - Python
Some sample codes for using selenium in Python just for fun.
python
crawler
crawling
selenium
python3
crawl
crawlers
selenium-tests
selenium-python
python-selenium
selenium-sample
selenium-example
selenium-py
selenium-website
-
Updated
Oct 31, 2019 - Python
-
Updated
Jul 30, 2021 - JavaScript
Repository of designing a crawler script to update a mirror database from Google Play on PHP.
-
Updated
Nov 6, 2020 - PHP
Improve this page
Add a description, image, and links to the crawlers topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the crawlers topic, visit your repo's landing page and select "manage topics."
An outage in myip.ms caused one of the lists to fail downloading - which caused error in testing, and was not recoverable.
Think of creating backup files for recovery from list providers outages