A collection of awesome web crawler,spider in different languages
-
Updated
May 24, 2020
#
web-scraper
Here are 340 public repositories matching this topic...
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
api
php
http
client
json
framework
curl
xml
proxy
restful
class
http-client
http-proxy
api-client
web-scraper
requests
web-scraping
php-curl
web-service
php-curl-library
-
Updated
May 23, 2020 - PHP
Web Scraper in Go, similar to BeautifulSoup
-
Updated
May 7, 2020 - Go
A framework for creating semi-automatic web content extractors
python
crawler
tutorial
extractor
scraping
web-scraper
selector
css-selector
web-scraping
scrapy
scrapers
beautifulsoup
xpath-expression
lxml
selector-expression
-
Updated
Oct 12, 2019 - Python
Faster requests on Python 3
python
curl
high-performance
cython
python-library
web-scraper
python3
speed
open-data
http-requests
web-scraping
scrapy
ndjson
python-requests
urllib
download-file
urllib3
faster-than-requests
requests3
requests-toolbelt
-
Updated
Jun 28, 2020 - Python
A list of scrapers from around the web.
-
Updated
Jun 25, 2020
A simple browser/client-side web scraper.
-
Updated
Apr 24, 2017 - TypeScript
NBA Stats API via Basketball Reference
-
Updated
Jun 29, 2020 - HTML
An app that downloads novels and generate e-books from online sources.
-
Updated
Jul 1, 2020 - Python
Bulk download your favourite anime episodes from your favourite anime websites
ffmpeg
anime
web-scraper
anime-search
anime-fans
anime-downloader
anime-scraper
9anime
hls-downloader
animeultima
animepahe-downloader
animepahe
4anime
monkey-dl
-
Updated
May 12, 2020 - Python
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
-
Updated
Jun 21, 2020 - Python
A collection of awesome web scaper, crawler.
-
Updated
Jan 21, 2020
MetaData html scraper and parser for Node.js (supports Promises and callback style)
-
Updated
Nov 20, 2019 - JavaScript
-
Updated
Jul 7, 2020 - JavaScript
Python web scrapers on commodity future exchanges including CME, LME, SHFE and influential media websites including BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
regex
regular-expression
web-scraper
web-scraping
newsletter
reuters
bloomberg
futures
web-scrapers
scrapper
news-websites
commodity
html-email
news-scraper
futures-historical-data
sraping
python-web-scraper
future-exchanges
financial-times
wall-street-journal
-
Updated
Jun 26, 2020 - Python
PHP Library for detecting CMS
-
Updated
Feb 24, 2020 - PHP
Open
Add Estate Agent
AnthonyBloomer
commented
Sep 3, 2018
Scrape estate agent information. Example estate agent page: https://www.daft.ie/dublin/estate-agents/dublin-2/herlihy/
Powerful web scraping framework for Crystal
-
Updated
Jun 21, 2020 - Crystal
Go cascadia package command line CSS selector
tsv
command-line
curl
extract
web-scraper
css-selector
web-scraping
html-source
command-line-tool
csv-table
cascadia
html-text
-
Updated
Apr 19, 2020 - Go
A pluggable, simple and powerful web scraper.
-
Updated
Jul 6, 2020 - JavaScript
louisgv
commented
Jul 3, 2017
It would be great if the application can handle default cases where:
song NAME OF SONG
will default to
song -y NAME OF SONG
sudiptosarkar
commented
Jun 5, 2018
README.md needs to be updated to document the new -m option to generate masscan command from ipinfo.io network (-n).
Web Scraping Craigslist's Engineering Jobs in NY with Scrapy
python
scrapy-spider
web-scraper
craigslist
web-scraping
scrapy
web-crawling
scrapy-crawler
scrapy-tutorial
-
Updated
Aug 5, 2017 - Python
9
arkon
commented
Apr 17, 2016
We should scrape the important dates info off of places like the Faculty of Arts & Science or UTM websites.
EDIT: This is a better list
Adult XXX Addons (18+) for the Kodi Media Center - Kodi is a registered trademark of the XBMC Foundation. We are not connected to or in any other way affiliated with Kodi - DMCA: legal@tvaddons.co
-
Updated
Jun 11, 2020 - Python
Yellowpages.com Web Scraper written in Python and LXML to extract business details available based on a particular category and location.
python
html
scraper
parsing
extract
web-scraper
lxml
yellow-pages
business-directory
yellow-pages-scraper
-
Updated
Jun 8, 2020 - Python
-
Updated
Jun 5, 2020 - JavaScript
State of the art object detection in real-time using YOLOV3 algorithm. Augmented with a process that allows easy training of the classifier as a plug & play solution . Provides alert if an item in an alert list is detected.
machine-learning
alerts
real-time
deep-learning
tensorflow
image-processing
web-scraper
automatic
yolo
image-recognition
webcam
convolutional-neural-networks
object-detection
webscraping
darknet
yolo2
video-recognition
yolov2
google-image-search
yolov2-model
-
Updated
Feb 17, 2019 - Python
Improve this page
Add a description, image, and links to the web-scraper topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the web-scraper topic, visit your repo's landing page and select "manage topics."
when run with following code :
with MyConnectionScraper(cookie='AQEDAS9oddoAec7fAAABcD_bsSwAAAFwY-g1LFEAR5RwykzJFoxZQ1ZjaMH2vcXgsasLMFb0GwyGbqgh_guqW-K122YvSwg2_zhDnX_gbpdXrYjPqY5Mq9U2o3KmrfQCYbImYSAmUOTVDuPpUoBPkS') as scraper:
connections=scraper.scrape()
people=connections.to_dict()
no such element: Unable to locate element: {"method":"css selector","selector":".mn-connections > h