#

crawl

Here are 164 public repositories matching this topic...

jhao104 / proxy_pool

Star

Python爬虫代理IP池(proxy pool)

redis flask crawler spider proxy crawl proxypool ssdb

Updated Sep 2, 2020
Python

kangvcar / InfoSpider

Star

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。

chrome automation spider selenium python3 crawl tkinter outlook wxpython csdn hotmail

Updated Oct 21, 2020
Python

wkunzhi / Python3-Spider

Star

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

python crawler spider selenium crawl scrapy splash geek taobao scrapy-crawler meituan dianping pyppeteer

Updated Jul 24, 2020
Python

game-helper / weixin

Star

微信小游戏辅助合集（加减大师、包你懂我、大家来找茬腾讯版、头脑王者、好友画我、悦动音符、我最在行、星途WeGoing、猜画小歌、知乎答题王、腾讯中国象棋、跳一跳、题多多黄金版）

nodejs game python socket helper robot mongodb crawl weixin charles fiddler anyproxy

Updated Jul 24, 2018
JavaScript

ArchiveTeam / grab-site

Star

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

crawler spider archiving crawl warc

Updated Aug 7, 2020
Python

liip / TheA11yMachine

Star

The A11y Machine is an automated accessibility testing tool which crawls and tests pages of any web application to produce detailed reports.

accessibility test crawl wcag

Updated Dec 17, 2019
JavaScript

rugantio / fbcrawl

Star

Open

Add crawling for events?

7

littlewine commented Apr 8, 2019

Is there an option to crawl events out of Facebook?
If not, would it be easy to implement? I could assist if there is interest for that.

Read more

enhancement good first issue help wanted

zkqiang / zhihu-login

Star

知乎模拟登录，支持提取验证码和保存 Cookies

python crawler spider login cookie zhihu crawl execjs

Updated Apr 11, 2020
Python

darbra / geetest

Star

geetest，滑动验证码

spider crawl geetest

Updated Dec 4, 2017
Python

zhangslob / awesome_crawl

Star

腾讯新闻、知乎话题、微博粉丝，Tumblr爬虫、斗鱼弹幕、妹子图爬虫、分布式设计等

python redis golang awesome tumblr websockets zhihu crawl scrapy weibo tencent douyu scrapy-redis tumblr-bot

Updated Apr 9, 2020
Python

bitextor / bitextor

Star

Bitextor generates translation memories from multilingual websites.

Updated Oct 22, 2020
Python

LoseNine / Crack-JS-Spider

Star

破解JS反爬虫加密参数，已破解中国裁判文书网（2020-06-30更新），淘宝密码，天安保险登录，b站登录，房天下登录，WPS登录，微博登录，有道翻译，网易登录，微信公众号登录，空中网登录，今目标登录，学生信息管理系统登录，共赢金融登录，重庆科技资源共享平台登录，网易云音乐下载，一键解析视频链接，财联社登录。

spider js crawl crack

Updated Jun 30, 2020
JavaScript

Kevin-Cherish / geetest

Star

滑动验证码，希望对你们有所帮助❤️

spider python3 crawl bilibili geetest

Updated Feb 3, 2020
Python

glouw / andvaranaut

Star

A dungeon crawler

crawler crawl dungeon

Updated Mar 4, 2019
C

zhao94254 / pspider

Star

一个简单的分布式爬虫框架

python spider crawl celery flask-api spider-framework

Updated Jul 20, 2020
Python

FLZ101 / dl_coursera

Star

A simple, fast, and reliable Coursera crawling & downloading tool

mooc download coursera crawl

Updated Sep 17, 2020
Python

monkey-soft / Scrapy_IPProxyPool

Star

免费 IP 代理池。Scrapy 爬虫框架插件

crawler spider schedule crawl scrapy proxypool ipproxy

Updated Sep 17, 2018
Python

Swader / diffbot-php-client

Star

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

nlp bot php machine-learning scraper ai scraping crawling artificial-intelligence crawl scrape scraped-data diffbot

Updated Jul 4, 2018
PHP

fetchurls

adamdehaven / fetchurls

Star

A bash script to spider a site, follow links, and fetch urls (with built-in filtering) into a generated text file.

website spider wget crawl urls shell-script bash-scripting

Updated Oct 22, 2020
Shell

zhshch2002 / gospider

Star

⚡ Light weight Golang spider framework | 轻量的 Golang 爬虫框架

go golang spider crawl spider-framework colly

Updated Oct 13, 2020
Go

handong0123 / cmd-toutiao

Star

摸鱼神器：在命令行中看今日头条

python keyboard news crawl toutiao

Updated Nov 21, 2019
Python

zkqiang / crawler-chrome-extensions

Star

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

python chrome-extension crawler scraper awesome spider scraping crawl awesome-list chrome-extensions

Updated Sep 18, 2019

Bin-Huang / NodeSpider

Star

Simple, flexible, delightful web crawler/spider package

crawler node web spider pipeline async promise crawl

Updated Jul 16, 2020
TypeScript

C0D3D3V / Moodle-Downloader

Star

A Moodle Crawler that downloads course content from Moodle (eg. lecture pdfs)

content crawler assets download dhbw crawl moodle downloads moodle-crawler donwnloader moodle-downloader assets-downloader moodle-downlaader moodle-download

Updated Sep 3, 2020
Python

ShichaoMa / structure_spider

Star

组合多请求，抓取结构化数据，基于scrapy组件

spider structure crawl scrapy

Updated Sep 27, 2019
Python

wxxiong6 / crawl

Star

php爬虫系统

php spider crawl

Updated Sep 5, 2018
PHP

crackcomm / crawl

Star

Lightweight library for scalable crawlers in Go.

go crawler crawl

Updated Jul 24, 2017
Go

gaoyang-china / CDPcrawl-pychrome

Star

弥补python的Requset库无法处理动态网页的问题,chrome debug procotol支持的所有内容

python chrome crawl chromedebugprotocol

Updated May 2, 2020
Python

ArchiveTeam / wget-lua

Star

Modern wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

crawler spider lua crawling archiving wget crawl warc webarchiving archiveteam

Updated Sep 19, 2020
C

JUSTINMKAUFMAN / Crawl

Star

Keyword-based headline news crawl app for macOS

macos swift rss google news osx label cnn scroll feed crawl marquee ticker nstextfield headline

Updated Jul 14, 2019
Swift

Improve this page

Add a description, image, and links to the crawl topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the crawl topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.