#

warc

Here are 75 public repositories matching this topic...

ArchiveBox

ArchiveBox / ArchiveBox

Star

🗃 The open source self-hosted web archive. Takes browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

Updated Dec 6, 2020
Python

internetarchive / heritrix3

Star

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

java warc heritrix webcrawling

Updated Nov 12, 2020
Java

conifer

Rhizome-Conifer / conifer

Star

Collect and revisit web pages.

python docker archives warc web-archiving wayback webrecorder pywb

Updated Dec 1, 2020
Python

ArchiveTeam / grab-site

Star

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

crawler spider archiving crawl warc

Updated Nov 5, 2020
Python

webrecorder / webrecorder-player

Sponsor Star

Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)

electron warc web-archiving webrecorder pywb

Updated Sep 17, 2020
JavaScript

ipwb

oduwsdl / ipwb

Star

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

python docker service-worker ipfs memento warc web-archiving wayback memento-rfc

Updated Oct 15, 2020
Python

wail

machawk1 / wail

Star

🐋 Web Archiving Integration Layer: One-Click User Instigated Preservation

python gui warc web-archiving pyinstaller wayback heritrix openwayback

Updated Dec 3, 2020
Python

webrecorder / warcio

Sponsor Star

Streaming WARC/ARC library for fast web archive IO

python warc web-archiving web-archives pywb

Updated Nov 3, 2020
Python

bitextor / bitextor

Star

Bitextor generates translation memories from multilingual websites.

Updated Dec 4, 2020
Python

warcreate

machawk1 / warcreate

Star

Chrome extension to "Create WARC files from any webpage"

chrome-extension warc web-archiving

Updated Dec 4, 2020
JavaScript

commoncrawl / news-crawl

Star

News crawling with Storm-crawler - stores content as WARC

crawler news web-crawler apache-storm warc

Updated Sep 21, 2020
Java

cocrawler / cocrawler

Star

CoCrawler is a versatile web crawler built using modern tools and concurrency.

screenshot crawler concurrency async-python python3 aiohttp warc aiohttp-client pluggable-modules

Updated Nov 28, 2020
Python

helgeho / ArchiveSpark

Star

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

spark internet-archive warc web-archiving webarchive archivespark spark-framework

Updated Dec 13, 2019
Scala

N0taN3rd / wail

Star

🐋 One-Click User Instigated Preservation

electron warc web-archiving high-fidelity-preservation browser-based-presrevation

Updated Feb 3, 2019
JavaScript

CGamesPlay / chronicler

Star

Offline-first web browser

electron browser warc

Updated Jan 14, 2019
JavaScript

cocrawler / cdx_toolkit

Star

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

python warc web-archiving cdx web-archives commoncrawl cdx-api

Updated Nov 28, 2020
Python

N0taN3rd / node-warc

Star

Parse And Create Web ARChive (WARC) files with node.js

warc web-archiving webarchive web-archives webarchiving warc-files chrome-remote-interface pupeteer

Updated Sep 4, 2020
JavaScript

webrecorder / replayweb.page

Sponsor Star

Serverless Web Archive Replay directly in the browser

service-worker warc web-archiving wayback-machine web-archive replay-web-page web-replay

Updated Dec 3, 2020
JavaScript

archivesunleashed / warclight

Star

A Rails engine supporting the discovery of web archives.

ruby rails rails-engine solr discovery blacklight warc webarchives webarchive-discovery

Updated Dec 6, 2020
Ruby

centic9 / CommonCrawlDocumentDownload

Sponsor Star

A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika

java mime-types warc cdx-files commoncrawl

Updated Nov 29, 2020
Java

PromyLOPh / crocoite

Star

Web archiving using Google Chrome

devtools archiving chrome-browser warc

Updated Dec 30, 2019
Python

datatogether / warc

Star

Golang WARC (Web ARChive) Library

golang package archiving warc iipc

Updated Aug 6, 2019
Go

pirate / internet-archiving-talk

Sponsor Star

🎭 An introduction to the Internet Archiving ecosystem, tooling, and some of the ethical dilemmas that the community faces.

slideshow wget talks warc censorship web-archiving ethics internet-archiving archivebox

Updated Oct 19, 2020
JavaScript

hrbrmstr / warc

Star

📇 Tools to Work with the Web Archive Ecosystem in R

r rstats warc warc-files r-cyber warc-ecosystem

Updated Aug 20, 2017
R

Mixnode / mixnode-warcreader-php

Star

Read Web ARChive (WARC) files in PHP.

php warc webarchive

Updated Mar 10, 2017
PHP

ArchiveTeam / wget-lua

Star

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

crawler spider lua crawling archiving wget crawl zstd warc webarchiving archiveteam wget-lua wget-at

Updated Oct 29, 2020
C

antiufo / Shaman.Dokan.Warc

Star

Mounts WARC files on Windows

fuse dokan scraping mount warc web-archive

Updated Apr 20, 2019
C#

webrecorder / cdxj-indexer

Sponsor Star

CDXJ Indexing of WARC/ARCs

warc web-archiving

Updated Sep 24, 2020
Python

ArchiveTeam / WebArchiver

Star

Decentralized web archiving

python crawler web decentralized archiving archiver warc webarchiving

Updated Aug 7, 2018
Python

Mixnode / mixnode-warcreader-java

Star

Read Web ARChive (WARC) files in Java.

Updated Mar 18, 2017
Java

Improve this page

Add a description, image, and links to the warc topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the warc topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.