Popular repositories

1. ArchiveSpark Public

   An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

   Scala 125 19

2. Web2Warc Public

   An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)

   Scala 22 4

3. internetarchive-transfer-scripts Public

   Scripts to transfer archive.org collections, using https://github.com/jjjake/internetarchive

   Python 9 2

4. HadoopConcatGz Public

   A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz

   Java 9 4

5. HadoopWebGraph Public

   A Hadoop input format to use gaphs in WebGraph's BV format with Hadoop and Spark.

   Java 7 3

6. Exspec Public

   Don't write specs anymore, just save 'em while testing your code interactively. Specs will become a byproduct.

   Ruby 5