TeamHG-Memex

Repositories

• autopager

  Detect and classify pagination links
  HTML 15 61 4 0 Updated Sep 9, 2020

• sklearn-crfsuite

  scikit-learn inspired API for CRFsuite
  Python 143 339 27 7 Updated Aug 18, 2020

• eli5

  A library for debugging/inspecting machine learning classifiers and explaining their predictions

  python nlp data-science machine-learning scikit-learn xgboost lightgbm
  Jupyter Notebook MIT 282 2,123 121 (15 issues need help) 15 Updated Jul 23, 2020

• html-text

  Extract text from HTML
  HTML MIT 15 57 9 2 Updated Jul 22, 2020

• soft404

  A classifier for detecting soft 404 pages
  Jupyter Notebook 10 48 3 0 Updated Jul 7, 2020

• scrapy-rotating-proxies

  use multiple proxies with Scrapy

  proxy scrapy
  Python MIT 95 413 25 4 Updated Jun 7, 2020

• Formasaurus

  Formasaurus tells you the type of an HTML form and its fields using machine learning
  HTML 32 79 9 2 Updated Jun 2, 2020

• sitehound-frontend

  Site Hound (previously THH) is a Domain Discovery Tool

  topics domain-discovery
  HTML Apache-2.0 13 21 2 2 Updated Mar 30, 2020

• scrapy-crawl-once

  Scrapy middleware which allows to crawl only new content

  scrapy
  Python MIT 19 54 1 1 Updated Mar 14, 2020

• arachnado

  Web Crawling UI and HTTP API, based on Scrapy and Tornado
  Python 63 145 16 4 Updated Mar 12, 2020

• agnostic

  Agnostic Database Migrations
  Python MIT 14 37 3 1 Updated Feb 27, 2020

• docker-tor-rotator

  Forked from mfasanya/docker-tor-rotator

  A rotating socks proxy using Tor, Delegate and Haproxy
  Dockerfile 11 12 1 0 Updated Dec 19, 2019

• autologin

  A project to attempt to automatically login to a website given a single seed
  Python Apache-2.0 34 90 6 1 Updated Oct 23, 2019

• tensorboard_logger

  Log TensorBoard events without touching TensorFlow
  Python MIT 54 605 8 1 Updated Oct 21, 2019

• aquarium

  Splash + HAProxy + Docker Compose
  Python MIT 39 146 19 0 Updated Nov 29, 2018

• json-lines

  Read JSON lines (jl) files, including gzipped and broken
  Python MIT 8 24 1 0 Updated Nov 21, 2018

• scrapy-kafka-export

  Scrapy extension which writes crawled items to Kafka
  Python MIT 7 25 1 0 Updated Nov 8, 2018

• scrapy-cdr

  Item definition and utils for storing items in CDR format for scrapy
  Python MIT 5 6 0 0 Updated Oct 29, 2018

• sitehound-backend

  Sitehound's backend
  HTML Apache-2.0 3 4 0 0 Updated Oct 17, 2018

• autologin-middleware

  Scrapy middleware for the autologin
  Python 13 31 4 1 Updated May 29, 2018

• undercrawler

  A generic crawler
  Python 22 57 16 0 Updated May 29, 2018

• domain-discovery-crawler

  Broad crawler for domain discovery
  Python MIT 7 11 2 0 Updated May 29, 2018

• page-compare

  Simple heuristic for measuring web page similarity (& data set)
  HTML 10 54 1 0 Updated May 29, 2018

• hh-page-classifier

  Headless Horseman Page Classifier service
  Python MIT 5 5 0 0 Updated May 29, 2018

• deep-deep

  Adaptive crawler which uses Reinforcement Learning methods
  Jupyter Notebook 38 138 0 0 Updated May 29, 2018

• scrash-lua-examples

  A collection of example LUA scripts and JS utilities
  JavaScript 3 5 0 0 Updated May 30, 2018

• MaybeDont

  A component that tries to avoid downloading duplicate content
  Python MIT 13 24 2 0 Updated May 29, 2018

• sitehound

  This is the facade for installation and access to the individual components
  Shell Apache-2.0 6 14 0 0 Updated May 30, 2018

• sshadduser

  A simple tool to add a new user with OpenSSH keys.
  Python MIT 0 1 0 0 Updated May 29, 2018

• bk-string

  A BK Tree based approach to storing and querying strings by Levenshtein Distance.
  C MIT 7 3 0 1 Updated May 29, 2018