TeamHG-Memex

Repositories

• eli5

  A library for debugging/inspecting machine learning classifiers and explaining their predictions

  python nlp data-science machine-learning scikit-learn xgboost lightgbm
  Jupyter Notebook MIT 270 2,006 113 (15 issues need help) 15 Updated May 22, 2020

• sklearn-crfsuite

  scikit-learn inspired API for CRFsuite
  Python 132 320 24 4 Updated May 20, 2020

• soft404

  A classifier for detecting soft 404 pages
  Jupyter Notebook 8 45 3 1 Updated May 19, 2020

• sitehound-frontend

  Site Hound (previously THH) is a Domain Discovery Tool

  topics domain-discovery
  HTML Apache-2.0 9 19 2 2 Updated Mar 30, 2020

• scrapy-rotating-proxies

  use multiple proxies with Scrapy

  proxy scrapy
  Python MIT 81 371 21 3 Updated Mar 19, 2020

• scrapy-crawl-once

  Scrapy middleware which allows to crawl only new content

  scrapy
  Python MIT 16 51 1 1 Updated Mar 14, 2020

• arachnado

  Web Crawling UI and HTTP API, based on Scrapy and Tornado
  Python 61 142 16 4 Updated Mar 12, 2020

• agnostic

  Agnostic Database Migrations
  Python MIT 12 33 3 1 Updated Feb 27, 2020

• docker-tor-rotator

  Forked from mfasanya/docker-tor-rotator

  A rotating socks proxy using Tor, Delegate and Haproxy
  Dockerfile 10 10 1 0 Updated Dec 19, 2019

• html-text

  Extract text from HTML
  HTML MIT 14 50 9 3 Updated Dec 10, 2019

• autologin

  A project to attempt to automatically login to a website given a single seed
  Python Apache-2.0 32 86 6 1 Updated Oct 22, 2019

• tensorboard_logger

  Log TensorBoard events without touching TensorFlow
  Python MIT 51 602 8 1 Updated Oct 21, 2019

• Formasaurus

  Formasaurus tells you the type of an HTML form and its fields using machine learning
  HTML 27 76 8 1 Updated Dec 17, 2018

• aquarium

  Splash + HAProxy + Docker Compose
  Python MIT 35 134 19 0 Updated Nov 29, 2018

• json-lines

  Read JSON lines (jl) files, including gzipped and broken
  Python MIT 7 22 1 0 Updated Nov 21, 2018

• scrapy-kafka-export

  Scrapy extension which writes crawled items to Kafka
  Python MIT 6 24 1 0 Updated Nov 8, 2018

• scrapy-cdr

  Item definition and utils for storing items in CDR format for scrapy
  Python MIT 3 5 0 0 Updated Oct 29, 2018

• sitehound-backend

  Sitehound's backend
  HTML Apache-2.0 1 4 0 0 Updated Oct 17, 2018

• autologin-middleware

  Scrapy middleware for the autologin
  Python 11 30 4 1 Updated May 29, 2018

• undercrawler

  A generic crawler
  Python 20 56 16 0 Updated May 29, 2018

• domain-discovery-crawler

  Broad crawler for domain discovery
  Python MIT 5 10 2 0 Updated May 29, 2018

• page-compare

  Simple heuristic for measuring web page similarity (& data set)
  HTML 9 51 1 0 Updated May 29, 2018

• hh-page-classifier

  Headless Horseman Page Classifier service
  Python MIT 4 4 0 0 Updated May 29, 2018

• deep-deep

  Adaptive crawler which uses Reinforcement Learning methods
  Jupyter Notebook 35 130 0 0 Updated May 29, 2018

• scrash-lua-examples

  A collection of example LUA scripts and JS utilities
  JavaScript 2 5 0 0 Updated May 29, 2018

• autopager

  Detect and classify pagination links
  HTML 14 55 4 0 Updated May 29, 2018

• MaybeDont

  A component that tries to avoid downloading duplicate content
  Python MIT 10 24 2 0 Updated May 29, 2018

• sitehound

  This is the facade for installation and access to the individual components
  Shell Apache-2.0 5 11 0 0 Updated May 29, 2018

• sshadduser

  A simple tool to add a new user with OpenSSH keys.
  Python MIT 0 1 0 0 Updated May 29, 2018

• bk-string

  A BK Tree based approach to storing and querying strings by Levenshtein Distance.
  C MIT 6 3 0 1 Updated May 29, 2018