Trandoshan Crawler

Repositories

• dashboard

  Web UI to manage a Trandoshan instance

  angular websocket angular-material angular8
  TypeScript 5 10 3 4 Updated Jun 7, 2020

• scheduler

  Go orchestrator process used to schedule website parsing

  go docker golang nats-messaging scheduler-engine
  1 package Go GPL-3.0 3 6 3 0 Updated Dec 20, 2019

• trandoshan-parent

  Parent repository used to centralize tools to run the application, manage source code, ...

  shell docker docker-compose
  Shell 6 17 1 (1 issue needs help) 0 Updated Dec 20, 2019

• persister

  Go process used to abstract database interaction. It consumes messages from queue and put result in database allowing asynchronous saving operation

  go docker golang mongodb database-connector nats-messaging
  1 package Go GPL-3.0 3 8 0 (1 issue needs help) 0 Updated Dec 19, 2019

• api

  REST API used by other processes to gather data

  go docker golang mongodb rest-api websocket gorilla-mux
  1 package Go GPL-3.0 3 6 3 0 Updated Dec 19, 2019

• feeder

  Oneshot process used to put an URL in the todo queue. Useful to start crawling process

  go docker golang nats-messaging
  1 package Go GPL-3.0 1 2 0 0 Updated Dec 19, 2019

• crawler

  Go process used to crawl websites

  go docker golang crawler crawling nats-messaging
  1 package Go GPL-3.0 11 144 1 0 Updated Dec 19, 2019

• k8s

  Kubernetes configuration file to boostrap trandoshan cluster

  k8s kubernetes-configuration
  Shell 10 43 3 (1 issue needs help) 0 Updated Oct 15, 2019

• amqp Archived

  Go wrapper for streadway/amqp

  go golang golang-library amqplib
  Go 1 0 0 0 Updated Aug 2, 2019