common-crawl

Process Common Crawl data with Python and Spark

A passive reconnaissance tool for known URLs discovery - it gathers a list of URLs passively using various online sources.

News crawling with Storm-crawler - stores content as WARC

A python utility for downloading Common Crawl data

An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.

Statistics of Common Crawl monthly archives mined from URL index files

A dataset for knowledge base population research using Common Crawl and DBpedia.

Tools to construct and process webgraphs from Common Crawl data

German small and large versions of GPT2.

Various Jupyter notebooks about Common Crawl data

The website of one humongous corpus.

⛏Extract metadata of a specific target based on the results of "commoncrawl.org"

Various Common Crawl utilities in Clojure.

Sample code to grep Common Crawl WARC files in Go, Java, Node and Python.

Parsing Huge Web Archive files from Common Crawl data index to fetch any required domain's data concurrently with Python and Scrapy.

Perform big data analysis on New york times, Twitter and Common Crawl APIs

We explore data by using Big Data Analysis and Visualization skills. To obtain this, we perform 3 main operations. i.e. i)Data Aggregation through different sources. ii) Big Data Analysis using MapReduce and iii) Visualization through Tableau. Data Analysis is very critical in understanding the data, and what we can do with the data. For small datasets it is easier to process and obtain the results. But as for big companies, it becomes crucial for them to obtain the trends of the company for any changes need to be made. Hence we introduce Big Data Analysis to solve this problem. In this lab, we collect close to 20000 tweets, 500 articles on New York Times and 500 articles on Common Crawl Data about Entertainment, which is our main topic of discussion. Using this data, we perform preprocessing and feed it to a MapReduce to find the Word Count and Word Co-Occurrence. Using this, we find the trend of the data collected in this topic. We have used Python to perform Data Analysis.Data Analysis is very critical in understanding the data, and what we can do with the data. For small datasets it is easier to process and obtain the results. But as for big companies, it becomes crucial for them to obtain the trends of the company for any changes need to be made. Hence we introduce Big Data Analysis to solve this problem. In this lab, we collect close to 20000 tweets, 500 articles on New York Times and 500 articles on Common Crawl Data about Entertainment, which is our main topic of discussion. Using this data, we perform preprocessing and feed it to a MapReduce to find the Word Count and Word Co-Occurrence. Using this, we find the trend of the data collected in this topic. We have used Python to perform Data Analysis.

Application of topic models for information retrieval and search engine optimization.

Exploratory Data Analysis using MapReduce with Hadoop is a project developed as partial fulfillment of the requirements for the Data Intensive Computing (CSE 587) course at the University at Buffalo

Common Crawl's processing tools

Methods for working with text.

This repository contains mapreduce extractors to preprocess and extract websites from the common crawl corpus.

A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types for mass-testing of frameworks like Apache POI and Apache Tika

Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends

A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/

Hadoop streaming EMR job

Analyzing Common Crawl data (specifically) to classify fake/real based on trained deep learning models (LSTM, CNN)

Discourse Markers identification in French Language

Eventually a search engine, but currently a filtering pipeline for HTML and soon WARC files.