Skip to content
#

tokenizer

Star

A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.

Here are 765 public repositories matching this topic...

Language: All
Filter by language
All 765 Python 183 Jupyter Notebook 75 JavaScript 71 Java 69 C++ 54 Go 44 PHP 40 C 39 TypeScript 39 Rust 37
Sort: Best match
Sort options
Best match Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

cbaziotis / ekphrasis

Star 591

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization
  • Updated Sep 29, 2022
  • Python

CogComp / cogcomp-nlp

Star 455

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.

nlp natural-language-processing data-mining big-data tokenizer transliteration similarity named-entity-recognition pos lemmatizer ner pos-tagging dependency-parsing lemmatization relation-extraction natural-language-understanding cogcomp parts-of-speech-tagging
  • Updated Oct 4, 2022
  • Java
Wikipedia
Wikipedia

Related Topics

antlr generated-parser language ll lookahead lr-parser semantic-analysis symbol-table yacc