Tokenizer

What is Tokenizer?

Tokenizer is a wrapper around the token_get_all function. It's built up of three classes:

1. Token - Each token returned from token_get_all is wrapped in this.
2. TokenListIterator - Implements SeekableIterator and Countable with extra methods to allow easy moving through tokens.
3. Tokenizer - Calls token_get_all on the source, turns each result into a Token representation and returns a TokenListIterator object.

Installation

You can install Tokenizer by adding hippophp/tokenizer to your composer.json file.

License

HippoPHP Tokenizer is licensed under The MIT License (MIT)