tokenizer

Methods

(inner) tokenize(str) → {Array.<Token>}

Source:
Parameters:
Name Type Description
str string

The text input
Returns:

An array of tokens

Type
Array.<Token>