English

Verb

tokenize ‎(third-person singular simple present tokenizes, present participle tokenizing, simple past and past participle tokenized)

1. (transitive, computing) To reduce to a set of tokens by parsing.
2. (transitive) To treat as a token minority.

Derived terms

• tokenizable
• tokenizer

Related terms

• tokenization