tokenize
English edit
Etymology edit
Pronunciation edit
- (General American) IPA(key): /ˈtoʊ.kən.aɪz/
Verb edit
tokenize (third-person singular simple present tokenizes, present participle tokenizing, simple past and past participle tokenized)
- (transitive, computing) To reduce to a set of tokens by lexical analysis.
- (transitive, computing) To substitute sensitive data with meaningless placeholders.
- (transitive) To treat as a token minority.