tokenizer

English edit

Etymology edit

tokenize +‎ -er

Noun edit

tokenizer (plural tokenizers)

(computing) A system that parses an input stream into its component tokens.

Retrieved from "https://en.wiktionary.org/w/index.php?title=tokenizer&oldid=43036057"