Wiktionary

tokenizer

English

edit

Etymology

edit

From tokenize +‎ -er.

Noun

edit

tokenizer (plural tokenizers)

  1. (computing) A system that parses an input stream into its component tokens.
Retrieved from "https://en.wiktionary.org/w/index.php?title=tokenizer&oldid=81250776"