tokenized

Compare

Why “tokenized” is being discussed

The most recent Hacker News stories and comments contributing to this topic's mentions.

Interest

Proportion of Hacker News items mentioning "tokenized" over time.

Mentions

Total number of Hacker News items mentioning "tokenized" over time.

Lexical tokenization is conversion of a text into meaningful lexical tokens belonging to categories defined by a "lexer" program. In case of a natural language, those categories include nouns, verbs, adjectives, punctuations etc. In case of a programming language, the categories include identifiers, operators, grouping symbols, data types and language keywords. Read more on Wikipedia

© 2026 HN Trends. Built by Ben Evans. HN Trends is not affiliated with Hacker News or Y Combinator.