What is tokenization in AI and how it works? Tokenization in NLP is all about splitting text into smaller parts, known as tokens - whether they’re words, subwords, or characters.