Definition of tokenizer - Reverso English Dictionary

Noun

technology

TECH.

program that divides text into tokens

TECH.

The tokenizer splits the sentence into individual words.
A tokenizer is essential for text analysis.
The tokenizer helps in breaking down complex sentences.

Add a suggestion |

Feedback:

Origin of tokenizer

English, token (sign) + -izer (agent noun suffix)

View all translations of tokenizer

✨Click below to see the appropriate translations facing each meaning.

French:tokeniseur, ...
German:Tokenizer, ...
Italian:tokenizzatore, ...
Spanish:tokenizador, ...
Portuguese:tokenizador, ...
Chinese:分词器, ...
Japanese:トークナイザー, ...
Arabic:مُجَزِّئ, ...
Hebrew:תוֹכְנָה לְחִלּוּק טֶקְסְט, ...
Russian:токенизатор, ...
Ukrainian:токенізатор, ...
Hindi:टोकनाइज़र, ...
Bengali:টোকেনাইজার, ...
Catalan:tokenitzador, ...
Czech:tokenizér, ...
Danish:tokenizer, ...
Dutch:tokenizer-programma, ...
Greek:διαχωριστής λέξεων, ...
Hungarian:tokenizáló, ...
Khmer:កម្មវិធីបំបែកអត្ថបទជាឯកតាពាក្យ, ...
Korean:토크나이저 프로그램, ...
Persian:برنامه توکن‌سازی, ...
Polish:program tokenizujący, ...
Portuguese (BR):tokenizador, ...
Romanian:program de tokenizare, ...
Swedish:tokenizer, ...
Thai:ตัวแบ่งคำ, ...
Turkish:dizgeleyici, ...
Vietnamese:bộ phân tích từ, ...

Browse the dictionary entries starting with “t”:

Discover how Reverso redefines the dictionary

Go beyond “simple words”

100,000 expressions & idioms Understand any expression you come across: from corporate jargon like 💬 touch base or ↩️ circle back to everyday expressions such as 🌊 out of the blue or 🦆 lame duck, as well as idioms like 🍲 spill the beans and 🐎 hold your horses.

40,000 images When you search for a concrete noun or object, images help you grasp the meaning at a glance. Try searching for 🧵 puppytooth, 🎠 merry-go-round, ⚜️ fleur-de-lys, or 💤 bunk bed.

Informal language & slang Everyday conversations are packed with informal expressions and slang. Meanings are clearly tagged, so you'll always know when a word is casual. Look up hangry (😋 hungry + 😡 angry), 😎 rizz, 🕹️ noob, or 🔑 low-key.

Abbreviations & acronyms Quickly decode modern abbreviations and acronyms. Go from IDK to OMG in seconds and stay up to date with popular abbreviations such as 🐐 GOAT, 📜 tl;dr, 👀 POV, or 🖥️ AFK.

Regional usage (US / UK) Reverso highlights regional differences between American and British English. You’ll never have to wonder where a word is commonly used: try lift vs. elevator, courgette vs. zucchini, or popular British slang like dodgy or banter. What’s more, when you search for 'color', you’ll immediately see its British counterpart, 'colour'.

Get more than a simple definition

Translations for each meaning View translations alongside each meaning to see how a word is translated in each context. Simply toggle translations ON and choose among 25+ languages. You can also display the translation of definitions or examples into your native language to deepen your understanding.

Synonyms & related words by meaning View synonyms and related words for each meaning, not just for the word as a whole. This helps you choose the most accurate word for your context. Click the

icon to discover related terms: analogies, antonyms, common collocates, words with same roots, umbrella terms… Try with happening or hunch.

Multiple examples for every meaning Each meaning comes with three real-life examples to show you how the word is used in different contexts. Click the

icon next to the first example to see two additional options and learn to use new vocabulary naturally.

Go from understanding to mastery

Save words for later review Click the

icon to save words you want to revisit. You can also save a specific meaning by clicking on the

icon next to it. For example, you might want to save the word “table” as a verb, which means “to postpone”.

Practice with personalized flashcards Reverso automatically transforms your searches and saved words into personalized flashcards and quizzes, featuring high-quality audio and usage examples. This helps you transform terms you only recognize into words you actually use confidently.

Learn with Netflix, YouTube, and your favorite texts With Reverso Chrome extension, transform movies, TV shows, articles, or song lyrics into an interactive lesson. Look up words and expressions in one click on any website and learn with your favorite content.