Verb: tokenize 'tow-ku,nIz
- (computing) substitute sensitive data with meaningless or random identifiers
"The company tokenized credit card numbers to enhance data security during transactions.";
- tokenise [Brit]
- (computing) represent assets and services by tokens for integration of granular payments and charges
"in Web3, assets and services are tokenized so they can be automatically paid for as they are used";
- tokenise [Brit]
- (linguistics) parse text into linguistic or semantic units
"The natural language processing system tokenized the sentence for analysis";
- tokenise [Brit]
- (AI/machine learning) convert text into a sequence of tokens (words, subwords, or characters) for processing by language models
"The GPT model tokenized 'delightfully complicated' into 4 tokens";
- tokenise [Brit]
Derived forms: tokenizing, tokenizes, tokenized
Encyclopedia: Tokenize