Verb: tokenise 'tow-ku,nIz
Usage: Brit (N. Amer: tokenize)
Usage: Brit (N. Amer: tokenize)
- (computing) substitute sensitive data with meaningless or random identifiers
"The company tokenised credit card numbers to enhance data security during transactions.";
- tokenize
- (computing) represent assets and services by tokens for integration of granular payments and charges
"in Web3, assets and services are tokenised so they can be automatically paid for as they are used";
- tokenize
- (linguistics) parse text into linguistic or semantic units
"The natural language processing system tokenised the sentence for analysis";
- tokenize
- (AI/machine learning) convert text into a sequence of tokens (words, subwords, or characters) for processing by language models
"The GPT model tokenised 'delightfully complicated' into 4 tokens";
- tokenize
Derived forms: tokenised, tokenises, tokenising
Encyclopedia: Tokenise