Token
In artificial intelligence, a token is the basic unit of text when processed by an AI model — it can be a word, a punctuation mark, or even part of a word.
How it works
When an AI model (such as ChatGPT) reads or generates text, it first splits the text into tokens and processes them one by one. A model's text capacity (such as context length) and API pricing are usually directly tied to the number of tokens consumed.
Examples
- 1.In ChatGPT, both the question you type and the AI-generated answer are counted as a certain number of tokens.
- 2.When using OpenAI's API, fees are calculated based on the total number of tokens consumed by input and output combined.
- 3.A long document must be split into multiple tokens before it can be fed into an AI model.