Statistical or Sentient Part 2- Tokens and Thoughts

What drives the eerily human-like capabilities of large language models like GPT-3? Their artificial "minds" operate on a simple principle - predicting the next most likely token. Understanding these basic mechanics provides the key to unlocking their full potential through prompt engineering.

Statistical or Sentient Part 2- Tokens and Thoughts

In a previous article, we explored modelling the "mind" of large language models (LLMs) and how they process information. As debates continue about artificial general intelligence and whether LLMs could ever be truly sentient, it is important to dive deeper into understanding their core functions.

How do these artificial neural networks actually think and reason? What are the implications for properly steering their capabilities through prompt engineering?

This article will break down the fundamental nature of LLMs as next token predictors. Grasping this concept is key to utilizing them effectively and pushing AI advancements forward. By better understanding the mechanisms of how LLMs sequentially generate content, we can master techniques to guide them toward productive reasoning chains and meaningful output.

Statistical or Sentient? Understanding the LLM Mind - Part 1 - Memory
Memory makes us human. Yet modern language AIs like GPT Models exhibit remarkable fluency without any human-like memory. How do they generate coherent text without the episodic memory fundamental to our own cognition? This article illuminates the inner workings and memory limitations of LLMs.

Large language models (LLMs) like GPT-4 continue to demonstrate impressive capabilities in natural language processing. However, it is important to understand the fundamental nature of how these models work to fully utilize them and continue advancing AI.

At their core, LLMs are next token predictors - they predict the next word or token in a sequence. Keeping this concept in mind opens up techniques for steering LLMs toward useful results.

Understanding LLMs as Next-Token Predictors

The Core Mechanism

At their essence, LLMs operate by predicting the next token in a sequence. This process is akin to completing a sentence where the model anticipates the most likely following word or symbol based on the context provided. The 'token' here refers to the smallest unit of language processing, which could be a word, part of a word, or even punctuation.

The key is that each new token depends on the previous context. So LLMs intrinsically create chains of tokens, with new tokens extending the chain in plausible directions. The generated text is a probabilistic sample from the model's understanding of likely continuations.

The Significance of Prediction

This predictive ability is not just a mechanical process; it represents a complex, context-driven understanding. Each token prediction is influenced by the preceding content, allowing the LLM to build upon existing information and maintain coherence throughout a conversation or a piece of text.

This sequential nature of LLMs has important implications. It means techniques that allow LLMs to explore paths of tokens meaningful to the desired output can be very effective. Rather than treating an LLM like a black box, we can guide it by giving it the space to flesh out complete thoughts and chains of reasoning.

Advanced Techniques in Prompt Engineering

Several techniques take advantage of the next token prediction abilities of LLMs:

Chain of Thought: Guiding the LLM

The 'Chain of Thought' technique involves structuring prompts in a way that guides the LLM along a specific line of reasoning. By providing a sequence of logical steps, the LLM can better navigate towards a coherent and relevant conclusion. This method leverages the model's predictive nature to follow a thought process, much like connecting dots in a logical sequence.

Tree of Thought: Exploring Multiple Pathways

On the other hand, 'Tree of Thought' is a more exploratory approach where multiple potential paths are laid out for the LLM. This technique allows the model to explore different avenues of thought, much like how neurons form connections in the human brain. It's a more dynamic and flexible approach that can lead to diverse and creative outputs.

The Role of Quality Data

Leading the Model in the Right Direction

The effectiveness of these techniques heavily relies on the quality of data fed into the LLM context window. More useful and relevant data leads the model in the right direction, enhancing the accuracy and relevance of its predictions. This highlights the importance of thoughtful and well-structured prompts in guiding the LLM to desired outcomes.


Large language models' immense capabilities stem from their ability to sequentially predict interrelated tokens. Techniques that allow models to print out meaningful chains of tokens guiding them toward answers are often highly effective. Understanding the core nature of LLMs as next token predictors opens up many possibilities for prompt engineering and advancing AI.

Read next