From Markov Chains to Thinking Code Models

How did we get from simple probability models to modern large language models?

Introduction

Artificial intelligence is often discussed as if models "know" the answers or think like humans. The reality is more interesting and nuanced.

Modern large language models (LLMs) learn probabilistic continuations of text snippets. From this basic idea—with the right architecture, a large amount of data, and heavy computation—surprisingly complex capabilities can emerge.

In this article, we trace the main milestones along the journey:

Markov chains and next-token prediction
Neural networks and parameters
Tokens and embeddings
The Transformer and attention
Training LLMs
Reasoning, or complex inference
Code models and practical limitations

Important: The examples provided here are illustrative, general examples. They do not describe the internal workings of a specific model or confidential system information.

1. The Markov Chain: The Probability of the Next Step

Imagine a system that always tries to estimate what follows the current state. In a weather example:

Sunny → Sunny       0.7
Sunny → Cloudy      0.2
Sunny → Rainy       0.1

This is a simple Markov model: it estimates the probability of the next state primarily based on the current state. When generating a longer sequence, the system chooses again at every step:

Sunny → Sunny → Cloudy → Rainy → Cloudy → ...

The basic idea of language models is similar, except that the "next state" here is often the next token—a word, subword, or punctuation mark—and the history taken into account is much richer.

2. Why Isn't a Simple Markov Chain Enough?

Based on a short history, it is often impossible to properly predict the next word:

The book that I bought yesterday is very ...

Here, multiple previous elements matter: the topic of the sentence, the grammatical structure, the word "book", temporal references, and the tone of the entire text. Furthermore, in natural language, distant parts can also influence each other.

Neural networks are useful because they do not store pre-fixed rules for every possible sentence. Instead, they learn the patterns from many examples that help select the next token.

3. Parameters: Knowledge Distributed, Not in Records

A neural network consists of many numbers—parameters. These can be values such as:

0.001253
-1.872000
0.417801

A single parameter on its own usually does not represent a specific fact or concept. The model's capabilities emerge from the joint operation of many parameters.

Therefore, it is misleading to imagine that the model contains an entry like:

Budapest → Capital of Hungary

Instead, configurations form within a vast, interconnected number space that make responses similar to certain text contexts probable.

4. Tokens and Embeddings: Turning Text into Numbers

The model does not see "words" directly. It breaks the input into tokens:

"A Transformer interesting."
        ↓
["A", " Transformer", " interesting", "."]

Each token is associated with a series of numbers, called an embedding:

"dog" → [0.31, -0.88, 1.72, 0.04, ...]

This number sequence is not a dictionary definition. However, during training, tokens appearing in similar contexts often end up in similar positions within the vector space.

                 wolf
                    •
          dog •         • cat


                              screwdriver •

Embeddings can capture not only similarities between words. Certain directions or regions of the vector space may also carry grammatical, thematic, stylistic, and other relationships.

5. The Transformer: Handling Text Relationships

The Transformer architecture was introduced by the paper Attention Is All You Need. The key idea is attention, the attention mechanism.

When processing a sentence, not every token needs to pay the same amount of attention to every other token. The attention mechanism learns which relationships are important in a given situation.

"The dog saw the ball, so it picked it up."
                              ↑
                 What does "it" refer to?

Illustrated:

tokens → weighting relationships → context-dependent representation

Attention can be described using three commonly used concepts:

Query: What is the given token looking for?
Key: What information does another token offer?
Value: What content is worth taking from there?

In practice, many attention heads work in parallel. One head may handle grammatical relationships, another distant references, and a third formatting patterns—this is just an intuitive explanation, not a rigid role allocation.

Additional understandable, visual explanations:

6. How Does an LLM Learn?

Pre-training

During pre-training, the model practices predicting the next token across many text examples:

Input:   The sun today is very
Target:  bright

If the model gives poor probabilities, the training algorithm slightly modifies the parameters. By repeating this many times, the network gets better and better at modeling the statistical structure of the language.

Fine-Tuning and Feedback

A pre-trained model is not necessarily a good assistant yet. Additional training phases can help it to:

follow instructions;
provide structured, understandable answers;
indicate uncertainty;
avoid inappropriate or dangerous responses;
make better use of tools and formats.

These processes can use various datasets, human or automated evaluation, and safety tests. The details vary by model and developer.

A good starting point for OpenAI developer materials is the API documentation, and for models, the OpenAI Models page.

7. Reasoning: When the Answer Requires Multiple Steps

A simple question often requires just a quick association:

What is the capital of France? → Paris

However, for a complex task, intermediate steps are needed to reach the correct answer:

task
  ↓
making a plan
  ↓
intermediate result
  ↓
checking or fixing
  ↓
final answer

Technically, a language model still generates tokens in these cases. The difference is that the intermediate steps create a new context upon which subsequent steps can build.

This is as if the model were not trying to reach the goal with a single giant leap, but breaking the problem down into subtasks. Therefore, in training and using modern reasoning models, allocating more computing time, generating multiple candidate solutions, and verifying them can be important. Further background on this is provided by OpenAI's "Learning to reason with LLMs" overview.

It is important to distinguish between useful derivation and presenting the full internal operation of the model. A good response can provide concise, verifiable reasoning without literally displaying every internal intermediate state of the system.

8. What Makes Code Models Special?

Program code is also text, but it has a stricter structure. A code model must simultaneously pay attention to:

syntax;
relationships between variables and functions;
types and interfaces;
the project environment;
tests and error messages;
the business goal of the task.

A code generation process might look like this, for example:

task description
      ↓
understanding relevant files and dependencies
      ↓
solution plan
      ↓
modifying code
      ↓
testing
      ↓
debugging and review

A code model is not reliable because it always "knows" the correct program. Rather, it can be useful because it can quickly suggest patterns, compare alternatives, and make improvements based on feedback. Final verification—tests, code reviews, security scanning—remains important.

Andrej Karpathy's Neural Networks: Zero to Hero course demonstrates practically how neural networks and GPT-like models are built from scratch.

9. What Doesn't All This Mean?

Some common misconceptions:

"The model copies the answer from a database."

This is not the best general model. The system uses patterns distributed across its parameters and generates new tokens based on them. Even so, it may happen that it remembers a training example or known text too closely.

"The model is always right if it is confident."

A probable language continuation is not the same as factual truth. The model can make mistakes, have incomplete context, or phrase a flawed statement convincingly.

"Reasoning proves human consciousness."

Multi-step reasoning can result in strong problem-solving behavior, but this does not in itself imply human consciousness, intent, or experience.

10. The Big Picture

Briefly, the journey:

Markov chain
   ↓  probability of the next element
Neural network
   ↓  learned parameters
Embedding
   ↓  concepts in a number space
Attention
   ↓  context and relationships
Transformer
   ↓  scalable text processing
LLM
   ↓  pre-training + fine-tuning + verification
Reasoning and coding

Modern LLMs do not consist of a single "magical" technique. Multiple ideas build upon each other: probabilistic prediction, vector representations, attention mechanism, large-scale training, and feedback-based improvement.

Perhaps the most important takeaway is this: the model's capabilities cannot be found in a single parameter or a single database record. The behavior that outwardly appears as language comprehension, reasoning, or programming assistance emerges from the coordinated operation of many simple calculations.

Further Learning Resources

Vaswani et al.: Attention Is All You Need
Jay Alammar: The Illustrated Transformer
3Blue1Brown: Neural Networks
Andrej Karpathy: Neural Networks: Zero to Hero
OpenAI: API documentation
OpenAI: Available models and capabilities

Editor's note: This article is an educational, simplified explanation. The specific architecture, training process, and capabilities of specific models may vary, so before making a technical or business decision, it is always advisable to review the official documentation and test results of the given system.