LLM Inference Explained: What Actually Happens When You Serve a Model

As spending on frontier AI services like ChatGPT, Claude, and Gemini climbs, usage caps hit developers, and open-source models gain popularity, most platform teams eventually need to serve an LLM on their own infrastructure. At first glance, it looks like any other workload running over HTTP behind a load balancer, but that's where the similarities stop. The model itself is just a folder full of floating-point numbers with nothing you can execute. The API is stateless, and to keep a conversation going, the client has to resend the entire conversation history each time; what an API call typically takes milliseconds now takes seconds or even minutes.

All these differences come from a few basic facts about how a model turns a prompt into tokens, and shape every platform decision, from buying GPUs to setting up gateways and autoscaling. In this article, we'll walk through the whole process: the files you deploy, the engine that serves them, and what happens to a request from raw text to streamed tokens.

The model: a folder of numbers

A model is a file (or, more often, a set of files) holding billions of numerical values called weights, plus a tokenizer and some configuration. It is not a program, and there is nothing to execute. On its own, a model does exactly nothing. For example, the following are the files of Qwen3-4B:

3.7G model-00002-of-00003.safetensors
3.7G model-00001-of-00003.safetensors
 95M model-00003-of-00003.safetensors
 11M tokenizer.json
2.6M vocab.json
1.6M merges.txt
 32K model.safetensors.index.json
 16K README.md
 11K LICENSE
9.5K tokenizer_config.json
726B config.json
239B generation_config.json

That's the entire structure in one place: three shards of raw numbers make up more than 99% of the data, along with a few small JSON and text files that explain how to use them. These are the main files, following the standard Hugging Face layout:

• config.json is the blueprint. It carries the architecture information the engine needs to arrange the weights and size its memory.

{
  "architectures": [
    "Qwen3ForCausalLM"
  ],
  ...
  "max_window_layers": 36,
  "model_type": "qwen3",
  "num_attention_heads": 32,
  "num_hidden_layers": 36,
  "num_key_value_heads": 8,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.51.0",
  "use_cache": true,
  "use_sliding_window": false,
  "vocab_size": 151936
}

• tokenizer.json acts as the model’s dictionary. Models don’t read text directly; this file contains everything needed to turn human text into the integer IDs the model uses.

{
  "version": "1.0",
  "truncation": null,
  "padding": null,
  "added_tokens": [
    {
      "id": 151643,
      "content": "<|endoftext|>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false,
      "special": true
    },
    ...
  ],
  "normalizer": {...},
  "decoder": {...},
  "model": {
    "type": "BPE",
    "vocab": {
      "!": 0,
      "\"": 1,
      "#": 2,
      "$": 3,
      ...
    }
  },
  "merges": [...]
}

• tokenizer_config.json includes, among other settings, the chat template. This template is the recipe that turns a structured conversation (system, user, and assistant turns, plus tool calls) into the single token stream the model was trained to understand.

{
  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {{- messages[0].content + '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n    {%- set index = (messages|length - 1) - loop.index0 %}\n    {%- if ns.multi_step_tool and message.role == \"user\" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}\n        {%- set ns.multi_step_tool = false %}\n        {%- set ns.last_query_index = index %}\n    {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n    {%- if message.content is string %}\n        {%- set content = message.content %}\n    {%- else %}\n        {%- set content = '' %}\n    {%- endif %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n        {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is string %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- else %}\n            {%- if '</think>' in content %}\n                {%- set reasoning_content = content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n                {%- set content = content.split('</think>')[-1].lstrip('\\n') %}\n            {%- endif %}\n        {%- endif %}\n        {%- if loop.index0 > ns.last_query_index %}\n            {%- if loop.last or (not loop.last and reasoning_content) %}\n                {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content.strip('\\n') + '\\n</think>\\n\\n' + content.lstrip('\\n') }}\n            {%- else %}\n                {{- '<|im_start|>' + message.role + '\\n' + content }}\n            {%- endif %}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is false %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- endif %}\n{%- endif %}",
  ...
}

• generation_config.json gives the model authors’ recommended settings for running the model. Engines use these as default settings unless a request specifies something different.

{
    "bos_token_id": 151643,
    "do_sample": true,
    "eos_token_id": [
        151645,
        151643
    ],
    "pad_token_id": 151643,
    "temperature": 0.6,
    "top_k": 20,
    "top_p": 0.95,
    "transformers_version": "4.51.0"
}

• Finally, several model-*.safetensors shards, and a model.safetensors.index.json which acts as their table of contents. This JSON file maps each tensor name to the shard that contains it. Each shard has a simple structure: a small JSON header lists tensor names, data types, shapes, and byte offsets, followed by raw tensor bytes. For Qwen3-4B, that means about 4 billion bfloat16 values at two bytes each, totaling around 8 GB. If you open a shard, you'll see named arrays of numbers, layer by layer:

model.embed_tokens.weight
tensor([[-0.0287,  0.0117,  0.0104,  ..., -0.0055, -0.0270, -0.0096],
        [-0.0291, -0.0212,  0.0109,  ..., -0.0033, -0.0062, -0.0004],
        [ 0.0035,  0.0177,  0.0012,  ..., -0.0058,  0.0022,  0.0130],
        ...,
        [ 0.0060,  0.0131,  0.0190,  ...,  0.0068, -0.0049, -0.0040],
        [ 0.0060,  0.0131,  0.0190,  ...,  0.0068, -0.0049, -0.0040],
        [ 0.0060,  0.0131,  0.0190,  ...,  0.0068, -0.0049, -0.0040]],
       dtype=torch.bfloat16)
model.layers.0.input_layernorm.weight
tensor([0.0598, 0.0288, 0.0576,  ..., 0.0184, 0.0222, 0.0223],
       dtype=torch.bfloat16)
model.layers.0.mlp.down_proj.weight
tensor([[ 0.1396,  0.0056,  0.0054,  ..., -0.0084,  0.0781, -0.0693],
        [-0.0859, -0.0074, -0.0142,  ..., -0.0161, -0.0022, -0.0403],
        [ 0.0649, -0.0021,  0.0081,  ...,  0.0110,  0.0223,  0.0283],
        ...,
        [ 0.0106,  0.0208,  0.0176,  ...,  0.0060, -0.0349, -0.0060],
        [-0.0079, -0.0007, -0.0603,  ..., -0.0059, -0.0271,  0.0085],
        [-0.0221, -0.0306,  0.0121,  ..., -0.0089, -0.0233,  0.0089]],
       dtype=torch.bfloat16)
model.layers.0.mlp.gate_proj.weight
tensor([[ 0.0014,  0.0057, -0.0032,  ..., -0.0415, -0.0276, -0.0154],
        [-0.0178,  0.0078,  0.0043,  ..., -0.0186, -0.0028, -0.0018],
        [-0.0042, -0.0018, -0.0016,  ...,  0.0330, -0.0026,  0.0244],
        ...,
        [-0.0081, -0.0007, -0.0052,  ..., -0.0076, -0.0114,  0.0062],
        [-0.0061,  0.0118, -0.0116,  ..., -0.0164, -0.0015,  0.0237],
        [ 0.0130,  0.0046, -0.0036,  ..., -0.0105,  0.0297, -0.0347]],
       dtype=torch.bfloat16)
model.layers.0.mlp.up_proj.weight
tensor([[ 0.0137,  0.0089,  0.0046,  ..., -0.0339,  0.0061, -0.0117],
        [ 0.0077,  0.0078,  0.0036,  ...,  0.0134,  0.0216, -0.0040],
        [-0.0105,  0.0057,  0.0053,  ...,  0.0251,  0.0044,  0.0092],
        ...,
        [ 0.0067, -0.0003, -0.0082,  ..., -0.0049,  0.0247, -0.0181],
        [-0.0021,  0.0044, -0.0056,  ..., -0.0111,  0.0133,  0.0135],
        [ 0.0046, -0.0028,  0.0004,  ...,  0.0167,  0.0505,  0.0347]],
       dtype=torch.bfloat16)
model.layers.0.post_attention_layernorm.weight
tensor([-2.0862e-05,  2.7466e-04, -5.5313e-05,  ...,  2.1582e-01,
         2.3145e-01,  2.1484e-01], dtype=torch.bfloat16)
model.layers.0.self_attn.k_norm.weight
tensor([ 1.9453e+00,  1.1172e+00,  1.7031e+00,  1.7969e+00,  1.4688e+00,
         1.9141e+00,  1.9844e+00,  1.8281e+00,  1.8203e+00,  1.6406e+00,
         ...,
         6.4844e-01,  1.8203e+00,  2.3750e+00,  2.9062e+00,  4.0938e+00,
         2.0625e+00,  2.1094e+00,  3.0781e+00,  1.5234e+00,  2.3125e+00,
         1.9141e+00,  2.0469e+00,  2.0156e+00], dtype=torch.bfloat16)

Dense vs Mixture-of-Experts transformer architectures

Before we move on, there's one important difference to highlight, since the biggest open models in the industry have changed recently. For years, nearly every model you could download shared a single design, now called dense; the newest flagships use a different one, Mixture-of-Experts (MoE). The key difference between the two is how many of those tensors each token passes through.

• Dense models (the Llama family and most models you have served so far) multiply every token through every tensor in the shards. Look back at the Qwen3-4B listing: each layer has one mlp.gate_proj, mlp.up_proj, and mlp.down_proj, and every token's vector passes through all of them. There is no mechanism to skip a stored number, so compute per token scales with model size: a 70B dense model does roughly 9× the work per token of an 8B one.

• Mixture-of-Experts (MoE) models keep the same file structure but store many small copies of that feed-forward trio in each layer (tensors named like mlp.experts.0.gate_proj, mlp.experts.1.gate_proj, and so on) plus one tiny extra tensor: a learned router. For each token, the router scores all experts and only the top-scoring few (plus, in several modern designs, one always-on shared expert) actually enter the multiplication; the other experts' tensors sit untouched for that token. Mixtral-8x7B ships 47B parameters but multiplies each token through ~13B of them; DeepSeek-R1 ships 671B and uses ~37B. You get big-model quality at small-model compute cost per token. This, together with inference-layer innovations like compressed KV caching (MLA, which we'll come back to in the memory section), is a large part of how DeepSeek served frontier-class quality so cheaply.

The catch is that memory requirements don’t depend on which tensors are used. All parameters must be loaded and accessible, even though each token only uses roughly 3–30% of them, depending on the design (~28% for Mixtral‑8x7B, ~5% for DeepSeek‑R1). MoE models are inexpensive to run but costly to store. To serve these models efficiently, you often need to distribute experts across many GPUs and hosts, and route tokens between them quickly, which requires network- and topology-aware scheduling.

The serving engine is your new web server

A model is as passive as a website’s HTML file. It does nothing until a web server loads it and handles requests. We can think of the serving engine as the model’s web server. It loads the weights onto the accelerator, provides an HTTP API, and translates between client JSON and GPU matrix operations. Everything else it does, from queueing requests, batching them for the hardware, and streaming each token as soon as it’s ready, serves one main goal: keep the expensive accelerator busy without making any single request wait too long.

The engine landscape, as of this writing:

Two things are more important than the engine names themselves. Engines are model-specific: a new model architecture only works once an engine adds support for it. That’s why every engine publishes a list of supported models, and why you should always check if you can serve a model before making promises to the business. Almost all engines use the OpenAI API specification (/v1/chat/completions, /v1/models), so clients, gateways, and benchmarks can work across different engines.

Tokens and the context window

Models don’t see words; they see tokens, integer IDs produced by the tokenizer we found in tokenizer.json. Everything downstream is billed, benchmarked, and memory-budgeted in tokens. Here is Qwen3-4B’s tokenizer on three short sentences:

from urllib.request import urlopen
from tokenizers import Tokenizer
TOK_URL = "https://huggingface.co/Qwen/Qwen3-4B/resolve/main/tokenizer.json"
tok = Tokenizer.from_buffer(urlopen(TOK_URL).read())
for text in ["Why is the sky blue?", "我喜欢吃火锅!", "김치찌개를 좋아해요!"]:
    enc = tok.encode(text)
    pieces = [tok.decode([i]) for i in enc.ids]
    print(f"{text!r}")
    print(f"  {len(enc.ids):>2} tokens -> {pieces}")
 
'Why is the sky blue?'
   6 tokens -> ['Why', ' is', ' the', ' sky', ' blue', '?']
 
'我喜欢吃火锅!'
   4 tokens -> ['我喜欢', '吃', '火锅', '!']
 
'김치찌개를 좋아해요!'
   9 tokens -> ['김', '치', '찌', '개', '를', ' 좋아', '해', '요', '!']

As you can see, there’s no fixed ratio between characters and tokens: you might get six tokens for five English words, or nine for a Korean one. This matters because the model’s main limit is measured in tokens. The context window is the maximum number of tokens the model can handle in a single request (including both prompts and responses), and it’s the number you see when a hosted model advertises “a million tokens of context.”

There’s one more step before tokenization. Your application sends a structured array of messages, but the model was trained on a single flat stream with special tokens to show who is speaking. The chat template from tokenizer_config.json bridges this gap by flattening the array into a stream, which is then tokenized.

Prefill and decode: the two phases behind every response

Inference is the computation that turns those tokens into new ones: a prompt goes in, a completion comes out. Inside the engine it runs in two phases:

• Prefill ingests the entire prompt in one large parallel pass. For every prompt token, it computes and stores its key and value vectors in the KV cache with the GPU memory holding the attention state for that sequence. The final output is the prediction for the first response token, which stops the user's time-to-first-token (TTFT).

• Decode generates the rest of the response, one token at a time. For each token, it computes keys and values, adds them to the cache, and reads the stored entries for all earlier tokens instead of recalculating them. Without KV Cache, generating token 500 would require re-processing the previous 499 tokens. With the cache, each token's attention math is done just once, when it first enters the sequence.

Statelessness and the cost of a conversation

The KV cache stays on the server, but the APIs are stateless. There’s no session object or conversation ID the model can use. If a user asks “What is the capital of Belgium?” and then follows up with “How many people live in that city?”, the model can only connect the two if your application resends the whole conversation with each request.

POST /v1/chat/completions HTTP/1.1
Host: localhost:11434
Content-Type: application/json
{
  "messages": [
    { "role": "user", "content": "What is the capital of Belgium?" },
    { "role": "assistant", "content": "The capital of Belgium is **Brussels**..." },
    { "role": "user", "content": "How many people live in that city?" }
  ],
  "model": "qwen3:8b"
}
 
HTTP/1.1 200 OK
Content-Type: application/json
{
  ...
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "As of 2023, the **Brussels-Capital Region** has a population
                    of approximately **1.2 million people**..."
      },
      ...
    }
  ],
  "usage": { "prompt_tokens": 134, "completion_tokens": 843, "total_tokens": 977 }
}

Whether they cost progressively more compute is a subtler question, and it is where the KV cache re-enters. Statelessness means replaying the history is the application's job; it does not mean the server always re-computes it. If the engine still holds the KV cache from the conversation's earlier turns, and the replayed prefix matches token for token, it skips prefill for those tokens and jumps almost straight to decode. A request that lands on a replica already holding its conversation's cache can be an order of magnitude cheaper to serve than one that lands cold.

Batching: how engines earn their throughput

So far, we have followed only a single request. However, a production environment can quickly reach hundreds of thousands of requests per day. To maintain high utilization and reduce waste, engines are adopting two strategies:

• Static batching gathers requests until the batch is full, then processes them all at once. This gives great throughput, but the first request in the batch has to wait for the last one to arrive, which adds latency. This trade-off works well for offline tasks like summarizing documents, nightly embedding jobs, or batch classification, where no one is waiting for an immediate response.

• Continuous batching lets new requests join the running batch as others finish, processing them token by token. This approach greatly reduces per-request latency, which is important for interactive workloads like chatbots.

Observability and the four numbers that matter

When it comes to observability, everything above, from phases, caches, and batches surfaces in four metrics:

• TTFT (time to first token) is how long the user waits before anything appears, and it's dominated by queueing plus prefill.

• TPOT (time per output token) is the pace of generation once it starts, and it's dominated by decode.

• Tokens per second is throughput, per request or aggregate across the fleet.

• Requests per second matters mostly for capacity planning once you know your typical token counts.

A quick way to benchmark a model against your infrastructure is to use the official vLLM benchmark client, vllm bench serve, that fires a controlled load at any OpenAI-compatible endpoint.

vllm bench serve \
  --model Qwen/Qwen3-4B-AWQ \
  --base-url http://localhost:8000 \
  --dataset-name random \
  --random-input-len 512 --random-output-len 128 \
  --num-prompts 64 --max-concurrency 8

Its report is organized around exactly the four numbers above (ITL is TPOT measured gap by gap):

================= Serving Benchmark Result =================
Successful requests:                     64        
Failed requests:                         0         
Maximum request concurrency:             8         
Benchmark duration (s):                  38.83     
Total input tokens:                      32707     
Total generated tokens:                  8192      
Request throughput (req/s):              1.65      
Output token throughput (tok/s):         210.95    
Peak output token throughput (tok/s):    392.00    
Peak concurrent requests:                16.00     
Total token throughput (tok/s):          1053.20   
--------------------Time to First Token---------------------
Mean TTFT (ms):                          1632.54   
Median TTFT (ms):                        1355.88   
P99 TTFT (ms):                           5376.73   
P90 TTFT (ms):                           4301.80   
----------Time per Output Token (excl. 1st token)-----------
Mean TPOT (ms):                          25.34     
Median TPOT (ms):                        24.11     
P99 TPOT (ms):                           40.39     
P90 TPOT (ms):                           32.41     
--------------------Inter-token Latency---------------------
Mean ITL (ms):                           25.36     
Median ITL (ms):                         20.61     
P99 ITL (ms):                            29.10     
P90 ITL (ms):                            21.39     
---------------------End-to-end Latency---------------------
Mean E2EL (ms):                          4851.09   
Median E2EL (ms):                        4414.25   
P99 E2EL (ms):                           7989.19   
P90 E2EL (ms):                           7964.52   
============================================================

By running it with increasing concurrency, we can see continuous batching at work: aggregate tokens per second climbs several-fold while each individual request's TTFT and TPOT degrade only modestly.

When the batch outgrows the hardware, throughput plateaus and TTFT is the first casualty (requests queue), which is why queue depth, not CPU, is the routing signal that matters for LLM backends. And notice what these metrics ignore: CPU and memory utilization, the two numbers your existing autoscaling knows how to watch. That mismatch breaks more than dashboards; we dedicate a separate article to why LLM traffic defeats standard Kubernetes load balancing and autoscaling.

Weights, quantization, and the memory bill

Now the budget question: what does it take to hold all of this? For the weights, it's enough to multiply parameters by bytes and you have the footprint. For today's large language models, however, that cost is substantial. At 16-bit precision, a 70B-parameter model occupies roughly 141 GB before processing a single token, far exceeding the 80 GB available on an H100 GPU. The challenge is therefore not scaling with demand, but simply fitting the model into memory. This is where Quantization comes in. It stores the weights at lower precision reducing the bytes and the resulting memory, with an accuracy cost that modern 4-bit methods keep surprisingly small.

The KV cache consumes GPU memory on top of the weights, and it grows with every token of every concurrent conversation. Engines reserve a large share of VRAM for it. vLLM, for instance, works within a budget defined as a configurable fraction of total GPU memory (92% by default since v0.20, 90% before that), loads the weights, runs a profiling forward pass to measure activation overhead, and pre-allocates whatever remains of the budget as KV-cache space. More KV headroom means bigger batches and better throughput; less means earlier queueing. This one number decides how much of that headroom exists.

The table below runs both calculations for four representative models, taking each one's layer count, KV-head count, and head dimension from its config.json (per-token KV cache = 2 × layers × KV heads × head dim × bytes):

† Kimi K2.5 uses Multi-head Latent Attention (inherited from the DeepSeek-V3 lineage).

A practical recommendation

If you’re thinking about running a model on your own infrastructure, maybe because of token costs, usage limits, or compliance needs, try testing things out on your laptop before buying a GPU. Download a small model with Ollama, like qwen3:8b or llama3.2:1b. Then, serve with Ollama, point vllm bench serve at it, and gradually increase --max-concurrency from 1 up to 32 to see how your hardware handles the load. This will help you get familiar with metrics like TTFT, TPOT, KV Cache, and other important inference stats. The experiment is free and helps you avoid common mistakes while learning how the process works.