Position And Context

Self-attention compares token representations, but the dot product does not know whether a token appeared at the start, middle, or end of the sequence unless position information is injected. Learned absolute embeddings add a table lookup to token embeddings. Sinusoidal encodings add fixed waves. Modern decoder-only LLMs often use rotary position embeddings because the relative offset between query and key positions becomes visible inside the attention score.

RoPE treats pairs of head dimensions as 2D coordinates and rotates them by an angle determined by token position and frequency. Apply the same position-dependent rotation to queries and keys, not values. When a query at one position meets a key at another, their dot product carries information about the relative distance between the two positions.

def rotate_half(x):
    even = x[..., 0::2]
    odd = x[..., 1::2]
    return torch.stack((-odd, even), dim=-1).flatten(-2)

def apply_rope(x, cos, sin):
    # x, cos, sin broadcast over batch and heads.
    return x * cos + rotate_half(x) * sin

q = apply_rope(q, cos[position_ids], sin[position_ids])
k = apply_rope(k, cos[position_ids], sin[position_ids])

1. Shape check: head dimension must be even for this pairwise form, and cosine/sine tensors must broadcast to batch, heads, time, head_dim.
2. Cache check: during decode, use the absolute cache position of the new token, not position zero of the one-token step.
3. Value check: leave v unrotated; the position signal reaches values through the attention weights.

ALiBi does not add a position vector to token embeddings. It adds a head-specific linear penalty to attention scores based on how far the key is from the query. Nearby past tokens receive a smaller penalty than distant tokens, creating a recency bias that can extrapolate beyond the training length more naturally than a learned absolute table.

def causal_alibi_bias(num_heads, q_len, k_len, slopes, device):
    slopes = slopes[:num_heads].to(device)
    q_pos = torch.arange(k_len - q_len, k_len, device=device)[:, None]
    k_pos = torch.arange(k_len, device=device)[None, :]
    distance = q_pos - k_pos
    distance = distance.clamp_min(0)
    return -slopes[:, None, None] * distance[None, :, :]

scores = (q @ k.transpose(-2, -1)) / math.sqrt(q.size(-1))
scores = scores + causal_alibi_bias(q.size(1), q_len, k_len, slopes, q.device)

RoPE scaling can make a model run at a longer maximum sequence length, but it is not proof that the model reasons well over that range. Treat context extension as a model change: check perplexity on long text, retrieval accuracy at different depths, instruction following when the answer is near the middle, and serving memory after the longer KV cache is admitted.

1. Math gate: verify RoPE frequency parameters, cache positions, and max sequence settings agree across training, inference, and serving config.
2. Quality gate: test short-context regressions and long-context recall separately; a longer window can still dilute attention or hurt precision.
3. Systems gate: recompute KV-cache bytes from the new maximum prompt plus generation budget before increasing concurrency.

Config Drill: Version The RoPE Scaling Contract

Context extension is more than changing max_position_embeddings. YaRN and LongRoPE both extend RoPE-based models by changing how positions map to rotary frequencies, while Hugging Face documents multiple RoPE variants such as linear, dynamic, yarn, longrope, and llama3. vLLM's context-extension guidance points back to those RoPE parameters because the serving engine must load the same positional contract the model was trained or adapted with.

rope_extension_record = {
    "base_model": "repo/name@revision",
    "tokenizer": "repo/name@revision",
    "rope": {
        "rope_type": "yarn | longrope | llama3 | linear | dynamic",
        "factor": 8,
        "original_max_position_embeddings": 8192,
        "rope_theta": 500000.0,
    },
    "validated_lengths": [8192, 32768, 65536],
    "serving_limit": {"max_model_len": 65536, "kv_cache_budget_checked": True},
}

1. Pin the original length: record the pre-extension context length separately from the new serving limit.
2. Pin the variant: do not treat all RoPE scaling methods as interchangeable; their frequency schedules and short-context behavior can differ.
3. Replay short tests: LongRoPE explicitly targets preserving original-window quality, so short-context regressions belong in the launch gate.
4. Keep runtime aligned: model config, tokenizer/chat template, serving engine, cache positions, and KV budget must agree before long-context evals mean anything.

Evaluation Drill: Sweep Evidence Position, Not Just Length

A long-context smoke test should vary where the needed evidence appears inside the prompt. The Lost in the Middle study found that models can perform best when relevant information is near the beginning or end and degrade when it is buried in the middle. LongBench adds a broader check: long-context quality should be measured across single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code tasks rather than a single retrieval trick.

long_context_eval = {
    "context_lengths": [4_000, 16_000, 64_000],
    "evidence_depths": [0.05, 0.25, 0.50, 0.75, 0.95],
    "tasks": ["key_value_retrieval", "multi_doc_qa", "summarization", "code"],
    "slices": ["answer_near_start", "answer_middle", "answer_near_end"],
}

1. Hold the answer fixed: move the same evidence across depths so position, not content difficulty, explains the score change.
2. Separate retrieval from reasoning: a needle test catches exact lookup, while multi-document and code tasks catch synthesis over long inputs.
3. Report the trough: publish the worst depth bucket as well as the average, because the middle of the prompt is often the product-critical failure case.

Sources: Attention Is All You Need, RoFormer / Rotary Position Embedding, ALiBi: Train Short, Test Long, Lost in the Middle, LongBench, LongBench v2, YaRN, LongRoPE, Hugging Face RoPE utilities, Hugging Face TGI model preparation and RoPE scaling, vLLM context extension.