Hadal Research
PAPER
27 October 2026
9 pages

← Notes

Changing one word: how anchor direction breaks list indexing

Kyssta

long context · positional retrieval · paired analysis · small models

Paper (PDF, 9 pages)Novelty audit

A model that can find the fifth item of a list from the front may still fail to find the same item from the back. That asymmetry is well known. What is less clear is where it comes from, because in an ordinary forward/backward comparison two things change at once: the model counts in a different direction and it has to compute an offset it cannot simply read.

At the midpoint of an odd-length list the confound disappears. “5th from the beginning” and “5th from the end” name the same item, at the same ordinal, for the same list. The only difference between the prompts is the anchor word. This paper tests that contrast on a single small model, with paired items and a decision rule fixed in advance.

The contrast

Items:    L, O, J, M, R, P, N, K, Q
Question: What item is at the 5th position from the beginning?   →  R

Items:    L, O, J, M, R, P, N, K, Q
Question: What item is at the 5th position from the end?         →  R

Scoring is a single forward pass over the letters of that list, so the uniform baseline is 1/L — 11.1% at L=9 — not 1/26. Each cell holds 24 lists, and every comparison is paired by item: the same list is asked both ways, so the difference is within-item.

What was measured

  • Model. Qwen2.5-1.5B-Instruct, with a 0.5B control, both run in float32 on CPU.
  • Design. 24-item cells, paired by item, across list set (two seeds), measurement mode (forced choice and free generation), length (L = 7, 9, 11) and three question templates.
  • Statistics. Exact tests and method-10 intervals for paired proportions, computed from the 2×2 counts.
  • Decision rule. A ±15 practical margin and the 24-item cell size were fixed before the midpoint runs. The 95% interval method was not; it was adopted in v5 and corrected in v6, after the data existed. The paper says so, and keeps a correction log.

Results

Backward minus forward accuracy, paired by item. Negative means the backward wording is worse. Rows are backward correct/incorrect by forward correct/incorrect.

cell2×2fwd %bwd %Δ pp95% CIexact pverdict
L=9 · 71 · forced[[4,0],[8,12]]5017−33.3[−51.2, −11.6]0.00781margin not cleared
L=9 · 71 · generate[[4,0],[8,12]]5017−33.3[−51.2, −11.6]0.00781margin not cleared
L=9 · 101 · forced[[4,0],[9,11]]5417−37.5[−55.2, −14.8]0.00391margin not cleared
L=7 · 101 · forced[[2,0],[6,16]]338−25.0[−44.3, −5.0]0.03125margin not cleared
L=11 · 101 · forced[[2,2],[3,17]]2117−4.2[−24.5, +16.3]1.00000no reduction
L=9 · 101 · word ordinal[[3,6],[2,13]]2138+16.7[−7.2, +38.2]0.28906no reduction
L=9 · 101 · counting[[0,3],[3,18]]12120.0[−21.3, +21.3]1.00000no headroom
qwen05 · L=9 · 101[[0,3],[0,21]]012+12.5[−3.5, +31.0]0.25000no headroom
qwen05 · L=7 · 101[[0,2],[0,22]]08+8.3[−6.7, +25.8]0.50000no headroom
Full grid, v6; cells without a model prefix are Qwen2.5-1.5B-Instruct. Intervals are Newcombe method 10 with the n/2 covariance correction. “Margin not cleared” means the interval excludes zero but its upper end sits above the −15 pp practical margin. A cell whose forward accuracy is below 15% has no headroom and is not evidence either way.

What the data supports — and what it does not

Four cells show a reduction whose interval excludes zero: −33.3, −33.3, −37.5 and −25.0 points. None of them clears the 15-point practical margin — their upper ends are −11.6, −11.6, −14.8 and −5.0. So the supported claim is about sign, not size:

In this model, with this wording, changing the anchor word alone reduces accuracy. This study cannot bound the size of that reduction below 15 points.

Those two claims are kept as separate verdicts in every table, because they are separate claims. Collapsing them into “a 33-point drop” would be true of the estimate and false of the interval.

The effect depends on the template

Both wording cells ran on the same 24 lists, so each item contributes both effects. The baseline effect is −37.50 points, the word-ordinal paraphrase is +16.67 points, and their interaction is −54.17 points ( [−79.17, −33.33], exact permutation p = 0.00048828, i.e. 2/4096). The paraphrase changes several things at once — “What” to “Which”, the digit to a word, a dropped clause — so this is an anchor × question-template interaction, not a digit-versus-word effect. It was specified after the baseline cells existed, and is labelled exploratory.

At L=11 the contrast is −4.2 points ( [−24.5, +16.3]): no reduction detected at this sample size, and the interval still admits a penalty of about twenty points. Length generalisation is therefore not established, only not contradicted.

Not claimed a general direction effect; a mixture of two mechanisms; anything about frontier models; priority over prior work on prompt sensitivity, which is real and predates this study.

Reproducing it

The analysis path imports only the Python standard library — no numpy, no scipy, nothing to install. recompute_v5.py regenerates every interval and verdict from the raw JSONL in under a second. The repository ships the raw records, the per-item indicators, a timestamped correction log, a novelty audit, and a SHA-256 manifest that verifies byte-for-byte. Regenerating on a different libm can shift an interval endpoint by ~1e-16; compare with a tolerance, not byte equality.

Code is MIT; paper, docs and data are CC BY 4.0. It is an unpublished manuscript — not peer reviewed — and no priority claim is made.

Sources
Status

Unpublished manuscript. Not peer reviewed; no priority claim is made.