This notebook is a visual companion to the chronological research notes. The examples are deliberately small and synthetic; I use them to isolate one mechanism at a time and make some of the less intuitive parts of the research easier to see.
Unless a figure is explicitly labeled historical result, its values are illustrative. These plots explain ideas behind the experiments; they aren't evidence from the research program.
1. Representation can change coordinates without losing information
Related milestone: 001 — What if text were a signal?
A Fourier transform can preserve all of the information in a signal while making different structure visible. The waveform and its frequency representation below describe the same synthetic signal, but they make different properties easy to inspect.
import numpy as np
import matplotlib.pyplot as plt
t = np.linspace(0, 1, 256, endpoint=False)
x = np.sin(2*np.pi*5*t) + 0.45*np.sin(2*np.pi*17*t)
freq = np.fft.rfftfreq(len(t), d=t[1]-t[0])
mag = np.abs(np.fft.rfft(x))
plt.figure(figsize=(7,3))
plt.plot(t, x)
plt.xlabel('Position')
plt.ylabel('Signal value')
plt.title('The same information in the original coordinates')
plt.show()
plt.figure(figsize=(7,3))
plt.plot(freq, mag)
plt.xlim(0, 30)
plt.xlabel('Frequency')
plt.ylabel('Magnitude')
plt.title('A coordinate change exposes different structure')
plt.show()
Sticky note — coordinate system: an invertible change of coordinates can preserve all of the information in a state while changing how difficult that information is for a particular downstream operation to use.
The visible frequency peaks don't imply that a spectral language model should perform better. They illustrate the distinction that started this research sequence: information being preserved and information being easy for a particular consumer to use are different questions.
2. Phase-aware similarity responds directly to phase difference
Related milestones: 003 and 004
For unit-length complex values, the normalized real-Hermitian similarity used in the phase-aware attention experiment reduces to cos(Δθ). Relative phase therefore affects similarity in a predictable circular way.
delta = np.linspace(-np.pi, np.pi, 400)
similarity = np.cos(delta)
plt.figure(figsize=(7,3))
plt.plot(delta, similarity)
plt.axhline(0, linewidth=1)
plt.xlabel('Phase difference Δθ (radians)')
plt.ylabel('Normalized real-Hermitian similarity')
plt.title('Phase-aware similarity is circular')
plt.show()
Two values with aligned phase receive positive similarity, a quarter-turn difference produces a score near zero, and opposing phases produce negative similarity. The point isn't that this is the only useful way to compare phase-valued features; it's that the operation directly reflects the structure we chose to represent.
That was the distinction behind the coordinate-control experiments. Expressing information in phase-aware coordinates may help on its own, while applying operations that explicitly respect relative phase is a separate intervention.
steps = np.array([0,16,32,64,96,128,192,288])
shared = np.array([5.4,3.7,2.95,3.02,3.18,3.38,3.55,3.72])
unit = np.array([5.5,4.5,4.05,3.85,3.72,3.60,3.42,3.30])
plt.figure(figsize=(7,3))
plt.plot(steps, shared, marker='o', label='Shared recurrent control')
plt.plot(steps, unit, marker='o', label='Unit-hypersphere model')
plt.xlabel('Training update')
plt.ylabel('Illustrative validation NLL')
plt.title('Ranking depends on the checkpoint-selection question')
plt.legend()
plt.show()
A fixed endpoint asks which model is better at one specified training budget. The best checkpoint observed anywhere along each curve asks a different question, and a stopping policy frozen before seeing the outcome asks another one.
The same pair of learning curves can therefore support different rankings without any of the measurements being numerically wrong. This is why the unit-hypersphere anomaly eventually required a fair checkpoint-selection experiment rather than another comparison at a convenient endpoint.
4. Normalization removes radial motion and rescales tangent motion
Related milestone: 007 — Derive before training.
For
N(x) = x / ||x||
the first-order gain of a radial perturbation is zero, while the gain of a tangent perturbation is 1 / ||x||.
r = np.linspace(0.2, 4.0, 300)
plt.figure(figsize=(7,3))
plt.plot(r, 1/r, label='Tangent gain 1/r')
plt.plot(r, np.zeros_like(r), label='Radial gain 0')
plt.axvline(1.0, linestyle='--', linewidth=1)
plt.xlabel('Input radius ||x||')
plt.ylabel('First-order gain')
plt.title('Normalization is exactly anisotropic')
plt.legend()
plt.show()
At unit radius, tangent perturbations are preserved to first order. At larger radius they are attenuated, while close to zero they are strongly amplified. That last behavior is one reason the theorem explicitly requires a nonzero input.
✓ Formal checkpoint: this derivative is machine-checked in the public Theorem Library normalization proof.
The figure shows the mathematical behavior of normalization. It doesn't assign meaning to either direction; deciding whether radial or tangent variation matters for a task remains empirical.
5. Small per-step differences can compound across recurrence
Related milestone: 008 — What does the sphere actually do?.
For the simplest possible scalar example, if one recurrent step scales a perturbation by g, then repeating the same operation for T steps scales it by g^T.
depth = np.arange(0, 21)
plt.figure(figsize=(7,3))
for g in [0.98,0.9,0.7]:
plt.plot(depth, g**depth, marker='o', label=f'g={g}')
plt.xlabel('Recurrent depth')
plt.ylabel('Perturbation gain')
plt.title('Mild one-step contraction can become strong finite-horizon contraction')
plt.legend()
plt.show()
Even mild contraction can therefore become substantial after enough recurrent steps.
Real neural networks are more complicated because the directions being amplified or contracted can rotate between steps. Multiplying the actual Jacobians along the trajectory captures that interaction; independently inspecting the singular values of each step generally does not.
This is why the mechanism campaign moved from one-step derivatives to finite-horizon Jacobian products.
6. Radius and angle are coupled in ordinary Euclidean polar geometry
Related milestone: 011 — Geometry should follow invariance.
For two vectors separated by a fixed angle θ, their ordinary inner product is
r_q r_k cos(θ).
rq = np.linspace(0.5,3.0,80); rk = np.linspace(0.5,3.0,80)
RQ,RK = np.meshgrid(rq,rk)
S = RQ*RK*np.cos(np.pi/3)
plt.figure(figsize=(6,4))
plt.imshow(S, origin='lower', aspect='auto', extent=[rq.min(),rq.max(),rk.min(),rk.max()])
plt.colorbar(label='Raw inner product')
plt.xlabel('Query radius'); plt.ylabel('Key radius')
plt.title('At fixed angle, Euclidean similarity still changes with radius')
plt.show()
The angle is fixed throughout the figure, but changing either radius still changes the similarity.
If semantic similarity is supposed to ignore radius, this is a mismatch between the desired invariance and the operation being used. If radius is supposed to modulate similarity, the same coupling may be useful. The equation doesn't choose the interpretation; the task requirement does.
7. Longer contexts trade repeated support for specificity
Related milestone: 012 — Natural-source task distinctions.
The following counts are illustrative rather than historical benchmark measurements.
L = np.array([8,16,32,64,128])
repeated = np.array([1200,760,410,125,18])
branches = np.array([310,245,180,62,9])
plt.figure(figsize=(7,3))
plt.plot(L,repeated,marker='o',label='Repeated exact contexts')
plt.plot(L,branches,marker='o',label='Contexts with distinct continuations')
plt.xscale('log',base=2)
plt.xlabel('Exact context length L (bytes)')
plt.ylabel('Illustrative support count')
plt.title('Specificity and repeated support trade off')
plt.legend()
plt.show()
Longer exact contexts identify increasingly specific situations, but naturally occurring text repeats those exact situations less often. Eventually there may be too little repeated support to estimate a useful conditional distinction.
The natural-source benchmark qualified at L=32. Context length was frozen before favorable model outcomes were inspected because choosing it afterward would turn benchmark construction into another form of outcome-dependent tuning.
8. Probe recoverability changes continuously with signal strength
Related milestone: 013 — Accessible does not imply used.
This synthetic experiment changes only the strength of a linearly accessible class signal embedded in otherwise noisy states.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng=np.random.default_rng(12); strengths=np.linspace(0,1.5,13); scores=[]
for strength in strengths:
n,d=3000,24; y=rng.integers(0,2,n); h=rng.normal(size=(n,d))
v=rng.normal(size=d); v/=np.linalg.norm(v)
h+=(2*y[:,None]-1)*strength*v
Xt,Xe,yt,ye=train_test_split(h,y,test_size=.3,random_state=5,stratify=y)
scores.append(LogisticRegression(max_iter=1000).fit(Xt,yt).score(Xe,ye))
plt.figure(figsize=(7,3))
plt.plot(strengths,scores,marker='o')
plt.axhline(.5,linestyle='--',linewidth=1)
plt.xlabel('Injected linearly accessible signal strength')
plt.ylabel('Held-out affine-probe accuracy')
plt.title('Accessible information is not a binary property')
plt.show()
As the accessible signal becomes stronger, a simple affine probe recovers it more reliably. There isn't a universal point where information suddenly changes from “absent” to “present”; recoverability depends on signal strength, sample size, the consumer being tested, and the evaluation protocol.
A strong probe result therefore establishes accessibility only for the tested probe family and task. It doesn't establish native causal use. A weak result is similarly ambiguous: the signal may be weak, the dataset may be too small, the probe family may be mismatched, or the optimization/protocol may have failed.
How to use this atlas
Each section is designed so that one assumption or parameter can be changed without rebuilding the whole experiment. Before rerunning a cell, predict what should happen and why.
The questions I find most useful are the same ones that eventually shaped the research program: what is fixed, what changed, what else could produce the same result, and what observation would make me change the next experiment?
The figures are here to build intuition around those questions. The milestone notebooks and their underlying research artifacts carry the evidence.