Empirical Work / Loss Landscape Measurements

Loss Landscape Measurements

GPT-2 small · April 2026 · Google Colab

ONE UN-REPLICATED FIRST-PASS RUN — directional only. Not statistically defensible. Sample size: approximately five sentences per domain. Coupling probe: pairwise Pearson correlation of attention weight matrices on a single probe input — inference-time behavioral measurement, not Hessian off-diagonal structure. These measurements are illustrative pending replication with larger samples and formal PyHessian analysis.

Model Specification

Model

GPT-2 small

Parameters

124,439,808

Training corpus

WebText (Reddit outbound links, 2019)

Weights

Unmodified pretraining weights

Fine-tuning

None — no RLHF or instruction tuning

Implementation

GPT2LMHeadModel.from_pretrained('gpt2')

Runtime

Google Colab, CPU

Date

April 2026

Domain Perplexity Map

Lower = dense WebText coverage (laminar territory). Higher = sparse coverage (archaeological signal candidate).

Domain	Perplexity	Territory
Technical documentation	19.1	Laminar
Vernacular dialect	33.2	Low drag
Reddit tech discussion	40.1	Low drag
Non-Western cultural context	46.4	Moderate gap
Literary prose	49.2	Moderate gap
Poetry	58.6	Elevated
Non-English text (Spanish)	83.6	High — corpus fingerprint
Academic abstract	102.5	Archaeological territory

5x difference between lowest (19.1) and highest (102.5). Academic abstracts highest — paywalled journals not linked from Reddit.

Inter-Head Coupling by Layer

Pairwise Pearson correlation of attention weight matrices between heads in the same layer. Single probe input. Proxy for Hessian off-diagonal coupling — not a direct measurement.

Layer	Coupling	Note
0	0.610	Low — early layer independence
1	0.703
2	0.724
3	0.747
4	0.675	Anomalous dip — below trend. Unexplained.
5	0.903	Sharp jump — mid-network transition
6	0.880
7	0.906
8	0.911
9	0.936	Peak — predicted highest ablation resistance
10	0.945	Peak — highest in network
11	0.789	Drop — output projection decouples

Monotonic increase with depth. Peak at layers 9-10. Layer 4 anomalous dip unexplained — test across Pythia scales to determine if architectural or scale artifact.

Untested Predictions

Layers 9-10 should show highest resistance to head ablation

Not yet run

Test: Michel et al. (2019) head ablation infrastructure. Zero out attention heads one at a time, measure perplexity change. Compare impact of ablating layers 9-10 vs layers 0-2.

OPT-125M academic abstract perplexity should be substantially lower than GPT-2's 102.5 (The Pile corpus includes academic papers)

Priority 2 — not yet run

Test: Run identical domain perplexity calculation on OPT-125M using same domain sentences. Direct corpus fingerprint test.

RLHF reduces perplexity variance, with largest reduction in archaeological domains (poetry, academic abstracts, non-English) not laminar domains

Priority 3 — not yet run

Test: Mistral-7B BASE vs INSTRUCT paired comparison. If reduction is uniform or concentrated in laminar domains, remagnetization claim is not supported.

Next step: PyHessian run on GPT-2 small to compute actual Hessian eigenvalue spectrum and measure correlation against the attention probe values recorded here. See experiment queue for full priority order.