ANDREA-120M β€” BLAH Jacobian Lens Bundle

A blah-jacobian-lens (format version 1) bundle for russellbal/ANDREA-120M, consumed by evals.blah.dev to render layer-by-token Logit Lens and Jacobian Lens readouts.

This artifact is a readout instrument: fitted transport matrices that let internal representations be decoded through the model's own final decoding path, per site and per position. It is not a sparse autoencoder, and it makes no architectural claims (no workspace, broadcast, or causal-necessity claims) β€” those would require interventions and controls that are downstream work.

Base model

Repository russellbal/ANDREA-120M
Revision (pinned) 6f81f00f9669db77d9fe6e62b525fac0650b34d8
Checkpoint model.safetensors (98,698,752 params, float32)
Architecture AndreaForCausalLM β€” 12 pre-norm decoder blocks, 768 wide, 12 heads, ReLU MLP, learned absolute positions (1024), parameter-free RMSNorm, no final norm, untied lm_head
Tokenizer Harris successor-variety, byte-level with 8192 learned segments + BOS (vocab 8449)
Custom code Yes β€” trust_remote_code=True loads the repo's modeling_andrea.py / tokenization_andrea.py

Exact file fingerprints (weights, config, tokenizer, chat template) are in lens-manifest.json; a consumer must refuse to apply this lens to any model whose fingerprints differ.

Sites

13 token-aligned sites of width 768, layout BTD:

id meaning
embed.post residual entering block 0 (wte+wpe, after input RMSNorm)
block.NNN.post (000–011) residual after block NNN (post both residual adds)

Target site: block.011.post β€” the exact tensor consumed by the final decoding path. ANDREA applies no final norm, so exact_final_decode(h) = lm_head(h) with a float32 cast, and a direct Logit Lens is well-defined at every site (logit_lens_supported: true).

Lens

  • Estimator kind: same_position β€” J_s = E_t[βˆ‚ target[t] / βˆ‚ site_s[t]], cotangent injected at one target position at a time, gradient read at the same source position; positions [skip_first, T-2]; averaged over positions then prompts; float64 accumulation; dimension-batched VJP over (position, dimension) pairs.

  • Application is affine (lens.centering.mode = "affine"):

    transported = target_mean + (h_s βˆ’ source_mean_s) @ Jα΅€_s
    jacobian_logits = exact_final_decode(transported)
    

    Per-site means (mean.<site>, float32) and the target mean (mean.target) are stored in transports.safetensors alongside the matrices.

  • Transports: dense, transport.0000–transport.0012, each [768, 768], stored J[out, in], applied h @ Jα΅€. Published dtype float16 (means float32), exported only after parity measurement against the float32 master β€” see exported_dtype_parity in validation.json.

Fitting corpus

Prompts sampled deterministically (seeded reservoir, seed 42) from the public russellbal/smol-seeds dataset at a pinned revision β€” an interleave of chat.txt and smoltalk.txt, two core sources of the megachat composite this model trained on. Fully public provenance: fit-report.json records the dataset revision, corpus fingerprint, filtering rules, counts, device, dtype, and elapsed statistics. Fitting parameters: see lens-manifest.json lens block.

Validation

validation.json carries the full suite: fingerprints, tokenizer parity, final-logit parity (exact to 0.0 by construction β€” the decode path is the model's own lm_head), determinism, VJP finite-difference check, matrix orientation, transport shapes, final-site identity, exported-dtype parity, split-half convergence, and golden-fixture reproduction. fixtures/golden.json + fixtures/golden.safetensors pin one raw prompt and one chat-formatted example with exact token ids, byte-exact token identities (bytes_base64 β€” Harris token strings use the GPT-2 byte-display mapping, so text alone is not byte-exact), captured activations, and top-k readouts.

Execution

execution.mode = "huggingface": evals.blah.dev workers load the base model from its repository at the pinned revision (custom code required, no auth). There is no public runtime URL; blah-lens-http/1 is not served.

Reproduction

From the model's source repository (training.ai.unturf.com, directory microgpt/):

# fetch chat.txt + smoltalk.txt from russellbal/smol-seeds at the revision in
# fit-report.json, then: paste -d'\n' chat.txt smoltalk.txt | grep -v '^$' > mix.txt
python3 scripts/jlens-corpus.py --dataset mix.txt \
    --samples 256 --heldout 8 --seed 42 --out corpus.jsonl \
    --corpus-name 'smol-seeds public chat+smoltalk interleave' \
    --identifier 'russellbal/smol-seeds (chat.txt + smoltalk.txt)' \
    --visibility public
python3 -m jlens fit --checkpoint <hf-snapshot-dir> --prompts corpus.jsonl \
    --samples 256 --max-seq-len 128 --skip-first 16 --dim-batch 96 \
    --estimator-kind same_position --dtype float32 --output dist/blah-lens
python3 -m jlens finalize --checkpoint <hf-snapshot-dir> \
    --repo-id russellbal/ANDREA-120M \
    --revision 6f81f00f9669db77d9fe6e62b525fac0650b34d8 \
    --corpus-meta corpus-meta.json --heldout corpus-heldout.jsonl \
    --output dist/blah-lens
npx @blahai/lens validate ./dist/blah-lens

Known limitations

  • Fitted at max_seq_len 128 of a 1024 learned-position window; transports average position effects only over fitted positions.
  • Fitting-corpus lines cap at 256 characters (a build-time cap in the seed datasets), so fitted sequences are short; 256 prompts compensate.
  • The chat template has no system role β€” ANDREA trained without one.
  • Readout instrument only: steering, suppression, ablation, and concept swaps are not part of this bundle.

License

AGPL-3.0, matching the base model. The fitting corpus is public (russellbal/smol-seeds, same project, same license family).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for russellbal/ANDREA-120M-jlens

Finetuned
(1)
this model