LlamaForCausalLM

LlamaForCausalLM at 32 layers and hidden size 4096. 11 published checkpoints share this shape.

Implementing it

How to read this sheet: method.

From GneissWeb.7B_ablation_model_on_350B_FineWeb.Edu.seed1

Also covers GneissWeb.7B_ablation_model_on_350B_FineWeb.Edu.seed2, GneissWeb.7B_ablation_model_on_350B_FineWeb.Edu.seed3, GneissWeb.7B_ablation_model_on_350B_FineWeb.seed1, GneissWeb.7B_ablation_model_on_350B_FineWeb.seed2, GneissWeb.7B_ablation_model_on_350B_FineWeb.seed3, GneissWeb.7B_ablation_model_on_350B_GneissWeb.seed1, GneissWeb.7B_ablation_model_on_350B_GneissWeb.seed2, GneissWeb.7B_ablation_model_on_350B_GneissWeb.seed3.

x = embed[ids]                    # [T, 4096]

for i in 0 .. 31:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05)
    q = h @ layers[i].self_attn.q_proj.T       # [T, 32*128]
    k = h @ layers[i].self_attn.k_proj.T       # [T, 32*128]
    v = h @ layers[i].self_attn.v_proj.T       # [T, 32*128]
    q, k = rope(q, k, theta=10000)
    a = softmax(q @ k.T / sqrt(128), mask=causal) @ v
    a = a @ layers[i].self_attn.o_proj.T
    x = x + a

    h = rmsnorm(x, layers[i].post_attention_layernorm)
    g = silu(h @ layers[i].mlp.gate_proj.T)           # [T, 10928]
    u =       h @ layers[i].mlp.up_proj.T
    y = (g * u) @ layers[i].mlp.down_proj.T
    x = x + y

x = rmsnorm(x, model.norm)
logits = x @ lm_head.T

Differs from the block above

The same arithmetic with these values changed. Everything not listed is as above.

Notes

head_dim is not published; 4096 / 32 = 128 is used.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Other shapes of LlamaForCausalLM

Identical across all of them: heads 32.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• GneissWeb.7B_ablation_model_on_350B_FineWeb.Edu.seed1, GneissWeb.7B_ablation_model_on_350B_FineWeb.Edu.seed2, GneissWeb.7B_ablation_model_on_350B_FineWeb.Edu.seed3, GneissWeb.7B_ablation_model_on_350B_FineWeb.seed1, GneissWeb.7B_ablation_model_on_350B_FineWeb.seed2, GneissWeb.7B_ablation_model_on_350B_FineWeb.seed3, GneissWeb.7B_ablation_model_on_350B_GneissWeb.seed1, GneissWeb.7B_ablation_model_on_350B_GneissWeb.seed2, GneissWeb.7B_ablation_model_on_350B_GneissWeb.seed3: out of scope, not-assessed — out of scope; no implementation is claimed and none is graded
• granite-7b-instruct, granite-7b-base: runs, shape-pack — stored checkpoints of this shape run; this one's own is not among them

Parts of the author's unpublished implementation that exist for this shape:

• a decoder implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture

Recorded pass: passes.