GPTBigCodeForCausalLM

GPTBigCodeForCausalLM at 88 layers and hidden size 6144. 2 published checkpoints share this shape.

Implementing it

How to read this sheet: method.

Generated from granite-34b-code-base-8k, which covers all 2 checkpoints of this shape — they agree on every field below.

x = wte[ids] + wpe[0..T]                 # [T, 6144], learned absolute positions

for i in 0 .. 87:
    h = layernorm(x, layers[i].ln_1, eps=1e-05)
    qkv = h @ layers[i].attn.c_attn.T + layers[i].attn.c_attn.bias
    q, k, v = split(qkv, [48*128, 128, 128])
    # Multi-query: 48 query heads, one key/value head — and one
    # fused projection. The configuration says so with
    # `multi_query: true` and publishes no `num_key_value_heads`.
    a = softmax(q @ k.T / sqrt(128), mask=causal) @ v
    x = x + a @ layers[i].attn.c_proj.T + layers[i].attn.c_proj.bias

    h = layernorm(x, layers[i].ln_2)
    y = gelu_pytorch_tanh(h @ layers[i].mlp.c_fc.T + layers[i].mlp.c_fc.bias)   # [T, 24576]
    x = x + y @ layers[i].mlp.c_proj.T + layers[i].mlp.c_proj.bias

x = layernorm(x, ln_f)
logits = x @ wte.T                        # tied to the input embedding

Notes

This is the older code lineage and it differs from the Granite classes at every point a reader might carry an assumption across: LayerNorm, not RMSNorm; learned absolute positions, not rotary; a GELU MLP with no gate, not SwiGLU; one fused c_attn rather than three projections; and a bias on every projection, where the Granite classes carry none. Each is in modeling_gpt_bigcode.py at transformers v4.57.1: LayerNorm, position table, fused c_attn, its q/k/v split, tied head.

wpe has 8,192 rows (n_positions), which is the hard context bound: there is no rotary base to extend.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Outside the stack: transformer.wte.weight (the token embedding, which the output also reads), transformer.wpe.weight (the learned position table), and the final norm transformer.ln_f.weight, transformer.ln_f.bias.

Other shapes of GPTBigCodeForCausalLM

Identical across all of them: width 6,144, heads 48, KV heads 1, FFN width 24,576, vocabulary 49,152, context 8,192.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• each of the 2 checkpoints of this shape: runs, recorded-forward — this checkpoint was loaded and completed a forward pass

Parts of the author's unpublished implementation that exist for this shape:

• a decoder implemented from the published configuration
• a configuration parser that refuses what it cannot express
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture

Recorded pass: passes.