Granite4VisionForConditionalGeneration

Granite4VisionForConditionalGeneration at 40 layers and hidden size 2560. One published checkpoint has this shape.

Layer map

One character per layer, input on the left, read from the published configuration.

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Implementing it

How to read this sheet: method.

No forward was recorded for this shape; the block is read from the published configuration and from the modelling code in the checkpoint's own repository at commit bf108f3 (modeling.py, downsampling.py), which the configuration's auto_map names.

Generated from granite-4.0-3b-vision.

# Eight projections of the tower, each added into the residual stream at
# the input of a different decoder layer. Nothing is scattered at the
# embedding: the image positions are zeroed there instead.

tiles = [base_tile] + anyres_grid(image)  # 384x384 tiles (`image_grid_pinpoints`)
hs = vision_tower(tiles).hidden_states    # patch 16: a 24x24 grid, 576 rows a tile

# Where each projector reads and writes, straight from the configuration:
#   `deepstack_layer_map`    tower -19, -13, -7, -1  ->  decoder 9, 6, 3, 0
#   `spatial_target_layers`  tower -1, four ways     ->  decoder 12, 15, 18, 21
for (depth, dest, sampler) in eight_projectors:
    g = layernorm(hs[depth], eps=1e-06)   # each projector has its own norm
    keys = window(g, 8) + image_positions # 3x3 windows of 64 rows, plus a
                                          #   learned [64, 1152]
    small = sampler(g)                    # 24x24 -> 12x12: area interpolation
                                          #   (the four deepstack projectors), or
                                          #   one fixed cell of every 2x2 block,
                                          #   a different one per spatial projector
    q = query + window(small, 4)          # learned [16, 1152] plus 3x3 windows
                                          #   of 16 rows: the queries carry the image
    out = qformer(q, keys)                # one BLIP-2 Q-Former layer, width 1152
    r = unwindow(out) @ out_linear.T + out_linear.bias   # [144, 2560] a tile
    rows[dest] = pack(r)                  # weave, unpad, image_newline, base
                                          #   tile first, as on the LlavaNext sheet

#   The SigLIP tower, as published:
#     layers                     27   `num_hidden_layers`
#     width                    1152   `hidden_size`
#     heads                      16   `num_attention_heads`
#     ffn width                4304   `intermediate_size`
#     patch / image          16 / 384 `patch_size`, `image_size`
#     activation  gelu_pytorch_tanh   `hidden_act`
#     norm eps                1e-06   `layer_norm_eps`

x = embed[ids]
x[ids == 100352] = 0                        # image positions zeroed (`image_token_index`)
x = x * 12                                # <- embedding_multiplier
for i in 0 .. 39:
    if i in rows:                         # 0, 3, 6, 9, 12, 15, 18, 21
        x[ids == 100352] += rows[i]        # at the layer's input, before it runs
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05)
    q = h @ layers[i].self_attn.q_proj.T       # [T, 40*64]
    k = h @ layers[i].self_attn.k_proj.T       # [T, 8*64]
    v = h @ layers[i].self_attn.v_proj.T       # [T, 8*64]
    q, k = rope(q, k, theta=10000000)
    k, v = repeat_kv(k, v, 5)            # 40 query heads share 8 key/value heads
    a = softmax(q @ k.T * 0.015625, mask=causal) @ v      # <- attention_multiplier
    a = a @ layers[i].self_attn.o_proj.T
    x = x + a * 0.22        # <- residual_multiplier

    h = rmsnorm(x, layers[i].post_attention_layernorm)
    g, u = split(h @ layers[i].shared_mlp.input_linear.T, 2)   # [T, 8192] each
    y = (silu(g) * u) @ layers[i].shared_mlp.output_linear.T
    x = x + y * 0.22

x = rmsnorm(x, model.norm)
logits = (x @ embed.T) / 10        # tied by the configuration (see Notes); <- logits_scaling

Notes

head_dim is not published; 2560 / 40 = 64 is used.

Output head: the configuration says tied, the checkpoint ships a separate one. granite-4.0-3b-vision sets tie_word_embeddings: true, which is what the block follows, but its checkpoint also ships a separate lm_head.weight [100353, 2560]. Its class subclasses LlavaNextForConditionalGeneration (modeling.py L39) and inherits its tying: a transformers load reads that tensor and then, because the configuration asks for tying, replaces it with the input embedding (modeling_utils.py L2994, L5065), so the logits it computes are the ones written here. The published headers carry shapes, not values, so whether the shipped head equals embed is not known; if it does not, a loader that uses the shipped lm_head.weight computes different logits (findings §10).

The decoder's multipliers are placed as in modeling_granitemoehybrid.py at transformers v4.57.1: embedding, attention scale, residual, logits; the repository's own forward applies the embedding and logits factors itself (modeling.py L541, L129).

Where each step is in the repository code at bf108f3. Four deepstack and four spatial projectors are built from the configuration (modeling.py L259-L270). Each is a WindowQFormerDownsampler: a LayerNorm, windows of 8 for the keys plus learned image_positions, a 12 × 12 downsample windowed by 4 and added to the learned query, one Q-Former layer, and out_linear to the decoder width (downsampling.py L61-L151). The deepstack projectors downsample by area interpolation (L9-L25), the spatial ones by taking one cell of each 2 × 2 block (L28-L58). Each projector's rows are packed like LlavaNext's, with image_newline (modeling.py L413-L453).

Image positions are zeroed at the embedding, and the eight maps are added at eight layer inputs. The embedding rows at the image token are set to 0 (modeling.py L537), and each map is added to the residual stream at those positions immediately before its destination layer runs (L579-L584). A port that scatters the rows into the embedding and then adds all eight applies one map twice; one that adds each map after its layer runs is one layer off. Both keep every shape.

The eight destinations are not consecutive. Each map needs its own destination recorded; the rule "feature k after layer k" puts the rows in layers 0 to 7 with every shape still correct.

The row count per tile is 144, not 576. The processor counts the base tile at the downsampled 12 × 12, plus unpadded rows and one newline per raster row (processing.py L33-L54).

The decoder is GraniteMoeHybridForCausalLM with every layer attention, and its mamba_* keys are unused. The configuration publishes a full Mamba block (mamba_d_state 256, 128 heads, chunk 256) and a layer_types list that is attention at all 40 positions; transformers builds a Mamba mixer only where that list says mamba (modeling_granitemoehybrid.py L1125-L1129). Its MLP is one fused input_linear split in half, gate first (L901-L908).

The repository also ships a LoRA adapter (adapter_config.json, adapter_model.safetensors; rank 256, on the decoder's attention and MLP projections). When the adapter is loaded, the repository's generate enables it for input with an image and disables it for text-only input (modeling.py L62-L71), so the decoder that reads the image is the base weights plus that adapter.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Other shapes of Granite4VisionForConditionalGeneration

Identical across all of them: layers 40, width 2,560, heads 40, KV heads 8, FFN width 8,192, context 131,072.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• granite-4.0-3b-vision: runs, shape-pack — stored checkpoints of this shape run; this one's own is not among them

Parts of the author's unpublished implementation that exist for this shape:

• a windowed multi-depth connector
• a vision tower implemented from the published configuration
• a decoder implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture