GraniteVisionEmb

GraniteVisionEmb at 40 layers and hidden size 2048. One published checkpoint has this shape.

Implementing it

How to read this sheet: method.

No forward was recorded for this shape; the block is read from the published configuration and from the modelling code in the checkpoint's own repository at commit baa53d1 (modeling_granite_vision_embedding.py, processing_granite_vision_embedding.py), which subclasses transformers' LlavaNextForConditionalGeneration.

Generated from granite-vision-3.3-2b-embedding.

# granite-vision-3.3-2b's image path and decoder, with the base tile
# moved to the end, the language head unused, and a 128-wide
# projection read from the last hidden state.

# Image path: as on the LlavaNext sheet (four tower depths, projector,
# weave, unpad, image_newline), except for where the base tile goes:
rows = concat(flatten(grid), r[0])        # `base_image_feature_location:
                                          #   last`: the base tile's 729
                                          #   rows come after the grid

#   The SigLIP tower, as published:
#     layers                     27   `num_hidden_layers`
#     width                    1152   `hidden_size`
#     heads                      16   `num_attention_heads`
#     ffn width                4304   `intermediate_size`
#     patch / image          14 / 384 `patch_size`, `image_size`
#     activation  gelu_pytorch_tanh   `hidden_act`
#     norm eps                1e-06   `layer_norm_eps`

x = embed[ids]
x[ids == 49155] = rows                    # masked scatter, as on the LlavaNext sheet

x = x * 12                                # <- embedding_multiplier
for i in 0 .. 39:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05)
    q = h @ layers[i].self_attn.q_proj.T       # [T, 32*64]; heads not published (Notes)
    k = h @ layers[i].self_attn.k_proj.T       # [T, 8*64]
    v = h @ layers[i].self_attn.v_proj.T       # [T, 8*64]
    q, k = rope(q, k, theta=300000)
    k, v = repeat_kv(k, v, 4)            # 32 query heads share 8 key/value heads
    a = softmax(q @ k.T * 0.015625, mask=causal) @ v      # <- attention_multiplier
    a = a @ layers[i].self_attn.o_proj.T
    x = x + a * 0.22        # <- residual_multiplier

    h = rmsnorm(x, layers[i].post_attention_layernorm)
    g = silu(h @ layers[i].mlp.gate_proj.T)           # [T, 8192]
    u =       h @ layers[i].mlp.up_proj.T
    y = (g * u) @ layers[i].mlp.down_proj.T
    x = x + y * 0.22

x = rmsnorm(x, model.norm)
# No logits: the last hidden state is read directly, so the published
# `logits_scaling: 8` never applies.

# A page image keeps only its last 729 image positions; every other row,
# text included, is dropped. With the base tile last, those are the base
# tile's rows.
doc = x[sorted(last_729(positions where ids == 49155))]
                                          # 729 is a literal in the code
e = doc @ custom_text_proj.T + custom_text_proj.bias   # [*, 2048] -> [*, 128]
e = e / (norm(e) + 1e-8)                  # L2, per row
e = e * attention_mask                    # padding rows zeroed

# A text query keeps every position. Scoring is late interaction:
score = sum over query rows i of  max over doc rows j of  (q_i . d_j)

Notes

The text configuration publishes neither num_attention_heads nor hidden_act. The block uses GraniteConfig's defaults, 32 heads and silu (configuration_granite.py L137-L139), so head_dim is 2048 / 32 = 64. Those are the library's values, not this checkpoint's statement.

The decoder's four multipliers are placed as in modeling_granite.py at transformers v4.57.1: embedding, attention scale, residual, logits.

Where each step is in the repository code at baa53d1. The base tile is placed last when base_image_feature_location is last (L79-L82). The model reads the decoder's last hidden state rather than logits, which GraniteModel records after the final norm (modeling_granite.py L464-L468 at transformers v4.57.1) (L122-L123), keeps the last 729 image-token positions in order (L126-L139), and applies custom_text_proj, a 2,048 → 128 nn.Linear with bias (L112), then an L2 normalisation and the attention mask (L143-L149). Multi-vector scoring is a max over document rows summed over query rows (processing L386-L431).

The projection is after the decoder, not in the connector. custom_text_proj reads the final hidden state; the connector's output is 2,048 wide, as the decoder needs.

The placeholder count is fixed by the processor before the tower runs, as on the LlavaNext sheet.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• granite-vision-3.3-2b-embedding: runs, shape-pack — stored checkpoints of this shape run; this one's own is not among them

Parts of the author's unpublished implementation that exist for this shape:

• a retrieval model implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture