ModernBertForSequenceClassification

ModernBertForSequenceClassification at 22 layers and hidden size 768. One published checkpoint has this shape.

Implementing it

How to read this sheet: method.

No forward was recorded for this shape; the block is read from the published configuration and from ModernBertForSequenceClassification at transformers v4.57.1.

Generated from granite-embedding-reranker-english-r2.

x = layernorm(embed[ids], embeddings.norm, eps=1e-05)   # [T, 768], no position table

for i in 0 .. 21:
    glob = (i % 3 == 0)                   # `global_attn_every_n_layers: 3`
    h = x if i == 0 else layernorm(x, layers[i].attn_norm)
                                          # layer 0 has no attn_norm tensor
    q, k, v = split(h @ layers[i].attn.Wqkv.T, 3)
    q, k = rope(q, k, theta=160000 if glob else 40000)
    a = softmax(q @ k.T / sqrt(64), mask=padding) @ v    # bidirectional; a
                                          #   local layer also masks |m - n| > 64
                                          #   (query position m, key position n)
    x = x + a @ layers[i].attn.Wo.T
    h = layernorm(x, layers[i].mlp_norm)
    g, u = split(h @ layers[i].mlp.Wi.T, 2)   # [T, 1152] each; the first half
    x = x + (gelu(g) * u) @ layers[i].mlp.Wo.T   #   is the activated one

x = layernorm(x, final_norm)
# Every LayerNorm here has a weight and no bias (`norm_bias: false`); no
# projection has a bias.
h = x[0]                                  # `classifier_pooling: cls`
h = silu(h @ head.dense.T)                # `classifier_activation: silu`;
                                          #   no bias (`classifier_bias: false`)
h = layernorm(h, head.norm, eps=1e-05)    # weight only
score = h @ classifier.T + classifier.bias   # [768] -> 1
# sentence-transformers' CrossEncoder then applies sigmoid(score):
# `sentence_transformers.activation_fn` is torch.nn.Sigmoid.

Notes

The published score passes through a sigmoid in sentence-transformers. The configuration sets sentence_transformers.activation_fn to torch.nn.modules.activation.Sigmoid (config.json L48–L51), which a CrossEncoder applies to the logit above; transformers alone returns the raw logit.

The head is dense → silu → LayerNorm → classifier, on the CLS row. ModernBertForSequenceClassification pools by classifier_pooling and runs ModernBertPredictionHead — norm(act(dense(x))) — before the classifier (modeling_modernbert.py L946-L955, L1118-L1127, L1204-L1213). The checkpoint holds exactly head.dense.weight, head.norm.weight, classifier.weight and classifier.bias (model.safetensors). This is not RoBERTa's tanh head, and it has no dense bias.

The attention is bidirectional: the mask is built from the padding mask alone, with no causal term, and local layers add a window of local_attention // 2 = 64 positions either side (modeling_modernbert.py L929-L943). Layer i is global when i % global_attn_every_n_layers == 0, and local layers use local_rope_theta (L464-L471). A port that makes every layer global computes a different function once a sequence is longer than the window, not just a slower one.

Layer 0 has no attn_norm. transformers builds it as nn.Identity(), because the embedding norm has just run (L520-L523), so layer 0 carries five tensors and every other layer six. A loader that assumes uniform layers either fails to find the tensor or inserts a norm the model does not have.

The MLP's Wi output is split in half and the first half is activated: act(input) * gate (L241-L242).

position_embedding_type is not read. These configurations say absolute or relative_key_query, and modeling_modernbert.py at v4.57.1 never refers to the key: the model always rotates. The keys that decide the arithmetic are the two rope bases, global_attn_every_n_layers, local_attention, hidden_activation and norm_eps, and siblings differ in them.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• granite-embedding-reranker-english-r2: runs, own-pack — a stored copy of this checkpoint exists; nothing recorded from it

Parts of the author's unpublished implementation that exist for this shape:

• an encoder implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture