GraniteSpeech5ForCTC

GraniteSpeech5ForCTC at 16 layers and hidden size 1024. 2 published checkpoints share this shape.

Implementing it

How to read this sheet: method.

No forward was recorded for this shape, so the block is read from the published configuration and from granite_speech5 in transformers v5.16.0, the first release that contains this class — not transcribed from a run.

Generated from granite-speech-5.0-470m-turboctc, which covers both checkpoints of this shape — they agree on every field below.

# The input width is not in config.json. It publishes `num_mel_bins: 80`
# and the encoder reads 320 = 80 x 2 x 2, and the two factors are not
# the same reason:
#     x2   each frame carries its own delta
#     x2   frames are then stacked in pairs
# (`deltas: true`, `stack_factor: 2` in preprocessor_config.json.)

feat = stack_pairs(with_deltas(mel_80(audio)))   # [T, 320]
x    = input_linear(feat)                 # -> 1024

for i in 0 .. 15:                          # conformer, five modules
    x = x + 0.5 * ff1(layernorm(x))       # silu, 1024 -> 4096 -> 1024
    x = x + attn_blocked(layernorm(x), window=128)
    #     ^ block-diagonal over non-overlapping 128-frame windows (the
    #       tail is padded and masked); Shaw relative positions added to
    #       the scores. No rope in this tower.
    stride = 2 if i in [0, 1] else 1       # `subsample_layers`
    c = conv_module(layernorm(x), stride) # lin1 -> GLU -> depthwise k=7
                                          #   -> BatchNorm -> silu -> lin2
    x = c + (x if stride == 1 else mean_pairs(x))
    #  ^ the frame count halves here, so the two modules below already
    #    run at the shorter length.
    x = x + 0.5 * ff2(layernorm(x))
    x = layernorm(x)                      # `norm_out`
    if i == 7:                            # num_hidden_layers // 2, after
        x = x + out_mid(softmax(out(x)))  #   the 8th block: self-conditioned
                                          #   CTC, through the same `out`

# T' = T / 4: stride 2 at each of layers [0, 1]. One CTC row per
# surviving frame.

#   The conformer tower, as published:
#     layers            16   `num_hidden_layers`
#     width           1024   `hidden_size`
#     heads              8   `num_attention_heads`
#     kv heads           8   `num_key_value_heads`
#     head dim         128   `head_dim`
#     ffn width       4096   `intermediate_size`
#     conv kernel        7   `conv_kernel_size`
#     activation      silu   `hidden_act`
#     vocab          16384   `vocab_size`
#     rel-pos table    512   `max_position_embeddings` — sizes the Shaw
#                            table (2 * 512 + 1 rows) and offsets the
#                            distances into it; distances inside a
#                            128-frame block stay within ±127
#     bias            True   `attention_bias` — on the feed-forward
#                            linears; q/k/v carry no bias, o_proj does

logits = ctc_head(x)                      # [T', 16384]
#        ^ `ctc_head` is tied to `encoder.out` (`tie_word_embeddings`), so
#          there is one matrix to load and two names for it.

# CTC, not autoregression: one distribution per frame, then collapse.
ids = argmax(logits, axis=-1)
ids = drop_consecutive_duplicates(ids)    # in that order —
ids = drop(ids, blank_id)                 # collapse first, then drop
#     blank = id 0 (`pad_token_id`, `<|blank|>`)

Sources: transformers v5.16.0 input_linear at num_mel_bins * 4 and the feature extractor's delta and stacking factors, with the published front-end settings; the block and subsampling block; block attention with the Shaw table and compute_attention_dists; the conv module and feed-forward; the mid-layer self-conditioning; the tied head; the CTC decode in ParakeetTokenizer._decode, the tokenizer class the checkpoint names.

The input is 320 wide and config.json says 80. The factor of four is a delta expansion and a pair-stack, in that order, and both live in preprocessor_config.json, not in the model configuration. A mel-only stack of four has the right width and the wrong content.

There is no decoder and no text config. This class transcribes by committing a token per frame, which is why its vocabulary is 16,384 and not a language model's.

The blank is dropped after the repeat collapse. The other order merges two real repeats that a blank was separating.

Time is subsampled inside the convolution module, at layers [0, 1]. The depthwise conv runs at stride 2 and the residual is mean-pooled over each pair, so the second feed-forward and the output norm of those layers already run at the shorter length. x = x[::2] after the block is a different function with the same output shape. Any per-frame alignment has to be rescaled by 4.

The middle of the tower reads its own CTC posteriors. After the eighth block the hidden state passes through out, a softmax and out_mid (16,384 → 1,024) and is added back. A port that skips it loads every tensor except out_mid and still transcribes.

Geometry

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• each of the 2 checkpoints of this shape: runs, shape-pack — stored checkpoints of this shape run; this one's own is not among them

The implementation meant here and below is an unpublished independent inference implementation by the author.

What stands behind the block above, beyond the published configuration it is read from:

• a conformer tower and projector implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Where an independent implementation and the published configuration disagree, the configuration is what this page reports and the disagreement is what it says.

Checkpoints with this architecture