Skip to content

Add single-dispatch layer-by-layer multi-head attention#91

Draft
andrej wants to merge 22 commits intoamd:develfrom
andrej:mha-lxl-sd
Draft

Add single-dispatch layer-by-layer multi-head attention#91
andrej wants to merge 22 commits intoamd:develfrom
andrej:mha-lxl-sd

Conversation

@andrej
Copy link
Copy Markdown
Collaborator

@andrej andrej commented Apr 6, 2026 •
edited
Loading

"Naive" alternative implementation for multi-head attention from the currently checked-in data-flow design. This is a simple layer-by-layer implementation, but it uses the single-dispatch mechanism to fuse it all into one MLIR file and save on CPU roundtrips and XRT overheads.

Includes two variants:

  1. "core": Only does the core matmuls and softmax; assumes projected and repeated inputs Q, K, V. This matches the functionality of the checked-in dataflow MHA.
  2. "projected": Performs the Q, K, V projections, applies a RoPE positional embedding and repeats K and V matrices for grouped-query attention. Takes an embedding vector and RoPE angles as input.

andrej
andrej commented Apr 6, 2026
View reviewed changes
Comment thread iron/operators/mha_prefill_lxl_sd/reference.py
Copy link
Copy Markdown
Collaborator Author

@andrej andrej Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reuse the reference from the existing mha? (Note: does not include RoPE and Q, K, V projections, but some code reuse should be possible.)

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 7, 2026

📊 Test Results for Test Example Applications

1d87fe8 (2026_04_07_21_05_39)

IRONCLAD

Tested on 2026_04_07_21_05_39 at commit 1d87fe8.

Test Checks TTFT (mean)TPS (mean)
llama_3.2_1b_prompt_1024_tokens_1 ✅ 5/5 2.13 n/a
llama_3.2_1b_prompt_1024_tokens_40 ✅ 5/5 2.18 4.31
llama_3.2_1b_prompt_13_tokens_1 ✅ 5/5 2.09 n/a
llama_3.2_1b_prompt_13_tokens_40 ✅ 5/5 2.09 4.31
📈 Trends (vs main branch) for Test Example Applications

1d87fe8 (2026_04_07_21_05_39)

IRONCLAD Trends

llama_3.2_1b

Commit/Date Num Tokens (max)Num Tokens (mean)Num Tokens (median)Num Tokens (min)Num Tokens (stddev)TPS (max)TPS (mean)TPS (median)TPS (min)TPS (stddev)TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)Total (max)Total (mean)Total (median)Total (min)Total (stddev)
130b6ea — 2025-12-05 21:33:1240.00 (+0.00%)40.00 (+0.00%)40.00 (+0.00%)40.00 (+0.00%)0.00 (n/a)4.71 (-0.42%)4.64 (-0.09%)4.64 (+0.65%)4.55 (-0.22%)0.05 (-17.66%)4.41 (-0.34%)4.39 (-0.19%)4.38 (-0.33%)4.37 (-0.15%)0.01 (-25.90%)12.96 (-0.00%)12.80 (+0.07%)12.80 (-0.23%)12.67 (+0.44%)0.09 (-21.12%)
0a6c11c — 2025-12-03 23:35:1540.00 (n/a)40.00 (n/a)40.00 (n/a)40.00 (n/a)0.00 (n/a)4.73 (n/a)4.64 (n/a)4.61 (n/a)4.56 (n/a)0.06 (n/a)4.42 (n/a)4.40 (n/a)4.40 (n/a)4.37 (n/a)0.02 (n/a)12.96 (n/a)12.79 (n/a)12.83 (n/a)12.62 (n/a)0.12 (n/a)

llama_3.2_1b_prompt_1024_tokens_1

Commit/Date TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)
1d87fe8 — 2026-04-07 21:00:002.15 (+0.09%)2.13 (+0.08%)2.13 (-0.42%)2.12 (+0.62%)0.01 (-31.21%)
912e6bc — 2026-04-07 19:08:432.15 (n/a)2.13 (n/a)2.13 (n/a)2.11 (n/a)0.02 (n/a)

llama_3.2_1b_prompt_1024_tokens_40

Commit/Date TPS (max)TPS (mean)TPS (median)TPS (min)TPS (stddev)TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)
1d87fe8 — 2026-04-07 21:00:004.33 (+2.90%)4.31 (+3.44%)4.31 (+3.58%)4.29 (+3.77%)0.01 (-46.93%)2.29 (+0.48%)2.18 (+0.83%)2.15 (+0.80%)2.13 (+0.61%)0.07 (-4.73%)
912e6bc — 2026-04-07 19:08:434.21 (n/a)4.17 (n/a)4.16 (n/a)4.14 (n/a)0.03 (n/a)2.28 (n/a)2.16 (n/a)2.13 (n/a)2.12 (n/a)0.07 (n/a)

llama_3.2_1b_prompt_13_tokens_1

Commit/Date TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)
1d87fe8 — 2026-04-07 21:00:002.10 (-0.10%)2.09 (+0.11%)2.09 (+0.19%)2.09 (+0.00%)0.01 (+8.87%)
912e6bc — 2026-04-07 19:08:432.10 (n/a)2.09 (n/a)2.09 (n/a)2.09 (n/a)0.01 (n/a)

llama_3.2_1b_prompt_13_tokens_40

Commit/Date TPS (max)TPS (mean)TPS (median)TPS (min)TPS (stddev)TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)
1d87fe8 — 2026-04-07 21:00:004.36 (+4.23%)4.31 (+3.57%)4.30 (+3.44%)4.29 (+3.23%)0.03 (+128.30%)2.09 (-0.38%)2.09 (-0.04%)2.09 (+0.00%)2.08 (+0.44%)0.01 (-34.93%)
912e6bc — 2026-04-07 19:08:434.18 (n/a)4.16 (n/a)4.16 (n/a)4.15 (n/a)0.01 (n/a)2.10 (n/a)2.09 (n/a)2.09 (n/a)2.07 (n/a)0.01 (n/a)

llama_3.2_1b_prompt_2048_tokens_1

Commit/Date Num_Tokens (max)Num_Tokens (mean)Num_Tokens (median)Num_Tokens (min)Num_Tokens (stddev)TPS (max)TPS (mean)TPS (median)TPS (min)TPS (stddev)TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)
897d04e — 2026-03-06 22:56:071.00 (+0.00%)1.00 (+0.00%)1.00 (+0.00%)1.00 (+0.00%)0.00 (n/a)0.00 (n/a)0.00 (n/a)0.00 (n/a)0.00 (n/a)0.00 (n/a)2.68 (-1.06%)2.68 (-1.06%)2.68 (-1.06%)2.68 (-1.06%)0.00 (n/a)
84d3478 — 2026-02-17 23:16:231.00 (n/a)1.00 (n/a)1.00 (n/a)1.00 (n/a)0.00 (n/a)0.00 (n/a)0.00 (n/a)0.00 (n/a)0.00 (n/a)0.00 (n/a)2.70 (n/a)2.70 (n/a)2.70 (n/a)2.70 (n/a)0.00 (n/a)

llama_3.2_1b_prompt_2048_tokens_40

Commit/Date Num_Tokens (max)Num_Tokens (mean)Num_Tokens (median)Num_Tokens (min)Num_Tokens (stddev)TPS (max)TPS (mean)TPS (median)TPS (min)TPS (stddev)TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)
897d04e — 2026-03-06 22:56:0740.00 (+0.00%)40.00 (+0.00%)40.00 (+0.00%)40.00 (+0.00%)0.00 (n/a)4.00 (-1.72%)4.00 (-1.72%)4.00 (-1.72%)4.00 (-1.72%)0.00 (n/a)2.70 (-0.44%)2.70 (-0.44%)2.70 (-0.44%)2.70 (-0.44%)0.00 (n/a)
84d3478 — 2026-02-17 23:16:2340.00 (n/a)40.00 (n/a)40.00 (n/a)40.00 (n/a)0.00 (n/a)4.07 (n/a)4.07 (n/a)4.07 (n/a)4.07 (n/a)0.00 (n/a)2.71 (n/a)2.71 (n/a)2.71 (n/a)2.71 (n/a)0.00 (n/a)

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 15, 2026 •
edited
Loading

CI Test Results

ea275b5 (2026_04_20_20_26_40)

IRONCLAD - CI Summary

Examples

Test Krackan Phoenix
llama_3.2_1b_prompt_1024_tokens_1 pass -
llama_3.2_1b_prompt_1024_tokens_40 pass -
llama_3.2_1b_prompt_13_tokens_1 pass -
llama_3.2_1b_prompt_13_tokens_40 pass -

Small

Test Krackan Phoenix
GPT2-Small-256seq no pass no pass
H2 no pass no pass
Llama3.2-256seq no pass no pass
M_128-K_128-num_aie_columns_1-tile_size_input_32-tile_size_output_128 pass pass
M_1792-K_896-N_1152-num_aie_columns_8-b_col_maj_False-c_col_maj_True-m_64-k_32-n_48-trace_size_0-partition_N_1 pass -
M_192-K_384-N_64-num_aie_columns_4-b_col_maj_False-c_col_maj_False-m_48-k_96-n_16-trace_size_0-partition_N_1 pass pass
M_192-K_384-N_64-num_aie_columns_4-b_col_maj_True-c_col_maj_True-m_48-k_96-n_16-trace_size_0-partition_N_1 pass pass
M_2048-K_2048-N_2048-num_aie_columns_1-b_col_maj_False-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1 pass pass
M_2048-K_2048-N_2048-num_aie_columns_2-b_col_maj_True-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1 pass pass
M_2048-K_2048-N_2048-num_aie_columns_8-b_col_maj_True-c_col_maj_True-m_64-k_64-n_64-trace_size_0-partition_N_1 pass -
M_2048-K_8192-num_aie_columns_1-tile_size_input_1-tile_size_output_2048 pass pass
M_2048-K_8192-num_aie_columns_2-tile_size_input_1-tile_size_output_1024 pass pass
M_2048-K_8192-num_aie_columns_4-tile_size_input_1-tile_size_output_512 pass pass
M_2048-K_8192-num_aie_columns_8-tile_size_input_1-tile_size_output_256 pass -
M_384-K_1536-N_1792-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_32-k_48-n_64-trace_size_0-partition_N_1 pass pass
M_64-K_512-N_256-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_16-k_64-n_64-trace_size_0-partition_N_4 pass pass
M_8192-K_2048-num_aie_columns_1-tile_size_input_4-tile_size_output_1024 pass pass
M_8192-K_2048-num_aie_columns_2-tile_size_input_4-tile_size_output_1024 pass pass
M_8192-K_2048-num_aie_columns_4-tile_size_input_4-tile_size_output_1024 pass pass
M_8192-K_2048-num_aie_columns_8-tile_size_input_4-tile_size_output_1024 pass -
M_896-K_1792-N_640-num_aie_columns_8-b_col_maj_False-c_col_maj_True-m_32-k_64-n_80-trace_size_0-partition_N_1 pass -
input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048 pass pass
input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-group_size_32 pass pass
input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024 pass pass
input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024-group_size_32 pass pass
input_length_2048-num_aie_columns_1-tile_size_2048 pass pass
input_length_2048-num_aie_columns_1-tile_size_2048-scalar_factor_3.0 pass pass
input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024 pass pass
input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024-group_size_32 pass pass
input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512 pass pass
input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512-group_size_32 pass pass
input_length_2048-num_aie_columns_2-tile_size_1024 pass pass
input_length_2048-num_aie_columns_2-tile_size_1024-scalar_factor_3.0 pass pass
input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512 pass pass
input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512-group_size_32 pass pass
input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256 pass pass
input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256-group_size_32 pass pass
input_length_2048-num_aie_columns_4-tile_size_512 pass pass
input_length_2048-num_aie_columns_4-tile_size_512-scalar_factor_3.0 pass pass
input_length_2048-num_aie_columns_8-num_channels_1-tile_size_256 pass -
input_length_2048-num_aie_columns_8-num_channels_1-tile_size_256-group_size_32 pass -
input_length_2048-num_aie_columns_8-num_channels_2-tile_size_128 pass -
input_length_2048-num_aie_columns_8-num_channels_2-tile_size_128-group_size_32 pass -
input_length_2048-num_aie_columns_8-tile_size_256 pass -
input_length_2048-num_aie_columns_8-tile_size_256-scalar_factor_3.0 pass -
input_length_2048-num_cores_1-num_channels_1-bypass_False-tile_size_2048 pass pass
input_length_2048-num_cores_16-num_channels_2-bypass_False-tile_size_128 pass -
input_length_2048-num_cores_2-num_channels_1-bypass_False-tile_size_1024 pass pass
input_length_2048-num_cores_2-num_channels_2-bypass_False-tile_size_1024 pass pass
input_length_2048-num_cores_4-num_channels_1-bypass_False-tile_size_512 pass pass
input_length_2048-num_cores_4-num_channels_2-bypass_False-tile_size_512 pass pass
input_length_2048-num_cores_8-num_channels_1-bypass_False-tile_size_256 pass -
input_length_2048-num_cores_8-num_channels_2-bypass_False-tile_size_256 pass pass
seq_len_16384-dim_64-num_heads_1-num_pipelines_8-num_kv_heads_0 pass -

Extensive

Test Krackan Phoenix
(no data) - -
Krackan - Small

IRONCLAD

Tested on 2026_04_20_20_26_40 at commit ea275b5.

Test Checks Latency (mean)Bandwidth (mean)Throughput (mean)
GPT2-Small-256seq ❌ 0/5 2480.58 n/a 81.68
H2 🟠 1/6 44626.98 n/a 14.76
Llama3.2-256seq 🟠 1/6 49892.22 n/a 96.69
M_128-K_128-num_aie_columns_1-tile_size_input_32-tile_size_output_128 ✅ 5/5 n/a 0.22 0.22
M_1792-K_896-N_1152-num_aie_columns_8-b_col_maj_False-c_col_maj_True-m_64-k_32-n_48-trace_size_0-partition_N_1 ✅ 5/5 2108.90 4.49 1765.05
M_192-K_384-N_64-num_aie_columns_4-b_col_maj_False-c_col_maj_False-m_48-k_96-n_16-trace_size_0-partition_N_1 ✅ 5/5 229.92 1.02 43.32
M_192-K_384-N_64-num_aie_columns_4-b_col_maj_True-c_col_maj_True-m_48-k_96-n_16-trace_size_0-partition_N_1 ✅ 5/5 277.40 0.86 36.65
M_2048-K_2048-N_2048-num_aie_columns_1-b_col_maj_False-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1 ✅ 5/5 49180.30 0.51 349.33
M_2048-K_2048-N_2048-num_aie_columns_2-b_col_maj_True-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1 ✅ 5/5 28612.38 0.88 600.46
M_2048-K_2048-N_2048-num_aie_columns_8-b_col_maj_True-c_col_maj_True-m_64-k_64-n_64-trace_size_0-partition_N_1 ✅ 5/5 7288.82 3.46 2359.56
M_2048-K_8192-num_aie_columns_1-tile_size_input_1-tile_size_output_2048 ✅ 5/5 n/a 12.92 12.92
M_2048-K_8192-num_aie_columns_2-tile_size_input_1-tile_size_output_1024 ✅ 5/5 n/a 24.45 24.43
M_2048-K_8192-num_aie_columns_4-tile_size_input_1-tile_size_output_512 ✅ 5/5 n/a 40.38 40.36
M_2048-K_8192-num_aie_columns_8-tile_size_input_1-tile_size_output_256 ✅ 5/5 n/a 41.92 41.90
M_384-K_1536-N_1792-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_32-k_48-n_64-trace_size_0-partition_N_1 ✅ 5/5 2375.38 3.46 907.80
M_64-K_512-N_256-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_16-k_64-n_64-trace_size_0-partition_N_4 ✅ 5/5 3353.26 0.38 20.54
M_8192-K_2048-num_aie_columns_1-tile_size_input_4-tile_size_output_1024 ✅ 5/5 n/a 12.64 12.63
M_8192-K_2048-num_aie_columns_2-tile_size_input_4-tile_size_output_1024 ✅ 5/5 n/a 24.40 24.38
M_8192-K_2048-num_aie_columns_4-tile_size_input_4-tile_size_output_1024 ✅ 5/5 n/a 38.39 38.36
M_8192-K_2048-num_aie_columns_8-tile_size_input_4-tile_size_output_1024 ✅ 5/5 n/a 42.24 42.21
M_896-K_1792-N_640-num_aie_columns_8-b_col_maj_False-c_col_maj_True-m_32-k_64-n_80-trace_size_0-partition_N_1 ✅ 5/5 1533.50 4.40 1360.36
input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048 ✅ 10/10 159.88 0.05 n/a
input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-group_size_32 ✅ 5/5 173.40 0.03 n/a
input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024 ✅ 10/10 164.44 0.05 n/a
input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024-group_size_32 ✅ 5/5 177.28 0.03 n/a
input_length_2048-num_aie_columns_1-tile_size_2048 ✅ 10/10 185.17 0.07 n/a
input_length_2048-num_aie_columns_1-tile_size_2048-scalar_factor_3.0 ✅ 5/5 166.74 0.08 n/a
input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024 ✅ 10/10 162.03 0.05 n/a
input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024-group_size_32 ✅ 5/5 177.96 0.03 n/a
input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512 ✅ 10/10 167.47 0.05 n/a
input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512-group_size_32 ✅ 5/5 168.62 0.03 n/a
input_length_2048-num_aie_columns_2-tile_size_1024 ✅ 10/10 184.63 0.07 n/a
input_length_2048-num_aie_columns_2-tile_size_1024-scalar_factor_3.0 ✅ 5/5 154.96 0.08 n/a
input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512 ✅ 10/10 188.16 0.05 n/a
input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512-group_size_32 ✅ 5/5 178.16 0.03 n/a
input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256 ✅ 10/10 184.86 0.05 n/a
input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256-group_size_32 ✅ 5/5 191.26 0.03 n/a
input_length_2048-num_aie_columns_4-tile_size_512 ✅ 10/10 182.83 0.07 n/a
input_length_2048-num_aie_columns_4-tile_size_512-scalar_factor_3.0 ✅ 5/5 194.28 0.07 n/a
input_length_2048-num_aie_columns_8-num_channels_1-tile_size_256 ✅ 10/10 189.16 0.05 n/a
input_length_2048-num_aie_columns_8-num_channels_1-tile_size_256-group_size_32 ✅ 5/5 187.34 0.03 n/a
input_length_2048-num_aie_columns_8-num_channels_2-tile_size_128 ✅ 10/10 210.58 0.04 n/a
input_length_2048-num_aie_columns_8-num_channels_2-tile_size_128-group_size_32 ✅ 5/5 227.62 0.02 n/a
input_length_2048-num_aie_columns_8-tile_size_256 ✅ 10/10 221.76 0.06 n/a
input_length_2048-num_aie_columns_8-tile_size_256-scalar_factor_3.0 ✅ 5/5 238.18 0.06 n/a
input_length_2048-num_cores_1-num_channels_1-bypass_False-tile_size_2048 ✅ 5/5 133.84 0.06 n/a
input_length_2048-num_cores_16-num_channels_2-bypass_False-tile_size_128 ✅ 5/5 199.66 0.04 n/a
input_length_2048-num_cores_2-num_channels_1-bypass_False-tile_size_1024 ✅ 5/5 159.96 0.05 n/a
input_length_2048-num_cores_2-num_channels_2-bypass_False-tile_size_1024 ✅ 5/5 148.44 0.06 n/a
input_length_2048-num_cores_4-num_channels_1-bypass_False-tile_size_512 ✅ 5/5 176.88 0.05 n/a
input_length_2048-num_cores_4-num_channels_2-bypass_False-tile_size_512 ✅ 5/5 157.28 0.05 n/a
input_length_2048-num_cores_8-num_channels_1-bypass_False-tile_size_256 ✅ 5/5 196.48 0.04 n/a
input_length_2048-num_cores_8-num_channels_2-bypass_False-tile_size_256 ✅ 5/5 165.44 0.05 n/a
seq_len_16384-dim_64-num_heads_1-num_pipelines_8-num_kv_heads_0 ✅ 5/5 40623.88 0.21 n/a

Trends:

IRONCLAD Trends

GPT2-Small-256seq

Commit/Date Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:362862.90 (n/a)2480.58 (n/a)2435.60 (n/a)2254.90 (n/a)229.08 (n/a)89.28 (n/a)81.68 (n/a)82.66 (n/a)70.32 (n/a)7.02 (n/a)

H2

Commit/Date Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:36140317.70 (n/a)44626.98 (n/a)1592.80 (n/a)1414.50 (n/a)67026.10 (n/a)23.72 (n/a)14.76 (n/a)21.07 (n/a)0.28 (n/a)11.10 (n/a)

Llama3.2-256seq

Commit/Date Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:36142449.00 (n/a)49892.22 (n/a)4072.35 (n/a)3917.90 (n/a)71072.63 (n/a)137.03 (n/a)96.69 (n/a)131.84 (n/a)3.81 (n/a)58.62 (n/a)

M_128-K_128-num_aie_columns_1-tile_size_input_32-tile_size_output_128

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:360.29 (n/a)0.22 (n/a)0.21 (n/a)0.16 (n/a)0.06 (n/a)0.28 (n/a)0.22 (n/a)0.21 (n/a)0.16 (n/a)0.06 (n/a)

M_1792-K_896-N_1152-num_aie_columns_8-b_col_maj_False-c_col_maj_True-m_64-k_32-n_48-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:364.98 (n/a)4.49 (n/a)4.45 (n/a)4.02 (n/a)0.39 (n/a)2341.20 (n/a)2108.90 (n/a)2115.00 (n/a)1888.20 (n/a)185.11 (n/a)1959.18 (n/a)1765.05 (n/a)1749.11 (n/a)1580.14 (n/a)155.24 (n/a)

M_192-K_384-N_64-num_aie_columns_4-b_col_maj_False-c_col_maj_False-m_48-k_96-n_16-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:361.34 (n/a)1.02 (n/a)1.07 (n/a)0.67 (n/a)0.25 (n/a)329.70 (n/a)229.92 (n/a)207.60 (n/a)165.20 (n/a)62.84 (n/a)57.12 (n/a)43.32 (n/a)45.46 (n/a)28.63 (n/a)10.62 (n/a)

M_192-K_384-N_64-num_aie_columns_4-b_col_maj_True-c_col_maj_True-m_48-k_96-n_16-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:361.09 (n/a)0.86 (n/a)0.98 (n/a)0.59 (n/a)0.25 (n/a)376.20 (n/a)277.40 (n/a)226.20 (n/a)203.30 (n/a)87.37 (n/a)46.41 (n/a)36.65 (n/a)41.72 (n/a)25.08 (n/a)10.47 (n/a)

M_2048-K_2048-N_2048-num_aie_columns_1-b_col_maj_False-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:360.51 (n/a)0.51 (n/a)0.51 (n/a)0.51 (n/a)0.00 (n/a)49329.20 (n/a)49180.30 (n/a)49211.10 (n/a)48888.00 (n/a)170.75 (n/a)351.41 (n/a)349.33 (n/a)349.11 (n/a)348.27 (n/a)1.22 (n/a)

M_2048-K_2048-N_2048-num_aie_columns_2-b_col_maj_True-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:360.89 (n/a)0.88 (n/a)0.88 (n/a)0.87 (n/a)0.01 (n/a)28786.00 (n/a)28612.38 (n/a)28707.40 (n/a)28336.20 (n/a)201.33 (n/a)606.29 (n/a)600.46 (n/a)598.45 (n/a)596.81 (n/a)4.24 (n/a)

M_2048-K_2048-N_2048-num_aie_columns_8-b_col_maj_True-c_col_maj_True-m_64-k_64-n_64-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:363.63 (n/a)3.46 (n/a)3.47 (n/a)3.28 (n/a)0.13 (n/a)7678.70 (n/a)7288.82 (n/a)7244.60 (n/a)6938.90 (n/a)268.52 (n/a)2475.89 (n/a)2359.56 (n/a)2371.41 (n/a)2237.35 (n/a)86.43 (n/a)

M_2048-K_8192-num_aie_columns_1-tile_size_input_1-tile_size_output_2048

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:3613.54 (n/a)12.92 (n/a)12.67 (n/a)12.43 (n/a)0.52 (n/a)13.53 (n/a)12.92 (n/a)12.66 (n/a)12.42 (n/a)0.52 (n/a)

M_2048-K_8192-num_aie_columns_2-tile_size_input_1-tile_size_output_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:3625.13 (n/a)24.45 (n/a)24.29 (n/a)24.01 (n/a)0.44 (n/a)25.12 (n/a)24.43 (n/a)24.27 (n/a)24.00 (n/a)0.44 (n/a)

M_2048-K_8192-num_aie_columns_4-tile_size_input_1-tile_size_output_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:3641.39 (n/a)40.38 (n/a)40.08 (n/a)39.95 (n/a)0.60 (n/a)41.36 (n/a)40.36 (n/a)40.05 (n/a)39.93 (n/a)0.60 (n/a)

M_2048-K_8192-num_aie_columns_8-tile_size_input_1-tile_size_output_256

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:3643.59 (n/a)41.92 (n/a)41.74 (n/a)40.61 (n/a)1.33 (n/a)43.56 (n/a)41.90 (n/a)41.71 (n/a)40.59 (n/a)1.33 (n/a)

M_384-K_1536-N_1792-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_32-k_48-n_64-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:364.17 (n/a)3.46 (n/a)3.57 (n/a)2.74 (n/a)0.53 (n/a)2945.70 (n/a)2375.38 (n/a)2256.10 (n/a)1933.20 (n/a)381.98 (n/a)1093.47 (n/a)907.80 (n/a)936.98 (n/a)717.64 (n/a)140.29 (n/a)

M_64-K_512-N_256-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_16-k_64-n_64-trace_size_0-partition_N_4

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:360.50 (n/a)0.38 (n/a)0.36 (n/a)0.32 (n/a)0.07 (n/a)3843.00 (n/a)3353.26 (n/a)3492.60 (n/a)2505.90 (n/a)567.57 (n/a)26.78 (n/a)20.54 (n/a)19.21 (n/a)17.46 (n/a)3.91 (n/a)

M_8192-K_2048-num_aie_columns_1-tile_size_input_4-tile_size_output_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:3613.48 (n/a)12.64 (n/a)13.13 (n/a)10.78 (n/a)1.09 (n/a)13.47 (n/a)12.63 (n/a)13.12 (n/a)10.78 (n/a)1.09 (n/a)

M_8192-K_2048-num_aie_columns_2-tile_size_input_4-tile_size_output_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:3624.83 (n/a)24.40 (n/a)24.28 (n/a)23.80 (n/a)0.43 (n/a)24.82 (n/a)24.38 (n/a)24.27 (n/a)23.79 (n/a)0.43 (n/a)

M_8192-K_2048-num_aie_columns_4-tile_size_input_4-tile_size_output_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:3641.19 (n/a)38.39 (n/a)38.47 (n/a)34.19 (n/a)2.65 (n/a)41.17 (n/a)38.36 (n/a)38.45 (n/a)34.17 (n/a)2.64 (n/a)

M_8192-K_2048-num_aie_columns_8-tile_size_input_4-tile_size_output_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:3645.74 (n/a)42.24 (n/a)44.05 (n/a)35.31 (n/a)4.29 (n/a)45.71 (n/a)42.21 (n/a)44.02 (n/a)35.29 (n/a)4.29 (n/a)

M_896-K_1792-N_640-num_aie_columns_8-b_col_maj_False-c_col_maj_True-m_32-k_64-n_80-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:20:365.10 (n/a)4.40 (n/a)4.60 (n/a)3.62 (n/a)0.59 (n/a)1836.50 (n/a)1533.50 (n/a)1445.80 (n/a)1305.40 (n/a)213.69 (n/a)1574.37 (n/a)1360.36 (n/a)1421.50 (n/a)1119.09 (n/a)181.62 (n/a)

input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.06 (n/a)0.05 (n/a)0.05 (n/a)0.04 (n/a)0.01 (n/a)210.20 (n/a)159.88 (n/a)161.20 (n/a)128.20 (n/a)24.48 (n/a)

input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.03 (n/a)0.03 (n/a)0.03 (n/a)0.02 (n/a)0.00 (n/a)214.90 (n/a)173.40 (n/a)161.60 (n/a)158.10 (n/a)24.07 (n/a)

input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.08 (n/a)0.05 (n/a)0.05 (n/a)0.04 (n/a)0.01 (n/a)228.30 (n/a)164.44 (n/a)164.50 (n/a)101.40 (n/a)40.39 (n/a)

input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.03 (n/a)0.03 (n/a)0.03 (n/a)0.03 (n/a)0.00 (n/a)203.40 (n/a)177.28 (n/a)180.70 (n/a)155.90 (n/a)18.32 (n/a)

input_length_2048-num_aie_columns_1-tile_size_2048

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.08 (n/a)0.07 (n/a)0.07 (n/a)0.05 (n/a)0.01 (n/a)238.10 (n/a)185.17 (n/a)179.35 (n/a)152.10 (n/a)24.72 (n/a)

input_length_2048-num_aie_columns_1-tile_size_2048-scalar_factor_3.0

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.09 (n/a)0.08 (n/a)0.08 (n/a)0.06 (n/a)0.01 (n/a)203.50 (n/a)166.74 (n/a)160.10 (n/a)129.40 (n/a)29.08 (n/a)

input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.07 (n/a)0.05 (n/a)0.05 (n/a)0.04 (n/a)0.01 (n/a)203.70 (n/a)162.03 (n/a)161.60 (n/a)122.90 (n/a)24.23 (n/a)

input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.05 (n/a)0.03 (n/a)0.03 (n/a)0.02 (n/a)0.01 (n/a)228.60 (n/a)177.96 (n/a)190.40 (n/a)113.40 (n/a)44.83 (n/a)

input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.07 (n/a)0.05 (n/a)0.05 (n/a)0.04 (n/a)0.01 (n/a)197.50 (n/a)167.47 (n/a)168.25 (n/a)125.50 (n/a)20.73 (n/a)

input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.04 (n/a)0.03 (n/a)0.03 (n/a)0.02 (n/a)0.01 (n/a)211.00 (n/a)168.62 (n/a)169.70 (n/a)117.70 (n/a)33.34 (n/a)

input_length_2048-num_aie_columns_2-tile_size_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.09 (n/a)0.07 (n/a)0.06 (n/a)0.06 (n/a)0.01 (n/a)213.00 (n/a)184.63 (n/a)189.70 (n/a)132.00 (n/a)26.38 (n/a)

input_length_2048-num_aie_columns_2-tile_size_1024-scalar_factor_3.0

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.11 (n/a)0.08 (n/a)0.09 (n/a)0.06 (n/a)0.02 (n/a)212.40 (n/a)154.96 (n/a)143.10 (n/a)116.10 (n/a)38.42 (n/a)

input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.07 (n/a)0.05 (n/a)0.05 (n/a)0.02 (n/a)0.01 (n/a)358.50 (n/a)188.16 (n/a)178.90 (n/a)124.10 (n/a)66.71 (n/a)

input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.04 (n/a)0.03 (n/a)0.03 (n/a)0.03 (n/a)0.01 (n/a)208.60 (n/a)178.16 (n/a)181.20 (n/a)130.40 (n/a)28.99 (n/a)

input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.07 (n/a)0.05 (n/a)0.04 (n/a)0.04 (n/a)0.01 (n/a)219.60 (n/a)184.86 (n/a)198.05 (n/a)123.00 (n/a)31.13 (n/a)

input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.03 (n/a)0.03 (n/a)0.03 (n/a)0.02 (n/a)0.00 (n/a)246.60 (n/a)191.26 (n/a)188.90 (n/a)153.10 (n/a)35.17 (n/a)

input_length_2048-num_aie_columns_4-tile_size_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.10 (n/a)0.07 (n/a)0.07 (n/a)0.05 (n/a)0.01 (n/a)236.40 (n/a)182.83 (n/a)185.20 (n/a)127.40 (n/a)35.23 (n/a)

input_length_2048-num_aie_columns_4-tile_size_512-scalar_factor_3.0

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.08 (n/a)0.07 (n/a)0.06 (n/a)0.05 (n/a)0.01 (n/a)255.80 (n/a)194.28 (n/a)191.40 (n/a)147.00 (n/a)41.73 (n/a)

input_length_2048-num_aie_columns_8-num_channels_1-tile_size_256

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.06 (n/a)0.05 (n/a)0.04 (n/a)0.03 (n/a)0.01 (n/a)258.10 (n/a)189.16 (n/a)186.25 (n/a)126.50 (n/a)42.61 (n/a)

input_length_2048-num_aie_columns_8-num_channels_1-tile_size_256-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.03 (n/a)0.03 (n/a)0.03 (n/a)0.03 (n/a)0.00 (n/a)198.50 (n/a)187.34 (n/a)194.10 (n/a)165.10 (n/a)13.89 (n/a)

input_length_2048-num_aie_columns_8-num_channels_2-tile_size_128

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.06 (n/a)0.04 (n/a)0.04 (n/a)0.03 (n/a)0.01 (n/a)287.80 (n/a)210.58 (n/a)216.20 (n/a)144.10 (n/a)40.72 (n/a)

input_length_2048-num_aie_columns_8-num_channels_2-tile_size_128-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.03 (n/a)0.02 (n/a)0.02 (n/a)0.02 (n/a)0.01 (n/a)285.40 (n/a)227.62 (n/a)225.20 (n/a)157.00 (n/a)46.56 (n/a)

input_length_2048-num_aie_columns_8-tile_size_256

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.08 (n/a)0.06 (n/a)0.06 (n/a)0.03 (n/a)0.01 (n/a)354.80 (n/a)221.76 (n/a)213.50 (n/a)149.30 (n/a)51.82 (n/a)

input_length_2048-num_aie_columns_8-tile_size_256-scalar_factor_3.0

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.08 (n/a)0.06 (n/a)0.05 (n/a)0.04 (n/a)0.02 (n/a)321.20 (n/a)238.18 (n/a)235.60 (n/a)144.90 (n/a)78.78 (n/a)

input_length_2048-num_cores_1-num_channels_1-bypass_False-tile_size_2048

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.07 (n/a)0.06 (n/a)0.06 (n/a)0.06 (n/a)0.01 (n/a)144.30 (n/a)133.84 (n/a)135.90 (n/a)114.60 (n/a)11.40 (n/a)

input_length_2048-num_cores_16-num_channels_2-bypass_False-tile_size_128

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.05 (n/a)0.04 (n/a)0.04 (n/a)0.04 (n/a)0.01 (n/a)219.60 (n/a)199.66 (n/a)206.90 (n/a)161.10 (n/a)23.58 (n/a)

input_length_2048-num_cores_2-num_channels_1-bypass_False-tile_size_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.07 (n/a)0.05 (n/a)0.05 (n/a)0.04 (n/a)0.01 (n/a)211.40 (n/a)159.96 (n/a)150.40 (n/a)124.70 (n/a)33.88 (n/a)

input_length_2048-num_cores_2-num_channels_2-bypass_False-tile_size_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.07 (n/a)0.06 (n/a)0.06 (n/a)0.04 (n/a)0.01 (n/a)212.10 (n/a)148.44 (n/a)144.70 (n/a)110.80 (n/a)38.63 (n/a)

input_length_2048-num_cores_4-num_channels_1-bypass_False-tile_size_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.06 (n/a)0.05 (n/a)0.05 (n/a)0.04 (n/a)0.01 (n/a)215.20 (n/a)176.88 (n/a)179.40 (n/a)146.40 (n/a)27.25 (n/a)

input_length_2048-num_cores_4-num_channels_2-bypass_False-tile_size_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.07 (n/a)0.05 (n/a)0.06 (n/a)0.04 (n/a)0.01 (n/a)212.80 (n/a)157.28 (n/a)143.60 (n/a)114.70 (n/a)39.05 (n/a)

input_length_2048-num_cores_8-num_channels_1-bypass_False-tile_size_256

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.05 (n/a)0.04 (n/a)0.05 (n/a)0.03 (n/a)0.01 (n/a)261.60 (n/a)196.48 (n/a)167.20 (n/a)149.60 (n/a)52.37 (n/a)

input_length_2048-num_cores_8-num_channels_2-bypass_False-tile_size_256

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.07 (n/a)0.05 (n/a)0.05 (n/a)0.04 (n/a)0.01 (n/a)190.90 (n/a)165.44 (n/a)176.80 (n/a)125.40 (n/a)29.44 (n/a)

seq_len_16384-dim_64-num_heads_1-num_pipelines_8-num_kv_heads_0

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:20:360.21 (n/a)0.21 (n/a)0.21 (n/a)0.21 (n/a)0.00 (n/a)40652.80 (n/a)40623.88 (n/a)40633.30 (n/a)40571.30 (n/a)32.59 (n/a)
Krackan - Examples

IRONCLAD

Tested on 2026_04_20_20_36_21 at commit ea275b5.

Test Checks TTFT (mean)TPS (mean)
llama_3.2_1b_prompt_1024_tokens_1 ✅ 5/5 2.14 n/a
llama_3.2_1b_prompt_1024_tokens_40 ✅ 5/5 2.16 4.30
llama_3.2_1b_prompt_13_tokens_1 ✅ 5/5 2.09 n/a
llama_3.2_1b_prompt_13_tokens_40 ✅ 5/5 2.09 4.29

Trends:

IRONCLAD Trends

llama_3.2_1b_prompt_1024_tokens_1

Commit/Date TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)
ea275b5 — 2026-04-20 20:30:422.16 (n/a)2.14 (n/a)2.13 (n/a)2.12 (n/a)0.02 (n/a)

llama_3.2_1b_prompt_1024_tokens_40

Commit/Date TPS (max)TPS (mean)TPS (median)TPS (min)TPS (stddev)TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)
ea275b5 — 2026-04-20 20:30:424.32 (n/a)4.30 (n/a)4.30 (n/a)4.27 (n/a)0.02 (n/a)2.26 (n/a)2.16 (n/a)2.14 (n/a)2.12 (n/a)0.06 (n/a)

llama_3.2_1b_prompt_13_tokens_1

Commit/Date TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)
ea275b5 — 2026-04-20 20:30:422.10 (n/a)2.09 (n/a)2.09 (n/a)2.07 (n/a)0.01 (n/a)

llama_3.2_1b_prompt_13_tokens_40

Commit/Date TPS (max)TPS (mean)TPS (median)TPS (min)TPS (stddev)TTFT (max)TTFT (mean)TTFT (median)TTFT (min)TTFT (stddev)
ea275b5 — 2026-04-20 20:30:424.31 (n/a)4.29 (n/a)4.30 (n/a)4.28 (n/a)0.01 (n/a)2.10 (n/a)2.09 (n/a)2.09 (n/a)2.09 (n/a)0.00 (n/a)
Phoenix - Small

IRONCLAD

Tested on 2026_04_20_20_28_27 at commit ea275b5.

Test Checks Latency (mean)Bandwidth (mean)Throughput (mean)
GPT2-Small-256seq ❌ 0/5 28547.52 n/a 8.63
H2 🟠 1/6 60467.83 n/a 2.27
Llama3.2-256seq ❌ 0/5 47582.68 n/a 12.26
M_128-K_128-num_aie_columns_1-tile_size_input_32-tile_size_output_128 ✅ 5/5 n/a 0.11 0.11
M_192-K_384-N_64-num_aie_columns_4-b_col_maj_False-c_col_maj_False-m_48-k_96-n_16-trace_size_0-partition_N_1 ✅ 5/5 546.42 0.41 17.57
M_192-K_384-N_64-num_aie_columns_4-b_col_maj_True-c_col_maj_True-m_48-k_96-n_16-trace_size_0-partition_N_1 ✅ 5/5 494.96 0.48 20.34
M_2048-K_2048-N_2048-num_aie_columns_1-b_col_maj_False-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1 ✅ 5/5 83580.58 0.30 205.69
M_2048-K_2048-N_2048-num_aie_columns_2-b_col_maj_True-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1 ✅ 5/5 24300.06 1.04 707.34
M_2048-K_8192-num_aie_columns_1-tile_size_input_1-tile_size_output_2048 ✅ 5/5 n/a 3.61 3.61
M_2048-K_8192-num_aie_columns_2-tile_size_input_1-tile_size_output_1024 ✅ 5/5 n/a 6.59 6.58
M_2048-K_8192-num_aie_columns_4-tile_size_input_1-tile_size_output_512 ✅ 5/5 n/a 11.16 11.16
M_384-K_1536-N_1792-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_32-k_48-n_64-trace_size_0-partition_N_1 ✅ 5/5 3749.98 2.54 666.02
M_64-K_512-N_256-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_16-k_64-n_64-trace_size_0-partition_N_4 ✅ 5/5 5639.10 0.23 12.41
M_8192-K_2048-num_aie_columns_1-tile_size_input_4-tile_size_output_1024 ✅ 5/5 n/a 3.78 3.77
M_8192-K_2048-num_aie_columns_2-tile_size_input_4-tile_size_output_1024 ✅ 5/5 n/a 6.89 6.89
M_8192-K_2048-num_aie_columns_4-tile_size_input_4-tile_size_output_1024 ✅ 5/5 n/a 11.81 11.81
input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048 ✅ 10/10 395.15 0.02 n/a
input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-group_size_32 ✅ 5/5 394.80 0.02 n/a
input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024 ✅ 10/10 384.01 0.02 n/a
input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024-group_size_32 ✅ 5/5 734.54 0.01 n/a
input_length_2048-num_aie_columns_1-tile_size_2048 ✅ 10/10 308.88 0.04 n/a
input_length_2048-num_aie_columns_1-tile_size_2048-scalar_factor_3.0 ✅ 5/5 393.30 0.03 n/a
input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024 ✅ 10/10 361.81 0.03 n/a
input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024-group_size_32 ✅ 5/5 395.28 0.01 n/a
input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512 ✅ 10/10 408.60 0.02 n/a
input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512-group_size_32 ✅ 5/5 464.36 0.01 n/a
input_length_2048-num_aie_columns_2-tile_size_1024 ✅ 10/10 414.00 0.04 n/a
input_length_2048-num_aie_columns_2-tile_size_1024-scalar_factor_3.0 ✅ 5/5 481.36 0.03 n/a
input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512 ✅ 10/10 442.49 0.02 n/a
input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512-group_size_32 ✅ 5/5 325.44 0.02 n/a
input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256 ✅ 10/10 447.32 0.02 n/a
input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256-group_size_32 ✅ 5/5 465.94 0.01 n/a
input_length_2048-num_aie_columns_4-tile_size_512 ✅ 10/10 518.35 0.03 n/a
input_length_2048-num_aie_columns_4-tile_size_512-scalar_factor_3.0 ✅ 5/5 466.28 0.03 n/a
input_length_2048-num_cores_1-num_channels_1-bypass_False-tile_size_2048 ✅ 5/5 648.24 0.02 n/a
input_length_2048-num_cores_2-num_channels_1-bypass_False-tile_size_1024 ✅ 5/5 464.38 0.02 n/a
input_length_2048-num_cores_2-num_channels_2-bypass_False-tile_size_1024 ✅ 5/5 368.64 0.02 n/a
input_length_2048-num_cores_4-num_channels_1-bypass_False-tile_size_512 ✅ 5/5 516.20 0.02 n/a
input_length_2048-num_cores_4-num_channels_2-bypass_False-tile_size_512 ✅ 5/5 514.50 0.02 n/a
input_length_2048-num_cores_8-num_channels_2-bypass_False-tile_size_256 ✅ 5/5 489.54 0.02 n/a

Trends:

IRONCLAD Trends

GPT2-Small-256seq

Commit/Date Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:5961466.00 (n/a)28547.52 (n/a)20008.00 (n/a)18529.80 (n/a)18477.14 (n/a)10.87 (n/a)8.63 (n/a)10.06 (n/a)3.28 (n/a)3.09 (n/a)

H2

Commit/Date Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:59185032.60 (n/a)60467.83 (n/a)11270.15 (n/a)9522.80 (n/a)78918.16 (n/a)3.52 (n/a)2.27 (n/a)2.99 (n/a)0.25 (n/a)1.48 (n/a)

Llama3.2-256seq

Commit/Date Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:5979489.30 (n/a)47582.68 (n/a)38823.50 (n/a)36444.50 (n/a)18096.84 (n/a)14.73 (n/a)12.26 (n/a)13.83 (n/a)6.75 (n/a)3.23 (n/a)

M_128-K_128-num_aie_columns_1-tile_size_input_32-tile_size_output_128

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:590.14 (n/a)0.11 (n/a)0.11 (n/a)0.10 (n/a)0.02 (n/a)0.14 (n/a)0.11 (n/a)0.11 (n/a)0.10 (n/a)0.02 (n/a)

M_192-K_384-N_64-num_aie_columns_4-b_col_maj_False-c_col_maj_False-m_48-k_96-n_16-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:590.52 (n/a)0.41 (n/a)0.39 (n/a)0.36 (n/a)0.06 (n/a)616.90 (n/a)546.42 (n/a)570.10 (n/a)425.50 (n/a)74.82 (n/a)22.18 (n/a)17.57 (n/a)16.55 (n/a)15.30 (n/a)2.74 (n/a)

M_192-K_384-N_64-num_aie_columns_4-b_col_maj_True-c_col_maj_True-m_48-k_96-n_16-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:590.63 (n/a)0.48 (n/a)0.44 (n/a)0.34 (n/a)0.14 (n/a)654.80 (n/a)494.96 (n/a)501.60 (n/a)350.80 (n/a)137.49 (n/a)26.90 (n/a)20.34 (n/a)18.81 (n/a)14.41 (n/a)5.80 (n/a)

M_2048-K_2048-N_2048-num_aie_columns_1-b_col_maj_False-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:590.31 (n/a)0.30 (n/a)0.30 (n/a)0.29 (n/a)0.01 (n/a)87826.10 (n/a)83580.58 (n/a)82968.50 (n/a)81279.30 (n/a)2508.12 (n/a)211.37 (n/a)205.69 (n/a)207.06 (n/a)195.61 (n/a)6.00 (n/a)

M_2048-K_2048-N_2048-num_aie_columns_2-b_col_maj_True-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:591.06 (n/a)1.04 (n/a)1.05 (n/a)0.99 (n/a)0.03 (n/a)25313.20 (n/a)24300.06 (n/a)24062.10 (n/a)23782.10 (n/a)616.06 (n/a)722.39 (n/a)707.34 (n/a)713.98 (n/a)678.69 (n/a)17.54 (n/a)

M_2048-K_8192-num_aie_columns_1-tile_size_input_1-tile_size_output_2048

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:593.85 (n/a)3.61 (n/a)3.69 (n/a)3.34 (n/a)0.22 (n/a)3.85 (n/a)3.61 (n/a)3.68 (n/a)3.33 (n/a)0.22 (n/a)

M_2048-K_8192-num_aie_columns_2-tile_size_input_1-tile_size_output_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:597.41 (n/a)6.59 (n/a)6.90 (n/a)5.27 (n/a)0.87 (n/a)7.41 (n/a)6.58 (n/a)6.90 (n/a)5.27 (n/a)0.87 (n/a)

M_2048-K_8192-num_aie_columns_4-tile_size_input_1-tile_size_output_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:5913.74 (n/a)11.16 (n/a)12.22 (n/a)7.09 (n/a)2.93 (n/a)13.73 (n/a)11.16 (n/a)12.22 (n/a)7.08 (n/a)2.93 (n/a)

M_384-K_1536-N_1792-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_32-k_48-n_64-trace_size_0-partition_N_1

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:593.85 (n/a)2.54 (n/a)2.18 (n/a)1.36 (n/a)1.14 (n/a)5934.60 (n/a)3749.98 (n/a)3701.00 (n/a)2091.90 (n/a)1656.01 (n/a)1010.53 (n/a)666.02 (n/a)571.18 (n/a)356.21 (n/a)298.50 (n/a)

M_64-K_512-N_256-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_16-k_64-n_64-trace_size_0-partition_N_4

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:590.30 (n/a)0.23 (n/a)0.20 (n/a)0.18 (n/a)0.05 (n/a)6781.40 (n/a)5639.10 (n/a)6361.20 (n/a)4130.40 (n/a)1230.35 (n/a)16.25 (n/a)12.41 (n/a)10.55 (n/a)9.90 (n/a)2.95 (n/a)

M_8192-K_2048-num_aie_columns_1-tile_size_input_4-tile_size_output_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:593.89 (n/a)3.78 (n/a)3.76 (n/a)3.66 (n/a)0.10 (n/a)3.89 (n/a)3.77 (n/a)3.75 (n/a)3.66 (n/a)0.10 (n/a)

M_8192-K_2048-num_aie_columns_2-tile_size_input_4-tile_size_output_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:597.51 (n/a)6.89 (n/a)7.45 (n/a)5.71 (n/a)0.84 (n/a)7.51 (n/a)6.89 (n/a)7.44 (n/a)5.71 (n/a)0.83 (n/a)

M_8192-K_2048-num_aie_columns_4-tile_size_input_4-tile_size_output_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Throughput (max)Throughput (mean)Throughput (median)Throughput (min)Throughput (stddev)
ea275b5 — 2026-04-20 20:24:5913.58 (n/a)11.81 (n/a)12.54 (n/a)9.70 (n/a)1.90 (n/a)13.57 (n/a)11.81 (n/a)12.53 (n/a)9.70 (n/a)1.90 (n/a)

input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.03 (n/a)0.02 (n/a)0.02 (n/a)0.01 (n/a)0.01 (n/a)564.50 (n/a)395.15 (n/a)408.05 (n/a)240.00 (n/a)115.28 (n/a)

input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.02 (n/a)0.02 (n/a)0.02 (n/a)0.01 (n/a)0.01 (n/a)625.30 (n/a)394.80 (n/a)296.70 (n/a)250.30 (n/a)162.97 (n/a)

input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.03 (n/a)0.02 (n/a)0.03 (n/a)0.01 (n/a)0.01 (n/a)630.80 (n/a)384.01 (n/a)339.65 (n/a)246.20 (n/a)149.48 (n/a)

input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.02 (n/a)0.01 (n/a)0.01 (n/a)0.00 (n/a)0.01 (n/a)2110.00 (n/a)734.54 (n/a)424.50 (n/a)289.30 (n/a)771.25 (n/a)

input_length_2048-num_aie_columns_1-tile_size_2048

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.07 (n/a)0.04 (n/a)0.04 (n/a)0.02 (n/a)0.01 (n/a)492.40 (n/a)308.88 (n/a)277.30 (n/a)187.30 (n/a)106.63 (n/a)

input_length_2048-num_aie_columns_1-tile_size_2048-scalar_factor_3.0

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.04 (n/a)0.03 (n/a)0.03 (n/a)0.02 (n/a)0.01 (n/a)559.30 (n/a)393.30 (n/a)377.30 (n/a)300.10 (n/a)104.99 (n/a)

input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.04 (n/a)0.03 (n/a)0.03 (n/a)0.01 (n/a)0.01 (n/a)679.90 (n/a)361.81 (n/a)281.35 (n/a)217.10 (n/a)163.02 (n/a)

input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.02 (n/a)0.01 (n/a)0.01 (n/a)0.01 (n/a)0.00 (n/a)457.60 (n/a)395.28 (n/a)384.00 (n/a)313.30 (n/a)59.29 (n/a)

input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.03 (n/a)0.02 (n/a)0.02 (n/a)0.01 (n/a)0.01 (n/a)632.30 (n/a)408.60 (n/a)387.15 (n/a)244.80 (n/a)143.29 (n/a)

input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.02 (n/a)0.01 (n/a)0.01 (n/a)0.01 (n/a)0.00 (n/a)560.70 (n/a)464.36 (n/a)534.90 (n/a)260.90 (n/a)124.51 (n/a)

input_length_2048-num_aie_columns_2-tile_size_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.06 (n/a)0.04 (n/a)0.04 (n/a)0.01 (n/a)0.02 (n/a)1069.90 (n/a)414.00 (n/a)336.20 (n/a)199.30 (n/a)259.74 (n/a)

input_length_2048-num_aie_columns_2-tile_size_1024-scalar_factor_3.0

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.03 (n/a)0.03 (n/a)0.02 (n/a)0.02 (n/a)0.00 (n/a)519.80 (n/a)481.36 (n/a)511.50 (n/a)395.50 (n/a)52.81 (n/a)

input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.03 (n/a)0.02 (n/a)0.02 (n/a)0.01 (n/a)0.01 (n/a)617.10 (n/a)442.49 (n/a)483.90 (n/a)274.40 (n/a)116.20 (n/a)

input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.02 (n/a)0.02 (n/a)0.02 (n/a)0.01 (n/a)0.01 (n/a)486.00 (n/a)325.44 (n/a)281.40 (n/a)210.90 (n/a)122.46 (n/a)

input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.03 (n/a)0.02 (n/a)0.02 (n/a)0.01 (n/a)0.01 (n/a)579.10 (n/a)447.32 (n/a)467.30 (n/a)256.00 (n/a)103.40 (n/a)

input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256-group_size_32

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.02 (n/a)0.01 (n/a)0.01 (n/a)0.01 (n/a)0.00 (n/a)548.50 (n/a)465.94 (n/a)468.40 (n/a)322.70 (n/a)87.74 (n/a)

input_length_2048-num_aie_columns_4-tile_size_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.07 (n/a)0.03 (n/a)0.03 (n/a)0.01 (n/a)0.02 (n/a)1726.30 (n/a)518.35 (n/a)405.25 (n/a)187.50 (n/a)447.13 (n/a)

input_length_2048-num_aie_columns_4-tile_size_512-scalar_factor_3.0

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.04 (n/a)0.03 (n/a)0.03 (n/a)0.02 (n/a)0.01 (n/a)655.40 (n/a)466.28 (n/a)460.40 (n/a)297.50 (n/a)142.34 (n/a)

input_length_2048-num_cores_1-num_channels_1-bypass_False-tile_size_2048

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.04 (n/a)0.02 (n/a)0.03 (n/a)0.00 (n/a)0.01 (n/a)1891.90 (n/a)648.24 (n/a)270.90 (n/a)220.20 (n/a)711.66 (n/a)

input_length_2048-num_cores_2-num_channels_1-bypass_False-tile_size_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.04 (n/a)0.02 (n/a)0.02 (n/a)0.02 (n/a)0.01 (n/a)544.30 (n/a)464.38 (n/a)535.60 (n/a)189.80 (n/a)154.16 (n/a)

input_length_2048-num_cores_2-num_channels_2-bypass_False-tile_size_1024

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.03 (n/a)0.02 (n/a)0.03 (n/a)0.02 (n/a)0.01 (n/a)532.40 (n/a)368.64 (n/a)287.60 (n/a)258.70 (n/a)136.98 (n/a)

input_length_2048-num_cores_4-num_channels_1-bypass_False-tile_size_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.02 (n/a)0.02 (n/a)0.02 (n/a)0.01 (n/a)0.00 (n/a)618.70 (n/a)516.20 (n/a)499.60 (n/a)464.30 (n/a)59.47 (n/a)

input_length_2048-num_cores_4-num_channels_2-bypass_False-tile_size_512

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.02 (n/a)0.02 (n/a)0.01 (n/a)0.01 (n/a)0.00 (n/a)659.00 (n/a)514.50 (n/a)550.20 (n/a)357.90 (n/a)127.00 (n/a)

input_length_2048-num_cores_8-num_channels_2-bypass_False-tile_size_256

Commit/Date Bandwidth (max)Bandwidth (mean)Bandwidth (median)Bandwidth (min)Bandwidth (stddev)Latency (max)Latency (mean)Latency (median)Latency (min)Latency (stddev)
ea275b5 — 2026-04-20 20:24:590.02 (n/a)0.02 (n/a)0.02 (n/a)0.01 (n/a)0.00 (n/a)610.30 (n/a)489.54 (n/a)516.90 (n/a)337.00 (n/a)123.90 (n/a)
Phoenix - Examples

IRONCLAD

Tested on 2026_04_20_20_20_58 at commit ea275b5.

Test Checks TTFT (mean)TPS (mean)

Trends:

IRONCLAD Trends

andrej added 22 commits April 15, 2026 12:11
@andrej
add single-dispatch layer-by-layer MHA
d52936f
@andrej
add GPT-2 sizes as test cases, make causal mask an option
dfe5f88
@andrej
as benchmarked
71237a2
@andrej
fix DMA dimension overflow
92e6607
@andrej
create separate attn_scores_scaled buffer
43e4d07
@andrej
move output GEMM out of core MHA
af75210
@andrej
remove symbol renaming after rebase to use link_with, other fixes
ee87e94
@andrej
format
ee02731
@andrej
make mha_prefill_lxl_sd use all available columns
abf37ab
@andrej
update test result CSV iteratively rather than all at once
4caac12
@andrej
make FusedMLIROperator work on Phoenix via multiple xclbin calls
8a65c6d
@andrej
make dispatch mode selectable, add tests
daf9162
@andrej
use partial softmax on long sequence lengths
ffe0515
@andrej
go up to 32768 sequence length for mha_lxl_sd benchmark tests
6ce23a2
@andrej
stochastic testing for large sequence lengths; split GEMM+softmax int…
0403ee2 
…o multiple invocations for large sequence lengths
@andrej
reuse buffers to avoid OOM
dab067c
@andrej
support longer sequence lengths and reduce buffer sizes: new AXPY mod…
bb47493 
…es for scale/add single scalar, allow more buffers to alias to reduce memory usage
@andrej
speed up causal masking
4abbae4
@andrej
oops --- softmax should use all cores!
a233ec2
@andrej
parallelization for AXPY when single blocks are too big
ada6c1f
@andrej
Phoenix support for partial softmax
859ee97
@andrej
reactivate sample-based verification for MHA
4375996
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jgmelber jgmelber Awaiting requested review from jgmelber jgmelber will be requested when the pull request is marked ready for review jgmelber is a code owner

@hunhoffe hunhoffe Awaiting requested review from hunhoffe hunhoffe will be requested when the pull request is marked ready for review hunhoffe is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andrej