Multi-Head Attention from Scratch

Asked at Oracle · ML Fundamentals & Algorithms · Difficulty: Medium · Last asked Jan 22, 2026

Implement multi-head scaled dot-product attention from scratch, using only array operations (NumPy or tensor math). Library attention modules such as `nn.MultiheadAttention` or `F.scaled_dot_product_attention` are not allowed. The function splits the model dimension into several heads, lets each head attend over the sequence independently, and joins the heads back together.

Loading the problem…