Each cell (i, j) in the score matrix is the dot product of token i's query with token j's key — how well token i's question matches token j's advertisement.

Scores are divided by √d_head = 16 to keep gradients stable before softmax.

This is where the architecture forces W_Q and W_K to specialize, despite both starting as random numbers. The dot product treats them asymmetrically: Q is always the rows (the seeker) and K is always the columns (the target). Because the math always processes them in this fixed order, gradient descent has no choice but to push them into complementary roles. The model is rewarded (lower loss) when attention pulls the right information from V — so W_Q learns to produce effective search requests and W_K learns to produce matching labels. If the model needs “the subject of a verb,” it makes the query for a verb align with the key for a noun.

The roles aren't interchangeable. Swapping Q and K would transpose this matrix, reversing which token attends to which and breaking the subsequent multiplication with V. If they shared a single weight matrix, the pattern would be symmetric — token A attending to B would force B to attend equally to A. Language doesn't work that way: a verb needs its object, but the object doesn't necessarily need the verb.

The Prediction Task