Calculate Attention With A Simple Example

We will now calculate 1 attention step using 3 supplied source positions, A, B and C. Their compatibility scores are 2, 1 and 0. These scores have already been scaled. We also supply small value vectors so every multiplication can be inspected.

What combination will pass forward, and what changes if position C becomes disallowed? This is a separate classroom construction following the scaled dot-product method described by Vaswani and colleagues in 2017. It is not a measured attention map from the package example. Query-key matching and its scaling

have already produced our given scores. Here an attention weight means a computed share for this step; the learned projection matrices are a different kind of weight, and this arithmetic does not update them. Begin with the scores. Softmax raises the constant e to each allowed score, then divides

by the sum of those exponentials. The constant e is about 2.718; the supplied approximate exponentials are 7.389 for score 2, 2.718 for score 1, and 1 for score 0. Their displayed sum is 11.107. Notice that score 0 still produces a positive contribution because e

raised to 0 equals 1. Divide each exponential by the total. To 3 decimal places, the attention shares are 0.665 for A, 0.245 for B, and 0.090 for C. They add to 1 under this displayed rounding. A gets the largest share,

while B and C still contribute. These shares are normalized over the supplied source positions, with no output-vocabulary choice made at this stage. Now attach the values. A has coordinates 2 and 0. B has coordinates 0 and 2. C has coordinates 1 and 1.

Multiply every coordinate in each value by its position’s share. Then add coordinate 1 across the contributions, and do the same for coordinate 2. Keeping the coordinate columns aligned prevents us from mixing unrelated entries. Using the displayed 3-decimal shares, coordinate 1 is 0.665 times 2,

plus 0.245 times 0, plus 0.090 times 1. That gives 1.420. Coordinate 2 is 0.665 times 0, plus 0.245 times 2, plus 0.090 times 1. That gives 0.580. The resulting vector is approximately 1.420

and 0.580 under this rounded working convention. The reading also records the unrounded calculation so the precision boundary stays visible. We have formed a mixture. The output is different from each of the 3 supplied value vectors. The attention shares sum to 1, while the value coordinates follow their own scale:

coordinate 1 of our result exceeds 1. That is valid for a representation vector. It is not a probability that a word is true or a signal that only A was selected. Several weighted contributions produced the result. Now hold the supplied scores and values fixed, but change the

allowed set so C is excluded. The mask gives C no contribution. Return to the exponentials for A and B and normalize over those allowed positions. Their displayed total is now about 10.107. The new shares are about 0.731 and 0.269, with 0 for the excluded position. This is a changed denominator and mixture,

rather than merely deleting C’s old contribution. Using those new 3-decimal shares, the output becomes about 1.462 and 0.538. Coordinate 1 is 0.731 times 2; coordinate 2 is 0.269 times 2. Removing C’s balanced value changes the mixture even though C previously had a smaller share.

A finite score of 0 and an excluded position therefore have different effects. The mask determines which information may contribute before normalization. Score scale also matters. Multiplying all 3 given scores by 10 would produce 20, 10 and 0, concentrating weight much more strongly on A.

The original Transformer’s dimension-dependent scaling addresses the numerical behavior of query-key scores before this step. Our given scores already include that operation. The 2 coordinates shown for the value vectors do not tell us an otherwise unspecified query-key dimension or justify applying another scale factor.

Pause for a checkpoint. All 3 positions are allowed, but now every score is 0. Keep the original values: 2 and 0, 0 and 2, then 1 and 1. What are the attention shares and combined output? Does score 0 exclude a position?

Finish your calculation before continuing. Each exponential is 1, so each share is 1/3. Combining the values gives coordinates 1 and 1. All 3 positions contributed, even though the result happens to equal C’s supplied value. A score of 0

did not exclude any position. Exclusion requires the defined mask, followed by normalization over what remains allowed. For a separate practice task, use 2 allowed positions with scores 0 and 0. Their values have coordinates 4 and 0, then 0 and 2. Calculate the mixture.

Next, exclude the 2nd position while keeping the 1st allowed. What are the new shares and result? Pause and complete the worksheet before consulting the key. The initial equal shares are 0.5 and 0.5, producing coordinates 2 and 1.

After excluding the 2nd position, the 1st receives share 1 and the excluded one receives 0. The output is therefore 4 and 0. The shares describe allowed-position contributions; the output coordinates remain a representation rather than a vocabulary probability distribution. You can now reproduce the normalized

value mixture, state the rounding you used, and recalculate it after a mask change. You can also distinguish a finite 0 score from exclusion and a computed attention share from a learned parameter. Next, we will place attention inside a complete transformer block, following the other transformations and connections that act on its output.