How Does the Dot Product Measure Similarity?

I was studying Build a Large Language Model (From Scratch) the other day when I came across the idea of using dot products to calculate similarity between two normalised vectors.

I found that intriguing.

The dot product is simply the sum of the pairwise products of two vectors’ components. How would that indicate similarity?

This article is my research written up. To be honest, proving why this works isn’t necessary to use it. If you’re like me, however, you value understanding why things are the way they are. Knowledge for knowledge’s sake ;)

Before we go on this adventure, let’s start with some basics.

What do we mean by similarity?

A vector can be represented as a collection of numbers. Geometrically, we can think of it as an arrow with a magnitude (length) and a direction.

Take a vector called $A$, defined as $(4, 5)$. This vector has two dimensions, although vectors can have many more. On a graph, vector $A$ looks like this:

Starting at the origin, it extends 4 steps along the x-axis and 5 steps along the y-axis.

Now imagine another vector, $B$, defined as $(2, 1)$.

If I wanted to calculate how similar $A$ is to $B$, a good first question would be: similar based on what metric?

I could, for instance, measure the distance between their endpoints. A smaller distance would indicate greater similarity. Notice what’s happening here, though: the lengths of the vectors matter, as well as their directions. If $A$ were shorter, that distance would change.

If I were interested only in direction, I could measure the angle $\theta$ (theta) between the vectors. That’s the approach we’ll use here.

On our two-dimensional graph, we could measure the angle with a protractor:

• At $0^\circ$, the vectors point in the same direction.
• At $90^\circ$, they’re orthogonal (perpendicular).
• At $180^\circ$, they point in opposite directions.

Cosine turns these angles into a similarity score:

$$\begin{aligned} \cos(0^\circ) &= 1 \\ \cos(90^\circ) &= 0 \\ \cos(180^\circ) &= -1 \end{aligned}$$

Plotting vectors and measuring angles isn’t a practical approach for computers, especially when vectors have hundreds of dimensions. We need a way to compute this score directly from their components.

Thankfully, maths can help us here.

giggles in trigonometry

Start with the law of cosines

There’s a mathematical rule that lets us calculate the cosine of an angle in a triangle, provided we know the lengths of all three sides. It’s called the law of cosines.

$$\begin{aligned} c^2 &= a^2 + b^2 - 2ab\cos(\theta) \\ 2ab\cos(\theta) &= a^2 + b^2 - c^2 \\ \cos(\theta) &= \frac{a^2 + b^2 - c^2}{2ab} \\ \theta &= \arccos\left(\frac{a^2 + b^2 - c^2}{2ab}\right) \end{aligned}$$

Here, $a$, $b$, and $c$ are the side lengths. The angle $\theta$ lies between the sides of lengths $a$ and $b$, opposite the side of length $c$.

The law of cosines generalises Pythagoras’ theorem:

$$c^2 = a^2 + b^2$$

Pythagoras’ theorem applies to right-angled triangles. When $\theta = 90^\circ$, the cosine term is zero, leaving us with the equation above.

How does this relate to our vectors $A$ and $B$? That’s what we’re about to find out.

Find the lengths of the vectors

We know the components of $A$ and $B$, but we haven’t calculated their lengths yet.

For $A$, the horizontal component is 4 and the vertical component is 5. Its length is the hypotenuse of the right triangle below.

Pythagoras’ theorem gives us a way to calculate that length:

$$\begin{aligned} z^2 &= x^2 + y^2 \\ z &= \sqrt{x^2 + y^2} \end{aligned}$$

Plugging in the components of $A$:

$$\begin{aligned} z^2 &= 4^2 + 5^2 \\ z &= \sqrt{16 + 25} \\ &= \sqrt{41} \\ &\approx 6.403 \end{aligned}$$

We write the length of $A$ as $|A|$, so:

$$|A| = \sqrt{41} \approx 6.403$$

More generally, we can write:

$$|A|^2 = A_x^2 + A_y^2$$

Here, $A_x$ and $A_y$ are the x and y components of $A$. Keep this relationship in mind; we’ll use it again shortly.

The same calculation for $B = (2, 1)$ gives $|B| = \sqrt{5} \approx 2.236$.

Find the third side

Okay, now that we have $|A|$ and $|B|$, how do we calculate the distance between their endpoints? Remember, we need all three side lengths to use the law of cosines.

It turns out there’s a neat way to do this. Subtract $B$ from $A$ to get a new vector, $C = A - B$:

$$A - B = (4, 5) - (2, 1) = (4 - 2, 5 - 1) = (2, 4)$$

This vector connects the endpoint of $B$ to the endpoint of $A$. That makes sense: to get from $B$ to $A$, move 2 steps along the x-axis and 4 steps along the y-axis.

We can now use Pythagoras’ theorem to calculate its length, $|C| = |A - B|$.

Let’s write the calculation in terms of the components, using $A = (A_x, A_y)$ and $B = (B_x, B_y)$:

$$\begin{aligned} |A - B|^2 &= (A_x - B_x)^2 + (A_y - B_y)^2 \\ &= (A_x - B_x)(A_x - B_x) \\ &\quad + (A_y - B_y)(A_y - B_y) \\ &= A_x^2 - 2A_xB_x + B_x^2 \\ &\quad + A_y^2 - 2A_yB_y + B_y^2 \\ &= (A_x^2 + A_y^2) + (B_x^2 + B_y^2) \\ &\quad - 2(A_xB_x + A_yB_y) \\ &= |A|^2 + |B|^2 - 2(A_xB_x + A_yB_y) \end{aligned}$$

A brief clarification on the last step: remember the relationship I asked you to keep in mind?

$$|A|^2 = A_x^2 + A_y^2$$

That’s how we replace the sums of squared components with $|A|^2$ and $|B|^2$.

Now we need to connect this result to the angle between the vectors.

Connect the two formulas

Remember the law of cosines:

$$c^2 = a^2 + b^2 - 2ab\cos(\theta)$$

For our triangle, the side lengths are $|A|$, $|B|$, and $|C|$, so:

$$|C|^2 = |A|^2 + |B|^2 - 2|A||B|\cos(\theta)$$

Since $C = A - B$, we can rewrite this as:

$$|A - B|^2 = |A|^2 + |B|^2 - 2|A||B|\cos(\theta)$$

We now have two expressions for $|A - B|^2$. One uses the components of the vectors; the other uses their lengths and the angle between them. Because they describe the same quantity, we can set them equal:

$$\begin{aligned} &|A|^2 + |B|^2 - 2|A||B|\cos(\theta) \\ &\qquad = |A|^2 + |B|^2 - 2(A_xB_x + A_yB_y) \end{aligned}$$

Subtract $|A|^2 + |B|^2$ from both sides, then divide by $-2$:

$$\begin{aligned} |A||B|\cos(\theta) &= A_xB_x + A_yB_y \\ A_xB_x + A_yB_y &= |A||B|\cos(\theta) \end{aligned}$$

Look at the left-hand side of the last equation. That’s exactly the dot product!

For $A = (A_x, A_y)$ and $B = (B_x, B_y)$, the dot product is $A \cdot B = A_xB_x + A_yB_y$. Therefore:

$$A \cdot B = |A||B|\cos(\theta)$$

The dot product equals the length of $A$ multiplied by the length of $B$, multiplied by the cosine of the angle between them.

Check it with our example

Let’s test the formula using our original vectors:

$$\begin{aligned} A &= (4, 5), \quad B = (2, 1) \\ A \cdot B &= 4 \times 2 + 5 \times 1 = 13 \\ |A||B| &= \sqrt{41}\sqrt{5} = \sqrt{205} \\ \cos(\theta) &= \frac{13}{\sqrt{205}} \approx 0.908 \\ \theta &\approx 24.8^\circ \end{aligned}$$

Now let’s check this with the law of cosines, keeping the squared lengths exact to avoid rounding too early:

$$\begin{aligned} |A - B|^2 &= 2^2 + 4^2 = 20 \\ \cos(\theta) &= \frac{41 + 5 - 20}{2\sqrt{41}\sqrt{5}} \\ &= \frac{13}{\sqrt{205}} \approx 0.908 \\ \theta &\approx 24.8^\circ \end{aligned}$$

We get the same answer.

Why normalisation makes the difference

If we’re dealing with normalised vectors—that is, vectors with a length of 1—the formula becomes:

$$\begin{aligned} A \cdot B &= |A||B|\cos(\theta) \\ &= 1 \times 1 \times \cos(\theta) \\ &= \cos(\theta) \end{aligned}$$

This is really beautiful, actually.

The dot product of two normalised vectors equals their cosine similarity. Without normalisation, the lengths of the vectors also affect the dot product.

We don’t need to calculate the angle first, or find the vector connecting the endpoints of $A$ and $B$.

Let’s normalise our original vectors. To normalise a non-zero vector, divide each component by the vector’s length:

$$\begin{aligned} A_{\mathrm{norm}} &= \frac{(4, 5)}{\sqrt{41}} \approx (0.625, 0.781) \\ B_{\mathrm{norm}} &= \frac{(2, 1)}{\sqrt{5}} \approx (0.894, 0.447) \end{aligned}$$

Their dot product gives us:

$$\begin{aligned} \cos(\theta) &= A_{\mathrm{norm}} \cdot B_{\mathrm{norm}} \\ &= \frac{4 \times 2 + 5 \times 1}{\sqrt{41}\sqrt{5}} \\ &= \frac{13}{\sqrt{205}} \approx 0.908 \\ \theta &\approx 24.8^\circ \end{aligned}$$

The same angle again! We’ve kept the exact values in the calculation; using the rounded components would give a slightly different approximation.

That, my friends, is the beauty of mathematics.

Let me stop now because my head hurts, lol!