The Dot Product: Similarity, Projections, and Attention

You now know a vector is just a list of numbers. Models love turning things into lists: a word becomes a list, a user becomes a list, a document becomes a list.

Once everything is a list, you need to answer questions like

All three reduce to one question

"Aligned" is fuzzy. You want a single number: big when the lists agree, zero when they're unrelated, negative when they push opposite ways. That number comes from one tiny operation: the dot product.

The whole operation in one line

For two vectors a, b ∈ Rⁿ:

a · b = Σᵢ aᵢbᵢ

Read the symbol · as "dot". The output is a scalar — one plain number, not a list.

Four properties that matter in ML:

• Symmetric: a · b = b · a — alignment doesn't care which comes first.
• Self product gives length: a · a = ‖a‖² — dot a vector with itself and you get its squared length (connects to Topic 8, Norms).
• Zero means orthogonal: the vectors point in independent directions; e.g., after centering, uncorrelated features have near-zero dot products.
• Matrix multiplication is just many dot products: the entry AB builds from dot products between rows of A and columns of B (Topic 4).

import numpy as np

a = np.array([1.0, 2.0, 3.0])
b = np.array([0.5, -1.0, 2.0])
print(a @ b)                       # 5.5  (also np.dot / np.inner)

cos = (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
print(cos)                         # ~0.798 : angle between embeddings

# Similarity of one query against 3 stored documents (vector-DB style)
docs = np.array([[1,0,1],[0,1,1],[1,1,0]], dtype=float)
query = np.array([1.0, 0.0, 0.5])
print(docs @ query)                # dot product for every doc at once

Take a = [1, 2, 3] and b = [0.5, -1, 2]. Line them up and go entry by entry.

• position 1: 1 × 0.5 = 0.5
• position 2: 2 × (-1) = -2
• position 3: 3 × 2 = 6
• now add them: 0.5 − 2 + 6 = 4.5

So a · b = 4.5. That's the entire algorithm: multiply, then add.

A second one that shows the "alignment" idea even better. Say you grade by three topics and two students have score vectors s₁ = [5, 3, 4], s₂ = [4, 4, 5]:

• s₁ · s₂ = 5·4 + 3·4 + 4·5 = 20 + 12 + 20 = 52
• A big dot product: the two profiles point the same way (similar strengths).
• If instead s₂ = [4, -4, 0]: 20 − 12 + 0 = 8 → small, the profiles disagree.
• If s₂ = [3, -5, 2]: 15 − 15 + 8 = 8. Change one sign more and you get s₁ · s₂ = 0 → perfectly unrelated directions.

Draw each vector as an arrow from the origin. The dot product is a score of "how much do these two arrows point the same way?" — measured by the shadow one arrow casts on the other.

code
Copy
  same direction      partial agree        perpendicular      oppose
     ╱   ╱               ╱                     │              ╲
    ╱   ╱   big +       ╱   medium +           │  zero         ╲   negative
   ●                    ●                      ●                 ●
   a≈b (long shadow)    a,·b small angle       a ⟂ b           a,·b > 90°

• When the arrows line up, each one's "shadow" on the other is long → large positive dot product.
• When they're at right angles, the shadow shrinks to nothing → zero.
• When they point opposite, the shadow is negative → negative dot product.

That shadow picture is exactly what "projection" means: the projection of a onto a unit direction u is (a · u)u — the dot product a · u is the shadow length.

Carry one image through the rest of this topic: a two-person survey scoreboard.

Imagine a survey with three questions, and each answer is a number. Your answers form a vector a, a friend's answers form b. The dot product a · b multiplies your answers question by question and adds them up — a single "how much did we agree?" score.

Now the payoff: in ML, that scoreboard generalizes.

• The "questions" become features or embedding dimensions.
• The "answers" become weights, word-embedding values, query and key vectors.
• The "agreement score" becomes similarity, neuron activation, or attention score — all of them literally the same multiply-and-add.

The algebra above and the shadow picture connect through the law of cosines, which gives the geometric identity:

a · b = ‖a‖ ‖b‖ cos θ

So the dot product measures alignment: positive when the angle is under 90°, zero when perpendicular, negative when opposing. Read it as: the agreement score (a·b) equals the length of a, times the length of b, times how aligned they are (cos θ).

Normalizing both vectors — dividing each by its own length — strips the magnitudes and leaves just the alignment, turning the dot product into cosine similarity ∈ [−1, 1] — the default relevance metric for embeddings:

• Text/image retrieval with sentence-transformers, CLIP, or OpenAI embeddings ranks by cosine similarity.
• Normalized (cosine) vector indexes in FAISS, pgvector, and Milvus store unit vectors so the dot product is the similarity — one multiply-add pass per candidate.
• Recommendation systems compute user–item affinity as inner products of learned latent vectors (classic matrix-factorization view, still visible in two-tower models).

The scoreboard is secretly the engine of neural nets.

Inside a neuron. A fully-connected neuron computes z = w · x + b then an activation. The dot product is the neuron's entire "opinion": the weight vector w encodes a preferred direction in input space; the neuron fires strongly for inputs aligned with w. A layer is a batch of such dot products — matrix multiplication again.

Inside attention. In Transformers (Vaswani et al., 2017, and every 2024–2026 LLM), attention scores are dot products between learned query and key projections:

Attention(Q,K,V) = softmax(QKᵀ / √d_k) V

Follow the scoreboard metaphor: the query q is "what I'm looking for", each key kᵢ is a stored answer sheet, q · kᵢ scores how well they agree, softmax turns the scores into weights, and the output is a weighted sum of value vectors. The Mermaid diagram at the top draws exactly this loop.

The /√d_k scaling keeps dot products from saturating the softmax as dimension grows. Variants like Grouped-Query Attention (GQA) and Multi-head Latent Attention (MLA, DeepSeek-V2/V3, 2024–2025) change how many K/V heads are stored, but the scoring operation remains the humble dot product — executed by FlashAttention kernels as fused matrix multiplies.

One dot product costs n multiplies and n−1 adds — O(n). Trivial… until you need it billions of times:

• Retrieving from 10 million 1,536-dim embeddings ≈ 1.5×10¹⁰ multiply-adds per query → why Approximate Nearest Neighbor (ANN) indexes (HNSW, IVF-PQ) exist to skip most dot products.
• Long-context inference: attention cost grows with the number of query–key dot products (quadratic in sequence length), motivating FlashAttention tiling, sparse/sliding-window attention, and KV caching.
• Dot products are the "embarrassingly parallel" kernel GPUs and TPUs are built for: SIMD FMA (fused multiply-add) units chew through them at teraflop rates.