ModelRefs / K-Nearest Neighbors — Tutorial

K-Nearest Neighbors — Tutorial

The simplest non-parametric classifier — and why it still wins on certain problems. Covers How KNN works, Pros, cons, and when to use KNN.

Overview

The simplest non-parametric classifier — and why it still wins on certain problems

Level: Intermediate. Estimated reading time: 25 minutes.

How KNN works

K-Nearest Neighbors is the most intuitive ML algorithm: to classify a new point, find the K training examples closest to it and take a majority vote.

No training phase — KNN is lazy: it memorises the entire training set and defers all computation to prediction time. This means training is instant but prediction is slow (O(n·d) per query, where n = training examples, d = features).

Distance metric: by default, Euclidean distance ||x - xᵢ|| = √Σ(xⱼ - xᵢⱼ)². For text, cosine distance works better. For mixed types (numeric + categorical), careful preprocessing is needed.

Choosing K: small K = more complex boundary (risk overfit), large K = smoother boundary (risk underfit). Use cross-validation to tune K. Odd K avoids ties for binary classification.

Pros, cons, and when to use KNN

Strengths: - Zero training time — useful when data changes frequently (online setting) - Works well with irregular decision boundaries that parametric models struggle with - Naturally handles multi-class classification - No assumptions about data distribution (non-parametric)

Weaknesses: - Slow at inference: must compare against every training example - Memory-intensive: stores all training data - Sensitive to irrelevant features and scale (always normalise) - Degrades in high dimensions (curse of dimensionality): in 1000 dimensions, all points become approximately equidistant

When to use KNN: recommendation systems (find similar users/items), anomaly detection (points with no close neighbours are anomalies), baseline comparisons, small datasets where training time doesn't matter.

For large datasets: approximate nearest-neighbour algorithms (FAISS, HNSW) make KNN practical at scale — this is exactly how vector databases retrieve similar embeddings.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to K-Nearest Neighbors — Tutorial.