Embeddings, Rerankers, and Classifiers: Similarity, Relevance, and Decisions

“Run BERT locally” does not yet specify a capability. The same encoder family can use different output heads and training objectives. Returning a vector, scoring a question–document pair, and selecting an action are different interfaces. Deployment location does not erase that distinction.

Identify the input and output

Role	Input and output	Scaling to a large corpus
Embedding model / bi-encoder	Text → fixed-dimensional vector	Documents can be encoded in advance and indexed
Reranker / cross-encoder	Question and candidate document → relevance score	Usually processes a small retrieved candidate pool
Classifier or decision model	Input and task definition → labels, scores, or candidate probabilities	Cost depends on how labels and candidates are represented

Sentence-BERT trains sentence representations for similarity comparisons. The issue is not whether a raw BERT model produces hidden states, but whether training makes distances useful for the task. Averaging arbitrary token vectors does not automatically produce a good retrieval model.

A bi-encoder processes questions and documents separately, so documents need not be recomputed for each question. A cross-encoder processes the pair together, allowing their tokens to interact within one forward pass. The retrieve-and-rerank example illustrates this division of work. Some models output a raw score with no upper or lower bound; others squash it into the range 0–1. Calling it a score does not establish probability calibration.

Open full-size image

Follow the arrows from bottom to top. On the left, each sentence becomes a reusable vector before cosine comparison. On the right, the pair enters BERT together, so a new candidate requires a new paired computation. This explains why retrieval can index document vectors while reranking usually scores a shortlist. The illustrated 0–1 output is a choice of scoring head, not proof of calibrated probability.

Sentence Transformers, Cross-Encoder vs. Bi-Encoder. · Apache 2.0

One workflow may need all three

Consider a fictional research assistant asked, “How can I transcribe meeting recordings offline?” Vector search retrieves 50 notes about speech and local inference. A reranker promotes 5 that explicitly discuss device requirements and offline limitations. An action classifier then selects “answer,” “search further,” or “clarify.” The candidates shrink from 50 notes to 5, and finally to one action.

What each stage learns in training differs: similarity relationships, question–passage relevance, and suitable actions in a task state. A highly relevant note may be obsolete. A predictable action may be unauthorized. Relevance cannot replace version checking, and action probabilities cannot replace tool authorization (see Tool Contracts for AI Agents).

An action classifier need not use fixed labels: the model can score each candidate action's description together with the input, so adding an option means writing one more description. Jev and Laya: Structured Decision Models covers this structured-decision setting and its limits.

Similarity is not a probability

This example computes dot products of unit vectors. The vectors are invented; no model is loaded.

from math import sqrt

def unit(v):
    length = sqrt(sum(x * x for x in v))
    return [x / length for x in v]

query = unit([1, 1, 0])
documents = [[1, 0, 0], [0, 1, 0], [0, 0, 1]]
scores = [sum(a * b for a, b in zip(query, unit(d)))
          for d in documents]
print([round(s, 3) for s in scores])  # [0.707, 0.707, 0.0]

The first two documents have the same angle in this representation. One could contain the correct procedure while the other merely mentions the topic. A score of 0.707 does not mean a 70.7% chance of correctness. Applying softmax to the scores makes the normalized values change when ten irrelevant candidates are added, even though the original documents have not changed.

Using scores to trigger actions requires task labels, error costs, and calibration checks (see Probability Calibration, Decision Thresholds, and Abstention). Ranking quality, classification accuracy, and reliable probabilities are separate properties.

For how metrics and normalization affect nearest neighbors, and how approximate search can miss them, see Vector Indexes: Exact Search, HNSW, and IVF.

Compare local models on the same job

For retrieval, use questions with relevant-document labels and measure recall and ranking. For action selection, use task states with allowed actions and outcomes, then measure wrong actions, abstention, and latency. Do not rank vector-encoding speed against the latency of scoring many question–document pairs as if they were the same operation.

Fix language, input length, candidate count, hardware, precision, batch size, and cold versus warm conditions. Small download size does not guarantee speed on long inputs or large candidate sets. Once the interface is clear, Retrieval Pipelines: Chunking, Recall, Reranking, and Citations and Inference Performance: Loading, First Tokens, Decoding, and Throughput turn “local BERT” into specific, comparable tasks.

As a starting point for mixed Chinese and English text, try a local multilingual model such as potion-multilingual-128M, which the baseline in retrieval pipelines uses; zvec-grep's model guide lists more candidates for code, English and multilingual text. Queries and documents must be encoded with the same model and the same normalisation; changing the model means re-encoding the whole collection.

My reading