Embeddings, Rerankers, and Classifiers: Similarity, Relevance, and Decisions
“Run BERT locally” does not yet specify a capability. The same encoder family can use different output heads and training objectives. Returning a vector, scoring a question–document pair, and selecting an action are different interfaces. Deployment location does not erase that distinction.
Identify the input and output
Sentence-BERT trains sentence representations for similarity comparisons. The issue is not whether a raw BERT model produces hidden states, but whether training makes distances useful for the task. Averaging arbitrary token vectors does not automatically produce a good retrieval model.
A bi-encoder processes questions and documents separately, so documents need not be recomputed for each question. A cross-encoder processes the pair together, allowing their tokens to interact within one forward pass. The retrieve-and-rerank example illustrates this division of work. A score may be a logit or a transformed value; calling it a score does not establish probability calibration.
Open full-size imageFollow the arrows from bottom to top. On the left, each sentence becomes a reusable vector before cosine comparison. On the right, the pair enters BERT together, so a new candidate requires a new paired computation. This explains why retrieval can index document vectors while reranking usually scores a shortlist. The illustrated 0–1 output is a choice of scoring head, not proof of calibrated probability.
One workflow may need all three
Consider a fictional research assistant asked, “How can I transcribe meeting recordings offline?” Vector search retrieves 50 notes about speech and local inference. A reranker promotes 5 that explicitly discuss device requirements and offline limitations. An action classifier then selects “answer,” “search further,” or “clarify.” These counts merely illustrate progressively smaller candidate sets.
The learning targets differ: similarity relationships, question–passage relevance, and suitable actions in a task state. A highly relevant note may be obsolete. A predictable action may be unauthorized. Relevance cannot replace version checking, and action probabilities cannot replace tool authorization.
Similarity is not a probability
This example computes dot products of unit vectors. The vectors are invented; no model is loaded.
from math import sqrt
def unit(v):
length = sqrt(sum(x * x for x in v))
return [x / length for x in v]
query = unit([1, 1, 0])
documents = [[1, 0, 0], [0, 1, 0], [0, 0, 1]]
scores = [sum(a * b for a, b in zip(query, unit(d)))
for d in documents]
print([round(s, 3) for s in scores]) # [0.707, 0.707, 0.0]
The first two documents have the same angle in this representation. One could contain the correct procedure while the other merely mentions the topic. A score of 0.707 does not mean a 70.7% chance of correctness. Applying softmax to the scores makes the normalized values change when ten irrelevant candidates are added, even though the original documents have not changed.
Using scores to trigger actions requires task labels, error costs, and calibration checks. Ranking quality, classification accuracy, and reliable probabilities are separate properties.
Fixed labels differ from supplied choices
A conventional classifier often maps a representation to a fixed label set. For “billing / technical / other,” the three output dimensions acquire their meaning during training. Adding a fourth class usually requires retraining or adapting the output head.
A candidate-conditioned model instead receives descriptions of this request's choices. Accepting new choices does not establish generalization to arbitrary tasks. Wording, overlapping choices, negation, and the absence of a correct choice can all change behavior. Jev and Laya address this structured-decision setting. Variable choice counts also do not imply that scoring hundreds of tools directly is effective. Retrieving candidates before selecting an action can make errors easier to locate.
Compare local models on the same job
For retrieval, use questions with relevant-document labels and measure recall and ranking. For action selection, use task states with allowed actions and outcomes, then measure wrong actions, abstention, and latency. Do not rank vector-encoding speed against the latency of scoring many question–document pairs as if they were the same operation.
Fix language, input length, candidate count, hardware, precision, batch size, and cold versus warm conditions. Small download size does not guarantee speed on long inputs or large candidate sets. Once the interface is clear, retrieval pipelines and inference performance turn “local BERT” into specific, comparable tasks.
Explore a built-in dataset in Embedding Projector: select a point, inspect its neighbors, and switch projection methods. Use it to distinguish neighborhoods in an embedding from the layout of a two- or three-dimensional view. A nearby label is a clue about the representation, not a calibrated relevance score.