Cohere Rerank: Cross-Encoder Reranking, Candidate Selection, Latency, and Cost
Cohere Rerank is a hosted cross-encoder scoring service that sits after a first-stage retriever in a RAG pipeline. It takes a query and a list of candidate documents pulled from a vector store, BM25 index, or hybrid search system, and returns a relevance score for each candidate by running the query and the document jointly through a transformer, rather than comparing precomputed embeddings. The trade-off is architectural: reranking is more accurate than bi-encoder similarity because it never compresses the document into a single vector, but it is also slower and priced per call, which forces a decision about how many candidates to send and how often to call it.
········
RERANK MODEL FAMILY: ARCHITECTURE, CONTEXT LENGTH, AND CHUNKING BEHAVIOR.
What each model version supports and how it handles documents longer than its context window.
Cohere currently offers five rerank models through its API, according to Cohere's own documentation: two legacy v3.0 models (English-only and multilingual), a v3.5 multilingual model, and a v4.0 generation split into a quality-optimized rerank-v4.0-pro and a latency-optimized rerank-v4.0-fast. All of them work the same way mechanically: for every document in the request, the query tokens and the document tokens are combined and counted against a single context limit. If that combined length exceeds the model's context window, Cohere's documentation states the document is automatically split into chunks and processed as multiple inferences, with the returned relevance score taken as the maximum across chunks — a behavior that matters because it means a very long document is not penalized for length, but it is billed for every chunk it produces.
Model | Languages | Context window | Chunk size when truncating | Positioned for |
rerank-english-v3.0 | English only | 4,096 tokens | 4,093 tokens | Legacy deployments, English-only corpora |
rerank-multilingual-v3.0 | 100+ languages | 4,096 tokens | 4,093 tokens | Legacy multilingual deployments |
rerank-v3.5 | 100+ languages | 4,096 tokens | 4,093 tokens | Current default for most RAG stacks |
rerank-v4.0-fast | 100+ languages | 32,768 tokens | 32,764 tokens | High-throughput, low-latency search |
rerank-v4.0-pro | 100+ languages | 32,768 tokens | 32,764 tokens | Long documents, semi-structured/JSON data, highest quality |
........
Query length is capped at half the context window: 2,048 tokens for the v3 family and v3.5, 16,384 tokens for v4.0, with longer queries truncated rather than rejected. Cohere's documentation also states a hard ceiling on request size — the product of document count and max chunks per document cannot exceed 10,000 — which in practice caps how many long documents can be sent in a single call before they need to be pre-chunked client-side. Cohere's stated capability for v4.0-pro to score semi-structured JSON documents directly (formatted as YAML strings per its best-practices guidance) is a vendor-described feature; independent benchmarking of its accuracy on structured data was not found in this research pass.
········
CANDIDATE SELECTION AND THE RETRIEVE-THEN-RERANK PIPELINE.
How many candidates to send to the reranker, and why that number is the main cost and latency lever.
Rerank is never the first-stage retriever. The standard pattern is to retrieve a broader candidate set with a cheap method — dense vector search, BM25, or a hybrid combination — and pass only that shortlist to the cross-encoder, which returns a reordered top-n. The reason for this split is computational: a cross-encoder requires a full transformer forward pass for every query-document pair, while a bi-encoder embedding comparison is a single dot product against precomputed vectors. Industry write-ups on the retrieve-then-rerank pattern illustrate the gap concretely — reranking tens of millions of records with a BERT-class cross-encoder on a single GPU has been estimated to take tens of hours, versus under 100 milliseconds for embedding-based retrieval over the same set — which is why reranking is applied to a shortlist of dozens to low hundreds of candidates, not the full corpus.
The size of that shortlist is the main tuning variable. A larger top_k passed into the reranker increases recall (the correct document is more likely to be in the candidate set at all) but scales cost and latency roughly linearly, since Cohere bills and processes each document in the request. A common starting point in worked examples is retrieving on the order of 25 to 100 candidates from the vector store and reranking down to a final top-n of 3 to 10 for the generation step; the right number depends on how noisy first-stage retrieval is for a given corpus, and is not a fixed value Cohere prescribes. Long documents should generally be chunked before retrieval rather than left whole and relying on Rerank's automatic chunking, both for cost predictability and because the first-stage retriever needs chunk-level granularity to find the right passage in the first place.
Score interpretation is a separate, non-obvious risk. Cohere's own best-practices documentation cautions that relevance scores are not linear or self-explanatory — a document scoring roughly 0.91 is not "twice as relevant" as one scoring 0.04 — and recommends running 30 to 50 representative queries against known borderline-relevant documents to calibrate a filtering threshold for a specific domain, rather than hardcoding a generic cutoff like 0.5. Skipping this calibration step is a common source of reranking pipelines that either drop relevant documents or let irrelevant ones through.
········
LATENCY AND PRICING ACROSS DEPLOYMENT ROUTES.
What reranking costs per query on each access path, and where the published numbers disagree.
Rerank is available directly from Cohere's API, through AWS Bedrock and AWS Marketplace/SageMaker JumpStart, and through Azure AI Foundry (Cohere's changelog confirms rerank-v3.5 and rerank-v4.0-fast listings on Azure AI Foundry). Pricing differs meaningfully by route and is billed per "search" — a unit generally described as one query scored against a bounded batch of documents — rather than a flat per-call fee, which means cost scales with candidate-set size regardless of which platform is used. The figures below come from third-party pricing trackers and marketplace listings rather than a single authoritative Cohere pricing page fetched directly in this research pass, and should be treated as indicative rather than exact at the time of purchase.
Access route | Model | Reported price | Observed latency / notes |
Cohere API / OpenRouter | rerank-v3.5 | ~$0.001 per search (OpenRouter); ~$2.00 per 1,000 queries reported for Bedrock | Sub-second for typical batch sizes |
Cohere API / OpenRouter | rerank-v4.0-fast | ~$0.002 per search | Optimized for high-throughput, low-latency search |
Cohere API / OpenRouter | rerank-v4.0-pro | ~$0.0025 per search | ~0.54s reported for a hosted search operation on one provider benchmark |
AWS Bedrock / Marketplace | rerank-english-v3.0, rerank-multilingual-v3.0 (legacy) | Provisioned throughput only, reported around $7.12/hour; no on-demand per-search rate | Requires committing to a running instance, not pay-per-call |
Cohere Model Vault (dedicated) | Any rerank model, dedicated instance | Reported range roughly $3,250–$6,500 per month | Dedicated capacity, not shared multi-tenant latency |
........
Trial API keys are reported as capped at 1,000 calls per month and explicitly disallowed for production traffic, which is enough for prototyping a reranking step but not for a live application. For teams sizing cost, the practical formula is: number of queries per day × average candidate-set size passed to Rerank × price per search-unit, with the reminder that a "search" bundles a bounded number of documents (commonly cited as up to 100), so a candidate set above that bundle size is billed as multiple search units per query, not one.
········
DECIDING WHETHER TO ADD A RERANKING STEP.
Reranking adds a network round trip and a transformer inference on the critical path of every query, which typically costs tens to a few hundred milliseconds depending on candidate-set size and model choice — an added latency budget that has to be justified by a measurable retrieval-quality gain, not assumed. It also introduces a hard dependency on an external API's availability and rate limits at query time, unlike a self-hosted embedding index that keeps working if a third-party service degrades. The gain it buys is real when first-stage retrieval is noisy: hybrid or dense retrieval over ambiguous, jargon-heavy, or multilingual corpora routinely returns a candidate set where the best answer is present but not ranked first, and a cross-encoder that reads the full query-document pair jointly corrects that ordering more reliably than tuning embedding similarity thresholds ever will. It is a weaker investment when the corpus is narrow, queries are short and well-matched to document vocabulary, and first-stage top-3 accuracy is already high, since the added cost and latency then buy little reordering benefit. The concrete rule: add Cohere Rerank when a measured evaluation on your own query set shows first-stage retrieval placing the correct passage outside the top 3 more than roughly 10-15% of the time, size the candidate set to the smallest value that keeps that failure rate low, and calibrate a relevance-score threshold against 30-50 real queries before using it to filter or gate downstream generation.
FOLLOW US FOR MORE.
·····
DATA STUDIOS
·····
