OpenSearch: Neural Search, Hybrid Retrieval, Vector Indexes, and RAG Pipelines
- 1 hour ago
- 8 min read
OpenSearch is the Apache 2.0-licensed search and analytics engine that AWS forked from Elasticsearch 7.10.2 in 2021, after Elastic relicensed Elasticsearch under the Server Side Public License. Governance now sits with the OpenSearch Software Foundation under the Linux Foundation, a vendor-neutral structure AWS handed the project to as adoption grew beyond its own managed service. On top of the original BM25 lexical engine, OpenSearch ships two plugins that turn it into a full retrieval stack for RAG applications: the k-NN plugin, which stores and searches dense vectors through a choice of approximate nearest neighbor engines, and the Neural Search plugin, which automates embedding generation, hybrid query fusion, and, more recently, retrieval-augmented generation itself through connectors to external LLMs. The result is a system that can run BM25 keyword search, vector similarity search, and an LLM-backed answer generation step inside a single query pipeline, which is the main reason teams already operating OpenSearch or Elasticsearch clusters evaluate it before adding a dedicated vector database.
········
OPENSEARCH'S K-NN ENGINE OPTIONS: LUCENE, FAISS, AND NMSLIB.
How the choice of engine, algorithm, and quantization method at index-creation time trades off memory footprint, recall, and query latency.
Vector fields in OpenSearch are declared as knn_vector mappings, and the engine backing that field is fixed at index creation — changing it later requires reindexing, not a live setting update. Three engines are available. Lucene is the native engine built directly into the Apache Lucene library that OpenSearch already depends on for lexical search: it runs in the JVM with no external native library, which simplifies operations and packaging, and it supports HNSW graphs plus a built-in scalar (int8) quantization codec. Faiss, Meta's similarity-search library, is loaded as a native (JNI) dependency and is the only engine that supports the IVF (inverted file index) algorithm alongside HNSW; it also carries the widest quantization set — scalar quantization at fp16 and int8 precision, product quantization (PQ), and binary quantization. NMSLIB was the original k-NN engine at OpenSearch's launch and is now considered legacy: it supports HNSW only, with no meaningful quantization path, and new indexes are generally pointed at Lucene or Faiss instead.
HNSW is available on all three engines and needs no training step — vectors are inserted into the graph incrementally. IVF, being Faiss-only, requires a separate training phase in which centroids are computed from a representative sample of the vector set before any vectors can be added; collections too small to produce a meaningful training sample should default to HNSW rather than force an IVF configuration. OpenSearch 3.0's move to Apache Lucene 10 changed the performance profile of the Lucene engine specifically: parallelized I/O for vector segments and an asynchronous fetch API were added, which OpenSearch's own benchmarking describes as making OpenSearch 3.0 up to 8.4x more performant in aggregate than OpenSearch 1.3 — a vendor-published, aggregate figure spanning more than just vector workloads, not a like-for-like k-NN query benchmark.
........
Engine | Algorithms | Quantization support | Operational notes |
Lucene | HNSW | Built-in scalar (int8) quantization | Pure JVM, no native library dependency; tied to the Lucene version OpenSearch ships; vector I/O parallelism improved from OpenSearch 3.0 onward (Lucene 10) |
Faiss | HNSW, IVF | Scalar (fp16, int8), product quantization (PQ), binary | Native JNI library; widest tuning surface; required for IVF and for the most aggressive memory compression |
NMSLIB | HNSW | None / limited | Original k-NN engine, now legacy; new deployments generally use Lucene or Faiss |
........
Maximum supported vector dimensionality is engine-dependent; third-party comparisons report OpenSearch's Faiss path accepting vectors up to roughly 16,000 dimensions, well above the 4,096-dimension ceiling attributed to Elasticsearch's Lucene-only HNSW implementation in the same comparisons — figures worth re-verifying against current documentation before relying on them, since both engines' limits have moved as their underlying Lucene versions changed.
········
NEURAL SEARCH AND HYBRID QUERY FUSION MECHANICS.
How embeddings are produced automatically at ingest and query time, and how lexical and vector scores are reconciled into one ranked result set.
The Neural Search plugin removes manual embedding calls from the indexing path with a text_embedding ingest processor: it is attached to an ingest pipeline, references a model registered through ML Commons — either hosted locally inside the cluster (ONNX or TorchScript models deployed on data nodes) or reached through a remote connector to an external embedding API — and writes the resulting vector into a knn_vector field as each document is indexed. At query time, a neural query clause references the same model ID and text query, and OpenSearch embeds the query and runs the k-NN search server-side; a raw knn query is also available for cases where the vector is already computed by the caller.
Combining that vector search with BM25 keyword search is done through a hybrid compound query, which submits multiple sub-queries — typically a match clause and a neural clause — against the same index in one request. The two sub-queries return scores on incompatible scales: BM25 scores are unbounded and corpus-dependent, while cosine or dot-product similarity scores are bounded, so combining them directly would let whichever sub-query happens to produce larger numbers dominate the ranking regardless of actual relevance. A search pipeline attached to the index handles this with a normalization processor — min-max or L2 normalization bring both score sets onto a comparable range — followed by a combination technique (arithmetic mean, geometric mean, or harmonic mean) that merges the normalized scores, with per-technique weights that must sum to 1.0 to control how much each retrieval method contributes.
OpenSearch 2.19 added reciprocal rank fusion (RRF) as an alternative to score normalization inside the same hybrid query framework. RRF ignores the underlying score values entirely and fuses results by rank position: rankScore(doc) = sum(1 / (k + rank)) across the contributing sub-queries, with a default rank constant k of 60. Because it never touches raw scores, RRF is insensitive to outlier values and mismatched score distributions between BM25 and vector search, at some cost to ranking quality — OpenSearch's own benchmark across six internal test datasets reported RRF scoring roughly 3.86% lower on NDCG@10 than normalization-based fusion, while improving p50 query latency by about 1.62%; both figures are dataset-specific and should not be read as guaranteed on other corpora. A separate OpenSearch optimization study, run on its own hybrid-search test set, found L2 normalization combined with arithmetic-mean fusion in the majority of its best-performing configurations, and found that tuning fusion weights per query rather than fixing one global weight improved DCG@10 by roughly 9–10% over the static baseline — again a result specific to that dataset and query mix, offered by OpenSearch as a starting point for tuning rather than a fixed rule.
········
RAG PIPELINES, ML COMMONS CONNECTORS, AND DEPLOYMENT COSTS.
How the retrieval-augmented-generation processor, external model connectors, and hosting choice determine what running OpenSearch for RAG actually costs.
Beyond retrieval, OpenSearch's search pipelines include a retrieval_augmented_generation response processor for conversational search: it takes the hits already retrieved by a query — lexical, vector, or hybrid — forwards them together with the user's question and prior conversation turns to a large language model reached through an ML Commons connector, and returns the generated answer alongside, or instead of, the raw hit list. Conversation state is kept in a dedicated memory index so multi-turn context persists across requests. This capability shipped as an experimental feature in the neural-search and ML Commons plugins and has continued to change across OpenSearch's frequent 2.x and 3.x releases, so its exact configuration surface should be checked against the version actually deployed rather than assumed stable from one release to the next. The LLM connector layer is provider-agnostic: OpenSearch's own documentation and tutorials show connectors configured against OpenAI, Cohere, Anthropic Claude, Amazon Bedrock, and DeepSeek, which means the generation step's cost and rate limits are set by whichever provider is connected, entirely separate from OpenSearch's own billing.
OpenSearch itself carries no license fee under Apache 2.0, but that only removes one cost line. Self-managed clusters shift node sizing, scaling, backups, and security patching entirely onto the operating team. Amazon OpenSearch Service is the most common managed path and bills per instance-hour by instance type plus attached storage and data transfer — a memory-optimized r6g.large.search instance (2 vCPU, 16 GiB memory) runs approximately $0.167 per hour on demand in US East (N. Virginia) as of this writing, before storage and any reserved-instance discount, and rates vary by region and change over time. OpenSearch Serverless offers a compute-unit-based billing alternative for teams that would rather not size clusters manually, at the cost of less direct control over version and engine choices. Third-party cost comparisons report Amazon OpenSearch Service running 30–50% cheaper than Elastic Cloud for equivalent workloads — a claim from independent comparison sites rather than from AWS or Elastic directly, and one that depends heavily on the specific instance mix and data volume being compared.
........
Deployment path | License / governance | Billing model | Operational burden |
Self-managed OpenSearch | Apache 2.0; OpenSearch Software Foundation (Linux Foundation) | Infrastructure cost only, no license fee | Full: sizing, patching, scaling, backups |
Amazon OpenSearch Service | Same OSS core, AWS-operated | Per instance-hour (e.g., ~$0.167/hr for r6g.large.search on demand) plus storage and data transfer | Low-medium: AWS handles provisioning and patching |
OpenSearch Serverless | Same OSS core, AWS-operated | Compute-unit-based, auto-scaled | Lowest: no cluster sizing, but less version/engine control |
........
········
WHEN OPENSEARCH IS THE RIGHT RETRIEVAL LAYER.
Where the combined lexical, vector, and RAG stack earns its place over Elasticsearch or a dedicated vector database, and where its trade-offs show up.
OpenSearch's strongest case is a team that already runs OpenSearch or Elasticsearch for logging, observability, or existing keyword search and needs to add vector retrieval without introducing a second database and a second operational surface. Because hybrid queries run BM25 and vector search in the same request against the same index, teams get lexical-plus-semantic fusion — useful when queries mix exact terms such as product codes, names, or error strings with conceptual similarity — without maintaining a synchronization pipeline between a search engine and a separate vector store. The Apache 2.0 license and Linux Foundation governance also matter to teams that specifically moved off Elasticsearch to avoid the Elastic License's restrictions on offering the software as a competing service.
The trade-offs sit on the operational and maturity side. Engine choice is fixed at index creation, so a wrong initial choice between Lucene and Faiss means a full reindex, not a setting change. IVF requires enough training vectors to build usable centroids, which makes it a poor fit for small or fast-growing collections that should start on HNSW instead. Quantization settings — scalar, PQ, binary — cut memory substantially, but each step down in precision needs to be validated against the target corpus's own recall rather than assumed from published defaults. The RAG-specific layer, the retrieval_augmented_generation processor, conversational memory, and the newer agent-orchestration features arriving in the 3.x line, is still moving fast enough between releases that teams should treat it as an actively evolving capability rather than a fixed API to build long-term automation against. Dedicated vector databases remain a narrower but more predictable choice when the workload is pure ANN search at very large scale, with no lexical component and no need to share a cluster with existing search infrastructure.
The decision rule: default to OpenSearch when vector search is being added to a workload that already lives in OpenSearch or Elasticsearch and needs BM25-plus-vector fusion or an in-pipeline RAG step under an Apache 2.0 license; choose a dedicated vector database instead when the vector workload would dominate cluster sizing on its own, or when the primary requirement is the lowest achievable ANN latency at large scale, which a general-purpose search engine built around lexical search first is not optimized to deliver ahead of a purpose-built vector store.
FOLLOW US FOR MORE.
·····
DATA STUDIOS
·····
