Retrieval at the Source: Embeddings and Search Near the Data

Retrieval-augmented generation, introduced by Lewis et al., combines a model's built-in knowledge with documents fetched at query time. It has become the default way enterprises connect language models to their own data. It also quietly creates new copies of that data: chunks, embeddings, indexes, caches and logs, often in a different system, a different cloud and sometimes a different country from the source.

This article argues that retrieval should run as close to the data as possible, and sets out an architecture for doing it.

Embeddings are personal data in disguise

A common assumption is that embedding vectors are an anonymised summary. They are not. Morris et al. showed at EMNLP 2023 that a method which iteratively corrects and re-embeds text could recover 92 percent of 32-token inputs exactly from their embeddings, and could recover full names from a dataset of clinical notes.

Two consequences follow:

• Residency rules apply to vectors. If the source text is personal or regulated data, treat its embeddings the same way for transfer, retention and erasure.
• The embedding service sees everything. Sending documents to a remote embedding API is a data transfer, with all the contractual and jurisdictional questions that implies.

The takeaway: an index is a copy of your data. Place and protect it as one.

The cost is in the harness, not the model call

Retrieval costs accumulate around the model: embedding generation, vector storage, index rebuilds, reranking and data egress. The FinOps Foundation reports that in production retrieval and agentic deployments, the infrastructure surrounding the model call routinely represents 40 to 60 percent of total feature spend. Moving documents across regions to embed them, and moving vectors back to search them, shows up as both egress and duplicate storage.

Choosing and sizing embeddings

Three choices dominate cost and quality:

Model. Benchmarks help but do not decide. MMTEB covers more than 500 evaluation tasks across over 250 languages, and found that while billion-parameter models lead on some languages and task categories, the best-performing publicly available model overall had only 560 million parameters. Evaluate on your own documents and languages; a small open model you can run next to the data often matches a large remote one.

Dimensions. Vector size drives storage and search cost linearly. Matryoshka representation learning trains embeddings so their leading dimensions remain usable on their own, reporting up to 14 times smaller embeddings at the same accuracy on ImageNet classification and up to 14 times faster real-world retrieval. Models trained this way let you store short vectors for a first pass and use full vectors only for reranking.

Rerank instead of over-retrieve. A cross-encoder reranker scoring a short list is usually cheaper than retrieving and stuffing many more chunks into a long prompt.

An architecture for retrieval near the source

1. Embed where the data lives. Run the embedding model on hardware in the same site or jurisdiction as the source system. Small open embedding models run on CPU servers or modest GPUs.
2. Index locally, federate queries. Keep an index per data domain or jurisdiction. A query fans out to the relevant indexes and merges results, instead of centralising all vectors.
3. Send only what the generator needs. Pass the top reranked passages to the generation model, not the corpus. If the generator must be remote, the exposure is limited to those passages, and they can go to a provider in an approved jurisdiction or inside a confidential enclave.
4. Gate every read. Apply the same access policy to an index entry as to the document it came from. A user who cannot open a file should not retrieve its chunks.
5. Erase end to end. Deleting a source document must delete its chunks, vectors and cached answers. Content addressing makes this traceable: every derived object can name the hash of its source.

Regional notes

Under the GDPR in the EU and the UK GDPR, embeddings of personal data are personal data processing, and transfers outside the region need a legal basis. Sector rules in finance and health add stricter localisation in many APAC and MENA jurisdictions. Federated, per-jurisdiction indexes avoid most of these questions by design.

What this means for data scientists and CTOs

• Classify indexes with the same sensitivity label as their source.
• Benchmark small open embedding models on your own corpus before defaulting to a remote API.
• Use shorter vectors where quality allows, and rerank.
• Budget for the harness: storage, rebuilds and egress, not just tokens.

Where Tenzro fits

Tenzro lets machines of every size serve models and hold data. For retrieval, what is built today:

• Embedding and reranking models. Providers serve text embeddings (Qwen3-Embedding 0.6B, 4B and 8B, EmbeddingGemma-300M with Matryoshka dimensions, BGE-M3, Snowflake Arctic Embed, ModernBERT-embed and a multimodal Qwen3-VL embedding model), vision embeddings, and cross-encoder rerankers. Embeddings bill per input token and reranking per query-document pair.
• Placement you can see. Providers declare a coarse region and jurisdiction, so an application can choose an embedding provider in the right place. An operator can also run a node as private (never announced, used only by its own callers) or gated (only callers its access policy names), keeping embedding next to the data on owned hardware.
• Storage with access control. Objects are content-addressed, every retrieval is checked against an access policy (owner only, an allowlist of identities, or a required capability), and sensitive objects can be encrypted at rest with one wrapped key per authorised identity. Providers are paid per byte-epoch only while they pass proof-of-retrievability challenges.
• Erasure of identity data. The identity layer supports GDPR Article 17 erasure for identities.

Not built as a protocol feature: a vector index or vector search service. Those are applications built over the embedding, rerank and storage surfaces. The network is live in preview. See tenzro.com.