Advancing the Science
of Retrieval
Our engineering is driven by rigorous research. We publish our findings to push the boundaries of what AI agents can understand and retrieve.
Four open problems, one goal
Every part of the stack — discovery, ranking, extraction, and scale — is an active area of research, not a solved problem we inherited off the shelf.
Neural Retrieval
Hybrid BM25 and dense embedding search, fused with reciprocal rank fusion and tuned for agent queries — not ten blue links for a human to skim.
Ranking & Link-Prediction
Next-link-prediction models that learn how the web actually connects, replacing brittle heuristics like classic PageRank and domain authority.
Structured Extraction
Turning raw HTML into typed, citable passages — boilerplate stripped, entities and provenance attached, ready for grounded generation.
Billion-Scale Indexing
Sharded, self-refreshing indexes that stay fresh across billions of pages with conditional re-crawl and zero manual operation.
Research that ships
Findings don't stay in preprints — they land in the production index within weeks. Here's what that's produced so far.
Fewer hallucinations on complex RAG tasks vs. flat-index search
Latency from publication to index for streaming sources
Pages targeted by our self-refreshing, independent crawl
Publications
A selection of our recent papers, spanning retrieval architecture, streaming infrastructure, and evaluation methodology.
Structured Retrieval for Autonomous Agents
Chen, A., Smith, B., et al.
We introduce a novel data lake architecture designed specifically for LLM consumption. By pre-structuring web data into typed entities, we demonstrate a 40% reduction in hallucination rates during complex RAG tasks compared to traditional flat-index search engines.
Low-Latency Indexing of Financial Streams
Patel, R., Chen, A.
This paper details the streaming infrastructure behind Serpex's 'Streams' zone. We propose a new method for real-time entity extraction and sentiment analysis on SEC filings and earnings call transcripts, achieving sub-100ms latency from publication to index.
Evaluating Search APIs for Code Generation
Smith, B., Lee, J.
A comprehensive benchmark of search APIs used as tools for code-generating LLMs. We show that semantic indexing of code repositories combined with API documentation significantly outperforms standard keyword search in resolving complex programming queries.
Partner with us on retrieval, grounding, and reasoning
We actively partner with academic institutions and research labs pushing the boundary of what agents can retrieve and reason over.