Python · Open Source · MIT · v1.0.1
Turn messy document folders into governed, hybrid-searchable datasets for AI agents.
Overview
Pure vector search misses exact terms. Keyword search misses semantic matches. ai-ready-data combines OpenAI-powered metadata enrichment with BM25 + dense hybrid retrieval — so agents can find the right document whether the query is precise or paraphrased.
For teams building agents over professional document folders: legal, property, due diligence, compliance, public-sector research, and operations.
Evidence
Measured on a clean technical documentation corpus and a mixed real-world document corpus. Hybrid mode is the default.
Corpus 1 — Pine Script v6 docs
11-query oracle · 303 chunks · text-embedding-3-small 512 dims
| Mode | Hit@1 | Hit@3 | MRR |
|---|---|---|---|
| vector | 10/11 | 8/11 | 0.745 |
| hybrid | 11/11 | 11/11 | 1.000 |
Three persistent failures in vector-only mode — “repainting”, “lookahead bias”, “plot shapes” — are resolved by hybrid. BM25 over the enriched searchable_text directly matches the exact terms.
Corpus 2 — Mixed documents
10-question oracle · 9 files · 8/9 extracted · 1 expected EmptyText
| Mode | Hit@1 | MRR | n |
|---|---|---|---|
| vector | 0.833 | 0.867 | 6 |
| hybrid | 0.900 | 0.925 | 10 |
n=6 shared questions for the hybrid/vector comparison. The scanned image-only PDF produced the expected EmptyText failure — confirming the vision extraction gap.
Pipeline
Extract
PDF (text-layer), DOCX, HTML, CSV, JSON, Markdown, plain text, and code files. Scanned PDFs with no text layer fail cleanly with an EmptyText signal.
Filter
OpenAI decides whether each document is relevant to the dataset goal before enrichment runs.
Enrich
OpenAI structured output extracts topics, document type, sensitivity, jurisdiction, effective date, routing summary, key questions, and a use_when routing hint.
Index
searchable_text — title + metadata + summary + key questions + chunk text — is indexed by BM25 and by a dense embedding model (text-embedding-3-small, 512 dims).
Search
Hybrid RRF: BM25 and dense search run independently. Their candidate lists are fused with Reciprocal Rank Fusion. Hybrid is the default mode.
Design
Four ideas that distinguish this from a plain embedding pipeline.
Enrichment metadata is concatenated into every chunk before BM25 indexing. BM25 over that enriched representation directly matches exact terms in the summary and key questions — not just the raw chunk text.
Dataset-level schema governance. Define which fields are extracted, their value types (string, number, boolean, multi-value), allowed values, and projection policy: include → searchable_text + index, metadata_only → index payload only, exclude → stored but not indexed.
Named pre-set filter views over a dataset. Constrain an agent's search to a governed subset without re-ingesting — create a slice with a filter expression, then search within it.
Independent BM25 and dense search candidates are fused with Reciprocal Rank Fusion. No alpha weights to tune. Vector, keyword, and hybrid modes are all available.
Surfaces
Every surface uses the same Retriever code path. The domain MCP template ships a pre-built wrapper for production agent integrations.
Limitations
Documented honestly. These are the known gaps as of v1.0.1.
No vision extraction yet
Scanned PDFs with no text layer produce an EmptyText failure and are excluded. A vision fallback (gpt-4o-mini, opt-in) is planned but not yet built.
No auth or multi-tenant enforcement
Tenants are stored separately but no request-level auth gates access. Not suitable for hosted multi-user deployments without adding an auth layer.
Synchronous ingestion
The pipeline is CLI-first and blocks the caller. Large corpora (thousands of files) will be slow.
No UI
Dataset management, metadata governance, and search are CLI, REST API, and MCP only.
Built and maintained by BOUCH. If you are building agents over professional document corpora and want help adapting the pipeline — book a call.
What ships with it