Index-free search

Every other search engine makes you ingest data before you can query it — which means the cost of keeping an archive searchable scales with the size of the archive, whether anyone searches it or not. Lucenia removes that step.

how it works

A codec, not a connector

Index-free search is implemented as a custom Lucene codec, not a separate query engine bolted on the side. Lucenia maps the physical layout of a Parquet file directly onto Lucene’s APIs: a row becomes a document, a column becomes a field, a file becomes a segment. Query execution reads the source file’s data pages directly.

The mounted file stays authoritative and is never modified. Deleting the mounted index leaves the source untouched.

s3://archive/2019-Q3/scenes.parquet
        │
        │  mount                    no copy, no pipeline
        ▼
read-only index                     range · term · _id · numeric aggs
        │
        │  POST /_reindex           when a slice earns it
        ▼
full index                          + full-text · + k-NN · + writes
what it changes

Cold data costs storage, not ingest

For archives measured in hundreds of terabytes, this changes what is affordable to keep queryable. Typical patterns:

PatternHow it plays out
Pre-ingest triageexplore a dataset with real queries — validate schema and data quality before committing to a full indexing pipeline
Cold-tier queriesrun occasional searches over archival data in object storage without building or paying for ingest
Hot-slice promotionwhen a slice of the archive earns full search, promote it with a standard _reindex — full-text, vectors, and writes from that point on
Source-of-truth preservationthe file in your bucket remains the system of record; search never forks it
// promote a mounted slice to a full index
POST /_reindex
{
  "source": { "index": "scenes_2019_q3" },
  "dest":   { "index": "scenes_2019_q3_full" }
}

Limitations

  • Mounted files support doc-values range, term/_id queries, and numeric aggregations — not full-text.
  • Read-only; one file maps to one segment in one shard.
  • Promotion to a full index is an explicit reindex, not automatic.

How big is your archive?

If the answer is measured in terabytes, this is the feature to evaluate first.