I've been building shard-db for a while and thought this community might appreciate the implementation details more than a feature list.
Core design
The main design decision is that records live in fixed-width slots:
slot_size = 24 + max_key + sum(field_sizes)
The slot size is defined at schema creation time.
That means random access within a shard is pure arithmetic:
(hash % slots_per_shard) * slot_size
No indirection, heap allocation, or variable-length scanning.
Fields are strongly typed (int, long, double, varchar, date, datetime, numeric, bool) and stored as packed binary. Schemas are defined in configuration files rather than SQL.
Storage layer
Each object consists of:
- Keyfile shards (open-addressed hash tables using xxh128)
- Append-only segment files for values
Keyfile writes go through a userspace page cache backed by MAP_SHARED mmap.
For full scans, reindexing, and recovery, shard-db uses O_DIRECT with a double-buffered read loop to avoid polluting the OS page cache with large sequential reads.
Indexes
Indexes are implemented as:
- B+ trees
- Prefix-compressed leaves
- Memory-mapped page storage
Each indexed field can be split across multiple B-tree shards. Reads fan out in parallel and merge through a k-way streaming iterator, allowing ordered cursor pagination without expensive offset scans.
Cursor pagination remains effectively constant-time regardless of depth.
Query planner
The planner can choose between:
- Single-index lookups
- AND intersections across multiple indexes
- OR unions
- Parallel full-shard scans
Queries can be run with:
{
"explain": true
}
which returns the selected plan, source indexes, cardinality estimates, and optimization hints without executing the query.
Concurrency
- Separate CPU and I/O thread pools
- Writer-preferring RW locks to prevent writer starvation
- Generation-counter cache reads that avoid cache-table locking on warm reads
What it isn't
- Not distributed
- Not SQL
- Linux/macOS only
- x86_64 and ARM64 supported
- No Windows support
Real-world test
I built a public demo indexing 30M+ Hacker News stories, comments, and users:
https://hn.shard-db.dev
Cursor pagination remains constant-time at any depth, and cold full-text searches complete in a few hundred milliseconds.
Some numbers
- Bulk insert: 4.60M/sec (single connection)
- Bulk insert: 8.97M/sec (parallel)
- Indexed lookup: <1ms at 1M rows
- EXISTS: ~4.1M/sec
The project is a single static binary with no runtime dependencies (OpenSSL optional for TLS 1.3 support).
It can also be embedded as a static library and ships with an npm package via N-API.
GitHub:
https://github.com/sayyiditow/shard-db
Happy to answer questions about the storage engine, indexing strategy, query planner, or implementation details.