Implementing Stores

This guide provides comprehensive instructions for implementing custom metadata stores, local block stores, and remote block stores for DittoFS. Whether you’re building a database-backed metadata store or a custom cloud storage integration, this document will walk you through the process with best practices and practical examples.

Overview
When to Implement Custom Stores
Understanding the Architecture
Implementing Metadata Stores
Implementing a Local Store
Implementing a Remote Store
Implementing a Remote Block Store (block-keyed)
Best Practices
Testing Your Implementation
Common Pitfalls
Integration with DittoFS

Overview

DittoFS uses a two-tier block store architecture with three distinct store types:

Metadata Stores: Simple CRUD operations for file/directory structure, attributes, permissions
Local Block Stores: Fast, per-share storage on local disk or in memory (L2 cache tier)
Remote Block Stores: Durable storage in S3 or compatible object stores (L3 tier, shared across shares via ref counting)

Key Design Principle: Each share gets its own *engine.BlockStore instance that composes a local store, optional remote store, and syncer. The engine orchestrates reads and writes across the tiers. Local stores provide fast access; remote stores provide durability.

This separation enables:

Independent scaling of metadata and block storage
Per-share isolation with shared remote backends
Different storage tiers (hot/cold storage, SSD/HDD)
Simple store implementations (just implement the interface, the engine handles coordination)

When to Implement Custom Stores

Metadata Store Use Cases

Implement a custom metadata store when you need:

Database-backed storage: PostgreSQL, MySQL, MongoDB, Cassandra
Distributed metadata: Multi-node coordination, consensus protocols
Advanced features: Full-text search, custom indexing, complex queries
Compliance: Audit logs, versioning, immutability guarantees

Example: A PostgreSQL-backed metadata store for enterprise environments requiring audit trails and high availability.

Local Store Use Cases

Implement a custom local store when you need:

Specialized local storage: NVMe-optimized, hardware-accelerated compression
Custom eviction: Access-pattern-aware eviction beyond simple LRU
Encryption at rest: Hardware-accelerated encryption for local blocks

Reference implementation: pkg/block/local/fs/ (filesystem-backed local store)

Remote Store Use Cases

Implement a custom remote store when you need:

Cloud storage integration: Azure Blob, Google Cloud Storage, custom object stores
Specialized storage: Tape archives, HSM systems, data lakes
Tiering: Automatic hot/cold data movement based on access patterns

Reference implementation: pkg/block/remote/s3/ (S3-backed remote store)

Understanding the Architecture

Each share gets its own *engine.BlockStore instance:

┌─────────────────────────────────────┐
│  engine.BlockStore (per-share)      │
│                                     │
│  ┌─────────────┐  ┌─────────────┐  │
│  │ LocalStore  │  │ RemoteStore │  │
│  │ (required)  │  │ (optional)  │  │
│  └──────┬──────┘  └──────┬──────┘  │
│         │                │          │
│         └───────┬────────┘          │
│                 │                   │
│          ┌──────▼──────┐            │
│          │   Syncer    │            │
│          │ (async xfer)│            │
│          └─────────────┘            │
└─────────────────────────────────────┘

LocalStore is required — all reads and writes go through local storage first
RemoteStore is optional — when configured, the Syncer asynchronously uploads local blocks to remote storage
Ref counting: Remote stores are shared across shares; when the last share using a remote store is removed, the connection is closed

File Handle and Block Resolution

Protocol handlers resolve the per-share block store via GetBlockStoreForHandle(ctx, handle):

File handle encodes the share name
Runtime extracts share name and returns the share’s BlockStore
Handler calls ReadAt / WriteAt on the BlockStore

Implementing Metadata Stores

The metadata store interface and implementation guide remains the same as before. See the pkg/metadata/Store interface and reference implementations:

pkg/metadata/store/memory/: In-memory (fast, ephemeral)
pkg/metadata/store/badger/: BadgerDB (persistent, embedded)
pkg/metadata/store/postgres/: PostgreSQL (persistent, distributed)

Conformance tests: pkg/metadata/storetest/

MetadataStore.EnumerateFileChunks

MetadataStore carries a mandatory cursor method that the mark-sweep garbage collector uses to enumerate every live block hash without loading the full file/block set into application memory.

Note: EnumerateFileChunks lives on MetadataStore, not FileChunkStore. Conceptually it iterates across files for the GC mark phase — a metadata-store-wide concern — so it sits above the narrow FileChunkStore surface (see FileChunkStore narrowing below).

// EnumerateFileChunks streams every FileChunk's ContentHash to fn.
// Implementations MUST:
//   - Iterate using a backend-native cursor (Badger prefix iterator,
//     Postgres server-side cursor with batched fetch, in-memory map
//     iteration) -- no full-set load.
//   - Honor ctx.Done(): return ctx.Err() promptly when the context is
//     cancelled.
//   - Emit zero-hash FileChunks the same way as non-zero-hash blocks;
//     the GC live-set ignores zero hashes.
//   - Abort iteration and return the fn error verbatim if fn returns
//     non-nil; do NOT swallow it.
//   - Be safe under concurrent writes: it is acceptable for the cursor
//     to miss FileChunks created mid-iteration; the next mark cycle
//     will pick them up.
EnumerateFileChunks(ctx context.Context, fn func(ContentHash) error) error

Conformance scenarios live in pkg/metadata/storetest/ and every backend MUST pass them:

Empty store: fn is never invoked; returns nil.
Single file: fn is invoked once per FileChunk for the file.
Large fanout (N files × M blocks): fn is invoked exactly N*M times in any order; no duplicates, no omissions.
fn-error mid-iteration: returning a non-nil error from fn aborts iteration and propagates the error.
Context cancellation: cancelling ctx mid-iteration causes the call to return ctx.Err() within the polling interval.

Memory-store reference: direct range over the in-memory map. Badger-store reference: txn.NewIterator over the FileChunk prefix. Postgres-store reference: server-side cursor (DECLARE + FETCH) with batches of 1000 rows.

FileChunkStore narrowing

pkg/block.FileChunkStore is the narrow public FileChunk surface. Backend implementations are simpler than the engine-internal helpers, which live on a separate wider interface.

type FileChunkStore interface {
    // GetByHash returns any FileChunk with the given content hash, or
    // (nil, nil) when absent (multiple rows may share a hash; best-effort).
    GetByHash(ctx context.Context, hash ContentHash) (*FileChunk, error)

    // Put creates or replaces a FileChunk by ID (upsert by ID, not hash).
    Put(ctx context.Context, block *FileChunk) error

    // Delete removes a FileChunk by ID. Returns ErrFileChunkNotFound if absent.
    Delete(ctx context.Context, id string) error

    // IncrementRefCount atomically bumps RefCount for the given FileChunk id.
    IncrementRefCount(ctx context.Context, id string) error

    // DecrementRefCount atomically decrements; returns the new count.
    DecrementRefCount(ctx context.Context, id string) (uint32, error)

    // DecrementRefCountAndReap atomically decrements and, if the count hits 0,
    // deletes the row in the same critical section. Returns the new count.
    DecrementRefCountAndReap(ctx context.Context, id string) (uint32, error)

    // AddRef atomically increments RefCount on the row indexed by hash
    // (the dedup LRU hit path). Returns ErrUnknownHash if no row exists.
    AddRef(ctx context.Context, hash ContentHash, payloadID string, blockRef BlockRef) error
}

Engine-internal companion interface: pkg/block.EngineFileChunkStore extends FileChunkStore with GetFileChunk(ctx, id) and ListFileChunks(ctx, payloadID) for the engine’s hot paths. All three built-in backends (memory, badger, postgres) satisfy it without changes — the narrow public surface is a documentation concern, not a runtime restriction. Custom backends implementing FileChunkStore SHOULD also implement the engine-internal helpers if they intend to slot into the *engine.BlockStore.

Internal storage shape is up to the backend. The built-in schema (id VARCHAR PRIMARY KEY + hash non-unique index) permits multiple rows per hash for older data; the public GetByHash surface hides that detail.

FileAttr.Blocks []BlockRef

FileAttr.Blocks []blockstore.BlockRef is the authoritative content list for every file. BlockRef is the 3-tuple (Hash, Offset, Size) — see pkg/block/types.go. The list MUST be sorted by Offset and is populated on every sync finalization.

Encoding requirements per backend:

Postgres: a separate file_block_refs join table keyed by (file_id, offset), with INCLUDE (size, hash) for index-only scans on the read hot path. Foreign key file_id REFERENCES files(id) ON DELETE CASCADE provides a safety net — the engine still decrements file_blocks.RefCount for every BlockRef BEFORE deleting the file; cascade catches engine-bug paths that miss the explicit decrement. Hash column is BYTEA (32 bytes), not hex TEXT.
Badger and Memory: inline-encode Blocks []BlockRef inside the existing FileAttr blob. Badger goes through pkg/metadata/store/badger/encoding.go (gob); Memory holds typed structs directly. Use omitempty so older blobs decode cleanly with an empty Blocks slice.

A new metadata-store method persists the list; in the built-in backends this is MetadataStore.SetFileChunks(ctx, handle, []BlockRef, authCtx) error. Custom metadata backends MUST persist atomically with the same transaction that updates Size/Mtime/Ctime — the engine relies on caller-side metadata-txn isolation rather than a per-chunk metadata roundtrip.

Conformance scenarios

The pkg/metadata/storetest/ suite includes:

BlockRef round-trip: SetFileChunks followed by GetFileAttr returns the same offset-sorted slice, byte-for-byte.
Empty / legacy compat: FileAttr blobs without a Blocks field decode to an empty slice without errors.
FK cascade (Postgres-only): deleting a file removes all matching file_block_refs rows.
Refcount reconcile: ∑ FileChunk.RefCount over the FileChunkStore equals ∑ len(FileAttr.Blocks) over the MetadataStore at every quiescent point.
Refcount concurrent fuzz (pkg/metadata/storetest/inv02_fuzz_test.go): 100-iteration property-based fuzzer creating, deleting, and copying files concurrently; asserts the invariant after each operation batch. Runs against all three built-in backends and any custom backend wired into the conformance harness.

FileAttr.ObjectID + FindByObjectID

FileAttr.ObjectID is a BLAKE3 Merkle root over BlockRef.Hash values sorted by Offset, prefixed by the domain-separation tag dittofs:objectid:v1\x00. Computed by blockstore.ComputeObjectID and persisted at every full quiesce in the same metadata transaction that updates Blocks/Size/Mtime.

Lifecycle: cleared (zeroed) on first dirty write that mutates Blocks, recomputed at next full quiesce (every block transitioned to Remote). Partial flushes leave ObjectID at zero so the lookup index never returns a half-quiesced file.

`FindByObjectID(ctx, ObjectID) ([]BlockRef, error)`

The file-level dedup short-circuit primitive. Looks up a file by its Merkle-root ObjectID. Returns (nil, nil) on miss; a non-nil result carries the canonical BlockRef list of the matching file (per-metadata- store scope, NOT per-share).

Backends MUST maintain a secondary index from ObjectID to file row:

Backend	Index
Postgres	Partial unique: `files_object_id_idx ON files(object_id) WHERE object_id IS NOT NULL`
Badger	Secondary key `obj:{hex} -> file_id`, maintained inside each `Put`/`Delete` write batch
Memory	`map[ContentHash]uuid`, guarded by the existing store mutex

Zero-valued ObjectID (legacy / pre-quiesce) MUST NOT match any row — implementations short-circuit and return (nil, nil) on zero input.

The unique constraint enforces first-committer-wins on a concurrent quiesce. On race, the loser surfaces a backend-specific unique-violation error that the runtime coordinator wraps into the shared metadata.ErrConflict sentinel.

A test-only optional capability ObjectIDIndexAccessor.CountObjectIDIndexRows is exercised by the storetest ConcurrentQuiesceRace scenario; backends implement it inline (e.g., SELECT count(*) for Postgres, txn.Get(keyObjectID(oid))-shape for Badger, direct map probe for Memory). Production code MUST NOT call it.

Conformance scenarios live in pkg/metadata/storetest/objectid_roundtrip.go and pkg/metadata/storetest/objectid_lookup.go. All built-in backends pass without per-backend t.Skip (the ObjectIDIndexAccessor capability is the only legitimate type-assertion-skip; backends without that accessor are still required to pass the functional scenarios).

Block Record Store

BlockRecordStore persists the bookkeeping needed by the blocks-only storage path. All four backends (memory, badger, sqlite, postgres) implement it and pass the corresponding conformance group. (There is no separate local-chunk-index metadata contract: the local journal owns its own (payloadID, offset)-keyed byte cache internally, and the per-file FileChunk manifest — written by the carve BlockSink — is the only metadata record of a carved chunk.)

BlockRecordStore

Tracks each packed remote block object: its content hash, byte length, live chunk count, and sync state. The carver uses this to decide which blocks are safe to upload; the GC uses it to decide which may be deleted.

type BlockRecord struct {
    BlockID        string
    BlockHash      block.ContentHash
    Length         int64
    LiveChunkCount uint32
    SyncState      block.BlockState // block.BlockStatePending = 0 while uploading
}

type BlockRecordStore interface {
    // PutBlockRecord writes or overwrites the record for rec.BlockID.
    PutBlockRecord(ctx context.Context, rec block.BlockRecord) error

    // GetBlockRecord retrieves the record for blockID.
    // Returns (_, false, nil) when no record exists — absence is not an error.
    GetBlockRecord(ctx context.Context, blockID string) (block.BlockRecord, bool, error)

    // DeleteBlockRecord removes the record for blockID. Idempotent.
    DeleteBlockRecord(ctx context.Context, blockID string) error

    // WalkBlockRecords calls fn for every stored block record.
    // Returns the first non-nil error from fn or from the store iterator.
    WalkBlockRecords(ctx context.Context, fn func(block.BlockRecord) error) error

    // DecrLiveChunkCount atomically decrements LiveChunkCount for blockID
    // by delta, flooring at 0. Returns the remaining count.
    // Returns an error if blockID does not exist.
    DecrLiveChunkCount(ctx context.Context, blockID string, delta uint32) (remaining uint32, err error)
}

Conformance: storetest group BlockRecordOps covers put/get round-trip, missing-key (returns false, not an error), delete idempotency, walk, and DecrLiveChunkCount with floor clamping.

DefaultCommitBlock — one fsync per block

metadata.DefaultCommitBlock atomically commits an entire block’s metadata in a single transaction — one fsync per block, not one per chunk:

func DefaultCommitBlock(
    ctx context.Context,
    s Transactor,
    rec block.BlockRecord,
    chunks []block.BlockChunkCommit,
    fileChunks []*block.FileChunk,
) error

BlockChunkCommit bundles the per-chunk commit data:

type BlockChunkCommit struct {
    Hash   block.ContentHash
    Remote block.ChunkLocator // remote locator recorded via MarkSynced
}

Everything commits in a single WithTransaction, so either the whole block is visible or none of it is — a commit error just propagates to the caller, whose requeue logic re-drives the batch. Inside the transaction, DefaultCommitBlock:

Calls GetBlockRecord — if the block record already exists the whole function is a no-op (idempotent restart path; no double-counting, locators untouched).
Calls PutBlockRecord with rec.
Writes each per-file FileChunk manifest row in fileChunks — the carver passes one per chunk (ID = {payloadID}/{offset}, Hash, DataSize); legacy callers pass nil and write no rows.
Records every chunk’s synced marker + remote locator. Locator writes are last-wins (delete-then-mark inside the tx), so the cas→blocks migration can rewrite a standalone locator to point into the new block.

Backends expose CommitBlock on the Store interface and SHOULD delegate to DefaultCommitBlock:

CommitBlock(ctx context.Context, rec block.BlockRecord, chunks []block.BlockChunkCommit) error

Conformance: group CommitBlockOps covers full commit with multiple chunks, idempotent re-commit, and MarkSynced retry after a simulated mid-commit crash.

Engine API surface

Custom block-store implementations that compose into *engine.BlockStore do not see the engine API directly — that surface is consumed by adapters via internal/adapter/common/. For reference, the signatures are:

ReadAt(ctx, payloadID, blocks []BlockRef, dest []byte, offset uint64) (int, error)
WriteAt(ctx, payloadID, currentBlocks []BlockRef, data []byte, offset uint64) ([]BlockRef, error)
Truncate(ctx, payloadID, currentBlocks []BlockRef, newSize uint64) ([]BlockRef, error)
Delete(ctx, payloadID, blocks []BlockRef) error
CopyPayload(ctx, srcPayloadID, srcBlocks []BlockRef, dstPayloadID) ([]BlockRef, error)

blocks is the CAS path with end-to-end BLAKE3 verification.

Implementing a Local Store

Local stores provide fast, per-share block storage. Each share gets an isolated local storage directory.

The LocalStore Interface

The pkg/block/local.LocalStore interface defines the contract. The local tier is the journal (pkg/block/journal/) — an append-only, log-structured write-back cache — so the surface is keyed by (payloadID, offset), not by content hash. See pkg/block/local/local.go for the authoritative definition and per-method contract; the representative methods are:

type LocalStore interface {
    // --- Data plane (payloadID + offset keyed) ---
    WriteAt(ctx context.Context, payloadID string, offset int64, data []byte) error
    ReadAt(ctx context.Context, payloadID string, offset int64, dst []byte) (n int, cold bool, err error)
    Hydrate(ctx context.Context, payloadID string, offset int64, data []byte) error // fill from remote on cold read
    Commit(ctx context.Context, payloadID string) error                             // fsync buffered writes
    FileSize(ctx context.Context, payloadID string) (int64, bool)
    DataExtents(ctx context.Context, payloadID string, fileSize int64) ([][2]uint64, error)
    Truncate(ctx context.Context, payloadID string, newSize int64) error
    Delete(ctx context.Context, payloadID string) error
    ListFiles(ctx context.Context) []string

    // --- Carve (local → remote) + eviction ---
    SetCarveTargets(deduper journal.Deduper, sink journal.BlockSink)
    Carve(ctx context.Context, opts journal.CarveOptions) (journal.CarveResult, error)
    UnsyncedBytes() int64
    Evict(ctx context.Context, targetBytes int64) (journal.EvictResult, error)
    SetEvictionEnabled(enabled bool)
    // ... plus lifecycle (Start, Close), Stats, Healthcheck, and a
    // no-op SetRetentionPolicy retained for admin-path compatibility.
}

A write buffers a dirty range and local-acks without fsync; Commit is the durability point. On a cold read ReadAt returns cold=true and the engine Hydrates the range back from the remote store. Carve packs a file’s dirty ranges into remote blocks via the injected BlockSink (see the carve pass in the architecture doc); Evict frees whole fully-synced segments under pressure.

Reference Implementation

The filesystem-backed store *fs.FSStore (pkg/block/local/fs/) is a thin adapter over *journal.Store: it bridges the string↔journal.FileID keyspace and forwards the data-plane calls; the journal owns segment layout, carve, eviction, and local GC. The in-memory store (pkg/block/local/memory/) is the other reference. There is no separate append-log or rollup tier and no metadata.RollupStore contract — those were removed when the journal replaced the two-tier local design.

Conformance Tests

The journal-native LocalStore surface is (payloadID, offset)-keyed, not the content-addressed block.Store surface, so the blockstoretest suites below (BlockStoreConformance / BlockStoreAppendConformance) do not apply to a local store — they target the CAS block.Store surface that remote stores implement (see Implementing a Remote Store). The in-tree local stores are exercised by their own package tests under pkg/block/journal/ and pkg/block/local/; model a new local store on those.

Implementing a Remote Store

Remote stores provide durable block storage shared across shares via ref counting.

The RemoteStore Interface

The pkg/block/remote.RemoteStore interface defines the contract. Like the local store, remote storage is content-addressed: the interface embeds the CAS block.Store surface (Put, Get, GetRange, Has, Delete, Head, Walk — keyed by block.ContentHash) and adds verification + health methods. See pkg/block/remote/remote.go for the authoritative definition:

type RemoteStore interface {
    block.Store // CAS Put/Get/GetRange/Has/Delete/Head/Walk

    // ReadBlockVerified GETs the object addressed by hash and verifies the
    // body's BLAKE3 hash matches `expected` before returning bytes. Returns
    // block.ErrCASContentMismatch on any verification failure.
    ReadBlockVerified(ctx context.Context, hash block.ContentHash, expected block.ContentHash) ([]byte, error)

    // HealthCheck is the legacy error-returning probe used by the syncer.
    HealthCheck(ctx context.Context) error

    // Healthcheck returns a structured health.Report (satisfies health.Checker).
    Healthcheck(ctx context.Context) health.Report

    // Close releases resources.
    Close() error
}

Implementation Pattern

package myremote

import (
    "context"

    "github.com/marmos91/dittofs/pkg/block"
)

type MyRemoteStore struct {
    client *MyCloudClient
    bucket string
}

func New(config Config) (*MyRemoteStore, error) {
    client, err := connectToCloud(config)
    if err != nil {
        return nil, err
    }
    return &MyRemoteStore{client: client, bucket: config.Bucket}, nil
}

// objectKey derives the storage key from the content hash. The CAS key is
// the hash; backends typically use its hex form, optionally sharded.
func (s *MyRemoteStore) objectKey(hash block.ContentHash) string {
    return hash.String()
}

func (s *MyRemoteStore) Get(ctx context.Context, hash block.ContentHash) ([]byte, error) {
    // Fetch the chunk addressed by hash. Return block.ErrChunkNotFound if absent.
    return s.client.GetObject(ctx, s.bucket, s.objectKey(hash))
}

func (s *MyRemoteStore) Put(ctx context.Context, hash block.ContentHash, data []byte) error {
    // Upload the chunk under its content-hash key (idempotent for identical bytes).
    return s.client.PutObject(ctx, s.bucket, s.objectKey(hash), data)
}

func (s *MyRemoteStore) Delete(ctx context.Context, hash block.ContentHash) error {
    // Remove the chunk (idempotent).
    err := s.client.DeleteObject(ctx, s.bucket, s.objectKey(hash))
    if err != nil && !isNotFoundError(err) {
        return err
    }
    return nil
}

func (s *MyRemoteStore) HealthCheck(ctx context.Context) error {
    // Verify connectivity (e.g., HEAD bucket)
    return s.client.HeadBucket(ctx, s.bucket)
}

func (s *MyRemoteStore) Close() error {
    return s.client.Close()
}

// ... implement the remaining block.Store methods (GetRange, Has, Head, Walk),
// ReadBlockVerified, and Healthcheck — see pkg/block/remote/s3 for a full example.

Ref Counting

Remote stores are shared across shares via ref counting:

When a share is created referencing a remote store, the ref count increments
When a share is removed, the ref count decrements
When the ref count reaches zero, Close() is called

This means your Close() implementation should release all resources (connections, goroutines, etc.).

Reference Implementation

See pkg/block/remote/s3/ for a production S3 remote store implementation with:

Configurable retry with exponential backoff
Health check via HEAD bucket
Efficient multipart uploads for large blocks

CAS contracts

All uploads go through a content-addressable keyspace cas/{hh}/{hh}/{hex}, and every byte downloaded from the remote is verified against the expected BLAKE3 hash. Two contract methods are required for any RemoteStore implementation that participates in the write path.

RemoteStore.WriteBlockWithHash

// WriteBlockWithHash uploads data under the CAS key derived from h
// and sets a backend-native object-metadata header carrying the hash.
//
// Semantics:
//   - The key MUST be derived from h via FormatCASKey (cas/{hh}/{hh}/{hex}).
//   - The backend-native object metadata MUST set "content-hash" to
//     "blake3:" + hex(h). For S3, this becomes the user-metadata header
//     x-amz-meta-content-hash. For other backends, set the equivalent
//     custom-metadata field.
//   - The PUT MUST be atomic: either the object exists at the CAS key
//     with the correct bytes AND the metadata header, or it does not
//     exist at all.
//   - The call MUST be idempotent: re-uploading the same h with the
//     same bytes is a no-op (or an overwrite that yields identical
//     state). This is what makes the syncer's restart-recovery janitor
//     safe.
//   - Errors are returned as typed values mapped through
//     internal/adapter/common/.
WriteBlockWithHash(ctx context.Context, blockKey string, hash ContentHash, data []byte) error

External tooling (e.g. aws s3api head-object) MUST be able to verify the header without DittoFS-specific tooling.

RemoteStore.ReadBlockVerified

// ReadBlockVerified reads the object at the CAS key derived from h
// and verifies its bytes against h end-to-end before returning them
// to the caller.
//
// Semantics:
//   - HEAD-style pre-check: if the backend exposes the content-hash
//     header cheaply (S3 GetObject returns it in the same response,
//     so no extra round-trip is needed), reject early with
//     ErrCASContentMismatch when the header does not match h.
//   - Streaming verification: the body is fed to a blake3.Hasher as
//     the caller reads it. On EOF, hasher.Sum(nil) MUST equal h or
//     the call returns ErrCASContentMismatch and the buffer is
//     discarded -- corrupt bytes MUST NOT be surfaced upstream.
//   - The streaming verifier sees bytes once (zero extra allocation).
//   - Verification is hard-required: there is no opt-out knob.
ReadBlockVerified(ctx context.Context, blockKey string, expected ContentHash) ([]byte, error)

Header pre-check + streaming recompute is “fail-closed twice” by design: the header alone is not sufficient (would trust the backend to never silently corrupt); recompute alone wastes a body read on a definitively-wrong object.

Conformance Tests

Test your remote store with the conformance suite:

package myremote_test

import (
    "testing"

    "github.com/marmos91/dittofs/pkg/block"
    "github.com/marmos91/dittofs/pkg/block/blockstoretest"
)

func TestMyRemoteStore(t *testing.T) {
    factory := func(t *testing.T) (block.Store, func()) {
        store, cleanup := createTestStore(t)
        return store, cleanup
    }
    blockstoretest.BlockStoreConformance(t, factory)
}

The BlockStoreConformance suite pins the CAS contract: Put + Get round-trip with no aliasing, idempotent re-Put of identical bytes, Get/GetRange/Has/Head/Delete semantics, and Walk enumeration. The dedicated ReadBlockVerified path on RemoteStore (round-trip succeeds; body-mismatch returns block.ErrCASContentMismatch; corrupt bytes never surface upstream) is exercised separately — see pkg/block/remote/s3 for the verification tests.

Implementing a Remote Block Store (block-keyed)

The pkg/block/remote.RemoteBlockStore interface is the block-keyed (non-CAS) remote store surface used by the live write path. Every new write is packed into block objects under the blocks/ prefix, separate from the legacy CAS cas/ namespace — the two namespaces never collide. New remote backends should implement RemoteBlockStore.

Legacy CAS reads: The per-chunk CAS object path (cas/<hash>, hash-keyed Get/GetRange on RemoteStore) is read-only and exists purely for backward compatibility with data written before the blocks-only flip. No new cas/ objects are ever written; a later migration re-packs the remaining ones, after which the legacy RemoteStore CAS surface is removed. A backend that only ever serves fresh DittoFS deployments does not need it.

The RemoteBlockStore Interface

type RemoteBlockStore interface {
    PutBlock(ctx context.Context, blockID string, r io.Reader) error
    GetBlock(ctx context.Context, blockID string) ([]byte, error)
    GetBlockRange(ctx context.Context, blockID string, offset, length int64) ([]byte, error)
    DeleteBlock(ctx context.Context, blockID string) error
    WalkBlocks(ctx context.Context, fn func(blockID string, meta block.Meta) error) error
}

See pkg/block/remote/remote.go for the authoritative definition and per-method docstrings. Current implementations: pkg/block/remote/s3/ and pkg/block/remote/memory/.

Key shape

Objects are keyed by an opaque blockID string. The on-disk/on-wire key is block.FormatBlockKey(blockID) = "blocks/<blockID>". Backends derive this key internally; callers and the engine never construct raw S3/object-store keys.

PutBlock

Writes the content of r under blocks/<blockID>. Implementations must stream r without buffering the full body — callers may supply an unbounded reader (e.g., an in-progress packing file). The call is idempotent: a second PutBlock for the same blockID overwrites silently.

GetBlock

Returns the full bytes of the named block object. Returns block.ErrChunkNotFound when the block is absent. The returned slice must be freshly allocated and must not alias internal storage.

GetBlockRange

Returns [offset, offset+length) bytes of the block object. Bounds validation:

Negative offset — must return block.ErrInvalidOffset (client-validated before any network call).
Past-EOF offset — a past-EOF offset cannot be detected here without a HEAD, so backends surface a native error (S3: HTTP 416) rather than ErrInvalidOffset. The contract only requires some error for offset >= EOF.
Non-positive length — must return block.ErrInvalidSize.
Past-EOF length — clamped to the object’s remaining bytes on backends that support partial-content (S3 Range header); no error.
Absent blockID — must return block.ErrChunkNotFound.

DeleteBlock

Removes the block object. Idempotent: deleting an absent blockID must return nil.

WalkBlocks

Enumerates every block object under the blocks/ prefix. The callback receives the blockID (with the blocks/ prefix stripped) and a block.Meta (size, last-modified timestamp). Ordering is unspecified.

Returning block.ErrStopWalk from the callback is a clean early exit — WalkBlocks returns nil.
Any other callback error halts enumeration and is returned wrapped as "walk halted at <blockID>: %w".
Context cancellation aborts immediately.

Block codec wire format

Block objects are written by pkg/block/blockcodec. A single block packs one or more chunk records into a continuous byte stream:

Block = [ preamble ][ record_0 ][ record_1 ] … [ record_{N-1} ]

preamble:
  magic      [4]byte = {'D','F','B','1'}   // "DFB1" — identifies a DittoFS block object
  flags      uint8                          // bit0 = 1 → record headers are AEAD-sealed
  blockID    uvarint(len) + len bytes (UTF-8)

record (plaintext, flags bit0 = 0):
  hash       [32]byte                       // chunk BLAKE3 content hash
  wireLen    uvarint
  wire       [wireLen]byte                  // enc(comp(chunk)) — already-transformed body

record (sealed, flags bit0 = 1):
  sealedHdrLen  uvarint
  sealedHdr     [sealedHdrLen]byte          // AEAD seal of {hash[32], wireLen uvarint}
                                            // AAD = blockID || recordIndex(uvarint)
  wire          [wireLen]byte               // wireLen recovered by opening sealedHdr

Sealed-header framing is used on encrypted shares. The AEAD tag covers hash + wireLen with blockID||recordIndex as additional authenticated data, while the wire body is already the encrypted chunk body (the encryption decorator applies its transform before Builder.Add is called). The chunk metadata is therefore authenticated even though the wire body is opaque.

Locators. Each call to Builder.Add(hash, wire) returns a block.ChunkLocator{BlockID, WireOffset, WireLength} — the byte range of the wire body within the block object. WireOffset and WireLength are over wire bytes (after the per-record header), not plaintext bytes. The engine stores these locators in the metadata store and recovers an individual chunk via GetBlockRange(blockID, WireOffset, WireLength) (surfaced through the optional ChunkReader.ReadChunk extension on the same backend).

Conformance suite

Every new RemoteBlockStore backend must pass blockstoretest.RemoteBlockStoreConformance:

package myremote_test

import (
    "testing"
    "github.com/marmos91/dittofs/pkg/block/blockstoretest"
)

func TestMyRemoteBlockStore(t *testing.T) {
    factory := func(t *testing.T) (blockstoretest.RemoteBlockStore, func()) {
        store, cleanup := createTestStore(t)
        return store, cleanup
    }
    blockstoretest.RemoteBlockStoreConformance(t, factory)
}

The suite covers: PutBlock/GetBlock round-trip, no-aliasing, GetBlockRange mid-range/past-EOF-clamped/invalid-offset/invalid-size/absent, DeleteBlock durability and idempotency, WalkBlocks enumeration and ErrStopWalk, idempotent PutBlock, zero-byte blocks, and concurrent same-ID PutBlock.

Best Practices

Thread Safety

All store implementations must be thread-safe. Multiple goroutines will access the store concurrently.

Context Handling

Always respect context cancellation, especially for remote stores where network calls can be slow:

func (s *MyStore) ReadBlock(ctx context.Context, blockID string) ([]byte, error) {
    if err := ctx.Err(); err != nil {
        return nil, err
    }
    // Proceed with operation
}

Error Handling

Local store errors should be wrapped with meaningful context
Remote store errors should distinguish transient (retry-able) from permanent failures
Delete operations should be idempotent (deleting a non-existent block is not an error)

Performance

Local stores: Minimize syscalls, use buffered I/O, consider memory-mapped files
Remote stores: Use connection pooling, implement retry with backoff, batch operations where possible

Testing Your Implementation

Conformance tests: Run the provided test suite (pkg/block/blockstoretest)
Concurrency tests: Verify thread safety with parallel reads/writes
Error handling tests: Test behavior with canceled contexts, network failures
Integration tests: Test with the full DittoFS stack (create share, mount, read/write)

Common Pitfalls

Not making Delete idempotent: Deleting a non-existent block should succeed
Ignoring context cancellation: Long operations should check ctx.Err() periodically
Unsafe concurrent access: Use proper synchronization for shared state
Resource leaks: Ensure Close() releases all resources (connections, goroutines, file handles)

Integration with DittoFS

Register Your Store

Add your store type to the configuration system:

func createLocalStore(config LocalStoreConfig) (local.LocalStore, error) {
    switch config.Type {
    case "fs":
        return fs.New(config.Path)
    case "memory":
        return memory.New()
    case "mylocal":
        return mylocal.New(config.Path)
    default:
        return nil, fmt.Errorf("unknown local store type: %s", config.Type)
    }
}

CLI Integration

Users can then create your store via CLI:

./dfsctl store block add --kind local --name my-store --type mylocal \
  --config '{"path":"/data/blocks"}'

Additional Resources

Interface Definitions: pkg/block/local/local.go, pkg/block/remote/remote.go
Reference Implementations:
- Local: pkg/block/local/fs/, pkg/block/local/memory/
- Remote: pkg/block/remote/s3/, pkg/block/remote/memory/
- Metadata: pkg/metadata/store/memory/, pkg/metadata/store/badger/, pkg/metadata/store/postgres/
Conformance Tests: pkg/block/blockstoretest/ (block stores), pkg/metadata/storetest/ (metadata stores)
Architecture: docs/ARCHITECTURE.md
Configuration: docs/CONFIGURATION.md
Contributing: docs/CONTRIBUTING.md

Implementing Stores

Table of Contents

Overview

When to Implement Custom Stores

Metadata Store Use Cases

Local Store Use Cases

Remote Store Use Cases

Understanding the Architecture

Per-Share BlockStore

File Handle and Block Resolution

Implementing Metadata Stores

MetadataStore.EnumerateFileChunks

FileChunkStore narrowing

FileAttr.Blocks []BlockRef

Conformance scenarios

FileAttr.ObjectID + FindByObjectID

FindByObjectID(ctx, ObjectID) ([]BlockRef, error)

Block Record Store

BlockRecordStore

DefaultCommitBlock — one fsync per block

Engine API surface

Implementing a Local Store

The LocalStore Interface

Reference Implementation

Conformance Tests

Implementing a Remote Store

The RemoteStore Interface

Implementation Pattern

Ref Counting

Reference Implementation

CAS contracts

RemoteStore.WriteBlockWithHash

RemoteStore.ReadBlockVerified

Conformance Tests

Implementing a Remote Block Store (block-keyed)

The RemoteBlockStore Interface

Key shape

PutBlock

GetBlock

GetBlockRange

DeleteBlock

WalkBlocks

Block codec wire format

Conformance suite

Best Practices

Thread Safety

Context Handling

Error Handling

Performance

Testing Your Implementation

Common Pitfalls

Integration with DittoFS

Register Your Store

CLI Integration

Additional Resources

`FindByObjectID(ctx, ObjectID) ([]BlockRef, error)`