08 · Segment files and rollover

Ch. 7 introduced the principle of "only ever appending to the end of a file." But what happens if you just keep growing a single file forever? Multi-gigabyte files, deletion headaches, backup nightmares. This chapter covers why and how Quipu-Log splits the log into multiple files (segments) — and the rollover, sealing, and reconstruction problems that splitting brings with it.

What you already know: SSTables and data files

If you've used a relational database, you probably know that data lives across multiple files. PostgreSQL splits a table into 1 GB chunks; LSM-Tree engines like RocksDB manage data in immutable units called SSTables. Why not one big file?

• Deletion is cheap. Dropping a time range means one unlink call per file. No expensive vacuum pass hunting down individual rows.
• Backups and replication are straightforward. Old files never change, so you can copy them without worrying that the file shifts under you.
• Corruption is contained. One bad file leaves every other file's data intact.

Quipu-Log splits the log into segment files for the same reasons. The difference: unlike SSTables there's no sorting or compaction — append-only means the write order is the file order.

Two states a segment can be in: active vs. sealed

A segment is always one of two things.

The active segment is the file currently being written — records keep getting appended to it.
A sealed segment is one that exceeded max_segment_bytes and was declared "done." Its contents are frozen — no further modifications, which makes it ideal for tamper-detection and backups.

File names and sequence numbers

Each segment file is named seg-NNNNNNNNNN.log, where N is a 10-digit zero-padded integer — the sequence number. In code:

crates/quipu-core/src/storage/table.rsconst SEGMENT_PREFIX: &str = "seg-";
const SEGMENT_SUFFIX: &str = ".log";

fn segment_path(dir: &Path, seq: u64) -> PathBuf {
    dir.join(format!("{SEGMENT_PREFIX}{seq:010}{SEGMENT_SUFFIX}"))
}

Because of the zero-padding, sorting filenames lexicographically gives you chronological order. A read_dir call followed by a name sort immediately gives you oldest-to-newest. The file with the highest number is the active segment.

Rollover: when a new segment opens

Every time a record is appended, the code checks: "is this file full enough?" Let's look at the code.

crates/quipu-core/src/storage/table.rs — Table::appendpub fn append(&mut self, row: &T, timestamp: u64) -> Result<()> {
    let payload = bincode::serialize(row)?;
    if !self.active.is_empty()
        && self.active.len() + payload.len() as u64 > self.max_segment_bytes
    {
        self.roll()?;
    }
    self.active.append(&payload, timestamp)?;
    self.spine.append(leaf_hash(&payload))?;
    Ok(())
}

If the active segment is non-empty and adding this record would push it past max_segment_bytes, roll() is called. The default is 64 * 1024 * 1024 = 64 MiB (configured in StoreConfig::new).

Inside roll(): ① fsync the current active file, ② write a metadata sidecar, ③ move it to the sealed list, and ④ open a new file at sequence number + 1. The order matters — the old file must be fully flushed before the new one opens.

The sidecar (.meta) and restart optimization

Scanning sealed segments from scratch on every restart to figure out their time ranges would be slow. So Quipu-Log writes a sidecar file (seg-NNNNNNNNNN.meta) at seal time. It holds the minimum and maximum timestamps, the record count, and the base_index.

crates/quipu-core/src/storage/table.rs#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
struct SegmentMeta {
    min_timestamp: u64,
    max_timestamp: u64,
    records: u64,
    base_index: u64,
}

base_index: record position that survives across segment files

base_index records which position in the overall log a segment's first record occupies. If segment 0 holds 100 records and segment 1 holds 80, segment 2's base_index is 180.

Why does this matter? When retention deletes segment 0, the file header of segment 2 still tells you that its records start at position 180 in the global log. That means Merkle inclusion proofs stay valid even after a segment is deleted. Ch. 21, inclusion proofs covers this in depth.

Restart: reading the directory to rebuild state

When the process restarts, Table::open reads the log directory and reconstructs state.

1. Collect all .log files via read_dir and sort by sequence number.
2. Designate the highest-numbered file as the active segment.
3. For the rest, read their sidecars and register them in the sealed list. If a sidecar is missing, skim the segment to reconstruct it.
4. Finally, call reconcile_spine() to sync the Merkle spine with the segments. Ch. 12, crash recovery

Deletion is one unlink call

When a retention policy fires ("delete anything older than 30 days"), what Quipu-Log does is remarkably simple: it calls remove_file (the OS unlink) on the target sealed segment. No vacuum pass finding and deleting rows, no space reclamation.

crates/quipu-core/src/storage/table.rs — Table::purge_older_thanfor seq in &doomed {
    if let Some(s) = self.sealed.remove(seq) {
        std::fs::remove_file(s.path)?;
        let _ = std::fs::remove_file(meta_path(&self.dir, *seq));
    }
}

The active segment is never deleted — the most recent records always survive.