04 · Reading & writing, and the atomicity of rename

Reading and writing files seems simple enough — "write some bytes, done," right? But "updating a file without corrupting it" is trickier than it looks. This chapter covers how Quipu-Log opens segments (OpenOptions) and the atomicity of rename(2) — the trick it uses for safe file replacement. What a DB engine handled for you, here you handle yourself.

Opening files: the options in OpenOptions

In Rust, you specify how to open a file with std::fs::OpenOptions. Looking at the code that opens a Quipu-Log segment makes each option's purpose clear.

crates/quipu-core/src/storage/segment.rs — Segment::open()let file = OpenOptions::new()
    .create(true)         // create it if it doesn't exist
    .truncate(false)     // don't clear it if it does — preserve existing content!
    .read(true)          // readable too (needed during recovery)
    .write(true)         // writable
    .open(path)?;

truncate(false) is the critical one. Opening with truncate(true) wipes the file's contents the moment it's opened. If this option were wrong when restarting a segment, every audit record accumulated so far would be gone. In an append-only store, never use truncate(true).

seek: reading and writing from anywhere in the file

A file descriptor remembers its "current position." seek() lets you change it. When Quipu-Log opens an existing segment, it seeks to the last known-valid position and starts appending from there.

crates/quipu-core/src/storage/segment.rs — resuming an existing filelet mut writer = BufWriter::with_capacity(256 * 1024, file);
writer.seek(SeekFrom::Start(s.valid_len))?;   // move to the last valid position
// subsequent appends start from here

SeekFrom::Start(n) is n bytes from the start of the file. SeekFrom::End(0) is the end of the file. SeekFrom::Current(n) is ±n from the current position. Quipu-Log always uses Start(valid_len) — after a crash, the tail may be corrupt, so the valid end and the file size can differ.

Torn writes and tail truncation

What happens if the power dies mid-append? The last record is left half-written in the file. This is called a torn write.

At startup, Quipu-Log scans the segment with skim() to find any frame with a broken CRC. Once it determines the last valid position (valid_len), it trims the file back to that point:

crates/quipu-core/src/storage/segment.rs — tail truncationif file.metadata()?.len() > s.valid_len {
    file.set_len(s.valid_len)?;   // trim the broken tail
}
let mut writer = BufWriter::with_capacity(256 * 1024, file);
writer.seek(SeekFrom::Start(s.valid_len))?;

set_len(n) sets the file size to exactly n bytes. Passing a value smaller than the current size truncates (trims); larger creates a hole (sparse space). In crash recovery, only the former is used.

The atomicity of rename(2)

Now the key point of this chapter. There's a pattern for replacing a file safely — write the new content to a temp file, then swap it in with rename.

Why not overwrite in place? If you open the existing file, start writing new content, and the power dies, you're left with a half-written file. The rename pattern avoids that:

1. Write the new content in full to a temporary file (target.rewrite).
2. fsync the temporary file.
3. rename(tmp, target) — on POSIX this operation is atomic. It either completes, or nothing happens at all. There is no intermediate state.

Quipu-Log uses this pattern when re-keying rewrites the entire table:

crates/quipu-core/src/storage/table.rs — rewrite_table()let tmp = dir.with_file_name(format!("{name}.rewrite"));
let backup = dir.with_file_name(format!("{name}.pre-rewrite"));
// ... write the new table in full to tmp ...
fresh.sync()?;          // fsync first
drop(old); drop(fresh); // release handles
std::fs::rename(dir, &backup)?;   // existing → backup
std::fs::rename(&tmp, dir)?;      // new → canonical path
std::fs::remove_dir_all(&backup)?; // delete backup