Write-Ahead Log (WAL) Pattern in Java: Ensuring Data Durability and Crash Recovery

Also known as

• Append-Only Log
• Redo Log
• Journaling

Intent of Write-Ahead Log Pattern

The Write-Ahead Log (WAL) design pattern ensures data durability and system recoverability in database engines, distributed consensus protocols, and transactional systems. It enforces a strict order of operations where any state mutation (e.g., insert, update, delete) must be written sequentially to an append-only log file on stable storage (disk) before it is applied to the main database state or in-memory storage structures.

Detailed Explanation of Write-Ahead Log Pattern with Real-World Examples

Real-world example

Imagine an accountant managing a company's ledger. Before modifying the main financial summary balance sheets, the accountant immediately records every incoming transaction line-by-line into a sequential physical logbook. If power cuts out mid-day or the summary balance sheets are damaged, the accountant can re-open the physical logbook, replay every recorded entry from the beginning, and perfectly recalculate the final financial state.

In plain words

Write-Ahead Log guarantees that no state mutation is lost during sudden system crashes by writing changes to a fast append-only disk log file before updating the in-memory store.

Wikipedia says

In computer science, write-ahead logging (WAL) is a family of techniques for providing atomicity and durability (two of the ACID properties) in database systems. In a system using WAL, all modifications are written to a log before they are applied. Usually both redo and undo information are stored in the log.

Class Diagram

classDiagram
    class OperationType {
        <<enumeration>>
        SET
        DELETE
        CHECKPOINT
    }

    class LogEntry {
        -long sequenceNumber
        -OperationType type
        -String key
        -String value
        +toLogString() String
        +fromLogString(String line)$ LogEntry
    }

    class WriteAheadLog {
        -File logFile
        -AtomicLong sequenceNumberCounter
        +append(OperationType type, String key, String value) LogEntry
        +readAll() List~LogEntry~
        +clear() void
    }

    class DatabaseStore {
        -WriteAheadLog wal
        -Map~String, String~ memTable
        +put(String key, String value) void
        +delete(String key) void
        +get(String key) String
        +checkpoint() void
        +simulateCrash() void
        +recover() void
    }

    DatabaseStore --> WriteAheadLog
    WriteAheadLog --> LogEntry
    LogEntry --> OperationType

Programmatic Example of Write-Ahead Log Pattern in Java

The WriteAheadLog class manages append-only sequential writes to disk:

public class WriteAheadLog {
  private final File logFile;
  private final AtomicLong sequenceNumberCounter = new AtomicLong(0);

  public synchronized LogEntry append(OperationType type, String key, String value) throws IOException {
    long nextSeq = sequenceNumberCounter.incrementAndGet();
    LogEntry entry = new LogEntry(nextSeq, type, key, value);

    try (BufferedWriter writer = new BufferedWriter(new FileWriter(logFile, true))) {
      writer.write(entry.toLogString());
      writer.newLine();
      writer.flush();
    }
    return entry;
  }
}

The DatabaseStore class coordinates writing to the log before modifying its in-memory MemTable:

public class DatabaseStore {
  private final WriteAheadLog wal;
  private final Map<String, String> memTable = new HashMap<>();

  public synchronized void put(String key, String value) throws IOException {
    wal.append(OperationType.SET, key, value);
    memTable.put(key, value);
  }

  public synchronized void delete(String key) throws IOException {
    wal.append(OperationType.DELETE, key, null);
    memTable.remove(key);
  }

  public synchronized void recover() {
    memTable.clear();
    List<LogEntry> entries = wal.readAll();
    for (LogEntry entry : entries) {
      if (entry.getType() == OperationType.SET) {
        memTable.put(entry.getKey(), entry.getValue());
      } else if (entry.getType() == OperationType.DELETE) {
        memTable.remove(entry.getKey());
      }
    }
  }
}

The App class demonstrates initialization, writes, crash simulation, and WAL recovery:

@Slf4j
public class App {
  public static void main(String[] args) {
    try {
      File logFile = File.createTempFile("wal_demo", ".log");
      WriteAheadLog wal = new WriteAheadLog(logFile);
      DatabaseStore store = new DatabaseStore(wal);

      store.put("user:101", "Alice");
      store.put("user:102", "Bob");
      store.delete("user:103");

      // Simulating system crash where in-memory state is lost
      store.simulateCrash();

      // System reboot & recovery from WAL log replay
      store.recover();

      LOGGER.info("MemTable post recovery: {}", store.getMemTableSnapshot());
    } catch (IOException e) {
      LOGGER.error("Error running WAL demo", e);
    }
  }
}

Program output:

15:45:00.100 [main] INFO com.iluwatar.writeaheadlog.App -- === 1. Initializing Storage Engine with WAL ===
15:45:00.105 [main] INFO com.iluwatar.writeaheadlog.WriteAheadLog -- WAL Entry appended & flushed to disk: LogEntry(sequenceNumber=1, type=SET, key=user:101, value=Alice)
15:45:00.106 [main] INFO com.iluwatar.writeaheadlog.DatabaseStore -- Applied SET operation to MemTable: user:101 = Alice
15:45:00.107 [main] INFO com.iluwatar.writeaheadlog.App -- === 3. Simulating Unexpected System Crash ===
15:45:00.108 [main] INFO com.iluwatar.writeaheadlog.DatabaseStore -- !!! SIMULATED SYSTEM CRASH: In-memory MemTable has been wiped !!!
15:45:00.109 [main] INFO com.iluwatar.writeaheadlog.App -- === 4. System Restart & Recovery from WAL ===
15:45:00.110 [main] INFO com.iluwatar.writeaheadlog.DatabaseStore -- Starting recovery process from WAL...
15:45:00.112 [main] INFO com.iluwatar.writeaheadlog.DatabaseStore -- Recovery completed. Replayed 5 log entries into MemTable.
15:45:00.113 [main] INFO com.iluwatar.writeaheadlog.App -- MemTable snapshot post recovery: {user:101=Alice, user:102=Bob Smith}

When to Use the Write-Ahead Log Pattern in Java

• Building storage engines or key-value data stores requiring ACID durability guarantees.
• Implementing fault-tolerant distributed consensus protocols (e.g., Raft, Paxos).
• System architectures where random disk I/O is expensive, allowing sequential append-only writes for maximum throughput.
• Message brokers or event streams requiring replayability after failure.

Real-World Applications of Write-Ahead Log Pattern in Java

• PostgreSQL / MySQL (InnoDB): Uses WAL / Redo Log for crash recovery and replication.
• SQLite: Write-Ahead Logging mode for concurrency and atomic commits.
• Apache Cassandra / RocksDB: Appends mutations to CommitLog / WAL before MemTable updates.
• Apache Kafka / Raft: Log replication across distributed nodes for consensus and state machine replication.

Benefits and Trade-offs of Write-Ahead Log Pattern

Benefits:

• High Performance: Sequential disk writes are significantly faster than random disk updates (e.g., updating B-Trees directly).
• Durability & Fault Tolerance: Guarantees no committed transaction is lost during sudden system crashes.
• Simplicity of Recovery: Replaying ordered log records deterministically restores the exact last-known state.

Trade-offs:

• Storage Overhead: Log files grow over time, requiring periodic checkpointing and log truncation.
• Recovery Time: Large log files without checkpoints can lead to slow startup/recovery times.

Related Java Design Patterns

• Event Sourcing: Captures state mutations as a sequence of events, similar to log replay.
• Command: Encapsulates requests as objects, which can be serialized into WAL entries.
• Memento: Stores state snapshots (checkpoints) to truncate logs.

References and Credits

• Designing Data-Intensive Applications (Martin Kleppmann)
• PostgreSQL Documentation: Write-Ahead Logging (WAL)
• Raft Consensus Algorithm Paper