WAL and recovery

Crash recovery

When PostgreSQL stops without a clean shutdown, the startup process replays the WAL written since the last checkpoint, page by page, until the data files are consistent again.

In plain words

Changed table pages sit in memory for a while before they are written to disk. If the server loses power, or a process is killed, those pages are simply gone. Nothing committed is lost though (with the default synchronous_commit = on), because every change was first written to the WAL.

When PostgreSQL starts again it notices it was not shut down cleanly. A helper called the startup process looks up where the last checkpoint began (that is written in a small control file) and re-applies every WAL record from there to the end of the log. For each record it checks the page: if the page already contains this change, it skips it; if not, it applies it.

When the log runs out, the database is exactly as it was at the moment of the crash, minus transactions that had not committed. A checkpoint is taken and connections are accepted again. The more WAL was written since the last checkpoint, the longer this takes.

For DBAs and operators

What starts recovery

If any backend dies abnormally (segfault, kill -9, the Linux OOM killer), the postmaster cannot trust shared memory any more. It terminates all other processes and reinitializes, which triggers crash recovery even though the server never "went down" (when restart_after_crash = on, the default):

LOG: server process (PID 4711) was terminated by signal 9: KilledLOG: terminating any other active server processesLOG: all server processes terminated; reinitializing

The log of a real recovery (PostgreSQL 18)

LOG: database system was interrupted; last known up at 2026-10-22 15:34:10 UTCLOG: database system was not properly shut down; automatic recovery in progressLOG: redo starts at 0/2713EC8LOG: invalid record length at 0/2D81088: expected at least 24, got 0LOG: redo done at 0/2D81058 system usage: CPU: user: 0.04 s, system: 0.00 s, elapsed: 0.04 sLOG: checkpoint starting: end-of-recovery immediate waitLOG: database system is ready to accept connections

The "invalid record length … got 0" line is normal: it is how recovery finds the end of WAL (zeroed or recycled space after the last record). For long recoveries the startup process logs progress every log_startup_progress_interval (10s): LOG: redo in progress, elapsed time: 10.01 s, current LSN: 0/7A31F28

How replay decides

pg_control gives the latest checkpoint record; its REDO pointer (where the checkpoint started) is where replay begins.
For each record and each block it references, the page is read. A page LSN points just past the last WAL record that changed the page, so if page LSN ≥ end LSN of this record the change is already on disk and is skipped; otherwise it is applied and the page LSN set to the record's end LSN.
A full-page image is restored unconditionally, which also repairs torn pages.
Commit records update pg_xact; transactions without a commit record are treated as aborted.

$ pg_controldata $PGDATA | grep -E "state|REDO|checkpoint location"
Database cluster state:               in crash recovery
Latest checkpoint location:           0/2713F20
Latest checkpoint's REDO location:    0/2713EC8
Latest checkpoint's REDO WAL file:    000000010000000000000002

Recovery time

Roughly proportional to the WAL since the last checkpoint's REDO point (the distance= in checkpoint log lines) and to how many pages must be read from disk. It is bounded by checkpoint_timeout and max_wal_size. Since PostgreSQL 15, recovery_prefetch reads ahead in WAL (wal_decode_buffer_size) and prefetches referenced blocks, which helps a lot on high-latency storage. Before replay, PostgreSQL fsyncs the data directory (recovery_init_sync_method; syncfs is much faster with many files).

Never delete postmaster.pid, pg_wal files or run pg_resetwal to "speed up" a recovery: that is how clusters lose data. If recovery fails, keep a copy of the whole data directory before trying anything.

PostgreSQL 18 source trail: xlogrecovery.c distinguishes the checkpoint record location from RedoStartLSN, the checkpoint's REDO location.

For developers: inside the source

Where it lives

startup.c: StartupProcessMain() just calls StartupXLOG() in xlog.c, which reads pg_control, runs SyncDataDirectory() if the cluster was not shut down cleanly (fd.c), and drives recovery.
src/backend/access/transam/xlogrecovery.c (linked in the source trail): InitWalRecovery() finds the checkpoint (ReadCheckpointRecord(), or backup_label), PerformWalRecovery() runs the redo loop, FinishWalRecovery() reports where WAL ended.
xlogreader.c: XLogReadRecord(), ValidXLogRecordHeader() and ValidXLogRecord() (CRC); a failed check is how the end of WAL is found. xlogprefetcher.c implements recovery_prefetch.
xlogutils.c: XLogReadBufferForRedoExtended(), the page-LSN test.

The redo loop

PerformWalRecovery()
  record = ReadRecord(xlogprefetcher, ...)      via XLogPrefetcherReadRecord()
  do {
      ProcessStartupProcInterrupts();
      ApplyWalRecord(xlogreader, record, &replayTLI)
          → GetRmgr(record->xl_rmid).rm_redo(xlogreader)   e.g. heap_redo()
      record = ReadRecord(...);
  } while (record != NULL);

(For archive recovery the same loop also checks recoveryStopsBefore()/recoveryStopsAfter().) Each redo routine asks XLogReadBufferForRedo() for its pages. It returns BLK_RESTORED after applying a full-page image, BLK_DONE when record->EndRecPtr <= PageGetLSN(page) (change already on disk), BLK_NEEDS_REDO otherwise, or BLK_NOTFOUND if the block no longer exists (for example, the relation is dropped or truncated later in the WAL). The typical pattern, from heapam_xlog.c:

if (XLogReadBufferForRedo(record, 0, &buffer) == BLK_NEEDS_REDO)
{
    /* re-apply the change to BufferGetPage(buffer) */
    PageSetLSN(page, lsn);
    MarkBufferDirty(buffer);
}
if (BufferIsValid(buffer))
    UnlockReleaseBuffer(buffer);

When redo ends, StartupXLOG() calls PerformRecoveryXLogAction(), which for crash recovery is RequestCheckpoint(CHECKPOINT_END_OF_RECOVERY | CHECKPOINT_IMMEDIATE | CHECKPOINT_WAIT), the checkpoint you see in the log. The "Writing a REDO routine" section of access/transam/README lists the locking rules for redo code.

Observe and experiment

TAP tests in src/test/recovery: 013_crash_restart.pl (backend killed, postmaster reinitializes), 022_crash_temp_files.pl, 039_end_of_wal.pl (all the ways WAL can end), 016_min_consistency.pl. Run with make -C src/test/recovery check PROVE_TESTS=t/013_crash_restart.pl in a build configured with --enable-tap-tests.
pg_stat_recovery_prefetch shows prefetch hits and skips; log_startup_progress_interval drives the "redo in progress" lines via ereport_startup_progress().
pg_controldata shows the state and REDO pointer; pg_waldump -s from that LSN lists what will be replayed.

Parameters that shape this

checkpoint_timeout max_wal_size full_page_writes recovery_prefetch restart_after_crash recovery_init_sync_method wal_decode_buffer_size log_checkpoints