Background processes

Startup process

The startup process is the one that replays WAL: once after a crash, until the target during point-in-time recovery, or forever on a standby.

In plain words

The startup process is PostgreSQL's replay engine. It reads WAL records one after another and re-applies each change to the data pages, exactly as the original transaction did.

It runs in three situations. After a crash, it replays the WAL written since the last checkpoint and then exits, and the server opens. During point-in-time recovery it fetches archived WAL files and stops at the moment you asked for. On a standby it never finishes: it keeps replaying the WAL that arrives from the primary, which is what keeps the copy up to date.

On a standby that also serves queries, replay sometimes has to remove rows a query is still reading. The startup process then waits a while and, if needed, cancels the query. Pick a situation with the selector.

For DBAs and operators

Who starts it

The postmaster forks it at every start (right after the checkpointer and bgwriter, which since PostgreSQL 15 also run during crash recovery); it reads pg_control and decides: clean shutdown and no signal file → nothing to do, exit at once; unclean shutdown → crash recovery; recovery.signal → archive recovery (PITR); standby.signal → standby mode. While it runs, no normal connections are allowed: crash recovery admits none until it is finished; during archive recovery or standby mode read-only connections are allowed once a consistent state is reached, if hot_standby = on.

The loop

Get the next record (from pg_wal, from the archive via restore_command, or from WAL streamed by the walreceiver); for each block it touches, read the page into shared buffers (or restore a full-page image), apply the change if the page LSN is older than the record, mark it dirty. The checkpointer and bgwriter write pages during recovery too; on a standby the checkpointer performs restartpoints. With recovery_prefetch = try (default) a look-ahead reader decodes up to wal_decode_buffer_size ahead and issues prefetches for blocks not yet cached, limited by maintenance_io_concurrency: see pg_stat_recovery_prefetch.

What it looks like

postgres: startup recovering 000000010000000000000012
postgres: startup waiting for 000000010000000000000004

The title says recovering <file> while it replays a segment and waiting for <file> while restore_command fetches that file from the archive; during a recovery-conflict wait waiting is appended. backend_type = 'startup'; wait event RecoveryWalStream while waiting for streamed WAL (the title keeps the last recovering file). Real log lines from a standby start:

LOG: entering standby modeLOG: redo starts at 0/30000D8LOG: consistent recovery state reached at 0/30001D0LOG: database system is ready to accept read-only connections

Crash recovery ends with LOG: redo done at 0/5A3F0B8 system usage: CPU: user: 0.21 s, system: 0.05 s, elapsed: 0.31 s followed by an end-of-recovery checkpoint. Long recoveries report progress every log_startup_progress_interval (10s): LOG: redo in progress, elapsed time: 12.04 s, current LSN: 0/7F2C1A8

Recovery conflicts on a hot standby

Replaying a VACUUM cleanup, an exclusive lock, a dropped database or tablespace can clash with a running query. The startup process waits up to max_standby_streaming_delay (or max_standby_archive_delay), 30s by default, measured from when the WAL was received, then cancels the query:

ERROR: canceling statement due to conflict with recoveryDETAIL: User query might have needed to see row versions that must be removed.

SELECT datname, confl_snapshot, confl_lock, confl_bufferpin, confl_deadlock
  FROM pg_stat_database_conflicts;

Options: hot_standby_feedback = on, a larger delay (more replay lag), or running long reports elsewhere. log_recovery_conflict_waits = on logs waits longer than deadlock_timeout.

If it dies

It is the most critical child on a standby. Any abnormal exit makes the postmaster stop everything:

LOG: startup process (PID 3302) was terminated by signal 9: KilledLOG: terminating any other active server processesLOG: shutting down due to startup process failure

So never kill it to "unstick" replay; find the cause (a conflict wait, missing WAL, I/O). A replay that seems stuck is often waiting for WAL: check pg_stat_wal_receiver and the startup process's wait event in pg_stat_activity.

For developers: inside the source

Where it lives

startup.c: the process shell: StartupProcessMain() sets up signals (SIGHUP, SIGTERM, SIGUSR2 for promote) and calls StartupXLOG(); ProcessStartupProcInterrupts() is called from the redo loop.
xlog.c: StartupXLOG() drives everything, from reading pg_control to the end-of-recovery checkpoint.
xlogrecovery.c: the recovery machinery; xlogprefetcher.c: recovery_prefetch; xlogutils.c: helpers for redo routines; standby.c: hot-standby conflicts.
Background: README (WAL, redo and the "WAL before data" rules).

Call flow

StartupXLOG()
  InitWalRecovery()      /* signal files, backup_label, find checkpoint */
  PerformWalRecovery()   /* the redo loop */
    ReadRecord() -> XLogPrefetcherReadRecord() -> XLogReadRecord()
    recoveryStopsBefore(), recoveryApplyDelay()
    ApplyWalRecord() -> GetRmgr(rmid).rm_redo(xlogreader)
  FinishWalRecovery()

PerformWalRecovery() loops record by record and checks CheckRecoveryConsistency(), which announces the consistent state and lets the postmaster admit hot-standby connections.
ApplyWalRecord() calls the resource manager's rm_redo (heap_redo, btree_redo, xact_redo …) under the rm_redo_error_callback context that prints the record being replayed.
Redo routines call XLogReadBufferForRedo(), which restores a full-page image (BLK_RESTORED), skips a page whose LSN is already newer (BLK_DONE), or returns BLK_NEEDS_REDO.
WaitForWALToBecomeAvailable() is a state machine over XLOG_FROM_ARCHIVE, XLOG_FROM_PG_WAL and XLOG_FROM_STREAM; it calls RequestXLogStreaming() to get a walreceiver and sleeps on recoveryWakeupLatch (wait event RecoveryWalStream) until WakeupRecovery(). XLogFileRead() sets the recovering/waiting for process titles.

Shared state and conflicts

XLogRecoveryCtlData (in xlogrecovery.c) publishes lastReplayedEndRecPtr, the pause state and the hot-standby flag under its info_lck spinlock; pg_last_wal_replay_lsn() reads it. Conflict handling lives in standby.c: ResolveRecoveryConflictWithSnapshot(), ResolveRecoveryConflictWithLock(), ResolveRecoveryConflictWithBufferPin() and others wait until GetStandbyLimitTime(), then signal the conflicting backends.

Experimenting and extending

TAP: 031_recovery_conflict.pl, 003_recovery_targets.pl, 013_crash_restart.pl, 039_end_of_wal.pl.
contrib/pg_walinspect and pg_waldump show the records the loop replays; wal_debug exists only in builds compiled with WAL_DEBUG.
Extensions with their own WAL register a custom resource manager with RegisterCustomRmgr() (xlog_internal.h) from shared_preload_libraries; its rm_redo then runs inside this process. Example: src/test/modules/test_custom_rmgrs.

Parameters that shape this

recovery_prefetch wal_decode_buffer_size maintenance_io_concurrency restore_command recovery_target_time recovery_target_action max_standby_streaming_delay max_standby_archive_delay hot_standby hot_standby_feedback recovery_min_apply_delay log_startup_progress_interval log_recovery_conflict_waits