WAL summarizer

For DBAs and operators

The process

Started by the postmaster when summarize_wal = on (reload is enough, no restart), also on a standby. ps: postgres: walsummarizer; backend_type = 'walsummarizer'. Wait events: WalSummarizerWal (waiting for more WAL), WalSummaryRead/WalSummaryWrite (file I/O), WalSummarizerError (sleeping after an error before retrying). It requires wal_level replica or logical.

What it writes

It decodes only the block references of WAL records (relation, fork, block number) into an in-memory block reference table, plus "limit blocks" for truncations. A summary file ends at a checkpoint's redo point (or at a timeline switch) and is named TTTTTTTTSSSSSSSSSSSSSSSSEEEEEEEEEEEEEEEE.summary: timeline, start LSN, end LSN in hex.

SELECT * FROM pg_get_wal_summarizer_state();
-- summarized_tli | summarized_lsn | pending_lsn | summarizer_pid

SELECT * FROM pg_available_wal_summaries() ORDER BY end_lsn DESC LIMIT 5;

SELECT * FROM pg_wal_summary_contents(1, '0/175ECB0', '0/A000028')
 LIMIT 10;   -- relfilenode, reltablespace, reldatabase, relforknumber, relblocknumber, is_limit_block

The summarizer itself removes summaries older than wal_summary_keep_time (default 10 days, 0 = never), once the WAL they cover is gone too. Keep it comfortably longer than the time between a backup and the next incremental backup based on it.

Incremental backup cycle

pg_basebackup -D /bk/full  -c fast             # writes backup_manifest
pg_basebackup -D /bk/incr1 --incremental=/bk/full/backup_manifest
pg_basebackup -D /bk/incr2 --incremental=/bk/incr1/backup_manifest
pg_combinebackup /bk/full /bk/incr1 /bk/incr2 -o /restore/data

The server reads the prior manifest's start LSN and timeline, waits until the summarizer has covered WAL up to the new backup's start, and then sends changed blocks as INCREMENTAL.<relfilenode> files (block list + blocks). Files that are new since the prior backup, empty, not relation files, free space map forks, or have more than 90 % of their blocks changed are sent whole. pg_combinebackup must get the chain in order, oldest first. The result is a normal full backup; recovery from it then needs WAL from its start as usual (archive or pg_wal included with -X).

Support notes

• Incremental backup fails if any summary in the needed LSN range is missing, for example because summarize_wal was off for a while, the summarizer lagged too far, or the summaries were already removed. Take a new full backup.
• If the summarizer cannot keep up, summarized_lsn lags behind and an incremental backup waits for it (with warnings in the server log).
• While summarize_wal is on, checkpoints do not remove WAL that has not been summarized yet. A summarizer that is stuck (look for WalSummarizerError and errors in the log) therefore makes pg_wal grow, like a stale replication slot.
• Tools like pgBackRest and Barman have their own incremental methods; the WAL summarizer serves the built-in pg_basebackup one.
• An ordinary ERROR does not end the process: it logs it, sleeps 10 s (WalSummarizerError) and retries. Any abnormal exit of the process (FATAL, SIGKILL) makes the postmaster perform a crash restart.