Replication

Logical decoding and ReorderBuffer

Logical decoding reads storage-level WAL, reconstructs row changes under their transaction IDs, and uses ReorderBuffer state to deliver coherent transaction changes to an output plugin. Large transactions can exceed logical_decoding_work_mem and be spilled or streamed.

In plain words

The write-ahead log (WAL) records changes in the order they happen, so changes from many transactions are mixed together. Logical decoding turns that mixed stream back into clean, per-transaction row changes ("insert this row", "update that row") that another system can apply.

To do that, PostgreSQL keeps each transaction's changes in a sorting area called the ReorderBuffer. When a transaction commits, its changes are sent out together, in commit order. Transactions that roll back are simply dropped.

The memory for all transactions being decoded together is limited by logical_decoding_work_mem. When the limit is reached, PostgreSQL takes the largest transaction and either saves it to disk temporarily or, if the subscriber supports it, starts streaming it before it commits. For built-in logical replication, the pgoutput plugin formats the changes and the subscriber applies them.

For DBAs and operators

What ReorderBuffer stores

ReorderBufferChange represents decoded changes such as INSERT, UPDATE and DELETE and points to its owning ReorderBufferTXN. The transaction state tracks the XID, snapshots, subtransactions, memory size and whether changes have been serialized or streamed.

Memory pressure

The implementation accounts the memory of all decoded transactions together against logical_decoding_work_mem and, when the limit is reached, evicts the largest transaction. ReorderBuffer exposes statistics (in pg_stat_replication_slots) for transactions/bytes spilled to disk, transactions/bytes streamed, and how many times spilling or streaming was invoked (spill_count, stream_count). The exact choice between serialization and streaming depends on whether the transaction/output path is streamable; it is not simply “all large transactions always spill”.

Built-in logical replication

A logical walsender performs decoding and loads pgoutput. On the subscriber an apply worker receives protocol messages. For streamed transactions PostgreSQL can use parallel apply workers when configured; if the requested parallel path is unavailable, PostgreSQL has fallback behavior rather than changing commit semantics.

Source trail

Logical decoding concepts and logical replication architecture.
reorderbuffer.h and reorderbuffer.c.
worker.c — subscriber apply paths.

For developers: inside the source

Call flow

XLogSendLogical()                        replication/walsender.c
  XLogReadRecord() → LogicalDecodingProcessRecord()   logical/decode.c
    heap_decode() → DecodeInsert() → ReorderBufferQueueChange()
    standby_decode() → SnapBuildProcessRunningXacts()  logical/snapbuild.c
    xact_decode() → DecodeCommit() → ReorderBufferCommit()
      ReorderBufferReplay() → ReorderBufferProcessTXN()
        rb->begin / apply_change / commit  (logical.c *_cb_wrapper)
          → pgoutput_begin_txn(), pgoutput_change(), pgoutput_commit_txn()

LogicalDecodingProcessRecord() dispatches on each resource manager's rm_decode callback. The same machinery serves the SQL interface (pg_logical_slot_get_changes() in logicalfuncs.c); logical.c builds the LogicalDecodingContext (CreateInitDecodingContext(), CreateDecodingContext()) and wires the ReorderBuffer callbacks to the plugin's OutputPluginCallbacks.

Data structures

ReorderBuffer: by_txn hash (XID → transaction), toplevel_by_lsn list, total size, a max-heap txn_heap for finding the largest transaction, the counters spillTxns/spillCount/spillBytes, streamTxns/streamCount/streamBytes, totalTxns/totalBytes (exported to pg_stat_replication_slots by UpdateDecodingStats()), and separate memory contexts for changes, transactions and tuples.
ReorderBufferTXN: xid, toplevel_xid, first_lsn/final_lsn, base_snapshot, subtxns, the changes list, size/total_size, and txn_flags bits such as RBTXN_IS_SERIALIZED, RBTXN_IS_STREAMED, RBTXN_HAS_CATALOG_CHANGES.
ReorderBufferChange: lsn, action (REORDER_BUFFER_CHANGE_INSERT …), the owning txn, and for DML data.tp.oldtuple/newtuple.

Memory limit: spill or stream

Every queued change updates the accounting (ReorderBufferChangeMemoryUpdate()) and calls ReorderBufferCheckMemoryLimit(). The limit applies to rb->size, all transactions of the decoding session together. While it is exceeded: if ReorderBufferCanStartStreaming() (plugin streaming enabled and the snapshot builder SNAPBUILD_CONSISTENT) finds a top-level transaction via ReorderBufferLargestStreamableTopTXN(), it calls ReorderBufferStreamTXN(); otherwise ReorderBufferLargestTXN() picks the largest (sub)transaction for ReorderBufferSerializeTXN(), which writes pg_replslot/<slot>/xid-<xid>-lsn-<X>-<X>.spill. At commit, ReorderBufferIterTXNInit() merges in-memory and restored (ReorderBufferRestoreChanges()) changes of the transaction and its subtransactions in LSN order with a binary heap.

Experiment

contrib/test_decoding is the reference output plugin (start from its _PG_output_plugin_init()); its sql/spill.sql, sql/stream.sql and isolation specs/ exercise this code. Try pg_logical_slot_peek_changes() on a test_decoding slot.
debug_logical_replication_streaming = immediate (developer option) streams or serializes every change at once; log_min_messages = debug2 shows "spill N changes in XID N to disk" and "restored N/N changes from disk".
TAP tests: src/test/subscription/t/015_stream.pl … 019_stream_subxact_ddl_abort.pl, src/test/recovery/t/006_logical_decoding.pl. Concepts: doc/src/sgml/logicaldecoding.sgml.

Parameters that shape this

wal_level logical_decoding_work_mem max_replication_slots max_wal_senders max_logical_replication_workers max_parallel_apply_workers_per_subscription