An outage rarely announces itself as a configuration problem. The one Jeremy Schneider documented this week looked like a network blip: streaming replication broke for a moment, the primary’s writes hung, and then stayed hung — thirty minutes of frozen TPS on a small cloud instance, hours on a larger dataset. The culprit was recovery_prefetch, a GUC almost nobody sets deliberately, sitting at its default of try on nearly every standby built since PostgreSQL 15.

What makes this worth an ops meeting is that two of the sharpest people writing about Postgres disagree. Michael Pettus walks through the internals and concludes you should leave it at try; Schneider, having reproduced a production outage deterministically, is leaning toward turning it off everywhere. Both argue from measurements. The useful work isn’t picking a side — it’s deciding which profile your cluster matches.

What the GUC Actually Does

WAL replay is single-threaded. The startup process reads a record, finds the page it changes, applies it, reads the next record — and blocks on storage whenever that page isn’t in shared_buffers. On a network-attached volume with roughly a millisecond of read latency, a standby that needs a million uncached pages spends a very long time asking for them one at a time.

recovery_prefetch wraps the WAL reader. Each time replay asks for the next record, the prefetcher decodes ahead, walks the block references, and issues posix_fadvise(WILLNEED) for main-fork blocks that aren’t already buffered, so the kernel starts the read early and the eventual read() is a memcpy. Values are off, try, on, default try; on is pointless because it refuses to start on platforms without read-ahead advice. Context is sighup, so you can flip it on a live standby with a reload and it takes effect mid-replay — an incident knob, not a restart.

Two partners matter as much as the GUC: wal_decode_buffer_size (how far ahead replay may decode, default 512kB, postmaster context) and maintenance_io_concurrency (how many prefetches may be in flight — 16 on 18 and later, 10 on 15 through 17).

Where the Read IOPS Go

Pettus’s numbers show the feature working: crash recovery of 309MB of WAL against a 3GB table on a simulated 1ms volume took 378 seconds with prefetching off and 30 seconds at maintenance_io_concurrency = 64, with time waiting on reads falling from 266 seconds to under half a second. That is why the default stuck.

Schneider’s contribution is the failure mode nobody had diagnosed. Attack it from two directions. When the primary does large sequential I/O, the prefetcher converts it into 8kB random reads — same bytes, up to 16x the IOPS. And because full_page_writes is on, as it should be, the first touch of every page after a checkpoint carries a full-page image that the prefetcher can’t see coming; those advisories are wasted calls that still consume an in-flight slot. A replica on a bounded-IOPS cloud disk gets pinned at its ceiling for tens of minutes after a write burst. Same bytes read, very different request count.

The Part That Turns Lag Into an Outage

Lag alone is survivable. The outage comes from the recovery state machine: read from archive or pg_wal, check for a promotion trigger, read from the primary via walreceiver, rescan timelines, sleep, repeat. When streaming breaks, replay falls back to local sources and does not resume streaming until it has caught up. If catching up is IOPS-bound, “caught up” can be hours away.

Now add synchronous replication: the primary can’t commit until a standby acknowledges. A momentary network glitch during a burst of writes becomes a full write outage on the primary, and the longer the lag, the longer the outage.

Schneider’s reproduction is deliberately boring — 500 IOPS disk, 2 vCPU, 8GB RAM, pgbench scale 640, a 5% update, then kill the walreceiver. TPS goes to zero and stays there for over 30 minutes. With recovery_prefetch = off the read-IOPS burst never happens and the standby applies the same update in under ten minutes.

Which Profile Are You?

• Standby on network-attached storage, working set bigger than RAM, routinely behind. Keep try, raise maintenance_io_concurrency to 32–64. Depth is the setting that matters.
• Working set fits in RAM, or replay is CPU-bound. Set off. On Pettus’s RAM-resident 1GB cluster, prefetching issued 783,000 advisories that saved zero reads and cost about a quarter of replay time (13.6 seconds versus 11.0).
• Direct I/O (debug_io_direct = data). The prefetcher silently does nothing; there is no page cache to advise into.
• Synchronous replication from a single standby. The GUC is the small problem. Two standbys with quorum commit — ANY 1 — is what removes “one network glitch equals write outage” from your risk register.

For planning: 19 RC1’s prefetcher is essentially 18.6’s, so no upgrade will change this answer for you.

Configuration, Two Profiles

# Profile A — standby on network-attached storage, large working set
recovery_prefetch = try
maintenance_io_concurrency = 64        # sighup; tune while watching io_depth
wal_decode_buffer_size = '4MB'         # postmaster context — restart required

# Profile B — database fits in RAM, or startup process pegged at 100% CPU
recovery_prefetch = off
maintenance_io_concurrency = 16

# Either way, if durability depends on a remote machine:
synchronous_standby_names = 'ANY 1 (pg_a, pg_b)'
synchronous_commit = on

Profile B applies without maintenance, which matters mid-incident:

psql -h standby -c "ALTER SYSTEM SET recovery_prefetch = 'off';"
psql -h standby -c "SELECT pg_reload_conf();"
# effective mid-replay; judge the result within a minute

Monitor io_depth, Not Just Lag

Replay lag tells you that you’re behind, not why — and the why decides whether more depth helps or hurts.

SELECT now() - pg_last_xact_replay_timestamp()            AS replay_lag,
       pg_last_wal_receive_lsn() - pg_last_wal_replay_lsn() AS backlog_bytes,
       prefetch, hit, skip_fpw, skip_rep,
       wal_distance, block_distance, io_depth
FROM   pg_stat_recovery_prefetch;

Read it like this: io_depth pinned at maintenance_io_concurrency while backlog_bytes grows means replay is genuinely read-bound — raise the concurrency. Backlog growing with io_depth well below the ceiling means replay waits on something else (lock waits, large FPI applies, one saturated core) and prefetch depth won’t fix it. Note also that prefetch counts advisories issued, not reads that happened.

On the primary, alert on wait_event = 'SyncRep' in pg_stat_activity alongside replication lag. It fires before the incident channel does.

Rehearse the Drill

Don’t discover this classification at 2am. On a staging pair with the same disk class and IOPS cap, run pgbench against a table larger than memory, apply a wide update, then kill the walreceiver. Measure two numbers: how long commits stall on the primary, and how long the standby takes to catch up. Repeat with recovery_prefetch = off. Whichever profile wins on your storage is the one you run — and the drill doubles as a test of whether your failover path keeps archive continuity, which breaks in the same moments replicas lag. Confirm your restore side too: a caught-up replica that has never been restored isn’t a backup, as our pgBackRest 2.60 page-checksum walkthrough shows.

What We’d Change This Week

Three things, in cost order. Check whether any standby is pinned at its IOPS ceiling during write bursts — a dashboard question, not a project. Write down which profile each cluster is in, because “we left the default” isn’t a decision. And if synchronous replication rides on a single standby, treat the second replica as availability spend; the alternative is discovering the coupling during your busiest hour, which is exactly when a replica is most likely to fall behind.

If you’d rather not run that drill yourself, our Postgres HA and failover team can plan and execute it with you, including the sync-rep topology review that decides whether recovery_prefetch is even your risk.