Buffer Pool / Page Cache — How a DB Manages Memory

The buffer pool (shared_buffers) is Postgres's in-RAM cache of 8 KB pages in shared memory. Every heap or index page a backend reads or writes goes through it — never straight to the file. When it fills, a clock-sweep algorithm picks a victim to evict; and the moment a dirty victim is written back is exactly where the write-ahead rule is mechanically enforced.

Intermediate buffer-pool shared-buffers buffer-manager buffer-tag clock-sweep usage-count pin dirty-page bgwriter ring-buffer double-buffering effective-cache-size Created 2026-06-30 · Updated 2026-06-30

Key Components

Buffer pool / shared_buffers: Postgres's in-RAM cache: an array of fixed 8 KB slots in shared memory, each holding one page. A slot's index is its buffer_id. shared_buffers is the setting that actually allocates this memory (default 128 MB; ~25% of RAM in production).
buffer_tag: A page's global identity: (tablespace, database, relation, fork#, block#). It turns "block 7 of this table's main fork" into something the manager can look up in O(1).
Buffer table: A hash map buffer_tag → buffer_id. Given a page's identity, it answers "which slot (if any) currently holds it?" in constant time.
Buffer descriptor: Per-slot state: the slot's current buffer_tag, a dirty flag (modified since load), refcount (the pin count — how many users have it open right now), and usage_count (popularity, driving eviction priority; capped at BM_MAX_USAGE_COUNT = 5).
Clock-sweep: The eviction algorithm — an approximate LRU. A pointer (nextVictimBuffer) walks the descriptors in a circle: skip pinned slots; decrement any usage_count > 0 and move on (a second chance); evict the first unpinned slot at usage_count == 0.
Pin (refcount): A pin (refcount > 0) marks a buffer as in-use and forbids its eviction. A backend pins on read (refcount++) and unpins when done (refcount--). Unlike a pin, usage_count persists after unpinning as a popularity residue.
Ring buffer (Buffer Access Strategy): A small set of buffers within shared_buffers (256 KB–16 MB) reused cyclically for large sequential scans and bulk ops (COPY, CREATE TABLE AS, VACUUM), so a giant scan churns its own few slots and can't evict the hot working set.
Double buffering: The fact that Postgres pages live in two caches at once — shared_buffers and the OS page cache. Deliberate: Postgres leans on the kernel's file cache rather than using O_DIRECT, which is why the pool is kept modest so RAM is left for the OS.
effective_cache_size: A planner hint that allocates nothing. It estimates total cache available (≈ shared_buffers + OS cache) and biases the cost model toward index scans. It changes plans, not memory — not to be confused with shared_buffers.

Concrete Example

Inspect the pool and what it currently holds:

SHOW shared_buffers;                       -- pool size (default 128MB; ~25% RAM in prod)
CREATE EXTENSION pg_buffercache;           -- inspect what's cached now
SELECT count(*) FROM pg_buffercache WHERE isdirty;   -- dirty pages awaiting flush

Now trace what happens when a backend asks for block 7 of a table. The request passes through the three buffer-manager structures in order — this is the lookup that runs on every page access:

1. buffer_tag  = (tablespace, database, relation, main-fork, block 7)
                 ↓  the page's global identity

2. buffer table  →  hash lookup: buffer_tag → buffer_id?
       HIT             → pin (refcount++), bump usage_count (cap 5), done
       MISS, free slot → take from free list, load from disk,
                         insert (tag → id) into buffer table, pin
       MISS, pool full → clock-sweep a victim (flush WAL if dirty),
                         load block 7 into it, insert into table, pin
                 ↓  answers "which slot?" in O(1)

3. buffer descriptor  →  the slot's state: buffer_tag, dirty flag,
                         refcount (pins), usage_count (popularity)

The buffer_tag is the key that makes this work: it converts a page's abstract identity into an O(1) hash-table probe, and a hit costs nothing but a lookup and a counter bump. A miss loads from disk into a free or freshly-evicted slot; if that evicted victim was dirty, its data page can only reach disk after its WAL is flushed — the write-ahead rule, enforced here.

Visual Model

Picture a librarian's desk with a fixed number of reading stands (the slots). Each stand has a sticky note recording who is reading it now (the pin / refcount), how popular it is (usage_count), and whether anyone has written in it (dirty). When every stand is full and a new book is requested, the librarian walks a circle with a pointer — the clock hand (nextVictimBuffer): she skips any book still being read, knocks each popular book's popularity down by one and moves on (a second chance), and evicts the first unpopular one she reaches. If that book was written in, she first copies the changes to the master journal (flush WAL) before reshelving it.

Step through one full clock-sweep below. Six buffer slots sit in a ring, each labelled with its usage_count and whether it is pinned. Watch the hand rotate and the counters decay.

Step 1 of N

Loading…

Deeper — Edge Cases & Gotchas

Why clock-sweep instead of true LRU?

True LRU keeps an exact access-ordered list and evicts the single oldest page. But maintaining that order requires moving a page to the front of the list on every access — and because the pool is shared by all backends, that shared list needs a lock on every read, which becomes a contention bottleneck. Clock-sweep drops the exact ordering: on access you only bump a per-page counter (no list, no global lock), and eviction approximates LRU via second chances. The trade is locked global ordering per access (LRU) versus a cheap per-page counter plus a rotating hand (clock-sweep). The small cap (5) guarantees even a hot page is eventually evictable rather than pinned in cache forever.

The WAL connection — the enforcement point

Before a dirty victim is written to its data file, the buffer manager must flush WAL up to that page's pd_lsn (XLogFlush). This is the write-ahead rule enforced mechanically: a data page can never reach disk before the WAL records describing it. Three actors flush dirty buffers — the checkpointer (flushes all dirty buffers at a checkpoint so old WAL can recycle), the background writer (trickles out soon-to-be-evicted dirty buffers so victims are usually already clean), and a regular backend that picks an un-cleaned dirty victim and must write it itself, synchronously, mid-query — a latency spike. Avoiding that spike is precisely why the bgwriter exists.

Cache pollution & ring buffers

A naive cache has a flaw: one SELECT * FROM huge_table would stream millions of never-reread pages through the pool and evict the entire hot working set. Postgres prevents this with a Buffer Access Strategy (ring buffer): large sequential scans (relation > 25% of shared_buffers) and bulk ops (COPY, CREATE TABLE AS, VACUUM) are confined to a small set of buffers within the pool, reused cyclically. So a giant analytical scan or a VACUUM churns its own few slots and cannot blow away the cache — which is why a big scan does not tank OLTP latency.

Anti-pattern: setting shared_buffers to 80% of RAM "to cache more pages."

-- Tempting but wrong on a 64 GB box:
shared_buffers = 52GB        -- "80% of RAM, cache everything!"

-- Sensible default:
shared_buffers = 16GB        -- ~25% of RAM; leave the rest for the OS cache

Why it backfires: Postgres pages live in two caches at once — shared_buffers and the OS page cache. (1) Double-buffering waste: oversizing the pool means hot pages are cached twice, so fewer distinct pages are cached overall. (2) Heavier checkpoints: a larger pool holds more dirty pages, so each checkpoint has more to flush, plus memory pressure on work_mem and the OS. Because Postgres deliberately shares caching with the kernel (unlike O_DIRECT engines such as Oracle/InnoDB), keep the pool near ~25% and leave RAM for the OS.

Anti-pattern: conflating effective_cache_size with shared_buffers and raising it to reserve memory.

Why it is a misconception: effective_cache_size allocates nothing. It is a planner hint (≈ shared_buffers + estimated OS cache) that nudges the cost model toward index scans over sequential scans. Changing it changes plans, not cache. Only shared_buffers allocates real memory. (Distinct again: work_mem sizes each sort/hash op; maintenance_work_mem sizes VACUUM and CREATE INDEX.)

Test Yourself

A backend wants block 7 of a table and it is not currently cached, and the pool is full. Which sequence correctly describes the miss path?

A colleague sets shared_buffers to 80% of RAM "to cache more," and throughput drops. What is the primary reason?

Explain clock-sweep: its three per-slot actions, why a hot page survives, and why Postgres uses it instead of true LRU.