Common Log File System

Overview

CLFS records a series of steps required for an action so they can be played back to commit a transaction to secondary storage or undone if needed. It first marshals log records to in-memory buffers, then writes them to log-files on stable media for permanent persistence. Built-in policies control when data is flushed, but a client application can override and force a flush. CLFS allows customizable log formats, expansion and truncation of logs according to defined policies, and simultaneous use by multiple client applications.

Log files can be stored anywhere on the file system. A device driver interface (DDI) lets physical storage system specific drivers plug into the CLFS API. The CLFS driver implements the ARIES recovery algorithm; custom drivers can support other algorithms. CLFS supports dedicated logs (single stream of log records) and multiplexed logs (multiple streams, each for a different application).

Even with multiple streams, logs are flushed sequentially in a single batch. Space for a set of log records can be allocated ahead-of-time to prevent failure due to lack of storage. A log record is first placed in a Log I/O Block in a system memory buffer; blocks are periodically flushed to stable storage. On the storage device, a log contains a set of contiguously allocated containers, each holding multiple Log I/O Blocks.

New records are appended to the present set. Each record is identified by a Log Sequence Number (LSN), an increasing 32-bit sequence number stored in the record header. The LSN encodes the container identifier, offset to the record, and record identifier, used to access the record later. Container identifiers are logical and must be mapped to physical containers by CLFS itself.