Ask.Cyberinfrastructure

What is a "Burst Buffer"?

storage
gpfs
lustre
programming-for-hpc

#1

What is a “Burst Buffer” and how is it different from other filesystems such as Lustre or GPFS?

CURATOR: Grace Wilson Caudill/Scott Valcourt


#2

Burst buffers are a name given to intermediate layers of custom software and hardware that sit between an I/O source (e.g. an application writing to some files) and an I/O end-point (e.g. a Lustre file system). Typically a burst buffer will stage and/or rearrange I/O requests from an application so that it writes to the final end-point file system in large, contiguous chunks. To make this work, the burst buffer system employs some form of high-speed intermediate storage (for example non-volatile RAM or SSD) and application I/O is transparently written to this intermediate layer.

Application code that has bursty (i.e. intermittent in time) I/O, especially with small files, can see effective I/O performance increases from burst buffer technologies. The high-speed buffer layer allows the application to progress through I/O rapidly, while the burst buffer system software has time during I/O free phases to move bytes from the burst buffer to the main storage.

Burst buffer technologies are most commonly developed as integrated solutions together with parallel storage backend systems. There is at least one backend storage-agnostic burst buffer experiment under development - see https://github.com/LLNL/burstfs