PetaBox

The PetaBox (also written Petabox) is a storage unit built by the Internet Archive and Capricorn Technologies. Its designers—Internet Archive staff and C. R. Saikley—aimed to store and process one petabyte (a million gigabytes) of information. The unit was designed to be inexpensive, easy to scale, and simple to maintain, with one system administrator per petabyte. It runs Linux as its standard operating system but supports multiple operating systems.

The PetaBox is colocation-friendly and can operate inside a 20-by-8-by-8-foot shipping container. Each rack consumes 6 kW of power, and the entire storage cluster uses 60 kW.

The system achieves a density of 100+ terabytes per rack, with a power consumption of 3 kW per petabyte. Instead of air conditioning, the unit uses its excess heat to help heat the building. The design also includes local computing via 800 low-end PCs and software that automates full mirroring.

The first 100-terabyte rack went live in June 2004 at the Internet Archive’s European branch in Amsterdam. Later that year, an 80-terabyte rack became operational at the Archive’s main San Francisco location. The Internet Archive then spun off PetaBox production to the newly formed Capricorn Technologies.

Between 2004 and 2007, Capricorn replicated the Archive’s deployment for major academic institutions, digital preservationists, government agencies, high-performance computing sites, medical imaging providers, digital image repositories, storage outsourcing firms, and other enterprises. Their largest product used 750-gigabyte disks. By 2007, the Internet Archive’s data center housed roughly three petabytes of PetaBox storage.

In 2010, the fourth version of the PetaBox began operation. Each unit provided 480 TB of raw storage, using 240 disks of 2 TB each, arranged with 24 disks per 4U-high rack unit and 10 units per rack, all running Linux.

As of December 2021, the Internet Archive’s PetaBox system comprised four data centers, 745 nodes, and 28,000 spinning disks. The Wayback Machine held 57 petabytes of information; book, music, and video collections added 42 petabytes, for a total of 99 petabytes of unique data. Because content was backed up in multiple locations, the system used 212 petabytes overall.