PetaBox
A high-density, low-power storage unit designed by the Internet Archive.
The PetaBox (also written Petabox) is a storage unit built by the Internet Archive and Capricorn Technologies. Its designers—Internet Archive staff and C. R. Saikley—aimed to store and process one petabyte (a million gigabytes) of information. The unit was designed to be inexpensive, easy to scale, and simple to maintain, with one system administrator per petabyte. It runs Linux as its standard operating system but supports multiple operating systems. The PetaBox is colocation-friendly and can operate inside a 20-by-8-by-8-foot shipping container. Each rack consumes 6 kW of power, and the entire storage cluster uses 60 kW. The system achieves a density of 100+ terabytes per rack, with a power consumption of 3 kW per petabyte. Instead of air conditioning, the unit uses its excess heat to help heat the building. The design also includes local computing via 800 low-end PCs and software that automates full mirroring.
The first 100-terabyte rack went live in June 2004 at the Internet Archive’s European branch in Amsterdam. Later that year, an 80-terabyte rack became operational at the Archive’s main San Francisco location. The Internet Archive then spun off PetaBox production to the newly formed Capricorn Technologies. Between 2004 and 2007, Capricorn replicated the Archive’s deployment for major academic institutions, digital preservationists, government agencies, high-performance computing sites, medical imaging providers, digital image repositories, storage outsourcing firms, and other enterprises. Their largest product used 750-gigabyte disks. By 2007, the Internet Archive’s data center housed roughly three petabytes of PetaBox storage.
In 2010, the fourth version of the PetaBox began operation. Each unit provided 480 TB of raw storage, using 240 disks of 2 TB each, arranged with 24 disks per 4U-high rack unit and 10 units per rack, all running Linux. As of December 2021, the Internet Archive’s PetaBox system comprised four data centers, 745 nodes, and 28,000 spinning disks. The Wayback Machine held 57 petabytes of information; book, music, and video collections added 42 petabytes, for a total of 99 petabytes of unique data. Because content was backed up in multiple locations, the system used 212 petabytes overall.
- Density
- 1.4 petabytes per rack
- Power consumption
- 3 kW per petabyte
- Power per rack
- 6 kW per rack
- Total cluster power
- 60 kW for the entire storage cluster
- First operational rack
- 100 terabyte rack in Amsterdam, June 2004
- Second operational rack
- 80 terabyte rack in San Francisco, 2004
- Largest product disk size
- 750 gigabyte disks
Lore & Background
The first 100 terabyte rack of the PetaBox became operational in Amsterdam at the Internet Archive's European arm in June 2004. Later that year, a second 80 terabyte rack became operational at the Internet Archive's main San Francisco location. The Internet Archive then spun off its PetaBox production to the newly-formed company Capricorn Technologies. Between 2004 and 2007, Capricorn replicated the Internet Archive's deployment for major academic institutions, digital preservationists, government agencies, high-performance computing sites, medical imaging providers, digital image repositories, storage outsourcing sites, and other enterprises. In 2007, the Internet Archive data center housed approximately three petabytes of PetaBox storage technology. In 2010, the fourth version of the PetaBox began operation, allowing for 480 TB of raw storage (240 disks of 2 TB each, set up with 24 disks per 4U high rack units and with 10 units per rack) running on Linux.
Reader's Guide
The PetaBox represents a significant approach to large-scale digital preservation and storage. Its design goals emphasized low power consumption (6 kW per rack, 60 kW for the entire cluster), high density (100+ TB per rack), and local computing to process data using 800 low-end PCs. It was designed to be colocation friendly and shipping container friendly, able to run in a 20' by 8' by 8' shipping container. Maintenance was intended to be easy, with one system administrator per petabyte, and software automated full mirroring. The system was designed to be inexpensive and easy to scale. As of December 2021, the Internet Archive's PetaBox storage system consisted of four data centers, 745 nodes, and 28,000 spinning disks. The Wayback Machine contained 57 petabytes of information; book, music and video collections contained an extra 42 petabytes, totaling 99 petabytes of unique data. Due to content being backed up in multiple locations, a total of 212 petabytes was used.
Did You Know?
- The PetaBox was designed to use excess heat to help heat the building, eliminating the need for air conditioning.
- The first 100 terabyte rack became operational in Amsterdam in June 2004.
More in Computer Storage, Part 2 1-24
Spotted an error? Know more?
Reader corrections go straight into our review queue. Suggest an edit · How this site is sourced