Vision processing unit

Lore & Background

Vision processing units emerged as a distinct class of AI accelerator, designed specifically to handle machine vision workloads. They differ from GPUs, which contain specialized hardware for rasterization and texture mapping for 3D graphics and whose memory architecture is optimized for manipulating bitmap images in off-chip memory. VPUs may include direct interfaces to take data from cameras, bypassing off-chip buffers, and emphasize on-chip dataflow between many parallel execution units with scratchpad memory, resembling a spatial architecture or a manycore DSP. Like video processing units, they may focus on low-precision fixed-point arithmetic for image processing.

Notable examples include Intel's Movidius Myriad X (third-generation VPU) and Myriad 2, used in Google Project Tango, Google Clips, and DJI drones. The Pixel Visual Core is a fully programmable image, vision, and AI processor for mobile devices. Microsoft HoloLens includes a holographic processing unit complementary to its CPU and GPU, aimed at interpreting camera inputs for environment tracking and augmented reality. Other designs include MIT's Eyeriss spatial architecture for CNNs, Yann LeCun's NeuFlow dataflow architecture implemented in FPGA, Mobileye EyeQ, and Nvidia's Programmable Vision Accelerator (PVA), a 7-way VLIW vision processor.

Some processors not described as VPUs are equally applicable to machine vision tasks, forming a broader category of AI accelerators, though as of 2016 there was no consensus on the name. Examples include IBM TrueNorth (neuromorphic), Qualcomm Zeroth Neural processing unit, and the Versatile Processor Unit (VPU) built into all models of Intel Meteor Lake processors for accelerating inference in computer vision and deep learning.

Reader's Guide

Vision processing units represent a significant shift in computing architecture, moving from general-purpose or graphics-oriented processors toward specialized hardware for machine vision. Their emphasis on performance per watt, rather than absolute performance, makes them particularly suitable for mobile and embedded applications such as robotics, IoT, smart cameras, and augmented reality devices. By integrating direct camera interfaces and on-chip dataflow with scratchpad memory, VPUs can process visual data more efficiently than GPUs, which are optimized for off-chip bitmap manipulation and 3D graphics rasterization. The emergence of VPUs has enabled new classes of digital cameras for virtual and augmented reality, and has brought machine vision acceleration to smartphones. While the category remains evolving—with some processors like IBM TrueNorth and Qualcomm Zeroth serving similar roles under different names—VPUs have established a distinct niche in the AI accelerator landscape, complementing CPUs and GPUs in handling real-time visual perception tasks.