ChatGPT’s Demand for High-Performance Memory Chips on the Rise

Overview of HBM

High Bandwidth Memory (HBM) is a DRAM memory chip based on a 3D stacking process that can be stacked vertically like floors in a skyscraper. Based on this design, the time to exchange information will be shortened. These stacked DRAM chips are connected to the CPU or GPU through Interposer’s super-fast interconnections, and the assembled modules can be connected to the circuit board.

HBM adjusts the power consumption efficiency of memory and can greatly improve the data processing speed. HBM is the fastest DRAM product at present. Its bandwidth per watt is more than 3 times higher than GDDR5, and HBM saves 94% of the surface area than GDDR5.

Currently, HBM is mainly installed on GPU, network switching and forwarding devices (such as routers and switches), AI accelerators, supercomputers, and high-efficiency servers.

As HBM prices soared, Orders from Samsung and SK Hynix also increased

According to industry sources, SK Hynix supplies Nvidia with third-generation HBM, which are paired with Nvidia’s A100 Gpus for ChatGPT. In addition, Nvidia has installed SK Hynix’s fourth-generation HBM into H100, which is already supplying ChatGPT servers.

In addition, Samsung has developed HBM with computing capability, which can not only store data but also calculate data. Samsung supplied the product to AMD in October 2022 for use in AI accelerators.

Now, the price of the third generation HBM has soared, which is five times the price of high-efficiency DRAM, and its market growth rate is more than double the original forecast of Samsung and SK Hynix.

High-compute AI applications are calling for high-performance memory chips

The popularity of high-powered AI applications like ChatGPT is accelerating AI chips

Kim Jae-Joon, Samsung’s vice president of memory, has said that the development of interactive AI applications based on natural language technology, such as ChatGPT, is conducive to improving memory requirements. The efficient and large amount of computing power and high memory capacity is the foundation of AI learning and inference models.

Generative AI applications such as ChatGPT need to learn from massive amounts of training data to achieve high-quality generated outputs. In order to realize functions such as keyword recognition and image recognition, AI models need to store a large amount of picture and audio information.

AI applications for C-end users must have the ability to process data quickly to output AI calculation results to users in real-time, so it also puts forward higher requirements for the data transmission speed of the memory chip.

With the further popularization of artificial intelligence technology, more and more AI applications occur on mobile devices and Internet of things terminal devices. Most of these devices are battery-powered or recharged, and are very sensitive to power consumption.

The ultra-high bandwidth memory technology represented by HBM is expected to become an inevitable choice for related acceleration chips. At the same time, the generation class model will also accelerate the further increase of HBM memory capacity and bandwidth.

In addition to HBM, new storage technologies such as CXL (Compute Fast Linking) coupled with software optimizations will also increase the capacity and performance of local storage in such applications, which is expected to gain more industrial adoption from the rise of generative models.