Introduction to Floating Point Precision

Computers don’t have infinite memory. Naturally, there is a limit to floating-point numbers because there is a hardware limit. Floating point precision refers to how numbers are represented in binary using a fixed number of bits. Half Precision (FP16), Single Precision (FP32), and Double Precision (FP64) are a few popular precision types. The benefits of each floating point vary, and the particular use case will determine which format is best.

Understanding FP16 and BF16

FP16 consists of one sign bit, five exponent bits, and ten mantissa bits. In total, FP16 can represent numbers roughly with about three to four decimal digits of precision.  BF16, or BFloat16, uses one sign bit but eight bits for the exponent and seven bits for the mantissa. BF16 actually uses the same exponent bias as FP32, which allows it to represent a very wide range of values. Consequently, it only has seven bits for the fraction, so it has two to three decimal digits of precision. In other words, BF16 sacrifices precision in exchange for a broader exponent range.

Dynamic Range vs. Precision

By comparing the two representations, we can see that BF16 can represent much larger and smaller numbers. Since BF16’s exponent of eight bits equals the exponent of FP32, it may be easier to convert an FP32 to BF16. This can be done by truncating the mantissa bits, and the exponent can be left unchanged, but rounding is usually preferred to preserve numerical accuracy. However, converting FP32 to FP16 may cause overflow or underflow issues if the value’s exponent is outside the FP16’s limited range. From this, we can see that picking between FP16 and BF16 will ultimately come down to your specific use case.

Memory Requirements and Efficiency

One of the biggest incentives for using reduced-precision formats such as FP16 and BF16 is the memory savings and the computational efficiency that they provide. Since both FP16 and BF16 only use 16 bits per number, that is half the size of the standard FP32 format and a quarter of the size of FP64. There are many benefits in saving memory.

Many computational workloads, especially in deep learning, are constrained by memory footprint and memory bandwidth. If we use FP16 or BF16 instead of FP32, we essentially need half the memory required to store model parameters, activation maps, gradients, and other tensors.

Performance

Using FP16 and BF16 also doubles the data throughput, making it much more efficient to transfer large amounts of data, such as large matrix multiplications. When comparing FP16 and BF16, there may be minor differences because some devices may be better optimized specifically for FP16 or BF16, but overall, they both provide similar performance boosts.

Modern GPUs often have specialized hardware units for lower precision arithmetic. For example, NVIDIA GPUs have Tensor Cores that can perform many FP16 or BF16 operations in parallel which reaches higher FLOPs, or floating point operations per second, than other formats such as FP32 or FP64. More and more hardware supports are optimized for half-precision formats which allows neural network training and inference to run several times faster.

Deep Learning Applications

The rise in popularity of FP16 and BF16 can be attributed to deep learning. FP16 was first widely used in neural network training and inference. Once GPUs like the NVIDIA Volta introduced specialized units for half-precision values, hardware support for FP16 became increasingly common. Researchers and engineers found that they could speed up the training of models, such as transformer models for translation, without a significant loss of accuracy.

FP16

FP16 is especially useful in tasks where moderate precision is needed, whether for small gradient updates or subtle weight differences, but the overall range of values is controllable.

The faster throughput of FP16 compared to FP32 has resulted in FP16 being used in inference tasks as it is particularly beneficial for real-time applications such as video processing or autonomous driving.

BF16

BF16 gained prominence through Google’s Tensor Processing Units (TPUs). Google found that many stability issues in training with FP16, such as training divergence due to underflow or overflow, could be mitigated by using a format with a wider range. BF16 was then adopted for training large models, such as in natural language processing and advanced vision models, especially as model sizes and batch sizes grew larger. Since then, BF16 has been supported in other hardware, such as in NVIDIA’s Ampere-generation GPUs and later.

The use cases of BF16 are slightly different from those of FP16. They often involve very deep networks or large-scale models where gradients and activations may vary greatly. For example, large language models with hundreds of billions of parameters may benefit from the increased range of BF16 because these models can have extremely large or small gradient values during training. BF16’s larger exponent range significantly reduces the likelihood of encountering unexpected zeros or infinities during computations, making it more stable than FP16 for large-scale models.

Mixed Precision Training

In practice, FP16 and BF16 are most commonly used in conjunction with other floating point precisions, known as mixed precision. The most important parts of the computation use higher precision, such as FP32, while the majority of the other parts use a lower precision.

A common strategy is to perform all matrix multiplications and convolutions in half precision but then accumulate the result of these operations in FP32. Especially in linear algebra operations, we need to sum up a large number of terms, such as computing a dot product. Since there are so many terms, a single multiplication’s tiny error won’t dominate the result, so to an extent, the errors may even themselves out. Accumulating the sum with higher precision ensures that our final result is still quite accurate despite the inputs being of lower precision.

Similarly, when updating weights, the update can be computed in FP32 even if the weights themselves are stored as FP16 or BF16. This way, many tiny gradient updates over time won’t be lost due to precision limits. Mixed precision allows us to use higher precision where stability and accuracy matter more and lower precision for the heavy lifting in linear algebra operations where efficiency is crucial.

Conclusion

Ultimately, both FP16 and BF16 serve the same goal of optimizing performance memory with the cost of some precision. FP16 contains more bits to the fraction, giving it finer precision, but its small exponent limits it to a relatively narrow range of representable values. On the other hand, BF16 has more bits in the exponent, giving it a vast range that covers practically all values in standard single precision at the cost of precision due to fewer fraction bits.

The choice between FP16 and BF16 depends on the hardware at hand and the specific demands of the application. It’s also important to understand the limitations of each floating-point format, so appropriate techniques can mitigate their disadvantages. Both formats help train larger models faster and deploy models more efficiently. As machine learning continues pushing toward even bigger models and datasets, efficiency becomes even more vital.