Honey, I shrunk the LLM! A beginner’s guide to quantization

Honey, I shrunk the LLM! A beginner’s guide to quantization

Jul 14, 2024

Hands on If you hop on Hugging Face and start browsing through large language models, you’ll quickly notice a trend: Most have been trained at 16-bit floating point of Brain-float precision.  FP16 and BF16 have become quite popular for machine learning –...