Optimum-Benchmark: How Fast and Memory-Efficient Is Your LLM?

Benchmarking quantization methods for Mistral 7B

When building applications with large language models (LLMs), knowing their running costs is critical. To optimize this cost, we must know precisely the hardware requirement for a given LLM. In other words, we need to know at least how much memory and the type of memory it will need, how fast the model is during inference, and several other performance-related metrics to evaluate its operational characteristics.

With optimum-benchmark, a framework developed by Hugging Face, we can generate a thorough assessment of the model's overall efficiency using key indicators such as memory usage, inference latency, and throughput.

In this article, I present optimum-benchmark and review its main features. Then, we will see how to use it to benchmark LLMs’ performance. For demonstration, I will especially focus on benchmarking quantization methods applied to Mistral 7B.

My notebook running optimum-benchmark for Mistral 7B is available here: