How about using a Mac Studio for the 4-bit quantized version? 192GB of LPDDR5 at 800GB/s and GPU with access to integrated memory... A little pricey, but perhaps effective and simple for this kind of work?
Sep 24, 2023·edited Sep 24, 2023Liked by Benjamin Marie
$6k for Studio with that config. Identical config on Pro is $9k. Not sure if there is some advantage to Pro worth the $3k delta. I figure $6k is not much different than a PC with good memory and some GPUs, but there is no PC with the equivalent GPU memory integration...
How about using a Mac Studio for the 4-bit quantized version? 192GB of LPDDR5 at 800GB/s and GPU with access to integrated memory... A little pricey, but perhaps effective and simple for this kind of work?
A Mac studio would definitely work very well!
But yes, it's an expensive configuration (around $8k if I'm not wrong)
$6k for Studio with that config. Identical config on Pro is $9k. Not sure if there is some advantage to Pro worth the $3k delta. I figure $6k is not much different than a PC with good memory and some GPUs, but there is no PC with the equivalent GPU memory integration...