The Kaitchup – AI on a Budget
Subscribe
Sign in
Share this post
The Kaitchup – AI on a Budget
Marlin: Nearly Ideal Inference Speed for 4-bit Models with vLLM (1k+ tokens/sec)
Copy link
Facebook
Email
Notes
More
Marlin: Nearly Ideal Inference Speed for…
Benjamin Marie
Mar 28, 2024
6
Share this post
The Kaitchup – AI on a Budget
Marlin: Nearly Ideal Inference Speed for 4-bit Models with vLLM (1k+ tokens/sec)
Copy link
Facebook
Email
Notes
More
20
1
This thread is only visible to paid subscribers of The Kaitchup – AI on a Budget
Subscribe to view →
Comments on this post are for paid subscribers
Subscribe
Already a paid subscriber?
Sign in
Share
Copy link
Facebook
Email
Notes
More
This site requires JavaScript to run correctly. Please
turn on JavaScript
or unblock scripts
Share this post
Marlin: Nearly Ideal Inference Speed for…
Share this post
This thread is only visible to paid subscribers of The Kaitchup – AI on a Budget
Comments on this post are for paid subscribers