Member-only story
A complete guide to running local LLM models
Meta just released Llama 2 [1], a large language model (LLM) that allows free research and commercial use. It’s expected to spark another wave of local LLMs that are fine-tuned based on it.
The open-source community has been very active in trying to build open and locally accessible LLMs as alternatives to ChatGPT after the first version of Llama came out in late February.
I have been closely following this trend — I have run and tested dozens of local LLMs that can be run on commercial hardware because I truly look forward to the future where everyone can have their own personal LLM assistants that are not constrained by a central provider. For businesses that build their products on LLMs, these models can also be more private, secure, and customizable choices.
In this post, I will share what I’ve learned about these local LLMs, including what your (best) choices are, how to run them, and how to pick the right setup for your use case. I will explain everything you need to know to get a model loaded and running, no matter if you have experience in programming or not. I will also share some ready-to-use Google Colab WebUI for these LLMs that you can try out yourself.
(If you’re just interested in trying out the models, you can visit this repo where I included the WebUI Colab Notebooks.)
Why do we need local LLMs?
If you’re unfamiliar with local LLMs, you might wonder, “If we already have the powerful ChatGPT, why would we want local LLMs?” I’ll admit that when I first heard about local alternatives to ChatGPT, I was skeptical too. But as I interacted more with ChatGPT and other hosted LLMs like Anthropic’s Claude or Google’s Bard, I realized that local LLMs offer some unique benefits beyond their text generation capability — data privacy, security, offline availability, customization, and reduced reliance on external services are all important aspects when you are using LLMs.
If you’re privacy-concerned or run a business that interacts with privacy-concerned users, it may not be a very wise idea to send you or your customer’s data to OpenAI or other providers. With local LLMs running on your own device or server, you maintain full control over your data.
