Diving into Self-Hosted LLM on Windows: My Initial Impressions

I've been experimenting with something new: running self-hosted AI on my Windows PC. I’m still very early in the process—just getting my feet wet—so this post won’t be a full tutorial or review just yet. I’ll save that for when I’ve had more time to dig in and really understand it. If the results are encouraging, I will do more with it. Otherwise I will use online services such as ChatGPT.

Why Host Your Own AI?

The biggest reason people go down this road is privacy. When you use self-hosted AI, the content you generate stays with you. It doesn’t get uploaded to the cloud or used to train someone else’s model. That kind of control is a huge plus if you’re working with sensitive or proprietary information.

What You'll Need

• I use an AM4 6800x processor. (Bigger is better)
• I have 64 Gig of DDR 4 memory. (Bigger is better)
• You’ll also need a decent NVIDIA graphics card because these AI models will work much better by using the **CUDA cores** to aid in the processing. I’m using an EVGA 3060 TI FTW, which works pretty well—but there’s still a little lag at times. A better card would improve the experience even more.
• The language model I selected is about 4 GB, which is fairly large.

What I Installed

There are three main software components involved in setting this up:

• OOLAMA – A tool for running large language models locally.
• LLAMA 3.1 – The actual language model.
• WSL (Windows Subsystem for Linux) – This lets you run a Linux environment directly on Windows.

All of these are free to download. I got everything installed without too much trouble using the online documentation. I can confirm—it does work! I think I had it running in about 45 minutes. This included the time it took to read the documentation. That wasn't bad considering this was all new to me.

First Impressions

I haven’t had a ton of time to play with it yet, but here’s what I’ve noticed so far:

• It works similarly to ChatGPT, but the responses feel a bit rougher around the edges.
• You need to phrase things a bit more carefully to get the best results.
• It's a smaller LLM when compared to online services, so your results aren't as comprehensive. I','m hopeful that a free LLM will be released at some point that will be in the 40-60 gig range.

My long-term goal is to use AI to help rewrite my documentation and parts of my website, to make things clearer and easier to read. If this setup can help with that, I’ll consider it a win. If the self-hosted doesn't work to my satisfaction, I will simply use ChatGPT or one of the others.

If there is enough interest I can share a more detailed walkthrough. For now, it's been a fun start into the world of DIY AI!