What this covers
Dave tests llama3.1 and llama3.2 using Ollama on a Raspberry Pi, a Herk Orion Mini PC, a 3970X, an M2 Mac Pro, and a 7995WX featuring the NVidia 6000 Ada GPU. See the results live! Check out my book on the autism spectrum! https://amzn.to/4elzfQv
I’m doing something that’s never been done before: we’re going to run a ChatGPT-style large language model locally on a wide range of hardware, from a $50 Raspberry Pi 4 all the way up to a $50,000 Dell AI workstation with dual Nvidia 6000 Ada cards.
If you saw my last video, you know I caught some heat for using top-tier hardware and running everything in WSL on Windows. This time, I’m doing things differently. We’re starting small and budget-friendly—no Linux shenanigans—and testing out local inference on everything from a Pi to a high-performance mini PC, a gaming rig, an M2 Mac Pro, and, of course, that beast of a Dell workstation.
Along the way, I’ll show you how to install Ollama on Windows, and we’ll compare how well each machine can handle models like Llama 3.1 and even the monstrous 405-billion parameter model. Which system will shine? Which one will falter? Can a Raspberry Pi even handle a large language model at all? And what happens when we push the $50,000 workstation to its limit?
If you’ve ever wondered what it takes to run a large language model locally, or just want to see how different hardware stacks up, this episode is for you! Be sure to stick around to the end for some surprising results.
💻 Hardware tested in this episode:
Raspberry Pi 4 (8GB RAM) Orion Herk Mini PC (Ryzen 9 7940HS) Desktop Gaming PC (Threadripper 3970X & Nvidia 4080) Apple M2 Mac Pro Dell Threadripper Workstation (96 cores & Nvidia 6000 Ada)
Check out Dave’s Attic for behind-the-scenes Q&A on episodes like this one.! HTTP://youtube.com/@UCtb6a_CnmGbSns9G8W2Ny0w
Follow me on Facebook for daily updates! HTTP://fb.com/davepl Twitter: @davepl1968davepl1968
Source description (no synthesized summary yet).
Dave demonstrates that running local large language models like Llama is feasible across a wide hardware spectrum from $50 to $50,000, but model size and hardware capability create a dramatic performance trade-off that determines practical usability more than raw machine cost.
- Raspberry Pi can run Llama but produces one word every 1-2 seconds, making it unusable for real-time interaction
- Sub-$400 Mini PC achieves ChatGPT-comparable speeds with smaller models, offering best price-performance ratio
- 405 billion parameter model on $50k workstation performs worse than 70B model on gaming GPU, proving model size dominates hardware advantage
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Llama can be installed and run on a Raspberry Pi 4 with 8GB RAM using the CPU only, but it produces responses at approximately one word every 1-2 seconds, making it impractical for real-time use despite technical feasibility.
“it's very very slow it's about one word a second if that maybe one word every two seconds nobody wants to watch it at that speed”
Ollama can be installed directly on Windows without requiring Linux or WSL2, providing a simpler installation path for Windows users wanting to run local LLMs.
“when we do it on the Windows systems I'll show you how to install it directly on Windows without any Linux or wsl2 Shenanigans”
WSL2 (Windows Subsystem for Linux 2) supports GPU passthrough, allowing Windows users to run Linux-based tools like Ollama with direct GPU acceleration without full virtualization overhead.
“one of the nicest things about the current Linux subsystem on Windows is that it supports passing the GPU through to Linux”
Model size and computational complexity have nearly equal or greater impact on LLM inference performance than the underlying hardware, meaning a poorly-sized model can neutralize the advantage of expensive, powerful machines.
“if we've learned nothing else the size of the model and the complexity of the calculations required to operate it and do inference on it have almost as much impact as the actual machine you're running it on so choose your model wisely”
The Llama 3.1 405 billion parameter model requires 228GB of download storage and approximately 200GB of RAM to load, but produces only one token every several seconds on a $50,000 Dell workstation with 512GB RAM and dual overclocked Thread Ripper 96-core CPU with RTX 6000 Ada GPU, taking approximately 30 minutes to generate a simple response.
“it is 228 GB which means you're looking at several minutes to several hours depending on your internet speed”
The comparison metaphor 'when a dog plays the piano, it's not about how well it does it, it's more that it does it at all' accurately describes the Raspberry Pi running Llama, where technical possibility matters more than practical performance.
“when a dog plays the piano it's it's not about how well it does it it's more that it does it at all and that's kind of how I feel about large language models on the pi I'm impressed that it works at all but while it's cool that it does so as a concept no one's going to put up with that kind of performance”
The Nvidia RTX 4080 GPU in a consumer gaming PC can run Llama 3.1 (70 billion parameters) at approximately ChatGPT speed with 16GB of host memory usage and averaging 75% GPU utilization with spikes to 100%, making this configuration faster than ChatGPT for local inference.
“we're using 16 GB of host memory and 100% of the GPU in brief spikes and now with the model running it's averaging around 75% GPU with spikes up to 100”
The 405 billion parameter Llama 3.1 model produces only one token every several seconds of real wall-clock time on a high-end workstation, making response generation effectively unusable for interactive dialogue.
“so yeah it's a very powerful model it's very big 405 billion parameters which it's very impressive that you can run that at home but as you can see you can barely run that at home this is almost as bad as the regular model running on the pie it's actually pretty close to it I would say so if we've learned nothing else”
The Hertz Mini PC (under $400 base model) running Ollama with Llama 3.1 produces responses at approximately the same speed as ChatGPT, making it the best price-performance value for local LLM inference among tested hardware.
“to me this seems like it's about the same speed as chat GPT and so it's fully usable and I think it's a pretty good deal for an under $400 machine”
The Hertz Mini PC's AMD Radeon 780M integrated GPU with 6GB of dedicated memory cannot run the full Llama 3.1 (70 billion parameter) model due to insufficient VRAM, forcing the system to use CPU inference instead, despite being marketed as an 'AI Mini PC'.
“since there's only 6 GB of dedicated GPU memory there's probably just not enough memory to load the model into the GPU is my guess it appears to have loaded the model into base memory and is using the CPU”
The Mac Pro with M2 Ultra chip and 128GB unified memory architecture (where all RAM is available as VRAM) produces very rapid inference responses for Llama 3.1, with GPU utilization spiking around 50%, making it a highly performant platform for local LLM inference.
“the really nice thing about the Apple architecture is that all of that Ram is also available to be allocated as video RAM meaning we should be able to run even large models with good performance”
Running Llama 3.2 on the high-end workstation produces inference responses at real-time speed that are fast enough to be 'incredibly quick' and 'Speedy', making the model practical for deployment.
“as you can see it rips off pretty quickly”
The Llama 3.2 model is significantly smaller (approximately 2GB) than Llama 3.1 (70B at ~5GB), but produces inference responses 'incredibly quickly' and is 'a very Speedy model' even on the high-end workstation, making it more practical for local inference despite potentially lower capability.
“the more efficient llama 3.2 it is significantly smaller I believe it's about 2 gab”
Direct Windows installation of Ollama eliminates the need for Linux abstraction layers or WSL2, providing a simpler setup path that addresses viewer concerns about configuration complexity.
“today we're going to remedy that as I show you how to do it on a wide range of systems starting with a Raspberry Pi and working our way up through a Mini PC a conventional gaming machine an M2 Mac Pro and then a top in $50,000 AI workstation from Dell and when we do it on the Windows systems I'll show you how to install it directly on Windows without any Linux or wsl2 Shenanigans”
Internet connection speed significantly affects Llama model download times, with a 5 gigabit connection downloading the 5GB Llama 3.1 model in a couple of minutes, whereas standard internet service would require significantly longer.
“even though at about 3 gbits a second it still takes a while so yours might take significantly longer if you're on regular internet service”
The Dell $50,000 AI workstation tested was provided as a 'gracious loan' from Dell, and Dave explicitly clarifies he did not personally purchase the expensive machine.
“the big machine is on gracious loan from Dell I think a lot of you assumed I ponied up the money for an outrageous machine like that but you know what they say I didn't get rich by writing a lot of checks”
Llama 3.1 comes in three parameter sizes: 8 billion, 70 billion, and 405 billion parameters, each with progressively larger model files and computational requirements.
“the Llama 3.1 model comes in three sizes 8 billion parameters 70 billion parameters and 405 billion up till now we've been running the 70 billion parameter version of this model”
Ollama requires a SHA digest verification step that must complete successfully before a downloaded model becomes usable, which adds substantial time to the model preparation process.
“I'll do a quick olama list to confirm the model is there and then we'll run llama 3.1 latest”
NVTop is a Linux tool that functions analogously to Windows Task Manager but specifically for monitoring Nvidia GPU usage, allowing real-time visualization of GPU memory allocation and utilization.
“I'm running NV top which is kind of like task manager for NVIDIA cards and allows you to monitor the progress and the use of the video card”
The Raspberry Pi CPU peaks at 84°C while running full-speed inference on all four cores at 100% utilization, using approximately 6GB of the available 8GB RAM.
“all four cores are pegged to 100% the CPU is getting up to about 84° and the most active task is of course AMA itself we're using about 6 GB of memory”
A 4-year-old Thread Ripper 3970x CPU with 32 cores and 128GB RAM paired with an Nvidia RTX 4080 GPU serves as a reasonable functional equivalent to a contemporary gaming PC despite its age.
“the astute Observer might notice in neofetch that the GPU is listed as a Microsoft GPU and not an Nvidia that's because in this case I'll still be doing this one under WSL 2 and one of the nicest things about the current Linux subsystem on Windows is that it supports passing the GPU through to Linux so let's see just how fast we can run local inference with a 480”
The Ollama model manifest verification process is time-consuming and requires checking the SHA digest of large model files (5GB+) before they become usable.
“verifying the shaw digest will also take some time”
Running Ollama under WSL2 on Windows displays the GPU as 'Microsoft GPU' in system information tools like neofetch, even though the actual hardware is an Nvidia GPU with direct GPU passthrough.
“the astute Observer might notice in neofetch that the GPU is listed as a Microsoft GPU and not an Nvidia that's because in this case I'll still be doing this one under WSL 2”
The high-end workstation's disappointing performance with the 405B model prompted Dave to 'redeem the machine somehow as it's unfair to burden it with a load that none of the other machines could even hope to lift'.
“I have to admit that made me a little sad to see a $50,000 workstation brought to its knees like that”
The episode received viewer criticism in comments for two specific choices: testing primarily on expensive hardware (Dell Thread Ripper with dual RTX 608 series GPUs) and using a complex setup (Linux on WSL2 on Windows).
“recently I did an episode showing you how to use a local large language model to provide chat GPT like functionality on your own home machine but I took a lot of heat in the video comments for two things one I demoed it on a Dell thread Ripper workstation with dual Nvidia 608 series gpus and two I used Linux on top of WSL on top of Windows to do it the major outcries were then that I should test it on more budget friendly Hardware than I should just install it on Windows”
Video model download and verification process can be sped up for presentation purposes using playback acceleration, allowing viewers to see the process without requiring them to watch real-time downloads lasting 'two or three minutes' or longer.
“this is going to take two or three minutes which I will speed up because even though at about 3 gbits a second it still takes a while”
Using a single console window for Ollama server operations and a separate window for model download and testing queries is an effective workflow organization strategy for local LLM deployment.
“I'm going to start with two console Windows one where I will install using the script for olama that I got from the olama website and we'll let it proceed through the download and I'll speed up those lengthy downloads so we can get right into the install”
When running Llama 3.1 on the RTX 4080, GPU utilization averaged around 75% with spikes to 100%, consuming 16GB of host system memory.
“now with the model running it's averaging around 75% GPU with spikes up to 100 the answer came out really quickly even with this 5 GB model”
The Thread Ripper workstation with 512GB of RAM has sufficient memory to load the full 405 billion parameter Llama 3.1 model, with peak memory usage reaching close to 200GB.
“This Thread Ripper is equipped with 512 GB of RAM which should be enough memory that we can load the 405 billion parameter version”
The Mac Pro M2 Ultra GPU utilization peaks around 50% while generating Llama 3.1 responses, indicating efficient processing with significant headroom for concurrent operations.
“it seems to spike around 50% and it produces an answer in very rapid fashion”
The high-end workstation configuration tested features an overclocked 96-core Thread Ripper CPU set to run at more than 800 watts TDP, consuming more than 1,200 watts total at the wall when combined with the Nvidia RTX 6000 Ada GPU.
“we're next going to step up to an overclocked 96 core thread Ripper with an Nvidia 6000 Ada card installed the CPU is set up to run at more than 800 watts of TDP and combined with the GPU this system pulls more than 1,200 Watts at the wall”
The Hertz Mini PC features a Ryzen 9 7940 HS processor with 4 GHz base clock and 5.2 GHz boost clock, and includes an AMD Radeon 780M RDNA3 iGPU running at 2800 MHz.
“this one spec at $676 features a ryzen 97940 HS chip with a 4 GHz base clock and a 5.2 GHz boost clock”
On a Raspberry Pi running Llama inference, the most active system process is Ollama itself, confirmed by task monitoring tools.
“the most active task is of course AMA itself”
The Hertz Mini PC uses integrated LPDDR5 SODIMM memory with a vapor chamber cooler for the CPU, can reduce power draw to 90 watts, and is equipped with a 140-watt external power supply.
“it uses lpddr5 sodm and has a real Vapor chamber cooler on the CPU it has dual M2 SSD slots Wi-Fi 6E and 2.5 GB networking”
The 405 billion parameter model download and verification process involves extensive disk I/O that creates significant disk activity that persists for an extended period.
“this can take quite a while and as you can see it burns through a lot of dis activity”
Dave purchased his Thread Ripper 3970x workstation approximately four years ago, and while it retains a single-core speed limitation from that era, it remains capable when paired with modern GPUs.
“this machine is a 3970x thread Ripper that I purchased about four years ago now it's rocking 32 cores and 128 GB of RAM but the single core speed is still that of a 4-year-old PC”
The Nvidia RTX 4080 GPU Dave tested represents the 'second best' consumer gaming GPU tier, below only the flagship RTX 4090.
“nothing but the second best for Dave now”