<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>I agree with this. I bought a data center quality GPU for fun and
testing. It's a referb, and I have 128 gb ram and I'm running
everything off of em.2 drives. Even with that much ram, context
windows don't last as long as the fronteer models. I suspect that
the fronteer models are cramming everything into vector db type
data storage services and using something to prefetch data when
needed, but I could be very wrong there.</p>
<p>Just in raw performance, I got a lot of this stuff cheap and on
sale. It would cost probably 3x the amount if not more right now.
You need a minimum of 64 gb ram, and that's on the low end.</p>
<p><br>
</p>
<p>It's worth doing, but you might do better at these current prices
buying a beefed up Mac server vs trying to build your own system.</p>
<p><br>
</p>
On 5/20/2026 11:35 PM, Paul York via NFBCS wrote:<br>
<blockquote type="cite"
cite="mid:CAMwL28JH92HirUPt-=AAt4hQwtz7U4M953PWDZNiK5QYgp3Y9g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div class="gmail_default" style="font-size:large">I've been
knee deep in local llm setup for the better part of the last
two weeks. To give you perspective on my hardware, I'm playing
with two moderately beefy "consumer" machines: a Windows
11-based Ultra 7, 64GB RAM, RTX 4070 w/ 12GB VRAM and a
linux-based Ryzen AI 9 HX370 mini pc with 64GB RAM (both
bought before prices went bonkers thankfully).</div>
<div class="gmail_default" style="font-size:large"><br>
</div>
<div class="gmail_default" style="font-size:large">TLDR: I'm
keeping my Claude and Gemini subscriptions.</div>
<div class="gmail_default" style="font-size:large"><br>
</div>
<div class="gmail_default" style="font-size:large">I think a
longer discussion will hinge on what you want to do with it.
Are you programming? Running OpenClaw/agentic stuff? Just
chatting? Creating documents and presentations? Doing
NotebookLM kind of things? Because here's the deal. After a
LOT of tweaking, I'm getting:</div>
<div class="gmail_default" style="font-size:large">
<ul>
<li>around 25 tokens per second output on my iGPU (Ryzen)
using some pretty high quality models (Qwen 3.6 35b and
Gemma 4 26b) by pushing the VRAM up to 48GB.</li>
<li>anywhere between 65 and 95 tokens per second output on
my RTX GPU using much lower quality models (Qwen 3.5 9b
and Gemma 4 4b).</li>
</ul>
<div>In both cases, if I don't take steps to optimize the
model such that it stays 100% in VRAM, it slows to an
entirely unusable rate.</div>
<div><br>
</div>
<div>UP FRONT WARNING--I'm a noob with this, so take my
explanation with a grain of salt.</div>
<div><br>
</div>
<div>What does that actually mean? Well especially if you use
a "reasoning" model like Qwen, then a simple query response
(like "tell me a funny dad joke") can take up to a minute to
respond. This is because approximately every word of every
"thought" is an output token. It "talks to itself" until if
decides it has found a reasonable answer. And it adds up
quick. Here are some basic results for this exact query on
all 4 models / hardware:</div>
<div>
<ul>
<li>Qwen on RTX 4070: required 1200 tokens and 18 seconds
to respond</li>
<li>Gemma on RTX 4070: required 300 tokens and 3.5 seconds
to respond</li>
<li>Qwen on iGPU: required 630 tokens and 23 seconds to
respond</li>
<li>Gemma on iGPU: required 450 tokens and took 18 seconds
to respond</li>
</ul>
<div>Again that's moderately beefy hardware and a lot of
tweaking. But I could also do far better if I accepted
much dumber models. Which may be just find for basic
agentic work. But much less good for coding or reasoned
synthesis. And the "smartest" model took 23 seconds to
reason through a dad joke. 7 seconds just to figure out
how to respond to "hello". Working on truly complex
reasoning can take a bathroom+coffee break to give you
back results.</div>
</div>
<div><br>
</div>
<div>Note too that this doesn't take into account context size
and context caching. Context is the LLM's active memory.
Models have maximums (I think they are tuned for these
sizes). But in most cases you'll likely have to accept
something lower. However, to be even moderately useful for
much of anything, you can't go terribly low. Coding tools
and agentic tools just blow up if they can't remember things
from one thought to the next.</div>
<div><br>
</div>
<div>The numbers I'm getting on the RTX are decent. Almost
usable. BUT the context sizes to achieve that make it
basically unusable for the kind of work I want to do. If I
bump up the context window to a usable level with these
models, I leak out into RAM (far slower than VRAM) and my
performance tanks to unusable levels (like < 5-10
tps...at least 80-90% or more slower).</div>
<div><br>
</div>
<div>The iGPU with huge VRAM is slower than the dedicated GPU,
but because I can crank up the context window, they actually
become usable for what I want to use them for.
However...speed. Claude Sonnet or Gemini Flash are easily
10x faster at everything. And more like 20x-30x faster for
most reasoning work. So at some point it's a question of how
much you value your time.</div>
<div><br>
</div>
<div>I will be using local models for some basic stuff, I
think. I'm starting down a personal knowledge management
path with AnythingLLM or something similar. I think it'll
pair perfectly with this. And I'll likely find more ways to
leverage it. But I won't be abandoning the big boys any time
soon.</div>
<div><br>
</div>
<div>And sadly, although your Ultra 7 w/ 32GB RAM is an
awesome PC, I fear your experience with local llms for
anything other than experimentation and learning will prove
frustratingly slow. And with prices the way they are right
now, getting your PC spec'd to perform moderately well will
certainly cost around the same as a full year of one of the
"ultimate" plans.</div>
<div><br>
</div>
<div>Hope this was helpful. And that I didn't show my
ignorance too badly.</div>
<div><br>
</div>
<div>Best,</div>
<div>Paul York</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, May 20, 2026 at
11:35\u202fPM Lewis Wood via NFBCS <<a
href="mailto:nfbcs@nfbnet.org" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">nfbcs@nfbnet.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11pt">I am
currently learning as well.</span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt">I am
now doing Ollama playlist lessons #2 currently.</span></p>
<p class="MsoNormal"><span style="font-size:11pt"><a
href="https://www.youtube.com/playlist?list=PLvsHpqLkpw0fIT-WbjY-xBRxTftjwiTLB"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.youtube.com/playlist?list=PLvsHpqLkpw0fIT-WbjY-xBRxTftjwiTLB</a></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt">I did
my initial research on Lm Studio before I learned
about Ollama CLI</span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt">This
was my first Lm Studio and it was an excellent one
regarding resources, models, agents, etc. Even
discussed how to load partial in differing areas gpu
and ddr.</span></p>
<p class="MsoNormal"><span style="font-size:11pt"><a
href="https://www.youtube.com/watch?v=UngVdAsQEiU"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.youtube.com/watch?v=UngVdAsQEiU</a></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt">You
can search youtube \u201clm studio\u201d</span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt">Lewis
Wood</span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<div>
<div
style="border-width:1pt medium medium;border-style:solid none none;border-color:rgb(225,225,225) currentcolor currentcolor;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span
style="font-size:11pt;font-family:Calibri,sans-serif"> NFBCS <<a
href="mailto:nfbcs-bounces@nfbnet.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">nfbcs-bounces@nfbnet.org</a>>
<b>On Behalf Of </b>Joe Orozco via NFBCS<br>
<b>Sent:</b> Wednesday, May 20, 2026 10:08 PM<br>
<b>To:</b> 'NFB in Computer Science Mailing
List' <<a href="mailto:nfbcs@nfbnet.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">nfbcs@nfbnet.org</a>><br>
<b>Cc:</b> Joe Orozco <<a
href="mailto:jsorozco@gmail.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">jsorozco@gmail.com</a>><br>
<b>Subject:</b> [NFBCS] Local AI</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><span style="font-size:11pt">Hello,</span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt">With
Google following in Claude\u2019s footsteps in terms of
usage restrictions, can anyone speak to their
experience using local LLM options? I\u2019m looking at
Jemma 4 and trying to understand how accessible this
route might be with JAWS on Windows.</span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt">I\u2019m on
a fairly decent machine: 32 GB RAM, Ultra 7
processor, 4 TB SSD. I see they\u2019re recommending GPU
for some of the more robust models, but I want to
think most of what I\u2019m doing shouldn\u2019t require
gaming machine specs. If you beg to differ though,
let me know.</span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt">If
anyone can speak to Jemma alternatives, I\u2019d also be
interested. I don\u2019t think I\u2019ll suspend my
subscriptions, but with these usage limitations
feeling like the new standard, I want to spread my
usage a little so that I don\u2019t feel like I need to
be hitting the top subscriptions just to get more
mileage out of the five-hour increments.</span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt">Thanks
in advance for any tips,</span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt">Joe</span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span
style="font-family:"Times New Roman",serif">--</span></p>
<p class="MsoNormal"><span
style="font-family:"Times New Roman",serif">Joe Orozco: Your
Message, My Mission</span></p>
<p class="MsoNormal"><span
style="font-family:"Times New Roman",serif"><a
href="https://joeorozco.com/services/"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://joeorozco.com/services/</a></span></p>
<p class="MsoNormal"> </p>
</div>
</div>
_______________________________________________<br>
NFBCS mailing list<br>
<a href="mailto:NFBCS@nfbnet.org" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">NFBCS@nfbnet.org</a><br>
<a
href="http://nfbnet.org/mailman/listinfo/nfbcs_nfbnet.org"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://nfbnet.org/mailman/listinfo/nfbcs_nfbnet.org</a><br>
To unsubscribe, change your list options or get your account
info for NFBCS:<br>
<a
href="http://nfbnet.org/mailman/options/nfbcs_nfbnet.org/paul%40yorkfamily.com"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://nfbnet.org/mailman/options/nfbcs_nfbnet.org/paul%40yorkfamily.com</a><br>
</div>
</blockquote>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
NFBCS mailing list
<a class="moz-txt-link-abbreviated" href="mailto:NFBCS@nfbnet.org">NFBCS@nfbnet.org</a>
<a class="moz-txt-link-freetext" href="http://nfbnet.org/mailman/listinfo/nfbcs_nfbnet.org">http://nfbnet.org/mailman/listinfo/nfbcs_nfbnet.org</a>
To unsubscribe, change your list options or get your account info for NFBCS:
<a class="moz-txt-link-freetext" href="http://nfbnet.org/mailman/options/nfbcs_nfbnet.org/tyler%40tysdomain.com">http://nfbnet.org/mailman/options/nfbcs_nfbnet.org/tyler%40tysdomain.com</a>
</pre>
</blockquote>
<div class="moz-signature">-- <br>
<div>
<p>
<strong>Ty Littlefield (he/him/his)</strong><br>
</p>
<ul>
<li> <a href="https://tysdomain.com" target="_blank">My
Website</a>|
</li>
<li><a href="https://www.linkedin.com/in/ty-lerlittlefield/">Linkedin</a>|</li>
<li><a href="https://github.com/tlfdev">Github</a></li>
</ul>
</div>
</div>
</body>
</html>