Menu
JAQForum Ver 26.09

Forum Index : Microcontroller and PC projects : Running AI models on your own computer: a detailed investigation

Posted: 06:17pm
14 Nov 2026
Copy link to clipboard
matherp
Guru

https://mather.host/ai/local-models/

Local language models

Which AI models can you run on your own hardware that are actually useful for embedded C and MMBasic work? Ten models were tested on a workstation with a 20 GB graphics card, on real tasks: writing C that is checked under AddressSanitizer, answering questions about the MMBasic source, and extending a BASIC interpreter.

The write-up covers the method, the results, how to set up Ollama, and what more money buys, from a used 24 GB card to a 128 GB Ryzen AI Max+ machine.
 
Posted: 07:52pm
14 Nov 2026
Copy link to clipboard
homa
Guru


Peter,

Very interesting—thanks for sharing the AI research and findings. Cool site overall! Was it made with AI?

Matthias
 
Posted: 07:59pm
14 Nov 2026
Copy link to clipboard
matherp
Guru

  Quote  Was it made with AI?

No. I did all myself in an hour (NOT)
 
Posted: 01:08pm
15 Nov 2026
Copy link to clipboard
robert.rozee
Guru

hi Peter,
   you wrote:
  Quote  Even £10,000 of hardware doesn't buy a top cloud model; the largest open models need hundreds of gigabytes. The same money pays for the most expensive individual coding plans, about $200 a month, for over four years. Local models win on privacy, working offline and having no running costs, and they're good for routine work.


when considering a local setup, one should probably also factor in electricity costs. here in new zealand i am paying 25 cents per unit (kWh) of electricity. if i were to run a home machine that consumed 1kW continuously for a month - ie, working non-stop - that would add up to around nz$180 for the electricity alone.

even more extreme, a fellow over on the eevblog forums put together a setup that i estimate would be consuming around 10kW when running, ie nz$2500/month if working non-stop. the forum discussion (from page 14 onwards) is worth a read, see:
https://www.eevblog.com/forum/testgear/eevblog-121gw-multimeter-firmware-details/325/

so if you factor in home electricity cost, how much does that offset the cost of a $200/month plan?


cheers,
rob   :-)
Edited 2026-11-15 23:12 by robert.rozee
 
Posted: 01:15pm
15 Nov 2026
Copy link to clipboard
matherp
Guru

True, I'll add something to the analysis. The card I use is the RTX 4000 ADA SFF which has 20GB of VRAM and a max power draw of 70W. Electricity would certainly be different with a pair of RTX4090s at 450W each. Of course, even with them you are unlikely to run a local environment 24/7
 
Posted: 08:15pm
16 Nov 2026
Copy link to clipboard
Bleep
Guru


Hi Peter,
I'm trying to run your bench tests on my lowly laptop.
Asking a general question of gemma 4 26b I get 12 to 14 Tps
selftest works and passes, but the other test seem to fail, any ideas?
{
"when": "2026-11-16 19:52",
"os": "Windows-11-10.0.26200-SP0",
"cpu": "AMD Ryzen 7 5825U with Radeon Graphics",
"ollama": "0.40.16",
"ollama_env": {
 "OLLAMA_FLASH_ATTENTION": "1",
 "OLLAMA_KEEP_ALIVE": "30m",
 "OLLAMA_KV_CACHE_TYPE": "q8_0",
 "OLLAMA_NUM_PARALLEL": "1"
},
"gpu": "",
"ollama_server": {
 "OLLAMA_FLASH_ATTENTION": "true",
 "OLLAMA_KV_CACHE_TYPE": "q8_0",
 "OLLAMA_NUM_PARALLEL": "1",
 "OLLAMA_CONTEXT_LENGTH": "0",
 "OLLAMA_KEEP_ALIVE": "30m0s",
 "OLLAMA_MAX_LOADED_MODELS": "0"
},
"ram": "32 GB @ 3200 MT/s"

python selftest_hard.py
printf   good=PASS  bad=FAIL (correct) FAIL multi got [-5-ab-ff-Q-   07|x     |%|00000ABC] want [-5-ab-ff-Q-   07|x
expr     good=PASS  bad=FAIL (correct) FAIL [6 AND 3 OR 8] got 6 err 0 want 10 err 0
}

python selftest_real.py
basic      reference PASS
          unchanged                              fails: FAIL case 5 (pass 1): returned 20 want 110
          RETURN keeps inner FOR loops           fails: FAIL case 17 (pass 1): returned 130 want 0
          FOR/NEXT ignore the GOSUB boundary     fails: FAIL case 18 (pass 1): returned 0 want 110
          skip test ignores the step sign        fails: FAIL case 7 (pass 1): returned 0 want 0
          NEXT always tests <=                   fails: FAIL case 7 (pass 1): returned 0 want 0
          GOSUBs and FORs share 8 slots          fails: FAIL case 15 (pass 1): returned 110 want 0
          GOSUB stack not reset                  fails: FAIL case 14 (pass 1): returned 20 want 0
progstore  reference PASS
          both bugs                              fails: ==3619==ERROR: AddressSanitizer: global-buffer-overflow on address 0x5af172e159c0 at pc 0x71ca73728854 bp 0x7f
          memmove length only                    fails: ==3632==ERROR: AddressSanitizer: global-buffer-overflow on address 0x5f9191ef4940 at pc 0x776e13328854 bp 0x7f
          capacity check only                    fails: FAIL op 176: prog_store(65, 41 chars) returned -1 want 0 (1004 bytes used before)
heap       reference PASS
          free runs into the next block          fails: FAIL op 33: heap_free_bytes 4608 want 4096, heap_largest 2816 want 2816
          move frees the old block first         fails: FAIL op 323: heap_realloc(293) returned offset 5376 want -1 (-1 = NULL)
          grow in place does not zero            fails: FAIL op 139: heap_realloc(233 -> 284) new pages not zeroed
          first fit off by one page              fails: FAIL op 0: heap_alloc(6829) returned offset 256 want 0 (-1 = NULL)
selftest OK


Regards Kevin.
 
Posted: 01:57am
17 Nov 2026
Copy link to clipboard
dMajo
Newbie


To run local models the way to go is to dedicate an Mac Studio M5Max w 128GB of unified ram (or M5Ultra & 256GB) to the task an use your PC/Workstation as client, without investing in gpu hw upgrades.
 
Posted: 12:45pm
17 Nov 2026
Copy link to clipboard
matherp
Guru

Think I'll buy a couple of these. Should be perfect for PicoMite development  
 
Posted: 01:27pm
17 Nov 2026
Copy link to clipboard
Mixtel90
Guru


But you don't like Linux...

That's put you off, hasn't it?  :)
 
Posted: 02:23pm
17 Nov 2026
Copy link to clipboard
JohnS
Guru

Maybe get half a dozen to round the price up nicely.

John
 
Posted: 08:26am
18 Nov 2026
Copy link to clipboard
Bleep
Guru


I went ahead and tried it anyway. :-) Seems to have done something!

python bench.py report --markdown
| Model | Coding | General | PicoMite repo | Hard | Real work | Minutes per run | Runs |
|-------|--------|---------|---------------|------|-----------|-----------------|-----|
|gptoss20b| 6.0/6| 6.0/6   |   5.0/5       | 0/2  |  5/9      |            66.7 |   1  |

| Model | Context | Threads | On GPU | Gen t/s | Prompt t/s (22k) | Gen t/s after 22k |
|-------|---------|---------|--------|---------|------------------|-----------------|
|gptoss20b|  32K  | default |  0%    |  13.7   |    9.4           |  2.8            |

Edited 2026-11-18 18:33 by Bleep
 


To reply to this topic, you need to log in.

The Back Shed's forum code is written, and hosted, in Australia.
© JAQ Software 2026