| Menu | JAQForum Ver 26.09 |
Forum Index : Microcontroller and PC projects : Running AI models on your own computer: a detailed investigation
https://mather.host/ai/local-models/ Local language models Which AI models can you run on your own hardware that are actually useful for embedded C and MMBasic work? Ten models were tested on a workstation with a 20 GB graphics card, on real tasks: writing C that is checked under AddressSanitizer, answering questions about the MMBasic source, and extending a BASIC interpreter. The write-up covers the method, the results, how to set up Ollama, and what more money buys, from a used 24 GB card to a 128 GB Ryzen AI Max+ machine. |
||||||
Peter, Very interesting—thanks for sharing the AI research and findings. Cool site overall! Was it made with AI? Matthias |
||||||
No. I did all myself in an hour (NOT) |
||||||
hi Peter, you wrote: when considering a local setup, one should probably also factor in electricity costs. here in new zealand i am paying 25 cents per unit (kWh) of electricity. if i were to run a home machine that consumed 1kW continuously for a month - ie, working non-stop - that would add up to around nz$180 for the electricity alone. even more extreme, a fellow over on the eevblog forums put together a setup that i estimate would be consuming around 10kW when running, ie nz$2500/month if working non-stop. the forum discussion (from page 14 onwards) is worth a read, see: https://www.eevblog.com/forum/testgear/eevblog-121gw-multimeter-firmware-details/325/ so if you factor in home electricity cost, how much does that offset the cost of a $200/month plan? cheers, rob :-) Edited 2026-11-15 23:12 by robert.rozee |
||||||
True, I'll add something to the analysis. The card I use is the RTX 4000 ADA SFF which has 20GB of VRAM and a max power draw of 70W. Electricity would certainly be different with a pair of RTX4090s at 450W each. Of course, even with them you are unlikely to run a local environment 24/7 |
||||||
Hi Peter, I'm trying to run your bench tests on my lowly laptop. Asking a general question of gemma 4 26b I get 12 to 14 Tps selftest works and passes, but the other test seem to fail, any ideas? { "when": "2026-11-16 19:52", "os": "Windows-11-10.0.26200-SP0", "cpu": "AMD Ryzen 7 5825U with Radeon Graphics", "ollama": "0.40.16", "ollama_env": { "OLLAMA_FLASH_ATTENTION": "1", "OLLAMA_KEEP_ALIVE": "30m", "OLLAMA_KV_CACHE_TYPE": "q8_0", "OLLAMA_NUM_PARALLEL": "1" }, "gpu": "", "ollama_server": { "OLLAMA_FLASH_ATTENTION": "true", "OLLAMA_KV_CACHE_TYPE": "q8_0", "OLLAMA_NUM_PARALLEL": "1", "OLLAMA_CONTEXT_LENGTH": "0", "OLLAMA_KEEP_ALIVE": "30m0s", "OLLAMA_MAX_LOADED_MODELS": "0" }, "ram": "32 GB @ 3200 MT/s" python selftest_hard.py printf good=PASS bad=FAIL (correct) FAIL multi got [-5-ab-ff-Q- 07|x |%|00000ABC] want [-5-ab-ff-Q- 07|x expr good=PASS bad=FAIL (correct) FAIL [6 AND 3 OR 8] got 6 err 0 want 10 err 0 } python selftest_real.py basic reference PASS unchanged fails: FAIL case 5 (pass 1): returned 20 want 110 RETURN keeps inner FOR loops fails: FAIL case 17 (pass 1): returned 130 want 0 FOR/NEXT ignore the GOSUB boundary fails: FAIL case 18 (pass 1): returned 0 want 110 skip test ignores the step sign fails: FAIL case 7 (pass 1): returned 0 want 0 NEXT always tests <= fails: FAIL case 7 (pass 1): returned 0 want 0 GOSUBs and FORs share 8 slots fails: FAIL case 15 (pass 1): returned 110 want 0 GOSUB stack not reset fails: FAIL case 14 (pass 1): returned 20 want 0 progstore reference PASS both bugs fails: ==3619==ERROR: AddressSanitizer: global-buffer-overflow on address 0x5af172e159c0 at pc 0x71ca73728854 bp 0x7f memmove length only fails: ==3632==ERROR: AddressSanitizer: global-buffer-overflow on address 0x5f9191ef4940 at pc 0x776e13328854 bp 0x7f capacity check only fails: FAIL op 176: prog_store(65, 41 chars) returned -1 want 0 (1004 bytes used before) heap reference PASS free runs into the next block fails: FAIL op 33: heap_free_bytes 4608 want 4096, heap_largest 2816 want 2816 move frees the old block first fails: FAIL op 323: heap_realloc(293) returned offset 5376 want -1 (-1 = NULL) grow in place does not zero fails: FAIL op 139: heap_realloc(233 -> 284) new pages not zeroed first fit off by one page fails: FAIL op 0: heap_alloc(6829) returned offset 256 want 0 (-1 = NULL) selftest OK Regards Kevin. |
||||||
To run local models the way to go is to dedicate an Mac Studio M5Max w 128GB of unified ram (or M5Ultra & 256GB) to the task an use your PC/Workstation as client, without investing in gpu hw upgrades. |
||||||
Think I'll buy a couple of these. Should be perfect for PicoMite development ![]() |
||||||
But you don't like Linux... That's put you off, hasn't it? :) |
||||||
Maybe get half a dozen to round the price up nicely. John |
||||||
I went ahead and tried it anyway. :-) Seems to have done something! python bench.py report --markdown | Model | Coding | General | PicoMite repo | Hard | Real work | Minutes per run | Runs | |-------|--------|---------|---------------|------|-----------|-----------------|-----| |gptoss20b| 6.0/6| 6.0/6 | 5.0/5 | 0/2 | 5/9 | 66.7 | 1 | | Model | Context | Threads | On GPU | Gen t/s | Prompt t/s (22k) | Gen t/s after 22k | |-------|---------|---------|--------|---------|------------------|-----------------| |gptoss20b| 32K | default | 0% | 13.7 | 9.4 | 2.8 | Edited 2026-11-18 18:33 by Bleep |
||||||
| The Back Shed's forum code is written, and hosted, in Australia. |