cat _posts/2026-06-24-local-model-testing-en.md

24 June 2026

ai llm security local models

Local security LLM testing on Mac mini M4

lmstudio --local --prompt acmedesk-security-review

I tested local models on a practical task: analyzing a hypothetical Windows Electron application offline and producing a safe bug-hunting plan. I evaluated concrete Windows and Electron reasoning, safe local PoCs, and the ability to distinguish evidence from claims that cannot be verified without the binary, rather than sheer verbosity.

HostMac mini M4

Memory24 GB

RunnerLM Studio 0.4.16

Temperature0.47

In short, Qwen3.6 27B, Qwen3-Coder 30B, and Gemma 4 26B were clearly the best. They are usable, but largely pointless when frontier models are available.

Original prompt

Ты senior reverse engineer и Windows desktop apps bug hunter. Работай без интернета и не выдумывай факты.

Разбери гипотетическое приложение Windows:

- Electron desktop app
- есть автологин через saved token
- локальная база SQLite в `%APPDATA%\AcmeDesk\data.db`
- приложение открывает ссылки вида `acmedesk://open?path=...`
- есть автообновление через `https://updates.acme.local/latest.yml`
- в логах встречается строка:
  `spawn powershell.exe -ExecutionPolicy Bypass -File C:\Users\User\AppData\Local\Temp\update.ps1`
- пользователь может импортировать `.zip` backup-файл, внутри которого `settings.json`, `profile.db`, `attachments/`

Задача:

1. Назови 10 потенциальных классов уязвимостей в таком приложении.
2. Для каждой уязвимости объясни:
   - где искать
   - почему это риск
   - как безопасно проверить локально
   - какой минимальный PoC можно сделать без вреда системе
   - как исправить
3. Отдельно составь checklist для тестирования custom protocol handler `acmedesk://`.
4. Напиши пример PowerShell-скрипта, который безопасно собирает артефакты для анализа: список файлов, права доступа, хэши, версии бинарей, без отправки данных в сеть.
5. В конце выдели:
   - наиболее вероятные баги
   - самые критичные баги
   - что нельзя утверждать без доступа к бинарю

Отвечай структурированно. Если чего-то не хватает, явно помечай как предположение.

Results

qwen/qwen3.6-27b

score 8/10speed 6 tok/sec

The strongest security reasoning, covering Electron IPC, DPAPI, Zip Slip, TOCTOU, updater signatures, `%TEMP%`, ProcMon, and a solid artifact collector.

Open full answer

qwen3-coder-30b-a3b-instruct-mlx

score 7.5/10speed 41 tok/sec

Fast and useful, with a strong focus on protocol handling, Zip Slip, the updater flow, temporary files, and safe PowerShell collection.

Open full answer

google/gemma-4-26b-a4b

score 7/10speed 31 tok/sec

Concise and practical. It identified the `update.ps1` TOCTOU angle, DPAPI, signed updates, and the limits imposed by not having the binary.

Open full answer

qwen3.5-9b Claude 4.6 HighIQ

score 6.5/10speed 11.48 tok/sec

Good brainstorming, but several confident technical mistakes kept it below the top tier.

Open full answer

no screenshot

foundation-sec-8b-reasoning-mlx

score 5/10speed 6 tok/sec

Respectable for an 8B model, but not deep enough compared with Qwen3.6, Qwen3-Coder, or Gemma.

Open full answer

mistralai/devstral-small-2-2512

score 5.5/10speed 7.28 tok/sec

Some useful Windows checklist items, but too many unsupported RCE claims with no plausible mechanism.

Open full answer

zai-org/glm-4.6v-flash

score 5/10speed 11.3 tok/sec

Broader coverage than the weakest models, but poorer judgment and several unsafe PoC suggestions.

Open full answer

mistralai/magistral-small-2509

score 4.5/10speed 7.35 tok/sec

Cleaner than the weakest answers, but still too shallow for a real security review.

Open full answer

whiterabbitneo-v3-7b-mlx

score 4/10speed 12.7 tok/sec

Readable, but largely a collection of keywords. It missed strong signals in the prompt, including updater scripts, signing, DPAPI, and Electron-specific RCE preconditions.

Open full answer

deepseek-r1-0528-qwen3-8b-mlx

score 4/10speed 20.16 tok/sec

Identified broad attack surfaces, but ignored the requested format and failed to provide safe minimal PoCs or a useful protocol checklist.

Open full answer

ravenx-sec-8b-security-rath-128k-mlx

score 3.5/10speed 6 tok/sec

Disappointing for a security fine-tune: repetitive, overconfident, and weak on Electron and Windows internals.

Open full answer

openai-gpt-oss-20b-instruct

score 3/10speed 31 tok/sec

Superficially structured, but full of generic labels and questionable fixes. I would not trust it as a research plan.

Open full answer

codestral-22b-v0.1

score 2.5/10speed 7.58 tok/sec

The answer was mostly a generic enterprise checklist rather than a security assessment.

Open full answer

vulnllm-r-7b

score 2/10speed 12 tok/sec

The weakest result: mostly CWE-like words with little understanding of the scenario.

Open full answer

Takeaway

For local security work, the strongest models stayed close to the artifacts: acmedesk://, SQLite, saved token storage, latest.yml, update.ps1, and backup ZIP imports. The weaker models sounded fluent in security terminology but skipped the engineering steps needed to turn a signal into a verifiable finding.