Can a small AI model help review cybersecurity evidence on a computer without a GPU? We compared two official Qwen2.5 instruction-tuned models under the same local conditions: 3B and 7B. We wanted to see whether they could identify security flaws in supplied material, how often they would make unsupported claims, and how long each review would take.
In this run, 7B identified the core mechanism in two of five planted vulnerabilities; 3B identified none. Both produced claims that did not hold up when we checked the source and local test controls.
How we ran the comparison
We used the models’ Q8_0 GGUF releases, which are eight-bit quantized versions rather than full-precision checkpoints. Both ran through the same pinned llama.cpp build on a CPU-only virtual machine with four AMD EPYC vCPUs and 32 GiB of provisioned RAM. We ran one model at a time, using the same eight cases in the same order.
The benchmark contained five deliberately planted vulnerabilities and three control cases with no demonstrated security flaw. Its materials included application source, SDK code, gateway configuration, a multi-file workflow, and a protocol capture. Each model reviewed one case’s supplied files at a time. Neither had a browser, code-execution tools, or access to the answer key. We separately ran local checks to verify what their answers claimed.
Both models used the same review prompt, four CPU threads, an 8,192-token context, temperature zero, and a 2,048-token answer limit. One 3B answer reached that limit, which matters when interpreting its result. We did not run a second pass without the limit.
Results at a glance
The 7B model found the central issue in a report-download handler: a report was looked up by its ID without binding that lookup to the organisation the user was allowed to access. In our synthetic case, that allowed a member of one organisation to receive another organisation’s report if they knew its ID.
It also spotted a flawed file-path check in a document preview feature. A user who could request previews could read a file in a sibling directory whose name shared the beginning of the intended document directory’s name.
Those were genuine vulnerabilities in the test fixtures, but the model’s explanations needed correction. In the report case, it also claimed that the organisation membership check was absent; the check was present. In the preview case, it claimed that any file could be read and suggested code execution. Our local proof showed a narrower read from a matching sibling path, with no execution.
Both models missed the other three planted mechanisms: a bearer token sent to a second origin after a redirect, a route authorised before its path was decoded, and a ZIP extraction sequence that used a symlink to redirect a later file write. Both correctly treated a failed password challenge as no demonstrated database access. On the other two control cases, they raised security claims that the supplied code and checks did not support.
Our rubric counted 14 distinct unsupported findings from 3B and 12 from 7B. Its mean scores were 1.12 and 1.62 out of 10 respectively. Those scores reflect this particular rubric’s penalties for incorrect findings; they are not general model ratings.
Speed on the same CPU machine
The 3B model took 21.98 minutes to complete all eight cases. The 7B model took 41.30 minutes. Their overall output rates, including prompt processing and request overhead, were 7.81 and 3.37 tokens per second. These durations are not a fixed-length speed test: 3B generated 10,306 output tokens, while 7B generated 8,359.
The highest observed process resident memory was 4.42 GiB for 3B and 9.08 GiB for 7B. Those are process high-water readings across each run, not per-case system-memory totals. Neither run failed or timed out.
What this tells us
On this CPU-only setup, 7B took longer and produced two useful security leads that 3B missed. It still missed three planted vulnerabilities and made unsupported claims alongside its correct findings. The 3B result is also qualified by the answer that hit the 2,048-token limit.
This was one run of eight synthetic cases, not a test against live products or a general ranking of the two models. The lesson for our research process is practical: a model can help point us toward a code path, but the proposed cause, attacker prerequisites, and impact must be checked before calling it a vulnerability.
