In our earlier article on the availability of powerful computers, we described how difficult it can be to obtain the hardware needed for large AI models. Scalattice offered a convenient way to test one without sourcing a machine: its hosted API let us send prompts to a catalog model and inspect the responses. Scalattice explains that API setup here.
Disclosure: Scalattice provided CyberS&D with a US$25 Kimi K3 API credit grant for this benchmark. We designed the tests, checked the answers and wrote the conclusions ourselves. We made no claim that this small set represents every use of the model.
Moonshot AI describes Kimi K3 as an open-weight model with 2.8 trillion total parameters and a native context capacity of 1,048,576 tokens. The Scalattice kimi-k3 route in our signed-in catalog advertised 131K context on 15 November 2026. Those are different specifications: our tests used the hosted route, and none approached the model’s native one-million-token maximum.
How we tested it
We used four sets of synthetic tasks with known answers:
Code debugging: 18 short cases, each sent twice as a fresh request.
Quantitative reasoning: 18 short cases, each sent twice.
Cybersecurity evidence review: five fixtures with a known security issue and three controls with no demonstrated issue. All eight completed once; seven completed an identical second run.
Archive reading: six shorter archives that required an exact record lookup, a highest-revision check or a correct “not found” result. All six completed once; one completed a second run.
Each scored request contained the full task in one message, with the answer key withheld. We used temperature zero and gave the model no tools or web access. We saved the request, answer, response time and reported token usage. Service errors were excluded from answer accuracy.
Results
Code debugging: The underlying keyed answer was correct in 36/36 completed responses after removing, where needed, a Markdown fence around the whole answer. 26/36 were valid raw JSON exactly as requested. The median end-to-end time was 2.12 seconds.
Quantitative reasoning: The underlying keyed answer was correct in 36/36 completed responses under the same narrow fence rule. 35/36 met the raw-JSON requirement. Median time was 2.09 seconds.
Cybersecurity review: In the first pass, the responses eventually identified all five planted issues and rejected all three controls. A provisional evidence-and-impact rubric gave 55/80 points across the eight cases. Seven cases had completed repeats: all seven again reached the right core verdict, but the paired rubric totals changed from 46/70 to 42/70.
Shorter archive reading: 0/7 completed replies obeyed the instruction to return only one raw JSON object. Each instead contained a long, invented back-and-forth, followed by one final JSON block with the correct seven fields and values. A separate, explicitly looser extraction found the correct final block in 7/7. Median response time was 55.6 seconds.
The chart keeps answer correctness, the requested output format, and service availability separate. Archive answers found at the end of invalid replies did not pass the JSON-only rule.
The short coding and reasoning cases were deterministic and small. Their perfect underlying scores are encouraging for those specific tasks, but they do not prove broad coding or mathematical ability. The cybersecurity result needs an even stronger qualification: every completed security response initially claimed that supplied files were missing, although the files were embedded in the request. In one case, the saved request already contained gateway.py and routes.json, but the answer said they were absent, then claimed they arrived later and showed a curl test transcript. The API record contains one user message and one assistant reply; our runner supplied no command tools and did not execute that test. The transcript is therefore a model-generated illustration, not observed test output. Other responses also overstated severity. A correct root cause at the end is useful; the complete response still needed careful human review before it could support a report.
The archive result tells a similar story from another angle. The model reached the right final answer on the shorter cases, yet failed the exact output instruction every time and invented a conversation in which records arrived in batches. The archive was actually supplied once, in full. One larger archive request completed but incorrectly said the archive was absent. We therefore cannot turn the seven correct extracted answers into a claim that this route reliably returns usable structured results, much less that we tested its 131K advertised limit.
Reliability and cost observations
Across 112 recorded API requests, 100 completed, 10 returned HTTP 503, and two streams ended incomplete. During part of the testing, Scalattice’s status page showed a partial outage. Shared capacity is one possibility; we could not determine whether capacity, routing, timeouts or another service condition caused the individual errors. We counted failed requests as availability observations, never as wrong model answers. Two longer coding and reasoning pilot requests also returned 503, leaving those cases without an accuracy result.
Early calibration revealed another limit to the setup. Requests specifying a 2,048-token response limit returned 5,126 and 12,278 billed output tokens. We removed the request-side limit for the scored benchmark. One of the two incomplete streams occurred later, without a response limit, so the original limit alone cannot explain that failure. Neither incomplete stream had a final completion marker or usage record. These observations do not establish whether the behavior came from the model, Scalattice’s serving system or their interaction.
The dashboard showed about AUD $7.24 of total test usage at the end of this pass, against a grant displayed as AUD $36.7125. That total includes calibration and service checks as well as the scored cases. The signed-in catalog listed AUD $3.74 per million input tokens and AUD $18.71 per million output tokens at the time; rates can change. We kept the raw usage records, including a few unexpectedly low reported prompt-token counts, rather than treating those anomalies as proven efficiency gains.
What we take from it
Scalattice made it straightforward for us to access a very large open-weight model without obtaining its hardware. On our small, short-answer tests, the kimi-k3 route often found the correct answer quickly. The harder lesson is about evidence and output discipline: correct final conclusions can arrive inside responses that invent a testing history, overstate impact or ignore a requested format. For cybersecurity research, we would use such output as a lead to verify, not as proof to submit.
These are results for the Scalattice kimi-k3 route as we observed it on one day. Scalattice’s terms explain that emergency model failover may occur; the returned model name alone does not independently prove which checkpoint served every call. We saw no direct evidence that failover occurred in this test.
