Reference
Measured results
Every number slowship quotes was measured, on real code, on one laptop. Here they all are, with what they don’t say.
#One real project
One project folder: the 174 dependencies of REA’s own npm package, plus 77 installed skills, 1 MCP server and a deliberately risky test build. A laptop with a 6 GB GPU, using gemma4:e2b.
| What | Measured |
|---|---|
| Things found | 275 (252 distinct) |
| Settled by rules alone | 202 |
| Same bytes as another copy, read once | 23 |
| Sent to the local model | 50, in 4 min 23 s of model time |
| Re-run with nothing changed | about 4 s |
| Code that runs, actually read | 3,827 of 3,829 files (99.95%); 454 test and example files left out on purpose |
| Compiled files read by REA | 4, about 2 to 4 s each: the MCP server (Rust, built without a crate list) and 3 native addons with no Go, Rust or .NET record |
| Who can spend your money | 9 of the agent’s tools hold 18 keys to 13 outside accounts: 3 read 10 keys in their own code, the rest are named in their instructions |
| Verdict | Held: the test build (an inert fixture with a fake key, risky Electron settings and an IPC handler that opens any address). 10 more worth a glance, 0 holds among real packages and skills |
The model isn’t repeatable bit for bit: between this run and the one before it, it moved 4 package verdicts by one step. Rules-only verdicts didn’t change at all.
#Compiled programs
Reading compiled programs installed on the same laptop, with REA 6.7.0. Nothing was started.
| File | What its build record says | Time |
|---|---|---|
ollama.exe | Go 1.26.0, 54 modules | 1.2 s |
uv.exe | Rust, 572 crates | 2.6 s |
PowerShell’s System.Management.Automation.dll | .NET, 62 references, 192 native imports, two flagged: CreateProcess and CryptUnprotectData | 5.6 s |
a C++ .node addon (node-pty) | No record: named, not read | 3.8 s |
VS Code’s Code.exe | Fuses as shipped: run as Node, NODE_OPTIONS and --inspect on, asar integrity off | 1.5 s |
| slowship’s inert .NET test loader, compiled | Held twice: decode-then-load in one method, a browser password file then a POST in another | 6.1 s |
#Against known malware
Measured on 200 agent skills from MalSkillBench: 100 malicious and 100 benign, a fixed and reproducible subset. Each rate has a range it is 95% likely to sit in, because 100 samples can’t pin it to the decimal.
| Rules only | Rules and gemma4:e2b | |
|---|---|---|
| Held | 12 of 100 malicious skills (7–20%), 2 of 100 benign | 13 malicious (8–21%), 3 benign |
| Held or worth a glance | 53 malicious (43–62%), 17 benign (11–26%) | 33–36 malicious, 8–9 benign, over two runs |
| Benign npm packages flagged (REA’s 174 dependencies) | 0 held, 18 worth a glance | 0 held, 3 worth a glance |
Put plainly
- A hold is usually right: 12 of 14 (between 60 and 96%).
- It catches about one malicious skill in eight as a hold. Most misses are attacks written as plain English in
SKILL.md, or code inside a markdown block, which slowship doesn’t read as code yet. - The model cuts false alarms by half or more, and it also lets more attacks through, because those attacks come with a description written to make them look expected.
- Malicious npm packages haven’t been measured yet.
#Rerun it yourself
The full write-up, every miss explained, and how to rerun it are in eval/EVAL.md. node eval/run.mjs compare lines two runs up skill by skill.