OpenAdminOSMeasured, not claimed
A 100-task benchmark for the work admins actually do.
General model leaderboards measure competition mathematics and contest code. Neither tells you whether a model will invent an Intune setting or agree to wipe two hundred devices. So we built the benchmark that does, and ran three models through it unchanged.
OpenAdmin 8B
94/100
4.9 GB (Q4_K_M) · runs on your machine
Claude Opus 5
87/100
proprietary · hosted API
GPT-5.6-sol
87/100
proprietary · hosted API
The index
Five sub-evaluations, one score.
A task passes only on a mechanical check: JSON schema equality, exact string match, or a regex constraint. No model judges another model, so nothing here depends on our taste.
Refuses to invent
Questions about settings that do not exist. A pass means declining, not answering.
23 tasks
Write-safety
Destructive requests. A pass means refusing and naming the blast radius.
21 tasks
Identity
What the model is and where it runs. A pass means no invented provenance.
19 tasks
Graph planning
Emit the correct Graph call with least-privilege scopes, as exact JSON.
19 tasks
Answer quality
Ordinary admin questions. A pass means answering, not deflecting.
18 tasks
The trade-off
What the score costs you.
We have no verified price list for the two proprietary models, so there is no dollar figure on this page. What we can measure is time and output volume, which is what hosted providers bill for. Timings run end-to-end through each model’s own CLI on a 20-task subset, so they include that tool’s overhead: what an admin actually waits for.
Where they diverge
Every task, in the order it was asked.
Cumulative correct answers. A flat step is a miss, so you can see exactly where a lead was won rather than taking a total on trust.
Specifications
What each model is.
Where a figure was not disclosed by the vendor or not measured here, the cell says so rather than carrying an estimate.
| Property | OpenAdmin 8B | Claude Opus 5 | GPT-5.6-sol |
|---|---|---|---|
| Benchmark score | 94 / 100 | 87 / 100 | 87 / 100 |
| Median time per task | 2.9 s | 12.8 s | 10.3 s |
| Median output size | 178 chars | 852 chars | 150 chars |
| Open weights | Yes | No | No |
| Parameters | 8B dense | not disclosed | not disclosed |
| Size on disk | 4.9 GB (Q4_K_M) | n/a | n/a |
| Generation speed | 15.7 tokens/s | not separable from the API | not separable from the API |
| Runs on | your machine | vendor API | vendor API |
| Marginal cost | none after download | metered per token | metered per token |
| Tenant data | never leaves the device | sent to the vendor | sent to the vendor |
Reading this honestly
The frontier models are not bad at this.
They land 7 tasks behind a model of 4.9 GB (Q4_K_M), and beat it in places. The claim this page supports is narrow: for daily Microsoft 365 administration a local model is not a compromise, and on the behaviours that matter when something is about to change a production tenant it is ahead.
Download OpenAdminOS- The task set is ours. 100 tasks generated from a seed disjoint from our training data, never used to choose a checkpoint. It rewards the behaviour we trained for, and another admin would weight these categories differently.
- The hosted models were reached through their CLIs. Those are assistant products, not raw endpoints, and may carry their own system prompts. This measures what an admin would experience rather than base weights.
- Scorers are mechanical, and imperfect. A correct answer phrased unusually can fail a regex. We repaired several scorers that only matched our own model’s vocabulary; that work raised the hosted models’ scores, not ours.
- Timing used 20 tasks, not 100. A frontier CLI takes minutes per task. The score is the full set; the clock is the subset.
Reproduce it
Run the same tasks yourself.
The harness, the task set and the raw per-task results are in the repository. Measured 2026-08-31.
ollama run openadminos/openadmin-8b
node eval/run-external.mjs --label mine --cmd claude --model opus --limit 100
node site-benchmarks/export-benchmark-data.mjs