We gave 8 frontier models 40 tiny, exact tasks — hash a string, cross a DST boundary, amortize a mortgage. Unaided they averaged 83%. With TinyFn's MCP tools, every single model scored 100%.
Same model, same prompt, same grader — the only difference is tool access.
Wrong answers aren't approximate — they're confidently, plausibly wrong.
it said7a1dc586498842e5f533ef0de434dc2e4c5cefae4c7fdddb2ac51d6d07b2fc0c truthe8cb3385659eb3eb361697cab6328852df7287cf27e890a5acab5fa232435e5d
it said3f2a9c1d84e7b60f5c1a8d93e4b7f206 truth0e429bcc2f1569c0daaf53dae426653a
it saidWm/DqyBzYXlzOiBjYWJDqSDimJUgY29zdHMg4oKsNC41MA== truthWm/DqyBzYXlzOiBjYWbDqSDimJUgY29zdHMg4oKsNC41MA==
it said2024-03-10 14:30 truth2024-03-10 15:30
it said121,931,746,278 truth121,932,799,878
How many of the 8 models solved each task — unaided vs. with TinyFn tools.
| Task | Category | Unaided | With TinyFn |
|---|---|---|---|
| md5 | hash | 0/8 | 8/8 |
| sha256 | hash | 0/8 | 8/8 |
| crc32 | hash | 4/8 | 8/8 |
| haversine | geo | 5/8 | 8/8 |
| regex-count | regex | 5/8 | 8/8 |
| unix-to-iso | datetime | 6/8 | 8/8 |
| nth-prime | number | 6/8 | 8/8 |
| collatz-length | number | 6/8 | 8/8 |
| count-letter | string | 6/8 | 8/8 |
| base64-encode | encode | 6/8 | 8/8 |
| hex-encode | encode | 6/8 | 8/8 |
| miles-to-km | convert | 6/8 | 8/8 |
| loan-payment | finance | 6/8 | 8/8 |
| compound-interest | finance | 6/8 | 8/8 |
| iso-to-unix | datetime | 7/8 | 8/8 |
| days-between | datetime | 7/8 | 8/8 |
| tz-convert-dst | time | 7/8 | 8/8 |
| business-days | time | 7/8 | 8/8 |
| rot13 | encode | 7/8 | 8/8 |
| liters-to-uk-gallons | convert | 7/8 | 8/8 |
| dec-to-hex | convert | 7/8 | 8/8 |
| big-multiply | math | 7/8 | 8/8 |
| contrast-ratio | color | 7/8 | 8/8 |
| pearson-correlation | stats | 7/8 | 8/8 |
| percentile-85 | stats | 7/8 | 8/8 |
| day-of-week | datetime | 8/8 | 8/8 |
| add-days | datetime | 8/8 | 8/8 |
| iso-week-number | datetime | 8/8 | 8/8 |
| day-of-year | datetime | 8/8 | 8/8 |
| tz-convert-summer | time | 8/8 | 8/8 |
| base64-decode | encode | 8/8 | 8/8 |
| url-encode | encode | 8/8 | 8/8 |
| f-to-c | convert | 8/8 | 8/8 |
| factorial-23 | math | 8/8 | 8/8 |
| gcd | math | 8/8 | 8/8 |
| hex-to-hsl | color | 8/8 | 8/8 |
| luhn-check | validate | 8/8 | 8/8 |
| sample-stddev | stats | 8/8 | 8/8 |
| last-usable-host | ip | 8/8 | 8/8 |
| bmi | health | 8/8 | 8/8 |
Design. Every model runs every task twice with identical prompts:
unaided (no tools) and with TinyFn (the task's TinyFn MCP category
server, e.g. api.tinyfn.io/mcp/hash/, attached as a toolset via
pydantic-ai, models served through OpenRouter).
Grading. Ground truth is computed with the Python standard library,
independently of the TinyFn API. Checkers normalize case, whitespace, and number
formats, with tolerances on rounded values, and answers must arrive on an explicit
ANSWER: line — so verbose models aren't penalized and lucky prose
doesn't count. Tool use is verified from message history.
Full transparency. The harness, all 40 task definitions, raw
per-evaluation results, and instructions to reproduce are on GitHub:
github.com/tinyfn-io/tinybench.
The benchmark keeps score on both sides of the API: the initial run's only
two tool-condition misses exposed a timezone bug in TinyFn's own
time/convert endpoint. We fixed it, redeployed, and reran
those evaluations — hence 100%.
One MCP URL. 500+ deterministic tools. No more plausible-looking hashes.