grainbox

npm package for Windows and Linux

evals-lab

Test what models, HTTP endpoints and APIs answer, side by side, and score the answers with evals.

npx evals-lab

Get it on npmRead the docs

The evals-lab home screen, with runs, history, the eval and prompt library, sources and model connections side by side.
The prompt library, listing prompt wordings with their version, how many runs used each, and when.
The evals-lab home screen at phone width, with runs and recent history stacked.

evals-lab is a bench for comparing what a model answers. Run two prompts side by side over a whole dataset, against the same model, and score each answer against one fixed in advance. Identical-looking output tells you nothing about which wording is better. A graded test does.

It started as the prompt bench for an Android app that tags photos on-device, where one word in a prompt moved the results more than any runtime setting. Nothing in it is specific to that app now. A model is any address that speaks OpenAI-style chat completions, so Ollama, llama.cpp, Anthropic and OpenAI-compatible endpoints all work. The lab can also grade HTTP endpoints and Power Automate flows.

Runs on your machine

The lab runs locally in your browser and keeps datasets, prompts, runs and API keys in one directory on your machine. By default only your own machine can reach it. A new lab opens on two demo datasets of public-domain images, each with a starter pipeline, so you can run something straight away.

npm install -g evals-lab
evals-lab

Then open http://localhost:8080. It needs Node.js 22.18 or newer, Python 3.10 or newer, and ImageMagick.

Licence

evals-lab is an early test release, licensed under the PolyForm Internal Use License 1.0.0. You and your company may use and change it for your own internal work.