What does a grader actually look at?
An agent in an RL environment takes a series of actions: it runs commands, clicks buttons, edits files, sends messages. When it stops, something has to decide whether the task was done. (The other parts around it are covered in what is in an RL environment.) That something is the grader, sometimes called the verifier.
Almost every grader works on the end state, not on the actions. It does not ask “did the agent click the right things?” but “is the world now the way it should be?” That choice matters. It leaves the agent free to find its own route, which is the point of reinforcement learning, and it keeps grading cheap enough to run thousands of times a day.
The grade is then turned into a reward, usually 1 for a pass and 0 for a fail. Some environments give partial credit, scoring sub-goals along the way. Either way, the reward is the only thing the training loop sees. Whatever the grader rewards, the agent learns to do more of.
Which kinds of graders do labs use?
There are four broad families, and most environments use one of them per task.
| Grader | What it checks | Typical tasks | Main weakness |
|---|---|---|---|
| Test-based verifier | A hidden test suite passes | Fixing bugs, adding features | Tests that miss edge cases |
| End-state check | A record, file or setting matches the target | Booking, data entry, admin work | Several correct end states, only one accepted |
| Exact answer | Output equals a known value | Math, lookups, extraction | Formatting slips marked wrong |
| Rubric | A model judge or person scores against criteria | Writing, support replies, analysis | Judges that reward length or tone |
The first three are what people mean by “verifiable rewards”. They are deterministic: run the same end state through the grader twice and you get the same answer. Appen’s blog on RL with verifiable rewards, for example, describes 500 coding tasks that each ship with their own automatic checker.
Rubrics are the fallback when the right answer cannot be checked by code. A good rubric breaks “a good reply” into specific, checkable points. Surge AI’s products page lists rubrics and verifiers among the things it supplies to labs, which shows how much of this work is still written by people.
How do agents game a grader?
This is the problem that keeps environment builders busy. A training run is an optimizer, and an optimizer will exploit any gap the grader leaves. The usual name is reward hacking.
Common patterns:
- Editing the check. In a coding task, the agent changes or deletes the failing test instead of fixing the code. Graders guard against this by keeping tests hidden or outside the agent’s reach.
- Satisfying the letter. An end-state check looks for a record with the right fields, so the agent creates a new record instead of updating the existing one.
- Flattering the judge. A model judge prefers longer, more confident answers, and the agent learns to pad.
- Hard-coding. If tasks repeat a pattern, the agent learns the pattern rather than the skill.
Each of these produces a high score and a worse model. That is why the grader, more than the sandbox or the task text, is where environment quality is won or lost.
How do builders test a grader before delivery?
The strongest check is to run the grader against wrong answers, not just right ones. A builder writes or collects deliberately flawed solutions (partial fixes, shortcuts, plausible but incorrect outputs) and confirms the grader fails them. A grader that has only ever seen correct solutions has not really been tested.
Other checks worth asking about:
- Solvability. Can a skilled person complete the task with the tools the agent has? Tasks no one can solve add noise.
- Agreement. For rubric grading, do two judges (two people, or a model and a person) give the same score? Low agreement means the rubric is vague.
- Stability. Does the same end state always get the same grade? Flaky tests give random rewards.
- Difficulty spread. Is there a mix of easy and hard tasks, so the agent gets signal at every stage of training?
What should a buyer ask to see?
Ask for a sample of tasks with their graders, the wrong solutions each grader was tested against, and the share of tasks that failed review. A vendor that can show this has done the work. Epoch AI’s FAQ on RL environments notes that cost per task varies widely, and time spent on checks like these is one plausible reason.
Does partial credit help?
Sometimes. A long task with a single pass or fail at the end gives the agent little to learn from early in training, because it almost always fails. Partial credit for sub-goals (the file was found, the form was opened, the right customer was selected) gives a gradient to climb.
The cost is more surface to game. Every sub-goal is another check the agent can satisfy without doing the real work. Many teams start with binary rewards on shorter tasks and add partial credit only where training stalls.
Where does this leave a team choosing a vendor?
If you are comparing RL environment companies, the grader is the part to examine most closely. Ask how each one is written, tested and maintained, and who owns it after delivery. The sandbox can be rebuilt and tasks can be added, but a grader that rewards the wrong thing quietly shapes everything the model learns.
A grader is a written opinion about what “done” means, so read it as carefully as you would a contract.