What is in an RL environment?
An RL environment is a resettable sandbox, a set of tasks to attempt inside it, the tools the agent may use, and a grader that turns each attempt into a reward. Vendors sell all four parts together, and the grader is where quality differs most.
What are the parts of an RL environment?
A sandbox
The world the agent acts in: a container with a code repository, a copy of a web app, a simulated desktop, a spreadsheet with a ledger in it. It resets to the same starting state for every attempt, so two runs of the same task can be compared.
Tasks
Each task is a starting state plus an instruction: "fix the failing test in this repo", "book the cheapest refundable fare", "reconcile these two invoices". A training run needs many tasks, varied enough that the agent learns the skill rather than memorizes the answers.
Tools
The actions the agent is allowed to take: a shell, a browser it can click and type in, an API for the app, a file editor. The tool set defines what the agent can and cannot do, and is part of what the vendor builds and documents.
A grader or verifier
Code (or a rubric applied by a model or a person) that looks at the end state and decides whether the task was done. For code, it is often a hidden test suite. For a booking task, it checks the database row. The grader is the hardest part to get right, because an agent will find any shortcut it leaves open.
Rewards
The number the training loop sees. Usually derived from the grader: 1 for a pass and 0 for a fail, or partial credit for steps along the way. How rewards are shaped decides what behavior is reinforced.
Logs and documentation
Trajectories (every action and observation), a written spec for each task, and notes on known failure modes. These let your team audit why a run passed and spot tasks that are broken or too easy.
What should I ask a vendor about each part?
| Part | Ask | A weak answer looks like |
|---|---|---|
| Sandbox | How fast does it reset, and can it run many copies at once? | Manual resets, or one copy per machine |
| Tasks | How are tasks written, reviewed and checked for duplicates? | No review step, or tasks generated from one template |
| Tools | Which actions are exposed, and are they documented? | "Whatever the app allows" |
| Grader | How was each grader tested against wrong answers? | Graders only tested on correct solutions |
| Rewards | Binary or partial credit, and who decided? | No written reward spec |
| Logs | Do we get full trajectories and a per-task spec? | Summary scores only |
What kinds of RL environments do companies buy?
Most work falls into a few families. Software engineering environments put the agent in a repository with failing tests. Computer-use environments give it a browser or desktop and a goal in a real-looking app. Business workflow environments copy the tools of a job (a CRM, a ticketing system, an ERP) and grade the end state. Reasoning environments pose math or science problems with checkable answers.
The family decides how hard grading is. A failing test is easy to check; whether a support agent handled a refund "well" needs a rubric. Expect rubric-graded work to cost more per task. The task costs page covers what has been published on price.
Questions about environment parts
- Is an RL environment the same as a benchmark?
- They share parts. Benchmarks hold a frozen task list plus a scorer, and exist to measure models. An environment is built to be trained in, so it needs many more tasks, fast resets and rewards a training loop can read. Many teams keep a held-out slice of their environment as an internal benchmark.
- What does "gym" mean here?
- It is shorthand for an environment an agent trains in, after the naming of early open-source RL toolkits that used it. Vendors use "gym", "environment" and "simulation" for much the same product.
- Who writes the grader, us or the vendor?
- Usually the vendor, to a spec you agree. Ask to see graders for a sample of tasks and how they were tested against wrong answers that should fail.
- What is reward hacking?
- An agent scoring well by exploiting a weak grader instead of doing the task, such as editing a test so it passes. Good environments are tested against this before delivery.
- Do we get the environment code or just access?
- It depends on the contract. Some vendors deliver code and tasks you run on your own infrastructure; others host the environment and you connect to it. Settle this before pricing, because it changes both cost and rights.
Related pages
- RL environment companies: who builds them
- Task costs: what one task costs to build
- How labs grade agents: graders and verifiers in more depth
Get a quote
Tell us the company ZIP code, a contact, and an outline of the environments you have in mind. We compare it with what each builder publishes and send you the shortlist by name.
How the site earns: we make introductions, and a builder you later contract with may pay us for yours.
Received. The next three things:
- Your outline is set against what each RL environment builder says it makes and the terms it publishes.
- A shortlist arrives by email: names, links, and a sentence on the fit for each.
- No builder hears about you until you answer yes. There is no mailing list to join.