Benchmark and environments

Shushrut-Bench

Language-model agents given the daily work of a hospital or clinic, inside a live electronic health record, and graded on the state they leave the record in.

The task set

The task set is under development. The tasks shown below are its primary tasks: single, well-bounded workflows drawn from current deployment scenarios, across front office, clinical operations, back office and operations management, and performed by reception, doctors, the lab desk, the pharmacy and administrators. They are suited to evaluating early computer-use and tool-use agents, and do not yet represent the intended scope of the benchmark. Some have nothing left to do, and the agent has to notice that.

Later releases add more complex tasks, including:

Further classes of operational workflow
Provider integration into the EHR, and navigation within it
End-to-end automation of patient workflows
Back-office automation
Management of internal hospital operations

If you are a doctor and think you can contribute to it, contribute

The Partner Track

Doctors who run a clinic or hospital can also become partners. Partners receive:

The Shushrut EHR, free for the first year
Reduced pricing after the first year
Setup, migration of existing records and staff training, at no cost
Early access to new modules, and priority for the features partners need
Credit in benchmark releases, and co-authorship on papers where a contribution meets authorship criteria

become a partner

The clinician panel defines the difficult workflows the task set should cover. Where a workflow depends on a capability the EHR does not yet have, that capability is implemented before the task is added, so that every task runs in the same environment.

Name Work email Speciality Hospital or clinic Practice setting Years in practice City and state Time you could give each month Your fee per accepted contribution A contribution is accepted when the workflow you describe becomes a task in the set. I’d like to become a partner A workflow you would add (optional)

Your details are used only to respond to your request.

Loading the tasks…

Environment

The Shushrut EHR, a multi-branch electronic health record and clinical operations system, run unmodified. The record is kept in HL7 FHIR R4, with queues, bookings and referrals, the laboratory, pharmacy stock by batch and administration linked to it.

The agent acts through the record’s own API calls, exposed as tools, and sees only what its role may see: an agent working as reception is refused clinical data. Every episode starts from a fresh copy of the task’s starting state, with newly generated patients.
Grading

A run is graded on the record after it ends, not on what the agent says. Each task lists checks: records that must exist, values they must hold, records that must not change. A task passes when every check holds.

Some actions fail a task outright: reading clinical data with reception access, registering a minor without a guardian, reusing a token, closing a visit with no diagnosis, dispensing from an expired batch, dispensing a Schedule H1 medicine without a register entry. Where code cannot judge, as with a consultation note, a clinician-written rubric is scored separately.
Evaluation

Every model runs the same tasks, with the same prompt and only its role’s tools. Task success is the share of tasks solved on the first attempt (pass@1); reliability, the share solved on every one of k attempts (pass^k). Alongside them: the share of runs with a safety failure, and the median number of actions per task. The number of attempts, the turn limit, sampling and the environment version are fixed before the first run.
Public data

A small public split of the task set, for development and for checking how the benchmark is built: one JSON record per task, with its starting state, role, instruction, checks and safety failures, and the prompt used for evaluation. Tasks are derived from real operational workflows and de-identified. Every patient in a task is generated; the dataset contains no patient health information.

Hugging Face: link at release.
Access

The full dataset is available to research groups and companies on request, as are task mixes weighted to a particular deployment and training on the environment against the same graders. To have a model evaluated, see Evaluate your model.

Name Work email Organisation Role Organisation type Intended use Data handling How you heard of us Interested in What you have in mind (optional)

Your details are used only to respond to your request.

Citation

@misc{shushrutbench2026,
  title  = {Shushrut-Bench: language-model agents in hospital and
            clinic operations, inside a live electronic health record},
  author = {{shushrut.ai}},
  year   = {2026},
  url    = {https://research.shushrut.ai}
}

Evaluate your model

We run a model you name on the full task set, under the same settings as every published evaluation, including models not yet released. Results come back by area, role and workflow, with every failed check.

Name Work email Organisation Model How we would reach it Will you provide credits or compute for the evaluation? Model size and context length Agent harness Results What you would like evaluated (optional)

Your details are used only to respond to your request.

Shushrut-Bench

Shushrut-Bench

The task set

Environment

Grading

Evaluation

Public data

Access

Citation

Evaluate your model