lamp.
ResourcesPricingAbout

Agent evaluation

Agent evaluation is testing an AI agent on a fixed set of known cases, comparing its answers with the correct ones, and repeating the test after every change.

What it means

An agent can sound confident and still be wrong, so a few good chats prove little. Collect real cases from your own business, write down the correct outcome for each, and count how many the agent matches. Money or statutory cases are marked critical and must always pass. Rerun after any change to instructions, tools or model.

Example

A distributor keeps 60 past supplier invoices with the ledger treatment its accountant chose. After the agent's instructions are edited, all 60 are rerun: 57 match, and the three misses are all freight bills.

Records and checks

  • Record the cases, expected answers, run date and score.
  • Verify each failure is read individually and critical cases score 100 percent.
  • The process owner approves the expected answers; someone other than the editor reviews results.

Common mistakes

  • Testing only typical cases, so rare ones such as credit notes or part deliveries fail in live use.
  • Tuning the agent on the same cases used to score it, which flatters the result.
Did not find your answer?Message the lamp team on WhatsApp. Ask about setup, your records, or anything we have not covered here.Ask us on WhatsApp →