lamp.
ResourcesPricingAbout

Evals

Evals are repeatable tests that score an AI system's outputs against expected results, so changes can be judged by evidence rather than impressions.

What it means

Short for evaluations. An eval is a list of inputs with the correct outcomes, run automatically and graded by what the system actually produced. Judge the resulting records or decisions, not how the agent worked internally. Tag the cases that matter most, such as money, stock or messages to customers, and require 100 percent on them. Run the evals before every release and whenever the underlying model changes.

Example

A stationery distributor writes 40 evals from past orders, including a part-shipment and a duplicate PO. A new instruction change breaks the duplicate case, and the failed eval blocks the release.

Records and checks

  • Record each case, the expected result, the version tested and the score history.
  • Verify that critical cases are tagged and that new bugs become new cases.
  • The owner approves the expected results; a second person reviews any case that is removed.

Common mistakes

  • Letting the pass mark slip downward to get a release out.
  • Writing cases only from imagination instead of from real past transactions.
Did not find your answer?Message the lamp team on WhatsApp. Ask about setup, your records, or anything we have not covered here.Ask us on WhatsApp →