Skip to main content
Evals score your real traffic automatically. An eval is a rule that samples matching traces or spans after they arrive and scores them, either with a fast, deterministic code check or with an LLM judge. Scores show up on the trace they came from, feed the Overview pass-rate card, and can drive alerts.

​
Creating an eval

A short wizard walks you through three steps:
  1. Target: what to run on. Choose trace (score the whole run) or span (score individual steps). Optionally filter by agent, trace name, and for spans, span type and model.
  2. Check: what to verify. Pick a preset (below).
  3. Score: how to score. For LLM judges, pick a judge model. For checks that take a parameter, set it (a substring, pattern, or max length). Set a sample rate (1% to 100%) to control how much matching traffic is scored.

​
Code checks

Deterministic, free, and run without any external calls:

​
LLM judges

Judges send the input and output to a model that returns a 0.00 to 1.00 score or a pass/fail verdict with a reason. Presets cover relevance, helpfulness, coherence, conciseness, instruction following, completeness, toxicity and safety, tool selection, and RAG checks (faithfulness, context relevance, and correctness against a reference).
LLM judges use your own provider keys. Add one (below) before creating a judge eval. An eval with no usable key shows the status needs key and doesn’t score until a key is added. The available judge models depend on the deployment.

​
Provider keys

The Provider Keys page stores the LLM provider API keys your judges use, encrypted at rest and scoped per project. Keys are write-only: once saved, the value is never shown again, and the page only shows which providers are set up. Add or replace a key, or delete it.
Provider-key encryption requires FOGLAMP_SECRETS_KEY (32+ chars) on the server. Without it, the page shows “Encryption not configured” and judge evals can’t run.

​
Eval detail

Opening an eval shows its recent activity: scored count, average score, pass rate, and judge spend over the selected range, plus a table of recent scores (target, pass/fail or numeric score, the reason, and when). Each scored run also shows which prompt version it used, so a change in scores can be read against a prompt change. Each enabled eval also has a status of ok, needs key, or error, and an inline on/off toggle.