# Instrument with an AI agent
Source: https://docs.foglamp.dev/ai-instrument
Instructions for a coding agent to add Foglamp tracing to an app.
This page is written for an AI coding agent (Claude Code, Cursor, and so on)
asked to add Foglamp tracing to a codebase. If you're a human, the
[Quickstart](/quickstart) is friendlier.
You are instrumenting a **Vercel AI SDK** app so its runs appear in Foglamp.
Foglamp captures model, tool, and step boundaries from the AI SDK and renders
them as traces with cost, tokens, and latency. Make the smallest safe change
that gets one real run flowing, then enrich.
## Rules
* **Check the AI SDK version first and pick the matching path** (step 2):
`wrap()` from `foglamp/wrap` on AI SDK v4 to v6, or `fog.integration(...)`
on v7. Both capture identical traces. Never upgrade the app's AI SDK just to
instrument it. Steps 3 to 5 apply to both paths.
* **Never refactor working AI code to instrument it.** `wrap()` covers
`generateText`, `streamText`, `generateObject`, `streamObject`, and the
`ToolLoopAgent` / `Experimental_Agent` classes. Wrap in place; do not
rewrite agent classes into `generateText` calls or restructure pipelines.
* Prefer the installed package's types and README over memory. Don't invent
SDK APIs or hand-wire ingest endpoints; only use `foglamp`'s public API
below.
* The SDK does nothing without `FOGLAMP_API_KEY`, so it's safe to add in every
environment. Nothing throws and no spans are sent until the key is set.
* **Names are static string literals.** `agentName`, `workflowName`, and
`traceName` must be written as literal strings in the source, never template
literals, concatenation, or variables. Anything dynamic (a slug, URL, id,
date) goes in `metadata`, `workflowRunId`, or `sessionId` instead. See
[the mapping rules](#3-map-the-codebase-to-foglamps-model).
* Instrument one real entry point first, verify a trace appears, then expand.
## 1. Install and configure
Install `foglamp` with the repo's own package manager (check the lockfile;
don't introduce a second one):
```bash theme={null}
npm i foglamp # or: pnpm add / yarn add / bun add
```
```bash .env theme={null}
FOGLAMP_API_KEY=fl_your_key_here
# Hosted ingest is the default. Only set this when self-hosting:
# FOGLAMP_INGEST_URL=http://localhost:4000/ingest
```
## 2. Wire up Foglamp for the installed version
Check the installed `ai` version first (read the lockfile or `package.json`),
then follow the matching path. Every traced call needs a `traceName` or an
`agentName`.
### AI SDK v4, v5, or v6: `wrap()`
Wrap the `ai` module once, then bind a context with `fog.with(...)`. The
returned functions keep the AI SDK's own, fully typed signatures. `wrap()` also
covers `generateObject`, `streamObject`, and the agent classes; instrument them
in place.
```ts theme={null}
import * as ai from "ai";
import { wrap } from "foglamp/wrap";
const fog = wrap(ai);
const { generateText } = fog.with({ agentName: "summarizer" });
await generateText({ model, prompt }); // traced automatically
```
See [AI SDK v4 to v6 (wrap)](/sdk/wrap) for `fog.run(...)` (ambient context)
and the per-call `foglamp: {...}` option.
### AI SDK v7: `fog.integration()`
Attach the integration to each `generateText` / `streamText` call via the
`telemetry` option.
```ts theme={null}
import { foglamp } from "foglamp";
import { generateText } from "ai";
const fog = foglamp();
await generateText({
model,
prompt,
telemetry: {
integrations: [fog.integration({ agentName: "summarizer" })],
},
});
```
## 3. Map the codebase to Foglamp's model
This is the step that makes the dashboard useful, so spend real effort here:
read the codebase and decide what its agents, workflows, and sessions actually
are before writing any context. The full context surface:
| Property | What it is | Good values |
| -------------------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agentName` | The named, reusable LLM behavior responsible for the call | Stable, low-cardinality names from the code's own vocabulary: `"support-triage"`, `"summarizer"`. Never per-request values. |
| `workflowName` + `workflowRunId` | A named multi-step process, and one execution of it | `workflowName` stable like an agent name (`"ticket-pipeline"`); `workflowRunId` an id the app already has for this execution: a request id, job id, ticket id. |
| `traceName` | Human label for a one-off call that isn't an agent | The call's purpose: `"classify-email"`. |
| `sessionId` | Groups traces in one conversation or user thread, across workflows and agents | The app's existing chat, conversation, or thread id. Conversations only; a batch, cron, or pipeline run id is a `workflowRunId`, not a session. |
| `customer` *(optional)* | The end customer this call serves, for per-customer cost. `{ id, name?, imageUrl? }` | Only set it if the app serves distinct customers or tenants. `id` is dynamic and high-cardinality (the customer's id); this is the one place such an id belongs in a first-class field rather than `metadata`. |
| `metadata` | Everything else, as string key/values | `userId`, tenant, environment, prompt version, A/B variant. |
Rules the SDK enforces, at the type level and at ingest:
* Every call needs `traceName` or `agentName` (both is fine; the trace belongs
to the agent and displays the `traceName`).
* `workflowName` and `workflowRunId` go together; one without the other is an
error. Calls sharing a `workflowRunId` are stitched into one run, so the id
must be shared by every call in the run and unique per execution.
How to decide, concretely:
* **Find the agents.** A class, module, or function that owns a system prompt
and is invoked from more than one place is an agent: give it an `agentName`
taken from what the code calls it.
* **Find the pipelines.** A request handler or job that makes several model
calls (or calls several agents) before producing its result is a workflow:
put the same `workflowName` + `workflowRunId` on every call in it, including
calls made by nested agents. Use `fog.run(context, fn)` at the handler or
job entry point. It sets the context for everything inside, however deeply
nested, so you don't pass a trace parameter through every function
signature:
```ts theme={null}
await fog.run(
{ workflowName: "ticket-pipeline", workflowRunId: ticket.id, metadata: { userId } },
() => handleTicket(ticket) // every instrumented call inside is attributed
);
```
On AI SDK v7, calls a tool makes back into the model inherit the parent
call's workflow and session context automatically (see the
[SDK overview](/sdk/overview#nested-calls-inside-tools)), so `fog.run()` is
mainly for the entry point and for the v4 to v6 [`wrap`](/sdk/wrap) path.
* **Find the threads.** If the app has conversations (a chat, a support
thread), pass its id as `sessionId` on every call serving that thread. A
session is a conversation where a user goes back and forth. If no human is
conversing, there is no session: a batch run, cron job, pipeline execution,
billing period, or engagement cycle is not a session, even though its id
would technically group traces. Group executions with
`workflowName` + `workflowRunId` and put longer-lived business ids
(campaign, cycle, tenant) in `metadata`. When in doubt, omit `sessionId`;
it is optional.
* **Don't overload the names.** High-cardinality values (user ids, slugs,
URLs, dates, ticket numbers) belong in `workflowRunId`, `sessionId`, or
`metadata`, never in `agentName`, `workflowName`, or `traceName`, which
should each have a small, stable set of values. The mechanical check: every
name must be a string literal at the call site. If you catch yourself
writing a template literal or passing a variable, the dynamic part is
metadata:
```ts theme={null}
// ❌ Wrong: every page produces a distinct "agent", ruining grouping
fog.integration({ agentName: `writeQa(${site}/${slug})` });
// ✅ Right: one agent, the page identified in metadata
fog.integration({ agentName: "writeQa", metadata: { site, page: slug } });
```
```ts theme={null}
fog.integration({
agentName: "retriever", // who: the reusable behavior
workflowName: "support-ticket", // what process this run is part of
workflowRunId: ticket.id, // which execution (shared, pre-existing id)
sessionId: conversation.id, // which user thread
metadata: { userId: user.id, env: process.env.NODE_ENV ?? "dev" },
});
```
The context fields are identical on both paths. On v4 to v6 the same object
goes into `fog.with({...})`, a per-call `foglamp: {...}` key, or
`fog.run({...}, fn)` for run-scoped context; see [wrap](/sdk/wrap).
## 4. Flush in serverless
Long-running servers (Node, Bun) flush on a timer automatically. Serverless is
detected automatically (the `VERCEL` / `AWS_LAMBDA_FUNCTION_NAME` env vars) and
switches to per-call flushing, so check what the deployment target needs before
adding flush plumbing:
* **Vercel**: nothing to do. Foglamp reads `waitUntil` from the runtime's
request context automatically (no `@vercel/functions` dependency needed).
* **Cloudflare Workers / other serverless**: pass `waitUntil` in the config
(`foglamp({ waitUntil: ctx.waitUntil })`), or `await fog.flush()` before
each handler returns.
## 5. Optional: live HUD (dev only)
If the app has a React UI and a local dev server, offer to wire up the live
HUD: a dev-only floating overlay that streams runs (steps, tool calls, tokens,
cost) on top of the app as the user develops. It needs no API key and does
nothing in production or on edge/serverless, so it's safe to leave in. Two
lines:
* **Server**: pass `hud: true` to the existing `foglamp({ ... })` call.
* **Client**: render `` from `foglamp/hud` once near the root of
the client app (for example the root layout).
```tsx theme={null}
import { FoglampHUD } from "foglamp/hud";
// near the app root, e.g. in the root layout:
```
The route or handler that creates `foglamp({ hud: true })` must run on the
Node runtime, not edge. Skip this step entirely if there's no React frontend.
Full reference: [Live HUD](/sdk/hud).
## 6. Hand off
A trace only exists once a real model call runs, and that's the user's job,
not yours. **Do not write smoke tests, test scripts, demo endpoints, or
synthetic "first trace" calls**; they add code the user has to delete and burn
real model tokens. Once the build and typecheck pass, finish by telling the
user exactly how to trigger their own instrumented flows: which command to
run, which page to hit, which job to kick off. Then they can run the app and
watch the first traces appear in the dashboard (Overview / Traces), and stream
live in the HUD if they enabled it.
See the [SDK reference](/sdk/overview) and [data model](/concepts/data-model)
for every option.
# Health check
Source: https://docs.foglamp.dev/api-reference/ingest/health-check
/api-reference/openapi.json get /health
Liveness probe. Returns the current size of the in-memory write buffer.
# Ingest a batch of traces
Source: https://docs.foglamp.dev/api-reference/ingest/ingest-a-batch-of-traces
/api-reference/openapi.json post /ingest
Validates the payload, resolves the API key, prices each span, and buffers the rows for a flush to ClickHouse. Fire-and-forget: a 202 means the batch was accepted, not yet durably written.
# API reference
Source: https://docs.foglamp.dev/api-reference/introduction
The Foglamp ingest API.
The **ingest API** is the write path: it receives batches of traces from the
SDK, prices each span, and stores them. Most users never call it directly, the
`foglamp` SDK does, but it's a plain HTTP and JSON API you can call from any
language.
## Base URL
| Deployment | Base URL |
| ----------- | ---------------------------- |
| Self-hosted | `http://localhost:4000` |
| Hosted | `https://ingest.foglamp.dev` |
The interactive playground below defaults to the self-hosted URL.
## Authentication
Authenticate every request with a Foglamp API key (`fl_...`). Two equivalent
forms are accepted:
```bash Authorization header theme={null}
curl https://ingest.foglamp.dev/ingest \
-H "Authorization: Bearer fl_your_key_here" \
-H "Content-Type: application/json" \
-d '{ "version": "v1", "traces": [ … ] }'
```
```bash x-api-key header theme={null}
curl https://ingest.foglamp.dev/ingest \
-H "x-api-key: fl_your_key_here" \
-H "Content-Type: application/json" \
-d '{ "version": "v1", "traces": [ … ] }'
```
Keys are stored as hashes, so only the prefix is ever shown again after
creation. A revoked key is rejected immediately.
## Responses
| Status | Meaning |
| ------------------------- | ---------------------------------------------------------------------------------------------- |
| `202 Accepted` | Batch accepted. Body: `{ "accepted": }`. |
| `400 Bad Request` | Invalid JSON, or the payload failed validation (see the `issues` array). |
| `401 Unauthorized` | Missing, invalid, or revoked API key. |
| `413 Payload Too Large` | Body exceeded the size cap (`INGEST_MAX_BODY_BYTES`, default 10 MiB). Rejected before parsing. |
| `429 Too Many Requests` | Either a per-key rate limit or a plan quota. See below. |
| `503 Service Unavailable` | Server is shutting down. Retry against another replica or after restart. |
A `202` means the batch was accepted into a write buffer that is stored
shortly after. Treat `202` as success.
### Two kinds of `429`
A `429` can mean two different things, and they need different handling:
* **Per-key rate limit**: too many requests per second for one key. The
response includes a `Retry-After` header. Back off and retry after it.
* **Plan quota exceeded**: the organization's monthly span quota is used up.
This is a billing condition, not rate limiting. It carries no `Retry-After`,
and retrying will not work until the plan is upgraded or the quota resets.
The `foglamp` SDK does not retry failed batches. A failed send goes to
`onError` and the batch is dropped, so telemetry never blocks or retries
inside your app. If you call the ingest API directly, add your own retry with
backoff for transport errors and rate-limit `429`s, and treat a quota `429`
as final.
## Payload shape
The body is a versioned batch: `version: "v1"` plus a `traces` array (1-1000
traces, each with 1-2000 spans). See the [data model](/concepts/data-model)
for what each field means, or the **Ingest** endpoints in the sidebar for the
full schema and a live playground.
# Changelog
Source: https://docs.foglamp.dev/changelog
What's new in Foglamp.
## Updates
**Calmer Foggy panel resize.** When you drag the [Foggy](/dashboard) chat
panel past its minimum or maximum width, the rubber-band stretch now caps at
28px instead of 48px, so the edge reads as a gentle hint rather than a
bounce. The min and max widths, and the spring-back on release, are
unchanged.
## Bug fixes
**The homepage demo matches the app again.** The interactive demo on
[foglamp.dev](https://foglamp.dev) now mirrors the [Traces](/dashboard/traces)
toolbar: **Errors only** sits with the default filters, and **+ Filter** is
the same icon-only button you see in the dashboard. Prompt version chips in
the demo also render as a single label, so "v" no longer splits from the
version number.
## Improvements
* **Prompt templates read as prose.** On the agent page, a prompt version's
template is laid out as a document — headings, lists, paragraphs — instead of
a block of monospace text, with a **Raw** toggle for the exact text. Each
`{…}` slot is now a chip; hover it to see what runs actually put there and
how often. An estimated token count sits above the template. See
[Prompt versions](/dashboard/agents#prompt-versions).
* **Prompt filter everywhere.** The traces list's **Prompt version** filter
lives under **+ Filter** and no longer needs an agent selected first;
picking a version selects its agent.
## New features
**Prompt versions (SDK 0.10.0).** Foglamp now knows which system prompt each
run used, and groups an agent's prompts into versions without you declaring
anything.
* **System prompt on every run.** The SDK records the `system` text (or an
agent's `instructions`) on the root span, plus free metadata: generation
settings (temperature, max output tokens, ...), the output schema, provider
warnings, and the served model id. The prompt shows in the
[trace detail](/dashboard/traces) side panel and the agent drawer. Capture
is on by default and follows `recordInputs`; set `recordSystemPrompt: false`
to keep the conversation but drop the prompt. See
[Configuration](/sdk/configuration).
* **Versions on the agent page.** A background job normalizes the recorded
prompts, folds content that varies between runs (a user's name, a retrieved
snippet) into `{…}` slots, and opens a new version when the text actually
changes. Each version shows its template, run count, first-seen time, and a
diff against the previous version. See
[Prompt versions](/dashboard/agents#prompt-versions).
* **`v3` chips and a traces filter.** Runs carry a prompt-version chip in the
trace detail sheet and agent drawer, and the traces list gains a
**Prompt** filter once an agent is selected.
* **Prompt version per scored run.** On an eval's page, each scored run
carries the version chip too, so a drop in scores can be read against the
prompt change that caused it.
Upgrade the `foglamp` package to **0.10.1** to start recording prompts.
Traces from older SDK versions keep flowing; they simply show no system
prompt and don't take part in versioning. Self-hosters: this release adds a
ClickHouse migration (`0019_system_prompt`) and Postgres tables for versions;
deploy ingest and server before upgrading clients.
## Bug fixes
**`foglamp` 0.10.0 could not be installed.** Its manifest shipped an
unresolved `catalog:` dependency range for `uuidv7`, so bun refused to install
it. 0.10.0 is deprecated on npm; 0.10.1 is the same code with the range
resolved.
## New features
**Edit setup decisions before approving.** When a coding agent submits an
instrumentation plan, every decision on the review page is now editable
before you approve. Rename agents, workflows, and conversation labels, change
where the run, thread, and customer ids come from, and toggle customer
attribution on or off. Edited rows show an **edited** badge with a one-click
reset, and clearing a field falls back to the detected value. Your agent
picks up the merged plan automatically, with no changes on the agent side.
See [Instrument with an AI agent](/ai-instrument).
**Verification report.** After your agent applies the plan, the review page
turns into a receipt: instrumentation coverage (including skipped calls and
notes), a before/after map diff, files changed, and any warnings the agent
raised. Once the plan is verified, it also shows how quickly the first trace
arrived in [Traces](/dashboard/traces).
## Updates
**Higher-quality architecture maps.** Foglamp now rejects degenerate maps at
plan upload (no model nodes, everything drawn as an agent, disconnected
groups) with actionable errors your agent can fix and retry. The setup prompt
ends with a matching pre-upload checklist, so most maps pass on the first
try.
**A smoother review page.** Approving closes the tab when the browser allows
it, and says the tab is safe to close when it doesn't. Rejecting takes a
two-step confirm, so one stray click can't end a plan. The decision list uses
the full viewport height, and the long "id comes from" fields wrap in an
auto-growing textarea instead of hiding text in a single line.
**Structured API errors.** The ingest and server APIs now return JSON error
bodies with `error`, `code`, and `hint` fields instead of plain text, so both
humans and agents get something they can act on.
**Markdown for agents on foglamp.dev.** Every marketing page on
[foglamp.dev](https://foglamp.dev) now has a markdown variant: send
`Accept: text/markdown` or append `.md` to the path. 404 pages include
recovery links instead of a bare shell.
## Bug fixes
**No more setup dead ends.** If your agent stops polling after you approve,
the review page now tells you how to resume it (send the agent any message).
A plan whose agent dies mid-apply now expires instead of hanging forever, and
the page nudges you to check the agent's output after 10 minutes of applying.
**Customer attribution can't be approved without an id source.** The Approve
button is disabled with an inline hint instead of returning a validation
error after the click.
## Updates
**Simpler alerts, with the context in view.** New alert names are generated
from their condition, and new rules focus on cost, p95 latency, error rate,
and eval pass rate. The Alerts table now shows the current value and last
firing, with a compact marker beside rules that are actively firing instead
of a separate status column. Open the edit dialog to change a rule or review
its recent fired and resolved history.
## Updates
**Alerts as a list.** The [Alerts](/dashboard/alerts) page swaps its card grid
for the standard list table used across the dashboard: search, a firing-only
chip, sortable name and window columns, status badges (with a pinging
indicator for alerts currently firing), per-row enable/disable and delete,
and a pagination footer. Finding and managing alerts now works the same way
as on Traces, Sessions, Agents, and Workflows.
**Tighter Evals page.** The stat-card strip at the top of
[Evals](/dashboard/evals) is retired. The toolbar is the first row now, so
you land straight on your evals. Rows that errored or are missing a judge
key use the same inline red triangle you already know from
[Agents](/dashboard/agents).
**Overview breathing room.** A little extra space between the
[Overview](/dashboard/overview) header and the first KPI row, so the page
opens less cramped.
## Updates
**Cleaner span inspector header.** When you open a span in the
[trace detail](/dashboard/traces) side panel, the redundant span-type badge
next to the title is gone. The identity chip already tells you whether you're
looking at a model, tool, or other span, so the header now leads straight
into the span's name and details.
## Updates
**Accurate trace structure (SDK 0.9.0).** The waterfall now reflects what
actually happened in a run. Three changes, all in the SDK collectors:
* **Tool spans nest under their step.** A tool call is now recorded as a
child of the LLM step that requested it (matched by tool-call id), on both
the v7 collector and the v4 to v6 `foglamp/wrap` adapter. Indentation in
the [trace waterfall](/dashboard/traces) is now real structure, not
decoration.
* **LLM steps measure generation only.** A step span ends when the model
finishes generating: at the measured model-call end on v7, or where its
first tool starts otherwise. Tool time lives entirely in the step's tool
child spans, so step duration means "time spent in the model" consistently
across streaming, non-streaming, and both SDK majors.
* **Fixed 0ms and inflated step durations.** Non-streaming multi-step calls
(`generateText` via wrap) that fired tools in parallel could collapse a
step to 0ms and absorb its time into the next one. Per-step tool
attribution fixes this, so every step now carries its real duration.
Upgrade the `foglamp` package to **0.9.0** to get the corrected shape. Traces
from older SDK versions keep flowing and render as before (flat tool rows).
No ingest changes or migrations required; self-hosters can deploy in any
order.
**Waterfall cleanup.** The span-type filter chips introduced on August 7 are
retired. With tools now nested under their steps, hiding a type mid-tree
created more confusion than focus. Subtree collapse and the repeated-siblings
fold (`×N`) remain the way to tame long traces.
## Updates
**Faster filtering on Traces.** Long filter dropdowns on
[Traces](/dashboard/traces) now have inline typeahead, so picking a model,
agent, workflow, or customer from a large list is a keystroke instead of a
scroll. A new **Metadata** filter lets you narrow by any `key = value` you've
tagged, with a free-text fallback when a key has too many values to list.
You can pin a metadata key as its own column, and every filter, sort, and
pinned column is reflected in the URL: copy the link and a teammate lands on
the same view.
**Trace detail upgrades.** The trace detail page now leads with a time
composition strip that breaks a run into model, tools, other, and idle time
at a glance, and shows a low rate-limit headroom banner when a provider is
close to your TPM/RPM ceiling. Token and request counters sit next to the
cost breakdown, and a workflow run chip jumps straight to the
[workflow run](/dashboard/workflows) the trace belongs to.
**Waterfall filtering and collapse.** The waterfall now has span-type filter
chips (model, tool, other) to focus on the spans you care about, and any
subtree can be collapsed. The collapsed row shows how many spans are hidden,
so long tool loops stay readable.
## Updates
* Faster tab switching in the dashboard. Recently visited routes across
[Traces](/dashboard/traces), [Sessions](/dashboard/sessions),
[Agents](/dashboard/agents), [Workflows](/dashboard/workflows), and the
[Overview](/dashboard/overview) now swap in on a single paint instead of
bouncing through a loading skeleton. Page data still refreshes in the
background so numbers stay fresh.
## Updates
* Tightened the page titles and social descriptions on
[foglamp.dev/scan](https://foglamp.dev/scan) and individual scan posters so
shared links unfurl with clearer, shorter copy on socials and in search
results.
## Updates
* The [foglamp.dev/scan](https://foglamp.dev/scan) landing page is now
streamlined to a hero and CTA. The "One prompt, from repo to map" story
section has been retired so you can go from arriving to scanning faster.
* The scan hero map now scales responsively on narrower viewports, so the
preview no longer spills off a laptop screen and the surrounding layout
stays intact.
## New features
**Foglamp Scan.** Point a coding agent at any repo and get a shareable,
unlisted `foglamp.dev/scan/` page that maps how the codebase works and
uses AI: triggers, agents, models, tools, and stores laid out on a flow map,
with a scored personality card for the project. The page unfurls on socials
with an image of the actual map, and the prompt is one copy-and-paste from
[foglamp.dev/scan](https://foglamp.dev/scan). Anonymous, no account required;
posters expire after 90 days.
## Updates
* The scan map now scores personalities on trait dominance (so different
codebases land on different cards), groups the legend by kind
(Triggers / Agents / Models / Tools / Stores / External) with a hover
spotlight, and uses brand marks for model nodes.
* Pan and zoom on the scan map now write transforms straight to the DOM for
smooth interaction on large graphs, and deep pipelines open height-fit on
the start of the flow.
* Inter is now self-hosted from the canonical `rsms/inter` v4.1 build across
the marketing site, restoring the optical-size axis and OpenType character
variants that Google Fonts strips.
## Bug fixes
* Fixed a 500 on `/scan/` pages under Vercel's serverless runtime
caused by an `elkjs` module-interop mismatch. The layout engine is now
lazy-loaded and probes every export shape.
* Fixed the OG image route occasionally returning a 500 on cold starts by
caching and retrying font loads.
* Contained the marketing footer's film-grain effect to the footer box so
noise no longer bleeds above the top border, and softened the cube field on
the landing page.
## Updates
**Customer on traces.** The [Traces](/dashboard/traces) table now shows a
customer chip (avatar and name) next to the session, agent, and workflow
columns, and the trace detail page shows a matching customer badge, so you
can spot which customer a run belongs to without leaving the trace. Powered
by the [customer field](/concepts/data-model#customer) you set on the SDK
call.
**"Not identified" on the Overview Customers card.** The
[Overview](/dashboard/overview) Customers card now includes a muted **Not
identified** row for unattributed spend, so calls without a `customer` tag
are visible instead of disappearing. Avatars across the customers UI also
pick up a filled glyph for better contrast.
## Bug fixes
* Shortened the **Clear filters** button on list-page filter bars to
**Clear**, so it stops wrapping in narrow toolbars.
## New features
**AI SDK v7 stable.** The `foglamp()` collector now tracks AI SDK v7 stable
(`ai@7.0.0`). No code changes needed on your side; the telemetry API is
unchanged from the v7 beta.
**First-class `aborted` status.** A stream that's cancelled before it
finishes (via AI SDK v7's `onAbort`: a caller cancellation, an `AbortSignal`,
or a timeout) is now recorded with a dedicated **`aborted`** status instead
of being swept later as an error. It renders amber across the
[traces](/dashboard/traces) view and is excluded from the error rate, since a
cancelled run isn't a failure. The steps that finished before the abort keep
their own status.
**Automatic nested-trace context.** On the v7 collector, a model call made
inside a tool's `execute` now inherits the parent call's workflow, session,
customer, and metadata automatically, so sub-agents land in the same workflow
run with no `fog.run()` plumbing. See the
[SDK overview](/sdk/overview#nested-calls-inside-tools).
Self-hosters: run ClickHouse migrations before relying on aborted rollups (a
new `aborted_count` column is added to the trace, workflow-run, and
per-minute summaries); ingest accepts the `aborted` status after the contract
upgrade.
## New features
**Live HUD overlay.** Drop `` from `foglamp/hud` into any
client component and pair it with `foglamp({ hud: true })` on the server to
watch your agents run live: steps, tool calls, tokens, and cost streaming on
top of your app as you develop. It's for local development only, needs no API
key, and adds nothing to your production bundle. The whole thing is two
lines. See [Live HUD](/sdk/hud).
## New features
**Per-customer cost attribution.** Tag a call with a `customer`
(`fog.integration({ customer: { id, name?, imageUrl? } })`) to attribute its
spend to the customer it serves. A new **Customers** card on the
[Overview](/dashboard/overview) ranks your top customers by cost, with their
avatar and name. Foggy can answer per-customer cost questions too. Only `id`
is required and the field is fully optional. See the
[data model](/concepts/data-model#customer).
Self-hosters: deploy ingest before upgrading the SDK, since older ingest
rejects the new `customer` field (and run migrations to create the customer
rollup tables).
## New features
**`npx foglamp login` CLI.** Authenticate from your terminal with the new
device-authorization flow: run `npx foglamp login`, approve the code in your
browser, and Foglamp creates an API key and writes `FOGLAMP_API_KEY` to your
`.env`. Zero runtime dependencies (Node 18+ built-ins only), so it drops
cleanly into agent-driven setups. See the [Quickstart](/quickstart).
**Animated API key reveal.** Newly created API keys now roll from the name
you gave them into the real value on reveal, so it's obvious which key just
appeared. The keys table also has pinned column widths and shows **Last
used** before **Created**. See
[Projects, keys & billing](/dashboard/account).
## Updates
**Dark-mode polish across the dashboard.** Refined shadows, borders, and
hover states on cards, dialogs, and tables for better contrast and less
visual noise in dark mode.
## Bug fixes
* Foggy's hidden composer now blurs when the panel closes, so keystrokes
stop landing in it and the `F` shortcut reopens the panel reliably.
* Product analytics no longer load in local and preview environments.
## New features
**Model vs. tool latency split.** LLM steps now record the pure
provider-call time (`modelCallMs`) separately from client-side tool
execution. The span still covers the whole step; the waterfall adds a sky
**model** segment (tool time is the remainder), and the span inspector shows
the split. Captured on AI SDK v7. The v4 to v6 (wrap) and non-reasoning
paths simply omit it, with no estimates.
**Grounding sources.** RAG and grounding citations a model reports
(`StepResult.sources`) are captured per step and listed in the span
inspector. Recorded only when output capture is on.
**Model-drift fingerprint and safety ratings.** The OpenAI-style
`system_fingerprint` is captured as a queryable column (spot a silent weight
change across otherwise identical calls), alongside provider safety ratings.
No logprobs are captured.
**Rate-limit headroom.** Provider rate-limit response headers (OpenAI
`x-ratelimit-*`, Anthropic `anthropic-ratelimit-*`) are normalized into a
cross-provider set: requests and tokens remaining, the limit, and time to
reset. "You're at 90% of your TPM" becomes queryable. Only rate-limit
headers are read; no other headers are stored.
The provider signals above (sources, fingerprint, safety, rate-limit) are
captured for text and object generation alike (`generateText`, `streamText`,
`generateObject`, `streamObject`) and flow through the
`ToolLoopAgent`/`Experimental_Agent` wrappers too. The model vs. tool
latency split is the one exception: it needs the AI SDK v7 model-call
lifecycle, so it lands on v7 text and agent steps and is absent on the
object path and on v4 to v6. It is never estimated.
Self-hosters: deploy ingest before upgrading the SDK, since older ingest
rejects the new span fields.
## New features
**Reasoning (extended thinking) capture.** Both SDK collectors now record
reasoning stream chunks: per-step thinking duration plus a reasoning-token
curve, sampled exactly like text chunks. The trace waterfall shows a violet
**thinking** segment on reasoning steps, and the span inspector's
time-to-first-token splits into thinking time plus time to first visible
text. Models that don't reason (or SDK majors that don't report reasoning
tokens) send nothing, with no estimates. Self-hosters: deploy ingest before
upgrading the SDK, since older ingest rejects the new span fields.
**Agent cost breakdown donut.** The agent page now shows where an agent's
spend goes (prompt, completion, reasoning, cache read/write, requests,
images, and web search), summed per pricing dimension over the selected
range. See [Agents](/dashboard/agents).
**Ambient trace context with `fog.run()`.** Wrap any call with
`fog.run(context, fn)` and every traced call inside picks up the workflow,
session, and metadata automatically. No parameter passing, and singletons
stay singletons. See the [SDK overview](/sdk/overview).
**Typed `with()` context binding and agent-class wrapping.** `fog.with(ctx)`
now returns wrapped functions typed exactly like the originals, and
`foglamp/wrap` ships drop-in wrappers for `ToolLoopAgent` and
`Experimental_Agent` with per-call tool attribution. See
[AI SDK v4 to v6 (wrap)](/sdk/wrap).
**Forgot-password and account recovery.** A new password-reset flow, plus a
redesigned accept-invitation page with explicit accept/decline and a
wrong-account recovery path.
**Shareable list views.** Search, filter, sort, and pagination on
[Traces](/dashboard/traces), [Sessions](/dashboard/sessions),
[Agents](/dashboard/agents), and [Workflows](/dashboard/workflows) are now
reflected in the URL. Copy the link and a teammate lands on the same view.
**Profile picture in the sidebar.** Google sign-in users now see their photo
in the sidebar; email and password users keep the initials avatar.
## Updates
**Confirmation dialogs for destructive actions.** Deleting a project,
provider key, pricing override, invitation, or access grant now asks for
confirmation first. Project deletes require typing the name to confirm.
**Louder SDK auth failures.** The transport now warns once on a 401 or 403
from ingest, even without debug mode, so a bad API key can't silently drop
every trace.
**Vercel `waitUntil` without the peer dependency.** The SDK reads the Vercel
runtime context directly, so serverless flushing works without adding
`@vercel/functions` to your project.
**Eval scores on a 0 to 1 scale.** Numeric judges now score on a normalized
0.00 to 1.00 scale (previously 1 to 5), with an **Avg score** stat card on
the [Evals](/dashboard/evals) page that reflects active filters.
**Deleted active project no longer strands the app.** Deleting the currently
selected project now falls back to another project instead of leaving the
dashboard blank.
**Newly created projects stay selected.** Creating a project now keeps it as
the active project instead of switching back to the first one in the list.
**Onboarding key stability.** The onboarding panel no longer revokes earlier
keys you may have already pasted into your code. The created key is cached
and reused while valid.
## Bug fixes
* Bedrock model IDs and dash-versioned Anthropic IDs (e.g.
`claude-haiku-4-5`) now resolve to pricing instead of recording null cost.
* Hardened model-ID normalization for `-fast` Anthropic variants, 3.x IDs,
and more Bedrock creators (OpenAI, Qwen, Luma, Stability, TwelveLabs,
DeepSeek R1).
* Fixed a 500 in the usage-by-day query caused by a column alias collision.
* Trace inspector card shadow is now visible in light mode.
* Brightened the default button hover state in dark mode.
* Invitation flow now survives sign-in: unauthenticated users bounce through
login and return to the invitation, and zero-org dashboards list pending
invitations with an **Accept** button.
* Disabled form controls while changes are saving, to prevent duplicate
submissions.
* Removed loading flashes on the billing plan and encryption banner.
* Stale project-scoped detail routes now redirect on project switch.
* Escape closes the Foggy panel.
## New features
**Sessions view.** Follow a single conversation or user thread end to end.
Set a `sessionId` on your calls and Foglamp groups every turn, across
workflows and agents, under one session. See [Sessions](/dashboard/sessions).
**AI SDK v4 to v6 support via `foglamp/wrap`.** A new wrap adapter
instruments older AI SDK versions without requiring an upgrade to v7.
Drop-in replacement for `generateText`, `streamText`, `generateObject`, and
`streamObject`, producing the same traces to the same ingest endpoint. See
[AI SDK v4 to v6 (wrap)](/sdk/wrap).
**Web-search usage in traces.** Provider-reported web-search calls are now
captured alongside token usage and cost, so search-heavy agents are billed
and analyzed accurately.
**Compile-time workflow safety.** The SDK now enforces at the type level
that `workflowName` and `workflowRunId` are passed together. Invalid
combinations fail at build time rather than producing orphaned spans.
**Better dashboard navigation.** Breadcrumb back navigation, card/table view
toggles, and column filtering and sorting are available across list pages,
including [Traces](/dashboard/traces), [Workflows](/dashboard/workflows),
and [Agents](/dashboard/agents).
**Editable project name and URL.** Change a project's name and URL from
**Org settings → General** without recreating it. The URL drives the favicon
shown in the project switcher. See
[Projects, keys & billing](/dashboard/account).
**Eval judge model picker.** Choosing a judge model when creating an
[eval](/dashboard/evals) now uses an icon dropdown so providers and models
are easier to tell apart at a glance.
**Copy-paste instrumentation in empty states.** Empty Agents and Workflows
pages now show ready-to-run snippets so new projects can ship their first
trace without leaving the dashboard.
## Updates
**Syntax-highlighted code blocks.** Code blocks across the dashboard are now
rendered with Shiki and follow the app theme: `github-light` in light mode
and `vesper` in dark mode.
## Bug fixes
* Hardened the ingest path against oversized span metadata and capped
onboarding-key accumulation.
* Resolved access-control and denial-of-service issues surfaced in a
security review.
* Fixed the onboarding telemetry API so first-trace detection fires
reliably.
# Data model
Source: https://docs.foglamp.dev/concepts/data-model
Traces, spans, workflows, runs, agents, sessions, and customers.
Foglamp's model maps directly onto how the Vercel AI SDK runs. A handful of
concepts cover everything you see in the dashboard.
## Trace
A **trace** is one top-level model call: a single `generateText`,
`streamText`, `generateObject`, or `streamObject` invocation. It carries the
call's identity (name, agent, workflow, run, session), free-form metadata, and
a list of spans.
| Field | Type | Notes |
| --------------- | ------- | --------------------------------------------------------------------------- |
| `traceId` | string | 1-128 chars, unique per call |
| `traceName` | string? | up to 256 chars; required if `agentName` is absent |
| `agentName` | string? | up to 256 chars |
| `workflowName` | string? | up to 256 chars |
| `workflowRunId` | string? | up to 128 chars; groups traces into one run |
| `sessionId` | string? | up to 128 chars |
| `customer` | object? | `{ id, name?, imageUrl? }`, the customer served (see [Customer](#customer)) |
| `metadata` | map? | string to string |
| `spans` | span\[] | 1-2000 spans |
Two rules are checked in the SDK and again at ingest:
* Every trace must set `traceName` or `agentName` (both is fine, see [Trace name](#trace-name)).
* `workflowName` and `workflowRunId` go together. Pass both or neither.
## Span
A **span** is a unit of work inside a trace: a model step, a tool call, or
anything else. Spans carry timing, status, model identity, token usage, and
optional input/output text.
| Field | Type | Notes |
| ----------------------- | ------- | -------------------------------------------- |
| `spanId` | string | 1-128 chars |
| `parentSpanId` | string? | builds the waterfall |
| `spanType` | enum | `agent`, `llm`, `tool`, `embedding`, `other` |
| `name` | string | up to 512 chars |
| `startTime` / `endTime` | int | epoch milliseconds; `endTime ≥ startTime` |
| `status` | enum | `ok` (default) or `error` |
| `errorMessage` | string? | up to 8192 chars |
| `provider` | string? | e.g. `openai` |
| `modelId` | string? | e.g. `gpt-4o` |
| `usage` | object? | token counts, see below |
| `ttftMs` | number? | time to first token (may be fractional) |
| `input` / `output` | string? | up to 1,000,000 chars each |
| `metadata` | map? | string to string |
### Usage
Every usage field is an optional non-negative integer. Each is priced on its
own at ingest.
| Field | Meaning |
| ----------------------- | ------------------------------- |
| `inputTokens` | prompt tokens |
| `outputTokens` | completion tokens |
| `totalTokens` | total reported by the provider |
| `reasoningTokens` | reasoning/thinking tokens |
| `cachedInputTokens` | prompt tokens served from cache |
| `cacheWriteInputTokens` | tokens written to cache |
| `imageCount` | images generated |
| `webSearchCount` | web search calls |
| `requestCount` | provider requests |
## Workflow and run
A **workflow** is a named, repeatable process (for example `deploy-digest`). A
**workflow run** is one execution of it, identified by `workflowRunId`. Every
trace sharing a `workflowRunId` belongs to the same run, and the dashboard
shows them as a single timeline.
## Trace name
A **trace name** (`traceName`) is the call's human label. Use it for a one-off
call that isn't an agent, like
`fog.integration({ traceName: "classify-email" })`, so the call is easy to find
and group in the dashboard.
The label a trace displays is `traceName ?? agentName`. If you set only
`agentName`, that's the label. If you set both, the call belongs to the agent
and shows the `traceName`. Every trace must set at least one of the two.
## Agent
An **agent** is a named actor (`agentName`) responsible for a call. Agents give
you per-agent totals for cost, latency, and errors across every trace they
produced, no matter which workflow they ran in.
## Session
A **session** (`sessionId`) groups the traces that belong to one conversation
or user interaction, across workflows and agents. Use it to follow a single
user thread end to end.
## Customer
A **customer** (`customer`) is the person or company your app is serving with a
call, such as a tenant or end user. It rolls cost up per customer (the
**Customers** card on the [Overview](/dashboard/overview)), which is the
building block for usage-based pricing on top of Foglamp.
| Field | Type | Notes |
| ---------- | ------- | ------------------------------------------------------------------------------------------------------------------ |
| `id` | string | **required**, 1-128 chars, the stable grouping key. Unlike names, this is meant to be dynamic: one id per customer |
| `name` | string? | up to 256 chars, display label; the latest value seen wins |
| `imageUrl` | string? | up to 2048 chars, avatar URL; falls back to a generated icon |
`name` and `imageUrl` are display-only and can change over time (Foglamp keeps
the latest). `customer` is optional; leaving it out changes nothing.
## How it streams
```
generateText() ─────────────▶ trace
├─ step (model call) ──────▶ llm span (tokens, cost, ttft)
├─ tool call ──────────────▶ tool span
└─ step (model call) ──────▶ llm span
```
The SDK opens a span per step and tool call, reads the AI SDK's own performance
metrics for time to first token, and closes the trace when the call ends.
**Embeddings aren't captured yet.** Both SDK paths trace `generateText`,
`streamText`, `generateObject`, and `streamObject`, plus the agent classes.
`embed`, `embedMany`, and `rerank` are not traced yet: an `embed` call today
produces a trace with a root span but no token usage or cost. The `embedding`
span type is reserved for when this lands.
# Projects, keys & billing
Source: https://docs.foglamp.dev/dashboard/account
API keys, team roles, plans, and usage quotas.
Everything is scoped to a **project** inside an **organization**. Switch
projects from the dropdown at the top of the sidebar; organization-wide
settings live under **Configs** in the nav.
## Signing in
The dashboard signs in with email and password by default. Magic-link email and
Google sign-in are optional (see
[Configuration](/self-hosting/configuration)). Forgot your password? Use
**Forgot password?** on the login screen to get a reset link by email; the link
works for one hour.
Password-reset and magic-link emails require
[email](/self-hosting/configuration) to be set up (`RESEND_API_KEY`). On a
self-hosted instance without it, password login still works but reset links
can't be sent.
## API keys
Each project has its own ingest keys. On the **API Keys** page you can:
* **Create** a key (admins and owners only). The full `fl_...` value is shown
once, in a modal. Copy it then: only a hash is stored, so it can never be
shown again. After that, only the prefix is displayed.
* **Revoke** a key. This takes effect immediately; the
[ingest API](/api-reference/introduction) rejects it on the next request.
Point the SDK at a key with `FOGLAMP_API_KEY` (see the
[Quickstart](/quickstart)).
## Team and roles
The **Organization** settings cover members, invitations, and projects. There
are three roles:
| Role | Can do |
| ---------- | ------------------------------------------------ |
| **Owner** | Everything, including deleting the organization. |
| **Admin** | Manage projects, API keys, members, and billing. |
| **Member** | Read project data. |
Invite teammates by email with a role, and manage or revoke pending
invitations from the same screen.
## Plans and billing
The **Billing** tab shows your current plan (free, pro, enterprise, or
unmetered on self-hosted instances with billing off) and lets owners and admins
upgrade or open the billing portal.
## Usage
The **Usage** tab shows your consumption against plan limits for the current
period:
* **Spans this period**: the metered quota, with the reset date.
* **Projects**, **Alerts**, and **Evals**: counts against their limits.
A bar turns red as it nears its limit. When the span quota is nearly used up,
an app-wide banner appears (amber at 90%, red at 100%).
Once the monthly span quota is used up, the ingest API rejects new spans with
a `429`. This is a billing condition, not rate limiting (see
[Two kinds of 429](/api-reference/introduction#two-kinds-of-429)). Upgrade
the plan or wait for the period to reset.
# Agents
Source: https://docs.foglamp.dev/dashboard/agents
Per-agent totals for cost, latency, and errors across every run.
An **agent** is a named actor responsible for a call (`agentName`). The
**Agents** view rolls every trace up by agent, no matter which workflow it ran
in, so you can answer "how is my `retriever` doing overall?"
Set `agentName` on a call's [integration](/sdk/overview) to fill this view.
## Agents list
A card per agent over the selected range: span count (with the LLM-span
subset), tokens, p95 latency, and cost, with an error badge when any of its
spans failed. A toggle switches between cards and a compact table.
## Agent detail
Opening an agent shows:
* **Stat cards**: spans, error rate, p95 latency, and total cost.
* **Spans & errors** and **Latency**: trend charts over the selected range.
* **Cost breakdown**: a donut of cost per pricing dimension.
* **Prompt versions**: the system prompts this agent has run with, grouped
into versions (see below).
* **Trace flow**: a graph of the agent's most recent trace, with its steps and
tool calls. Click through to the full [trace](/dashboard/traces).
* **Recent traces**: the agent's latest runs with spans, tokens, duration, and
cost. Click a row to open the trace.
### Prompt versions
Foglamp reads prompt versions off your runs — there is nothing to declare. When
the SDK records a system prompt (on by default, see
[`recordSystemPrompt`](/sdk/configuration)), a background job groups the
distinct prompts an agent has run with into versions:
* **Slots**: content that changes between runs (a user's name, a retrieved
snippet, a date) is folded into the version's template as a `{…}` line, so
a personalized prompt is still one version.
* **Edits**: when the prompt text changes and the new text keeps being used
for later runs, a new version opens. A one-off variation does not.
Each version shows when it was first seen, how many runs used it and its
share of all runs, and how many distinct prompt texts fold into it. Select a
version to read its template beside the list or diff it against the previous
version. To see the runs behind a version, use the **Prompt version** filter
on the [traces](/dashboard/traces) page; the `v3` chip next to a trace's
system prompt links back here.
The template reads as a document: headings, lists, and paragraphs are laid
out, and values the versioning replaced (`{date}`, `{id}`, `{n}`, …) show as
quiet tokens. Each slot is a **varies per run** chip; hover it to see the
text runs actually put there, with how many runs carried each value. An
estimated token count sits above the template. Use **Raw** to see the exact
template text instead.
Versions are inferred, so they can be re-grouped as more runs land — for
example, a new prompt held with the current version until it has run a few
times becomes its own version once it has.
Agents and workflows are independent views. An agent's totals cover every
workflow it took part in; a [workflow](/dashboard/workflows) groups whichever
agents ran together in one process. Set both `agentName` and `workflowName`
to get both views.
# Alerts
Source: https://docs.foglamp.dev/dashboard/alerts
Threshold rules on cost, latency, errors, and eval pass rate.
**Alerts** are threshold rules checked continuously against a rolling window of
your traffic. When a metric crosses its threshold, the rule starts **firing**
and sends an email. When it recovers, the rule returns to **ok** and sends a
resolve email. A rule that keeps firing re-notifies on a cooldown.
## Parts of a rule
| Field | Description |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Name** | Generated from the condition when you create the rule, such as `Cost above $500`. You can customize it later. |
| **Metric** | What to measure. See the table below. |
| **Condition** | Comparison operator: `>`, `≥`, `<`, `≤`. |
| **Threshold** | The number the metric is compared against. |
| **Over the last** | How much recent traffic to include each time the rule is checked: 5 minutes, 15 minutes, 1 hour, or 24 hours. It is a rolling lookback, not a fixed calendar period. |
| **Notify** | An email address to notify. |
| **Enabled** | Turn the rule on or off without deleting it. |
## Metrics
| Metric | Meaning |
| ---------------- | ------------------------------------------------------------------------- |
| `cost` | Total cost (USD) over the selected lookback. |
| `latency_p95` | p95 step latency. Entered in milliseconds. |
| `error_rate` | Share of spans that errored. Entered as a percentage. |
| `eval_pass_rate` | Pass rate for a chosen [eval](/dashboard/evals). Entered as a percentage. |
Existing rules that use a legacy metric continue to run and can still be
edited. New rules offer the four metrics above.
## How checking works
A background job sweeps every enabled rule on an interval
(`ALERT_EVAL_INTERVAL_MS`, default 60 seconds). For each rule it computes the
metric over the window and compares it to the threshold:
* **ok to firing**: threshold crossed. Records a `fired` event with the
observed value and threshold, and sends an email.
* **firing to ok**: metric recovered. Records a `resolved` event and sends an
email.
* **still firing**: re-notifies at most once per `ALERT_RENOTIFY_MS` (default
1 hour), so you aren't emailed every minute.
Each transition is stored as an alert event, so every rule has a history of
when it fired and resolved.
Alert emails require email to be set up on the deployment
(`RESEND_API_KEY`). Without it, rules still run and change state, but no
emails are sent. See
[self-hosting configuration](/self-hosting/configuration#email-optional).
## List view
The Alerts page lists every rule with its metric, condition, current value,
lookback, last evaluation, and most recent firing. A small red marker appears
beside the name only while an enabled rule is firing. Select **Edit** to change
the rule and review its recent fired and resolved history. Plans cap how many
rules an organization can create; the
[Usage](/dashboard/account#usage) panel shows your count against the limit.
# Cost & pricing
Source: https://docs.foglamp.dev/dashboard/cost
How Foglamp prices spans.
Foglamp prices every span the moment it arrives, using the token counts the
provider reported. Cost is never recomputed later, so what you see in a trace
is what was calculated when it landed.
## How pricing works
Each [usage dimension](/concepts/data-model#usage) is priced on its own
against a model price table sourced from
[OpenRouter](https://openrouter.ai/api/v1/models) and refreshed every 24 hours.
The priced dimensions are prompt, completion, request, image, web search,
reasoning, cache-read, and cache-write tokens.
The span detail panel shows a **pricing source** so you can see where each cost
came from.
A model Foglamp can't find a price for shows as **`(unknown)`** in cost
breakdowns, never as `$0`. The Overview's "% priced" number tells you what
share of LLM spans got a price; a low number usually means you're running a
model OpenRouter doesn't list yet.
## Where cost shows up
* **[Overview](/dashboard/overview)**: total cost, cost over time stacked by
model, and per-model and per-agent cost tables.
* **[Traces](/dashboard/traces)**: per-span cost and pricing source.
* **[Agents](/dashboard/agents)** and **[Workflows](/dashboard/workflows)**:
cost totals per agent and per run.
* **[Alerts](/dashboard/alerts)**: alert on `cost` over a time window.
# Evals
Source: https://docs.foglamp.dev/dashboard/evals
Score production traffic with code checks or LLM judges.
**Evals** score your real traffic automatically. An eval is a rule that samples
matching traces or spans after they arrive and scores them, either with a fast,
deterministic **code check** or with an **LLM judge**. Scores show up on the
[trace](/dashboard/traces#scores) they came from, feed the
[Overview](/dashboard/overview) pass-rate card, and can drive
[alerts](/dashboard/alerts).
## Creating an eval
A short wizard walks you through three steps:
1. **Target: what to run on.** Choose `trace` (score the whole run) or `span`
(score individual steps). Optionally filter by agent, trace name, and for
spans, span type and model.
2. **Check: what to verify.** Pick a preset (below).
3. **Score: how to score.** For LLM judges, pick a judge model. For checks that
take a parameter, set it (a substring, pattern, or max length). Set a
**sample rate** (1% to 100%) to control how much matching traffic is
scored.
## Code checks
Deterministic, free, and run without any external calls:
| Preset | Checks |
| ---------------------------- | --------------------------------------------------------- |
| **No PII** | Output has no emails, phone numbers, SSNs, cards, or IPs. |
| **No secret leak** | Output has no API-key, token, or private-key shapes. |
| **Valid JSON** | Output parses as JSON. |
| **No refusal** | Output isn't a refusal. |
| **Non-empty** | Output isn't empty. |
| **Max length** | Output is within a character budget. |
| **Contains / Excludes text** | Output does (or doesn't) contain a substring. |
| **Regex match** | Output matches a pattern. |
| **Tool args valid** | (spans only) A tool call's input is a valid JSON object. |
## LLM judges
Judges send the input and output to a model that returns a 0.00 to 1.00 score
or a pass/fail verdict with a reason. Presets cover relevance, helpfulness,
coherence, conciseness, instruction following, completeness, toxicity and
safety, tool selection, and RAG checks (faithfulness, context relevance, and
correctness against a reference).
LLM judges use your own provider keys. Add one (below) before creating a
judge eval. An eval with no usable key shows the status **needs key** and
doesn't score until a key is added. The available judge models depend on the
deployment.
## Provider keys
The **Provider Keys** page stores the LLM provider API keys your judges use,
encrypted at rest and scoped per project. Keys are write-only: once saved, the
value is never shown again, and the page only shows which providers are set up.
Add or replace a key, or delete it.
Provider-key encryption requires `FOGLAMP_SECRETS_KEY` (32+ chars) on the
server. Without it, the page shows "Encryption not configured" and judge
evals can't run.
## Eval detail
Opening an eval shows its recent activity: scored count, average score, pass
rate, and judge spend over the selected range, plus a table of recent scores
(target, pass/fail or numeric score, the reason, and when). Each scored run
also shows which [prompt version](/dashboard/agents#prompt-versions) it used,
so a change in scores can be read against a prompt change. Each enabled eval
also has a status of **ok**, **needs key**, or **error**, and an inline on/off
toggle.
# Overview
Source: https://docs.foglamp.dev/dashboard/overview
The dashboard home: key numbers, trends, and breakdowns.
The **Overview** is the dashboard home. One screen answers "what is my AI app
doing right now, and how does it compare to before?" Every number respects the
[date-range picker](#date-range) at the top of the page, and each card shows
the change compared to the previous window of the same length.
## KPI cards
| Card | What it shows |
| ------------------ | -------------------------------------------------------------------------------------------------------- |
| **Total cost** | Cost for the window, an estimated monthly rate, and the share of LLM spans that got a price. |
| **Error rate** | Errored spans divided by total spans. |
| **Eval pass rate** | Pass/fail ratio across scored traffic, with the number of checks scored (see [Evals](/dashboard/evals)). |
| **Latency p95** | p95 step latency, with p50 and time-to-first-token p95 underneath. |
| **Requests** | Span count, with the LLM-span subset. |
| **Tokens** | Total tokens, split into input and output. |
For cost, error rate, and latency, the change colors are flipped: red means the
number went up (worse), green means it went down.
## Trend charts
* **Cost over time**: a stacked line chart of cost per minute, split by model.
The top five models get their own line; the rest are grouped as **Other**.
* **Requests & errors**: spans per minute with errors overlaid.
* **Latency**: p50 / p95 / p99 per minute.
* **By model**: a table of requests, tokens, p95 latency, and cost per model.
* **By agent**: the same, grouped by `agentName`.
* **By workflow**: the same, grouped by `workflowName`. Traces with no workflow
show as **Ungrouped**.
* **By customer**: cost, requests, and errors grouped by `customer`, with the
customer's avatar and name. Only traces that set a `customer` appear here.
Models Foglamp can't price show as `(unknown)` rather than `$0`. A missing
price is never treated as free. You can add a rule on the
[Pricing](/dashboard/cost) page to price them.
## Date range
The range picker drives every view. Presets: last hour, last 24 hours (the
default), 7 / 30 / 90 days, today, this month, last month, plus a custom
calendar. Future dates are disabled.
# Sessions
Source: https://docs.foglamp.dev/dashboard/sessions
Follow one conversation or user thread end to end.
A **session** groups the traces that belong to one conversation or user
interaction, across workflows and agents. Set a `sessionId` on your calls,
typically a chat or thread id, and Foglamp groups every turn under it.
Only traces with a `sessionId` appear here. If you don't set one, the call is
still recorded under [Traces](/dashboard/traces); it just isn't grouped into
a session.
## Sessions list
A paginated table over the selected range: session id, the agent involved, turn
count, tokens, cost, and last activity. Sessions with errors carry a badge.
## Session detail
Opening a session shows stat cards (turns, tokens, duration, cost) and a
conversation timeline, one block per turn, in order. Each turn shows:
* the user message, with a control to view the full raw input if it was cut
off,
* the assistant's output for that turn,
* the turn's cost, tokens, and duration,
* a link to the underlying [trace](/dashboard/traces), plus workflow and error
badges where relevant.
A session reads like the conversation it represents, and the full execution
detail of any turn is one click away.
# Traces
Source: https://docs.foglamp.dev/dashboard/traces
Inspect a single run: waterfall, span detail, scores, and replay.
A **trace** is one top-level `generateText` or `streamText` call. The
**Traces** view lists them. Opening one shows the full execution as a waterfall
you can inspect span by span and replay in real time.
See the [data model](/concepts/data-model) for how traces and spans are
defined.
## Traces list
A paginated table, newest first, over the selected date range. Columns: trace
id, name (`traceName`, falling back to `agentName`, else **Untitled trace**),
span count, tokens, duration, cost, and when it ran. Errored traces get a red
badge; cleanly **aborted** traces (see below) get an amber one. Click a row to
open the trace.
Filter by agent, model, workflow, customer, or a metadata key/value. The
**+ Filter** menu also offers **Prompt version**, which narrows the list to
the runs that used one inferred
[prompt version](/dashboard/agents#prompt-versions). With an agent picked it
lists that agent's versions; otherwise every agent's, and choosing one also
selects its agent.
## Waterfall
Each span is a row, indented under its parent, so nested tool calls and steps
sit under the call that made them. Span types are color-coded:
| Type | Color | Meaning |
| ------- | ------ | ----------------------------- |
| `agent` | amber | the root span, the whole call |
| `llm` | violet | one model step |
| `tool` | blue | one tool call |
| `other` | grey | anything else |
Each bar's position and width match when the span started and how long it ran,
so you can see at a glance what ran in sequence and what overlapped.
A span's status also tints its bar and badge: errors are red, and **aborted**
spans are amber. An aborted span is a stream that was cancelled before it
finished (a caller cancellation, an `AbortSignal`, or a timeout, via AI SDK
v7's `onAbort`). Aborts are not counted in the error rate, since a cancelled
run isn't a failure, and the steps that finished before the abort keep their
own status. This is different from an `abandoned` trace, one that never
finished and was closed by [`maxTraceAgeMs`](/sdk/configuration), which does
count as an error.
## Span detail
Click any span to open the inspector panel next to the waterfall. Depending on
the span, it shows:
* **Timing**: start time, duration, and time to first token. On reasoning
models, that splits into thinking time plus time to first visible text.
* **Model**: provider and model id.
* **Tokens & cost**: input/output tokens, computed cost, the pricing source
behind it, and a cost breakdown per dimension (prompt, completion, cache
read/write, reasoning, image, web search, request) when more than one
applies.
* **Model call**: on v7 spans, how much time was the model itself versus tools.
* **Throughput**: for streaming spans, a tokens-per-second number.
* **Provider signals**: when captured, rate-limit headroom, the model build
fingerprint, safety ratings, and grounding sources.
* **Tools available**: the tools the model was offered for the call.
* **System prompt**: on the run's root span, the system prompt or agent
instructions the run started with, rendered as markdown, with a link to the
agent's [prompt version](/dashboard/agents#prompt-versions).
* **Payloads**: the captured `input` and `output`, pretty-printed, in a
scrollable block (subject to
[`recordInputs`/`recordOutputs`](/sdk/configuration)).
* **Metadata & errors**: any span metadata, and the error message if the span
failed.
Selecting the whole trace instead of a single span shows a summary: duration,
cost, tokens, span and LLM-call counts, and errors.
## Scores
If an [eval](/dashboard/evals) has scored a trace or span, the scores appear in
the inspector under **Evals**: one row per result, green for pass and red for
fail (or the numeric score), with the judge's reason inline and a link to that
run on the eval page.
## Replay
The waterfall can also replay the trace: press play to watch it rebuild on its
real timeline, switch between 1x / 2x / 4x speed, and drag the ruler to seek.
A throughput backdrop and a peak tokens-per-second readout sit behind the bars,
and the first-token moment is marked on each LLM bar. Replay uses the token
samples the SDK already records, so it needs no extra setup. Opening a trace
with `?replay=1` plays it automatically.
# Workflows
Source: https://docs.foglamp.dev/dashboard/workflows
Group multi-call processes into runs and inspect them as a flow.
A **workflow** is a named, repeatable process (for example `deploy-digest`). A
**workflow run** is one execution of it. Tag your calls with a `workflowName`
and a shared `workflowRunId`, and Foglamp groups them into a single run. See
the [data model](/concepts/data-model).
`workflowName` and `workflowRunId` go together: pass both or neither. Reuse
the same `workflowRunId` (a job id, request id, and so on) across every call
in one execution. Traces with no workflow land in an **Ungrouped** bucket.
## Workflows list
A card per workflow over the selected range, each showing run count, trace
count, tokens, cost, last run time, and an error badge when any run failed. A
toggle switches between cards and a compact table.
## Workflow detail
Opening a workflow shows two things:
* **Run flow**: a graph of the selected run, one node per trace, colored by
status and labeled with name, start time, and duration. Click a node to open
that [trace](/dashboard/traces).
* **Runs table**: every run of the workflow with its trace count, duration,
cost, and timestamp. Click a row to load it into the flow graph above.
# Introduction
Source: https://docs.foglamp.dev/introduction
Observability for apps built with the Vercel AI SDK.
Foglamp shows you what your AI app is doing: cost, speed, token usage, full
traces, and prompt/response logs. It works with agents built on the
[Vercel AI SDK](https://ai-sdk.dev) and takes two lines of code to set up.
Foglamp is TypeScript-only and supports AI SDK v4, v5, v6, and v7. You add the
`foglamp` package to your app, and traces are sent to a backend you can run
yourself or use as a hosted service.
On AI SDK v4, v5, and v6, Foglamp wraps the `ai` module with
[`foglamp/wrap`](/sdk/wrap). On v7 (beta), it uses the SDK's built-in
telemetry API. Both produce the same traces. The [Quickstart](/quickstart)
helps you pick.
## Why Foglamp
One user action in an AI app can trigger several model calls, tool calls, and
retries, each with its own cost and speed. Regular monitoring tools were not
built for that. Foglamp is.
Every span is priced as it arrives, split by input, output, reasoning, and
cached tokens. If a model has no known price, you see a dash, never a wrong
\$0.
Each `generateText` or `streamText` call becomes a trace, with its steps and
tool calls as spans. Group them into agents, workflows, and runs.
p50/p95/p99 latency and time to first token, read from the SDK's own
metrics.
Run the whole stack with `docker compose up`, or point the SDK at the
hosted service. The code is the same either way.
## How it works
```
your app ──foglamp──▶ ingest API ──▶ ClickHouse ──▶ dashboard
```
* **SDK** sits in your app, collects spans, and sends them in batches. If no
API key is set it does nothing at all. It never throws and never slows down
your model calls.
* **Ingest API** checks your API key, prices each span, and stores it.
* **Dashboard** shows traces, workflows, agents, sessions, cost, evals, and
alerts. See the [dashboard tour](/dashboard/overview).
## Next steps
Install the SDK and see your first trace in minutes.
Learn what traces, spans, workflows, runs, and sessions are.
Every option and integration method.
Traces, workflows, agents, sessions, evals, cost, and alerts.
Run the full stack on your own servers.
Not seeing traces? Common causes and fixes.
# Quickstart
Source: https://docs.foglamp.dev/quickstart
Add Foglamp to a Vercel AI SDK app and see your first trace.
## What you need
* The `ai` package installed (v4, v5, v6, or v7)
* A Foglamp API key (starts with `fl_`), created in the dashboard or printed by
the seed script when [self-hosting](/self-hosting/overview)
Not sure which AI SDK version you have? Run `npm ls ai` or check
`package.json`. There are two setup paths, one for v4 to v6 and one for v7.
Both produce the same traces.
## 1. Install
```bash npm theme={null}
npm i foglamp
```
```bash pnpm theme={null}
pnpm add foglamp
```
```bash bun theme={null}
bun add foglamp
```
```bash yarn theme={null}
yarn add foglamp
```
## 2. Configure
Set your key in the environment:
```bash .env theme={null}
FOGLAMP_API_KEY=fl_your_key_here
# Only needed when self-hosting. The hosted service is the default.
FOGLAMP_INGEST_URL=http://localhost:4000/ingest
```
If `FOGLAMP_API_KEY` is not set, Foglamp does nothing: no traces are sent and
nothing breaks. This makes it safe to keep the code in place in every
environment.
## 3. Add it to your code
Pick the tab for your AI SDK version.
Wrap the `ai` module once with `wrap()`, then use `fog.with(...)` to name
your calls. The functions you get back have the AI SDK's own types, so you
use them exactly as before.
```ts theme={null}
import * as ai from "ai";
import { openai } from "@ai-sdk/openai";
import { wrap } from "foglamp/wrap";
const fog = wrap(ai);
const { generateText } = fog.with({ agentName: "summarizer" });
const { text } = await generateText({
model: openai("gpt-4o"),
prompt: "Summarize the latest deploy.",
});
```
`wrap()` also covers the AI SDK's agent classes, and `fog.run(context, fn)`
lets you set context for a whole block of code at once. See
[AI SDK v4 to v6 (wrap)](/sdk/wrap) for details.
Pass the integration into each call's `telemetry.integrations` array. Every
call needs a `traceName` or an `agentName`. If you use `workflowName`, pass
`workflowRunId` with it.
```ts theme={null}
import { foglamp } from "foglamp";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
const fog = foglamp();
// One run of a workflow. Reuse this id for every call in the run.
const runId = crypto.randomUUID();
const result = await generateText({
model: openai("gpt-4o"),
prompt: "Summarize the latest deploy.",
telemetry: {
integrations: [
fog.integration({
agentName: "summarizer",
workflowName: "deploy-digest",
workflowRunId: runId,
}),
],
},
});
// For a one-off call that isn't an agent, just name it:
// fog.integration({ traceName: "summarize-deploy" })
```
To trace every call without editing each one, register once with
`registerTelemetry(foglamp())`. See the [SDK overview](/sdk/overview).
## 4. Flush
On long-running servers (Node, Bun), Foglamp sends data automatically on a
timer. In serverless functions, make sure the data is sent before the function
returns:
```ts theme={null}
await fog.flush();
```
See [Runtimes and flushing](/sdk/runtimes) for what each platform needs.
## 5. Look at the dashboard
Run your app, then open the dashboard. Your call shows up under **Traces** with
its spans, tokens, cost, and timing. Calls that share a `workflowRunId` appear
together on the **Workflows** page.
All options and integration fields.
How traces, spans, workflows, and runs fit together.
# Configuration
Source: https://docs.foglamp.dev/sdk/configuration
Every option accepted by foglamp().
`foglamp(config?)` accepts the options below. Every field is optional. The v4
to v6 entry point takes the same options: `wrap(ai, config)` accepts them all,
plus a `context` for defaults (see [wrap](/sdk/wrap)).
```ts theme={null}
const fog = foglamp({
apiKey: process.env.FOGLAMP_API_KEY,
endpoint: process.env.FOGLAMP_INGEST_URL,
flushIntervalMs: 5000,
maxBatchTraces: 50,
maxBatchSpans: 500,
maxPayloadChars: 100_000,
recordInputs: true,
recordOutputs: true,
recordSystemPrompt: true,
debug: false,
onError: (err) => console.error(err),
});
```
## Options
| Option | Type | Default | Description |
| -------------------- | -------- | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `apiKey` | string | `FOGLAMP_API_KEY` | Your `fl_...` key. If unset, the collector is off. |
| `endpoint` | string | `https://ingest.foglamp.dev/ingest` | Where to send data. Set this when self-hosting. |
| `flushIntervalMs` | number | `5000` | How often batches are sent on long-running servers. |
| `maxBatchTraces` | number | `50` | Send early once this many traces are buffered. |
| `maxBatchSpans` | number | `500` | Send early once this many spans are buffered. |
| `maxQueuedSpans` | number | `5000` | Max spans held in memory (for example when ingest is unreachable). Past it, the oldest traces are dropped and `onError` fires. |
| `maxTraceAgeMs` | number | `600_000` | A trace still open after this long (10 minutes) is closed as `abandoned` and counted as an error, so a crashed generation can't leak spans. A cleanly cancelled stream is handled right away instead, via the [`aborted` status](/dashboard/traces). |
| `maxPayloadChars` | number | `100_000` | Max length for each `input`/`output` field. Longer values are cut off. |
| `recordInputs` | boolean | `true` | Capture prompt text. |
| `recordOutputs` | boolean | `true` | Capture response text. |
| `recordSystemPrompt` | boolean | `true` | Capture the system prompt (`system`, or an agent's `instructions`) on the run. It powers the **System prompt** section and [prompt versions](/dashboard/agents#prompt-versions). Also off when `recordInputs` is off. |
| `waitUntil` | function | none | Serverless flush hook. Detected automatically on Vercel and Lambda; pass `ctx.waitUntil` on Cloudflare Workers. See [Runtimes](/sdk/runtimes). |
| `fetch` | function | global `fetch` | Custom fetch implementation. |
| `debug` | boolean | `false` | Log batching and flush activity. |
| `onError` | function | none | Called on transport or serialization errors instead of throwing. |
The ingest API caps any single `input` or `output` field at 1,000,000
characters. `maxPayloadChars` (default 100,000) cuts off earlier to keep
payloads small. Raise it only if you need fuller logs. Values above 1,000,000
are clamped to that cap.
## Privacy: keeping text out
To record traces, cost, and timing without ever sending prompt or response
text, turn capture off:
```ts theme={null}
const fog = foglamp({
recordInputs: false,
recordOutputs: false,
});
```
Token counts and cost still work. They come from the provider's usage report,
not from the text.
To keep the conversation but drop only the system prompt (for example when it
embeds proprietary instructions), set `recordSystemPrompt: false`. Generation
settings (temperature, max output tokens, ...) and provider warnings are always
recorded as span metadata; they contain no prompt text.
## Error handling
The collector never throws into your app. Failures go to `onError` if you
provide one, and are otherwise ignored. Turn on `debug` to watch batching and
flushing during development.
```ts theme={null}
const fog = foglamp({
debug: process.env.NODE_ENV !== "production",
onError: (err) => reportToSentry(err),
});
```
# Live HUD
Source: https://docs.foglamp.dev/sdk/hud
Watch your agent run live on top of your app with foglamp/hud.
The HUD is a floating overlay for development. It streams your agent's
execution live on top of your running app: steps, tool calls, tokens, and cost.
It uses the same telemetry Foglamp already collects, so you watch your tools
run (and fail, and recover) without leaving your app.
The HUD does not need an API key. With `hud: true` and no `apiKey`, traces
stream to the overlay but aren't sent to the backend. If you have a key, both
happen at once.
## What you need
* A React app (the overlay is a React component). Nothing to install beyond
`foglamp`.
* A server running on Node. The overlay connects to a small localhost event
server that `foglamp({ hud: true })` starts inside your process. It does not
work on edge or serverless (see [Caveats](#caveats)).
## 1. Install
If you already use Foglamp, you have everything. Otherwise:
```bash npm theme={null}
npm i foglamp
```
```bash pnpm theme={null}
pnpm add foglamp
```
```bash bun theme={null}
bun add foglamp
```
```bash yarn theme={null}
yarn add foglamp
```
## 2. Turn the HUD on (server)
Pass `hud: true` where you create the collector:
```ts theme={null}
import { foglamp } from "foglamp";
const fog = foglamp({ hud: true });
```
That's the first line. Everything else, like `fog.integration(...)` on your
calls, stays the same. The HUD uses the telemetry you already send.
## 3. Drop in the overlay (client)
Render `` once near the root of your client app, for example in
your root layout. Its styles are isolated from yours, and it does nothing
unless the local event server is running, so it's safe to leave in.
```tsx theme={null}
import { FoglampHUD } from "foglamp/hud";
export default function RootLayout({ children }) {
return (
<>
{children}
);
}
```
That's the second line. Run your app, trigger an AI flow, and watch it stream.
## `` props
| Prop | Type | Default | Description |
| ------------- | ------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `port` | number | `8517` | Event server port. Must match `foglamp({ hudPort })` if you changed it. |
| `url` | string | none | Full event endpoint (absolute URL or same-origin path like `/hud/events`). Overrides `port`. Useful when reaching the server through a proxy. |
| `defaultOpen` | boolean | `false` | Start expanded instead of as a collapsed pill. |
| `theme` | `"light" \| "dark" \| "system"` | `"system"` | Color theme. `"system"` follows your app's `.dark` class or the OS. |
| `redact` | boolean | `false` | Hide prompt, response, and tool payloads on screen. Turn on before recording or screen sharing. |
## Server options
Two fields on `foglamp(config)` control the event server. See
[Configuration](/sdk/configuration) for the full table.
| Option | Type | Default | Description |
| --------- | ------- | ------- | ----------------------------------------------------------------------- |
| `hud` | boolean | `false` | Start the local HUD event server. Can also be set with `FOGLAMP_HUD=1`. |
| `hudPort` | number | `8517` | Port for the event server. Also `FOGLAMP_HUD_PORT`. |
The client `port` and the server `hudPort` must match. If you run more than
one HUD-enabled process locally, give each its own port and point its
`` at it.
## Recording a demo
Set `redact` before you record or screen share. It hides every prompt,
response, and tool payload while keeping the timing, token, and cost view:
```tsx theme={null}
```
## Caveats
The HUD is for local development only. `hud: true` is ignored in production
and on edge or serverless runtimes, because the event server needs a
long-lived Node process. Your normal telemetry is unaffected; only the
overlay is turned off.
* **Node runtime, not edge.** On Next.js, the route or handler that creates
`foglamp({ hud: true })` must run on the Node runtime (the default), not
`export const runtime = "edge"`.
* **No production cost.** The HUD's server code is loaded lazily and never
enters your edge or browser bundle. The core `foglamp` entry stays free of
React and HTTP code.
* **Safe to commit.** `` does nothing unless the event server is
running, so leaving both lines in is harmless. To be explicit, gate it to
dev:
```ts theme={null}
const fog = foglamp({ hud: process.env.NODE_ENV !== "production" });
```
# SDK overview
Source: https://docs.foglamp.dev/sdk/overview
The foglamp() collector and its integration methods.
The `foglamp` package has two entry points, one per AI SDK generation. Both
batch spans and send them to your ingest endpoint, and both share the same
context fields, configuration, and flushing. They only differ in how they
attach to your calls.
Wraps the `ai` module. Use this on v4, v5, and v6.
Uses v7's built-in telemetry API. Documented below.
This page covers the v7 `foglamp()` collector. The v4 to v6 `wrap()` API has
its own [page](/sdk/wrap).
```ts theme={null}
import { foglamp } from "foglamp";
const fog = foglamp();
```
The SDK has one small runtime dependency (`uuidv7`) and does not require any
particular version of `zod`. Its only required peer dependency is `ai`
(`ai@^4 || ^5 || ^6 || ^7.0.0-beta.1`).
## `foglamp(config?)`
Creates a collector. All options are optional; if there is no API key, the
collector silently does nothing. See [Configuration](/sdk/configuration) for
the full option table.
```ts theme={null}
const fog = foglamp({
apiKey: process.env.FOGLAMP_API_KEY,
endpoint: process.env.FOGLAMP_INGEST_URL,
flushIntervalMs: 5000,
});
```
## Collector methods
### `fog.integration(context)`
Returns a telemetry integration to pass into a call's
`telemetry.integrations` array (v7 also accepts the older
`experimental_telemetry` name). The context labels every span the call
produces. It must include a `traceName` or an `agentName`; if it has neither,
`integration()` throws right away.
```ts theme={null}
fog.integration({
agentName: "summarizer",
workflowName: "deploy-digest",
workflowRunId: run.id,
sessionId: user.threadId,
customer: { id: account.id, name: account.name, imageUrl: account.logoUrl },
metadata: { environment: "production", region: "us-east-1" },
});
// A one-off call that isn't an agent: name it instead.
fog.integration({ traceName: "classify-email" });
```
| Context field | Type | Notes |
| --------------- | -------------------------------------------------- | ------------------------------------------------------------------------------- |
| `traceName` | string | Label for a one-off call. Required if `agentName` is absent. |
| `agentName` | string | The agent making the call. Required if `traceName` is absent. |
| `workflowName` | string | The named process. Pass with `workflowRunId`. |
| `workflowRunId` | string | Groups traces into one run. Pass with `workflowName`. |
| `sessionId` | string | Ties traces to one conversation. |
| `customer` | `{ id: string; name?: string; imageUrl?: string }` | The customer this call serves. Powers per-customer cost. Only `id` is required. |
| `metadata` | `Record` | Free-form labels. Values are stored as strings. |
Two rules, checked at compile time and again at ingest: every call needs a
`traceName` or an `agentName` (both is fine; the display label is
`traceName ?? agentName`), and `workflowName` and `workflowRunId` must be
passed together.
### `fog.flush()`
Sends any buffered spans right away and resolves when done. Call this before a
serverless function returns. Safe to call when the collector is disabled; it
resolves immediately.
```ts theme={null}
await fog.flush();
```
### `fog.shutdown()`
Stops the flush timer and sends everything left, including traces added while a
send was already in progress. Use this when a long-running server shuts down.
```ts theme={null}
process.on("SIGTERM", async () => {
await fog.shutdown();
});
```
`flush()` keeps the collector running; use it at the end of a serverless
handler. `shutdown()` is final; use it once when the process exits. Calling
only `flush()` at exit can leave behind traces that were added mid-send.
## Two ways to register
Pass `fog.integration(...)` into one call's telemetry. Fully typed, and
takes priority over global registration.
`registerTelemetry(foglamp())` traces every call. It reads `functionId` as
the `agentName` and known keys from `telemetry.metadata`.
## Nested calls inside tools
When a tool's `execute` function makes its own AI SDK call (for example a
sub-agent), the v7 collector automatically passes the parent call's grouping
context (`workflowName`, `workflowRunId`, `sessionId`, `customer`, `metadata`)
into that nested call. The inner call lands in the same workflow run with no
extra code.
```ts theme={null}
const fog = foglamp();
await generateText({
model,
tools: { researchAgent }, // its execute() calls generateText again
telemetry: {
integrations: [
fog.integration({
agentName: "planner",
workflowName: "deep-research",
workflowRunId: run.id, // the nested call inherits this
}),
],
},
});
// The sub-agent's trace joins workflow run `run.id` automatically.
```
Only grouping context is inherited. The inner call keeps its own name
(`agentName` / `traceName`), and it is its own trace in the run, not a child
span of the tool. A more specific inner `fog.run()` or `fog.integration()`
still wins. This only works on the v7 collector; on v4 to v6 use
[`fog.run()`](/sdk/wrap) to share context across nested calls.
Next: tune batching and text capture in [Configuration](/sdk/configuration),
and make sure spans leave serverless functions in
[Runtimes and flushing](/sdk/runtimes).
# Runtimes & flushing
Source: https://docs.foglamp.dev/sdk/runtimes
Make sure spans leave the process in every runtime.
Foglamp batches spans in memory and sends them in the background. On a
long-running server you never notice this. In serverless environments the
process can freeze or stop before a batch is sent.
Anything not flushed before the runtime suspends is lost. The buffer is
in-memory on purpose, so it never blocks your model calls.
The collector detects its runtime and picks a flush strategy on its own. You
can always override it with `fog.flush()` or by passing a `waitUntil` hook.
Flushing works the same on both entry points. The examples below use the v7
`foglamp()` collector, but a `wrap()` handle has the same `flush()` and
`shutdown()` methods, the same `waitUntil` option, and the same automatic
detection on AI SDK v4 to v6.
## Long-running servers (Node, Bun)
Nothing to do. Batches are sent every `flushIntervalMs` (default 5 seconds) and
earlier when batch limits are hit. For a clean shutdown, drain the buffer on
exit:
```ts theme={null}
process.on("SIGTERM", () => fog.shutdown());
process.on("SIGINT", () => fog.shutdown());
```
## Vercel functions
Nothing to do. Foglamp detects Vercel and reads `waitUntil` from the runtime's
request context to keep the function alive until the send completes. No extra
package needed. If detection ever fails, pass it yourself:
```ts theme={null}
import { waitUntil } from "@vercel/functions";
const fog = foglamp({ waitUntil });
```
## Cloudflare Workers
Workers keep `waitUntil` on the request `ctx`, so pass it in:
```ts theme={null}
export default {
async fetch(req, env, ctx) {
const fog = foglamp({ waitUntil: ctx.waitUntil.bind(ctx) });
const result = await generateText({
model: openai("gpt-4o"),
prompt: "…",
telemetry: {
integrations: [fog.integration({ traceName: "worker-handler" })],
},
});
return Response.json(result);
},
};
```
## AWS Lambda
Lambda freezes the process the moment your handler returns, so a background
send may never happen. Await the flush before returning. Do not rely on
`process.on("beforeExit")`; it won't fire.
```ts theme={null}
export const handler = async (event) => {
const result = await generateText({ /* … */ });
await fog.flush();
return result;
};
```
## Manual flush anywhere
`fog.flush()` works in every runtime and resolves when the send completes. It
is safe to call when the collector is disabled (it resolves immediately), so
you can leave it in unconditionally.
```ts theme={null}
await fog.flush();
```
# AI SDK v4 to v6 (wrap)
Source: https://docs.foglamp.dev/sdk/wrap
Instrument older AI SDK versions by wrapping the module.
The `foglamp()` collector needs the telemetry API introduced in AI SDK v7. On
v4, v5, or v6, use the `foglamp/wrap` entry point instead. It wraps the AI SDK
functions and produces the same traces to the same endpoint.
```ts theme={null}
import * as ai from "ai";
import { wrap } from "foglamp/wrap";
const fog = wrap(ai, {
context: { agentName: "support" }, // default context for every call
});
// Bind a context and get the AI SDK's own functions back, fully typed.
const { generateText } = fog.with({ agentName: "summarizer" });
// Use exactly like the AI SDK. Traces are captured automatically.
const { text } = await generateText({
model: openai("gpt-4o"),
prompt: "Summarize this ticket.",
});
```
`wrap()` supports AI SDK v4 and later. On v7, prefer the native
[`foglamp()`](/sdk/overview) collector. The package declares
`ai@^4 || ^5 || ^6 || ^7.0.0-beta.1` as a peer dependency.
## What it captures
Each wrapped call becomes one trace, with the same shape as the v7 path:
* **`generateText` / `streamText`**: a root span, one `llm` span per step, and
one `tool` span per tool call.
* **`generateObject` / `streamObject`**: a root span plus one `llm` span.
* **Agent classes**: `wrap()` also returns wrapped, drop-in versions of
`ToolLoopAgent` and `Experimental_Agent`. `agent.generate()` and
`agent.stream()` are traced like `generateText` and `streamText`. If the
agent has an `id` and no `agentName` was set, the `id` is used as the
`agentName`.
* **Exact tool timing**: `wrap()` times each tool's `execute` directly, so tool
spans have a real measured duration.
* **Streaming stats**: for `streamText`, Foglamp watches the stream through the
call's `onChunk` callback to record time to first token and the token curve
behind tokens/sec and replay. It never consumes or changes your stream.
* **Provider signals**: every `llm` span carries what the provider reports,
such as grounding sources, the OpenAI-style `system_fingerprint`, safety
ratings, and rate-limit headroom. One difference from v7: `wrap` measures
exact tool time but cannot separate out the model-only window, so it omits
`modelCallMs` rather than guess.
## Per-call context
Contexts stack in layers, and later layers win per field: `wrap(ai, { context })`
sets the default, `fog.run(context, fn)` sets context for a block of code,
`fog.with(context)` binds on top of both, and a call-time `foglamp` option wins
over all three. `metadata` maps merge across layers, with inner keys winning.
**`fog.run()` sets context for everything inside a callback.** Use it for
things scoped to one run: workflow run ids, session ids, request metadata. Every
wrapped call inside the callback picks up the context, no matter how deeply
nested, with no parameters to pass around. Module-level singleton agents stay
singletons.
```ts theme={null}
// agent module: static identity, bound once
const { ToolLoopAgent } = fog.with({ agentName: "brand-analysis" });
const brandAnalysisAgent = new ToolLoopAgent({ model, tools, output });
// request handler: run context, set at the boundary
await fog.run(
{ workflowName: "brand-onboarding", workflowRunId: brandId, metadata: { brandId } },
() => runOnboarding(brand) // every wrapped call inside is attributed
);
```
Nested `run()` calls merge, inner over outer. Works on Node, Bun, Deno, and
Vercel or Cloudflare edge runtimes (anywhere `node:async_hooks` exists). The v7
collector has the same method: `fog.run()` layers under `fog.integration()`.
On the v7 collector, a model call made inside a tool's `execute` inherits the
parent call's workflow and session context automatically (see the
[SDK overview](/sdk/overview#nested-calls-inside-tools)). The `wrap` path has
no such hook, so on v4 to v6 use `fog.run()` to share context with a tool
that calls back into the model.
**`fog.with()` keeps your types.** It returns the wrapped functions and agent
classes typed exactly like the AI SDK's originals, so generics, `Output.object`
result types, and tool typings all survive:
```ts theme={null}
const { generateText, ToolLoopAgent } = fog.with({
agentName: "retriever",
workflowName: "support-ticket",
workflowRunId: ticket.id,
customer: { id: account.id, name: account.name }, // optional, per-customer cost
});
```
You can also pass a `foglamp` option on any wrapped call; it is removed before
the arguments reach the AI SDK. Calls that use it lose the AI SDK's generic
result types, so prefer `with()` when you need the typed result:
```ts theme={null}
await fog.generateText({
model,
prompt,
foglamp: {
traceName: "classify-email",
sessionId: user.threadId,
metadata: { environment: "production" },
},
});
```
The context fields are the same as
[`fog.integration(context)`](/sdk/overview): `traceName`, `agentName`,
`workflowName` + `workflowRunId`, `sessionId`, and `metadata`.
## Agent classes
When the module exports `ToolLoopAgent` (v6/v7) or `Experimental_Agent` (v5),
`wrap()` returns wrapped versions with the same constructor and methods.
Instrument them in place; there is no need to rewrite agent code to
`generateText`:
```ts theme={null}
const { ToolLoopAgent } = fog.with({ agentName: "research" });
const agent = new ToolLoopAgent({
model: openai("gpt-4o"),
tools: { search },
stopWhen: stepCountIs(5),
});
await agent.generate({ prompt }); // traced: root + llm steps + tool spans
```
Your `onStepFinish` and `onFinish` callbacks still run; Foglamp composes with
them. One gap: agent streams expose no `onChunk`, so `agent.stream()` traces
have no time to first token or token-curve samples (plain `streamText` does).
## Your callbacks are preserved
If you pass `onChunk`, `onStepFinish`, `onFinish`, or `onError` to a wrapped
call, your callback always runs. Foglamp's telemetry runs alongside it and
never throws into your app.
## Flushing
`wrap()` returns `flush()` and `shutdown()` alongside the wrapped functions.
Use them exactly as on the collector (see
[Runtimes and flushing](/sdk/runtimes)). Serverless platforms are detected
automatically, and on Vercel the invocation is kept alive with `waitUntil` from
the runtime's request context, with nothing to install. On other serverless
platforms pass `waitUntil` in the config (for example Cloudflare's
`ctx.waitUntil`) or `await fog.flush()` before the handler returns.
```ts theme={null}
const fog = wrap(ai, { context: { agentName: "support" } });
// … after your handler's work …
await fog.flush();
```
## Configuration
`wrap(ai, options)` accepts every [configuration](/sdk/configuration) field the
collector does (`apiKey`, `endpoint`, `recordInputs`, `recordOutputs`, `recordSystemPrompt`,
`maxPayloadChars`, `waitUntil`, and so on), plus `context` for the default
context.
## Limits compared to v7
* A client-side tool (one with no `execute` function, run by your app) can't be
timed directly, so its time is attributed at step boundaries rather than as
an exact duration. Tools with an `execute` are timed precisely.
* `wrap()` instruments a module you pass in; there is no global
`registerTelemetry` on v4 to v6.
# Configuration
Source: https://docs.foglamp.dev/self-hosting/configuration
Environment variable reference for a self-hosted deployment.
All services are configured through environment variables, validated at
startup. Anything with a default is optional; everything else is required.
`apps/server/.env.example` mirrors this reference.
## Core
| Variable | Required | Default | Description |
| -------------------- | -------- | ------------- | --------------------------------------------------------------------------- |
| `DATABASE_URL` | yes | none | Postgres connection string. |
| `BETTER_AUTH_SECRET` | yes | none | Session-signing secret, at least 32 chars. |
| `BETTER_AUTH_URL` | yes | none | Public URL of `apps/server` (e.g. `http://localhost:3000`). |
| `CORS_ORIGIN` | yes | none | Dashboard origin allowed to call the API (e.g. `http://localhost:3001`). |
| `CORS_EXTRA_ORIGINS` | no | none | Extra allowed origins, comma or space separated (preview deploys, staging). |
| `PORT` | no | `3000` | Port for `apps/server`. Usually set by the host. |
| `NODE_ENV` | no | `development` | `development`, `production`, or `test`. |
## ClickHouse
| Variable | Default | Description |
| --------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CLICKHOUSE_URL` | `http://localhost:8123` | ClickHouse HTTP endpoint. |
| `CLICKHOUSE_USER` | `default` | Username. |
| `CLICKHOUSE_PASSWORD` | empty | Password. Empty is fine while ClickHouse is unreachable from outside your network (the base `docker-compose.yml` keeps 8123 off the host). The server warns if it's empty in production. |
| `CLICKHOUSE_DATABASE` | `foglamp` | Database name. |
## Ingest (`apps/ingest`)
| Variable | Default | Description |
| -------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `INGEST_PORT` | `4000` | Port for the ingest API. |
| `INGEST_FLUSH_INTERVAL_MS` | `1000` | How often the write buffer flushes to ClickHouse. |
| `INGEST_FLUSH_MAX_ROWS` | `1000` | Flush early once this many rows are buffered. |
| `INGEST_RATE_LIMIT_RPS` | `100` | Spans per second per API key. A request costs its span count. |
| `INGEST_MAX_BODY_BYTES` | `10485760` | Max request body in bytes. Larger requests get a `413`. |
| `REDIS_URL` | none | Optional shared Redis for rate limiting across ingest replicas (e.g. `redis://redis:6379`). Unset means per-instance limiting, fine for one replica. |
| `API_KEY_CACHE_TTL_MS` | `60000` | In-memory API-key cache lifetime. |
## Cost and pricing
| Variable | Default | Description |
| ----------------------- | ------------------------------------- | ---------------------------------------- |
| `OPENROUTER_MODELS_URL` | `https://openrouter.ai/api/v1/models` | Source for model pricing. |
| `FOGLAMP_PRICING_FILE` | none | Local pricing JSON for air-gapped hosts. |
## Alerts (`apps/server`)
| Variable | Default | Description |
| ------------------------ | --------- | --------------------------------------------------------- |
| `ALERT_EVAL_INTERVAL_MS` | `60000` | How often enabled rules are checked. |
| `ALERT_RENOTIFY_MS` | `3600000` | Cooldown between repeat emails while a rule keeps firing. |
## Evals and provider keys (`apps/server`)
Evals score your traces with judge models using your own provider keys. The
whole feature depends on `FOGLAMP_SECRETS_KEY`: without it, provider keys can't
be stored and scoring stays off.
| Variable | Default | Description |
| ---------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `FOGLAMP_SECRETS_KEY` | none | At least 32 chars. Encrypts provider API keys at rest. Generate with `openssl rand -base64 32`. Unset disables evals and provider keys. |
| `SCORING_EVAL_INTERVAL_MS` | `60000` | How often enabled evals are swept and jobs queued. |
| `SCORING_SETTLE_MS` | `60000` | Spans newer than this aren't scored yet, so late spans can settle first. |
| `EVAL_JUDGE_CONCURRENCY` | `5` | Max judge LLM calls in flight per eval job. |
| `EVAL_SCORING_BATCH` | `100` | Max targets scored per eval per sweep. |
| `EVAL_JUDGE_MAX_INPUT_CHARS` | `200000` | Character budget for a judge call's prompt fields. Larger payloads are cut off. |
| `EVAL_EXECUTOR_BATCH` | `5` | Max queued eval jobs claimed and run per tick. |
## Billing with Stripe (optional)
Most self-hosts leave this block unset. Billing turns on only when both the
secret key and webhook secret are present.
| Variable | Default | Description |
| ----------------------------- | --------- | ------------------------------------------------------------------------- |
| `STRIPE_SECRET_KEY` | none | Stripe API secret key. |
| `STRIPE_WEBHOOK_SECRET` | none | Webhook signing secret for the Stripe endpoint. |
| `STRIPE_PRICE_ID_PRO_MONTHLY` | none | Price ID for the Pro monthly plan. |
| `STRIPE_PRICE_ID_PRO_ANNUAL` | none | Price ID for the Pro annual plan. |
| `QUOTA_WARN_INTERVAL_MS` | `3600000` | How often organizations are checked for the 90% span-quota warning email. |
## Foggy, the in-app assistant (optional)
Foggy turns on only when `GOOGLE_GENERATIVE_AI_API_KEY` is set.
| Variable | Default | Description |
| --------------------------------- | -------------------------- | ------------------------------------------------------------ |
| `GOOGLE_GENERATIVE_AI_API_KEY` | none | Gemini API key. Unset disables Foggy. |
| `FOGGY_MODEL` | `gemini-3.1-flash-lite` | Model used for chat. |
| `FOGGY_DOCS_URL` | `https://docs.foglamp.dev` | Docs site Foggy fetches `llms.txt` / `llms-full.txt` from. |
| `FOGGY_RPM` | `15` | Per-user requests per minute. |
| `FOGGY_DAILY_LIMIT` | `200` | Per-user messages per day. |
| `FOGGY_MAX_STEPS` | `6` | Max tool-use steps per answer. |
| `FOGGY_MAX_OUTPUT_TOKENS` | `1500` | Output token cap per answer. |
| `FOGGY_PUBLIC_RPM` | `5` | Landing-page Foggy: requests per minute per IP. |
| `FOGGY_PUBLIC_DAILY_LIMIT` | `30` | Landing-page Foggy: messages per day per IP. |
| `FOGGY_PUBLIC_GLOBAL_DAILY_LIMIT` | `2000` | Landing-page Foggy: messages per day across all visitors. |
| `FOGGY_PUBLIC_MAX_STEPS` | `3` | Landing-page Foggy: max tool-use steps per answer. |
| `FOGGY_PUBLIC_MAX_OUTPUT_TOKENS` | `700` | Landing-page Foggy: output token cap per answer. |
| `FOGLAMP_API_KEY` | none | Optional: send Foggy's own LLM calls into a Foglamp project. |
| `FOGLAMP_INGEST_URL` | none | Ingest endpoint for that key. |
## Email (optional)
Enables magic-link login and alert emails. Without these, password login still
works.
| Variable | Description |
| ------------------- | -------------------------------- |
| `RESEND_API_KEY` | Resend API key. |
| `RESEND_FROM_EMAIL` | From address for outbound email. |
## Google sign-in (optional)
Turns on only when both are present. Set the OAuth redirect URI in the Google
Cloud console to `/api/auth/callback/google`.
| Variable | Description |
| ----------------------------- | --------------------------------------------------------------------------------------------------------- |
| `GOOGLE_CLIENT_ID` | OAuth client ID. |
| `GOOGLE_CLIENT_SECRET` | OAuth client secret. |
| `AUTH_DISABLE_EMAIL_PASSWORD` | Set to `true` to turn off email and password sign-in (e.g. Google and magic link only). Default: enabled. |
## Platform admin (optional)
| Variable | Description |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `PLATFORM_ADMIN_EMAILS` | Comma-separated emails allowed to open the cross-organization Platform stats page (instance totals, ingestion volume, ClickHouse storage). Unset hides it for everyone. |
## Seed bootstrap
Used once by the seed script. No defaults: if unset, a random password is
generated and printed once.
| Variable | Description |
| ---------------- | ----------------------- |
| `ADMIN_EMAIL` | Initial admin email. |
| `ADMIN_PASSWORD` | Initial admin password. |
## Web dashboard (`apps/web`)
| Variable | Description |
| ------------------------ | -------------------------------------------------------- |
| `NEXT_PUBLIC_SERVER_URL` | Public URL of `apps/server` the browser calls. |
| `NEXT_PUBLIC_APP_URL` | Public URL of the dashboard itself. |
| `INTERNAL_SERVER_URL` | Optional in-cluster server URL for server-side requests. |
# Self-hosting
Source: https://docs.foglamp.dev/self-hosting/overview
Run the full Foglamp stack on your own infrastructure.
Foglamp runs anywhere Docker runs. The same code powers the hosted service and
your self-hosted deployment. There is no separate "community edition" with
features removed.
## Architecture
```
┌──────────────┐
your apps ──────▶│ apps/ingest │──┐
(SDK) │ :4000 │ │ writes
└──────────────┘ ▼
┌────────────┐
│ ClickHouse │ spans + rollups
└────────────┘
┌────────────┐
│ Postgres │ orgs, projects, keys, alerts
└────────────┘
┌──────────────┐ ▲
dashboard ──────▶│ apps/server │──┘ reads + auth + alert cron
(browser) │ :3000 │
└──────┬───────┘
┌──────▼───────┐
│ apps/web │ Next.js dashboard
│ :3001 │
└──────────────┘
```
| Service | Port | Role |
| ------------- | ---- | ----------------------------------------------------------------------------------------- |
| `apps/ingest` | 4000 | Receives spans from the SDK, checks API keys, prices and stores spans. Scales on its own. |
| `apps/server` | 3000 | Dashboard API, auth, and the alert checker. |
| `apps/web` | 3001 | Next.js dashboard UI. |
| ClickHouse | 8123 | Span store and rollups. |
| Postgres | 5432 | Organizations, projects, API keys, alerts. |
There is no external queue. The ingest write buffer is in memory and flushes to
ClickHouse on an interval and on shutdown, and eval scoring jobs queue in
Postgres. Redis is optional: with `REDIS_URL` set, rate limiting is shared
across ingest replicas; without it, each instance limits on its own, which is
fine for a single replica. The default compose file includes Redis.
Which ports are reachable from the host depends on which compose files you
use. The base `docker-compose.yml` keeps ClickHouse's 8123 internal only. A
plain `docker compose up` also loads `docker-compose.override.yml`, which
publishes 8123, Postgres 5432, and Redis 6379 on `localhost` for local
development. For production, run without the override
(`docker compose -f docker-compose.yml up --build`) and set
`CLICKHOUSE_PASSWORD` before the network is reachable. An open ClickHouse
with no password exposes every span.
## Quickstart
```bash theme={null}
git clone https://github.com/foglamp-labs/foglamp.git
cd foglamp
docker compose up --build
```
On boot the stack runs migrations, sets up ClickHouse, and runs the seed
script. The seed prints an admin login and an API key **once**; copy them. If
you missed them, search the `migrate` service logs for `Save these now`
(`docker compose logs migrate`).
There are no default credentials. If `ADMIN_EMAIL` and `ADMIN_PASSWORD` are
unset, the seed generates a random password and prints it a single time.
Then open the dashboard at `http://localhost:3001` and log in with the seeded
email and password. Magic-link email and Google sign-in are optional; login
works out of the box without them.
## Pointing the SDK at your deployment
Set the ingest URL in your instrumented app:
```bash theme={null}
FOGLAMP_API_KEY=fl_seeded_key_here
FOGLAMP_INGEST_URL=http://your-host:4000/ingest
```
## Operations
* **Retention**: how long spans are kept depends on the plan. Each span is
stamped with a `retention_days` value when it arrives, and expires on its
own after that. Hosts with billing off keep spans effectively forever.
* **Pricing**: model prices come from the OpenRouter models API, cached and
refreshed every 24 hours. For air-gapped hosts, supply a local JSON file
with `FOGLAMP_PRICING_FILE`.
* **Email**: set `RESEND_API_KEY` to enable magic-link login and alert emails.
Without it, everything else still works via password login.
See [Configuration](/self-hosting/configuration) for the full environment
variable reference.
# Troubleshooting
Source: https://docs.foglamp.dev/troubleshooting
Why you might not see traces, and how to fix it.
Foglamp fails quietly on purpose: it never throws errors into your app and
never slows it down. The downside is that a wrong setup gives you silence
instead of an error. This page covers the usual causes.
## Turn on debug first
Debug logging tells you whether the collector is on, when batches are sent, and
whether sending failed:
```ts theme={null}
const fog = foglamp({ debug: true });
```
If you see `[foglamp] FOGLAMP_API_KEY not set — telemetry disabled (no-op).`,
that is your answer. See the first item below.
## No traces at all
Without `FOGLAMP_API_KEY` (or an explicit `apiKey`), Foglamp does nothing.
Set the key in the environment where the code actually runs. The usual
mistake is a `.env` file that isn't loaded, or a key set locally but not in
the deployed environment.
Serverless platforms can freeze the process as soon as your handler
returns, before the data is sent. Await `fog.flush()` before returning. On
AWS Lambda this is required. See [Runtimes and flushing](/sdk/runtimes).
The hosted endpoint is the default. If you self-host, set
`FOGLAMP_INGEST_URL` to your own ingest API, for example
`http://your-host:4000/ingest`, and make sure it ends with `/ingest`.
Debug logging shows failed responses from the endpoint.
Per-call tracing only works if the call actually includes the integration
in its `telemetry.integrations` array. Check that it does, or register
globally with `registerTelemetry(foglamp())`.
## Traces appear but something's off
Foglamp has no price for that model, so it shows nothing rather than a
wrong number. Prices come from
[OpenRouter](https://openrouter.ai/api/v1/models) and refresh every 24
hours, so this usually fixes itself once the model is listed. Token counts
and timing are not affected. See [Cost and pricing](/dashboard/cost).
Text capture may be off. Check that you haven't set `recordInputs: false`
or `recordOutputs: false`, and that `maxPayloadChars` isn't cutting off
more than you expect. See [Configuration](/sdk/configuration).
Your organization has used up its monthly span quota. The dashboard shows a
red banner when this happens. Upgrade the plan or wait for the period to
reset. See [Projects, keys and billing](/dashboard/account#usage).
When many streams run at the same time on one globally registered
collector, Foglamp can't always tell which stream a token belongs to, so it
drops the sample. Use a per-call `fog.integration(...)` for reliable
replay.
Embeddings aren't captured yet. `embed` and `embedMany` produce a trace
with a root span but no usage. See the
[data model](/concepts/data-model#how-it-streams).
## Still stuck?
Turn on `debug` and send transport errors somewhere you'll see them:
```ts theme={null}
const fog = foglamp({
debug: true,
onError: (err) => console.error("[foglamp]", err),
});
```
Foglamp never retries a failed batch. A failed send is reported to `onError`
and dropped, so `onError` is the place to catch transport problems.