telemetry.dev

Observability for AI agents

telemetry.dev Introduction

telemetry.dev is an observability platform for AI agents. It shows each agent run as a single trace, with every model call and tool call represented as a span that includes duration, tokens, cost, and errors. Teams can use the same traces for prompts and evals, edit prompts without a deploy, run them against datasets, and compare scores with production. Alerts notify users of errors, cost spikes, and agent loops. Integration is available via TypeScript or Python SDKs or an OpenTelemetry exporter.

Key Features

Traces of AI agent runs with model and tool call spans showing duration, tokens, cost, and errors
Prompt editing and evaluation on the same traces, with dataset runs compared against production scores
Alerts for errors, cost spikes, and agent loops
TypeScript and Python SDKs plus OpenTelemetry exporter support

Use Cases

Debugging AI agent runs by inspecting individual model and tool call spans
Evaluating prompt changes against datasets before shipping them
Monitoring production agents for errors, rising costs, and looping behavior

Why Startups Use It

Startups building AI agents get end-to-end visibility into agent behavior, costs, and errors, letting small teams debug, evaluate prompts, and monitor production agents without building in-house observability tooling.

Alternative Options

Langfuse, LangSmith, Arize Phoenix

Frequently Asked Questions

How do I integrate telemetry.dev with my application?

You can connect using the TypeScript or Python SDK, or via the OpenTelemetry exporter you may already be using.

Can I change prompts without redeploying?

Yes, prompts and evals use the same traces, allowing you to change a prompt without a deploy and run it on a dataset to compare scores with production.