Implementing 5-Signal Metrics Dashboard for AI Tutor Cost (and Auditability)

Implementing a metrics dashboard for an AI tutor is an accounting problem before it is a visualization problem. Attribute each agent-loop attempt to a stable interaction ID, record latency and billable usage as separate facts, and only then aggregate those facts into a live view. The decision rule is strict: choose the storage and dashboard path that preserves an auditable event trail when a retry, timeout, or late provider response arrives.

Short answer: what belongs in the first version?

For an edtech agent that helps a student work through an exercise, start with five signals per loop: end-to-end latency, model or tool latency, input and output usage, retry count, and terminal outcome. Add dimensions for course, lesson, agent version, model class, and an opaque interaction ID. Do not put raw prompts, answers, or student identifiers into metric labels.

The dashboard should answer two different questions without mixing them: "Is the tutor responding within its service objective?" and "What did this completed loop consume?" The first is a time-series question. The second is an attribution question that needs durable records and reconciliation. A counter that says 42,000 tokens were used cannot explain which loop owns 1,200 of them after a timeout.

This distinction is the architecture decision record for the system. Its invariants are: one logical attempt has one idempotency key; usage is additive only after a defined settlement point; every aggregate can be traced to immutable facts; and a missing provider response remains distinguishable from zero usage. Those rules matter more than whether the chart refreshes in one second or five.

Which tools help when building a real-time metrics dashboard?

The critical path is small enough to make explicit. An agent runner emits a start fact, records each model or tool span, then emits one terminal fact. A separate projector turns facts into dashboard series. The projector can be retried because its writes are keyed by event ID. The agent runner must not infer a successful completion from an HTTP connection closing.

Here is a deliberately generic Go shape for the event boundary. It contains no provider-specific route and keeps the student outside the telemetry payload.

package telemetry import ( "crypto/sha256" "encoding/hex" "time" ) type LoopEvent struct { EventID string `json:"event_id"` InteractionKey string `json:"interaction_key"` OccurredAt time.Time `json:"occurred_at"` AgentVersion string `json:"agent_version"` CourseID string `json:"course_id"` LessonID string `json:"lesson_id"` Kind string `json:"kind"` LatencyMS int64 `json:"latency_ms,omitempty"` InputUnits int64 `json:"input_units,omitempty"` OutputUnits int64 `json:"output_units,omitempty"` RetryCount int `json:"retry_count,omitempty"` Outcome string `json:"outcome,omitempty"` } func InteractionKey(studentSession, turnID string) string { h := sha256.Sum256([]byte(studentSession + ":" + turnID)) return hex.EncodeToString(h[:]) } 

This identifier is deterministic, but it is not a license to use a student ID as a label. Keep the key in the event store or trace context, restrict access to it, and expose only bounded dimensions to the metrics backend. This is also where compliance limits enter the design: telemetry should contain the minimum data needed for operations, with retention, access, and deletion behavior defined separately from the dashboard query.

For cost attribution, I would store usage in units supplied by the model boundary and attach a versioned cost policy outside the metric label set. That keeps a policy correction from rewriting historical observations invisibly. The dashboard can display estimated cost, but the ledger-like record should preserve the units, policy version, and settlement state that produced the estimate.

Which architecture survives retries and late results?

There are three practical arrangements. They are not interchangeable.

For this edtech case, the second arrangement is the decision. Metrics remain the fast read model; they are not the source of financial truth. OpenTelemetry can carry traces and metrics across instrumentation boundaries, Prometheus can provide a pull-oriented time-series store, and Grafana can render queries from supported data sources. Datadog provides an integrated commercial path with its own ingestion and retention boundaries. Those are meaningful differences in operating model, not a ranking. A 30-second freshness target is a reasonable starting criterion for the real-time view, but it does not justify weakening settlement or replay guarantees.

The rejected option is a single counter incremented after each model call. It is valid for approximate capacity signals, such as concurrent calls or request rate. It is a poor cost ledger because a retry creates a second call, a client disconnect can leave settlement ambiguous, and an exporter restart can make delivery semantics invisible. The exactly-once mindset belongs at the business event boundary; transport delivery itself may still be at-least-once.

A projector therefore needs an idempotent write key and an explicit state transition. The following critical path omits the database adapter, but the checks are the important part.

package projector import ( "context" "errors" ) type UsageEvent struct { EventID string InteractionKey string InputUnits int64 OutputUnits int64 PolicyVersion string Settled bool } type Store interface { InsertEventOnce(context.Context, UsageEvent) (bool, error) AddUsage(context.Context, UsageEvent) error } func Project(ctx context.Context, s Store, event UsageEvent) error { if event.EventID == "" || event.InteractionKey == "" { return errors.New("event identity is required") } if !event.Settled { return nil } inserted, err := s.InsertEventOnce(ctx, event) if err != nil { return err } if !inserted { return nil // A replay is safe. } return s.AddUsage(ctx, event) } 

In production, InsertEventOnce and AddUsage need a transaction or an equivalent outbox/inbox protocol. Otherwise the projector can accept the event and fail before adding usage, which is recoverable only if the pending state is queryable. A dashboard that hides projector lag turns this into a misleading green chart. Expose ingestion lag, projection lag, duplicate-event count, and unsettled-loop count beside the business series.

How do the tool choices change the failure model?

The tool comparison should begin with data ownership, query shape, and failure recovery. OpenTelemetry is a vendor-neutral instrumentation standard and collector ecosystem; it helps standardize signal production, but it does not by itself decide retention or cost policy. Prometheus is well suited to labeled time series and alert evaluation, but labels must stay bounded and its pull model shapes how short-lived workers expose data. Grafana is a visualization and query layer whose usefulness depends on the quality and availability of its data sources. Datadog offers a managed, integrated observability service, which reduces the amount of platform machinery a team operates while making its ingestion, query, retention, and access controls part of the service boundary.

Those distinctions answer different constraints. A small platform team may value managed operations. A regulated organization may require exportable records and a retention schedule it can prove. A team debugging agent behavior may need traces that connect a tool call to a terminal outcome. None of those requirements can be inferred from a chart screenshot.

I would test candidates with a replay fixture instead of a feature checklist: 1,000 loop attempts, 70 retries, 12 timeouts, 8 late settlements, and duplicate delivery of every terminal event. The expected result is not a pretty panel. It is one settled attribution per logical attempt, a visible unsettled count, and a reconciliation report whose total equals the event-store total under the selected policy version. The numbers are test inputs, not a performance claim.

What makes the result trustworthy over time?

Deploy the dashboard and projector together with a schema version, and alert on the data path as well as the tutor path. A useful panel set has a latency percentile, terminal outcomes, usage by agent version, settled usage by course and lesson, and a reconciliation delta. Keep raw event access narrower than aggregate access. An instructor may need an aggregate for a lesson; that does not imply access to interaction-level telemetry.

The first version should also make absence visible. Zero output units, an unknown usage response, and a dropped event are three different states. Encoding all three as zero is an accounting bug disguised as a metric. Likewise, a retry count is not a cost amount, and a model call duration is not a student-visible latency. Name each measurement according to its boundary.

A 2026 review should verify the current configuration, retention, and compliance obligations rather than treating a tool comparison as permanent advice. Standards evolve, hosted products change their limits, and the local data policy can change faster than either. The durable choice is the event contract: stable identity, explicit settlement, bounded dimensions, replayable facts, and an aggregate that can be challenged with evidence.

That contract is a limitation as well as a strength: teams that need only coarse request-rate alarms may find an event projector disproportionate, while teams that need exact attribution cannot safely replace it with a charting layer.

That is the dashboard I would ship for an AI tutor. It is useful in real time, but it earns trust from the audit trail behind the pixels.

References

• https://12factor.net/logs
• https://opentelemetry.io/docs/concepts/observability-primer/
• https://prometheus.io/docs/concepts/data_model/
• https://grafana.com/docs/grafana/latest/
• https://docs.datadoghq.com/getting_started/
• https://www.nist.gov/privacy-framework