A multi-agent workflow takes 45 seconds to answer, no exception was thrown, and every dashboard is green. What do you do next? That operations question is what AgentOps Center sets out to answer. It is a small GitHub project built for a SigNoz hackathon. I read it file by file without deploying it, so this article keeps three things apart: what the code does, what the README claims, and what I could not check.

The repository is more useful as a pattern than as a product. It shows how to name spans in a system of cooperating agents, where to send them, and how to make an assistant’s answer depend on real telemetry. We will walk through those three decisions, then the limits.

A note on method: the last commit I read is dated 26 July 2026. I quote no star count because I did not read one from the source. The README says MIT, yet I found no LICENSE file in the tree, so check before reusing anything.

The problem AgentOps Center targets: slow is not broken

Classic APM counts requests and HTTP errors. An agent workflow behaves differently. It loops, replays a tool, or waits on a model provider that is rate limiting it. Nothing breaks, everything slows down, and the token bill climbs.

The README lists the questions operators need to settle: which node looped, which tool returned a rate limit, what each sub-agent cost. Answering them takes spans per agent, per model call and per tool, all linked in one trace. That is what the code builds.

Decision 1: how to name spans in a multi-agent system

The file backend/instrumentation/agent_spans.py declares attribute constants. It reuses the gen_ai.* keys for model calls, then adds house keys for agents and tools.

Excerpt of agent\_spans.py: constants gen\_ai.system, gen\_ai.request.model, gen\_ai.usage.input\_tokens, then agent.name, agent.workflow\_id and tool.name

The split is sensible. Standard keys let you compare models and providers, while house keys carry business logic: which agent, which workflow. A workflow identifier on every span then groups everything together, even when the spans come from different libraries. To see what a span carries, read the anatomy of an OpenTelemetry span.

Two cautions apply:

  • The OpenTelemetry GenAI conventions are moving quickly. The specification page now points to a dedicated repository that I could not read (the server returned an error when I checked). gen_ai.system may have been replaced by a newer provider attribute: unverified.
  • gen_ai.usage.cost_usd is the project’s own invention, not a key I know to be standard. Dashboards built on it will stay specific to your code.

One more design detail: gen_ai.prompt and gen_ai.completion are cut to 500 characters. Full prompts in spans are expensive and expose data, so truncation is a fair compromise. Still, set your own policy before production. See also OpenTelemetry attributes in Dynatrace.

Decision 2: where the telemetry goes

The setup.py module creates a tracer provider and a meter provider, exporting over OTLP gRPC to localhost:4317 by default, which OTEL_EXPORTER_OTLP_ENDPOINT overrides. It turns on FastAPI and httpx instrumentation, plus LangChain through OpenInference, which also covers LangGraph according to a code comment. Sampling is fixed at 100 %, flagged in the code as a demo setting.

The repository’s Collector receives OTLP and writes straight to ClickHouse using SigNoz exporters, one pipeline per signal.

Excerpt of otel-collector/config.yaml: exporters clickhousetraces, signozclickhousemetrics and clickhouselogsexporter, then the traces pipeline with memory\_limiter and batch

This layout is for self-hosted SigNoz. With another backend, Dynatrace for instance, only the exporter changes: the OTLP protocol and the attributes stay. Our OpenTelemetry Collector to Dynatrace guide shows that variant. I have not tested this repository against Dynatrace.

For operations, note the memory_limiter and the batch of 10,000 items every 10 seconds. Good instinct: an agent system can emit thousands of spans per run. The README’s smoke tests report between 2,340 and 3,559 spans per run.

Decision 3: let an assistant diagnose, but only from evidence

This is the most interesting idea. The Root Cause Copilot does not ask a model cold. An evidence engine first calls the SigNoz MCP server to fetch traces, error logs, metrics and alerts. It then builds a context labelled by source tool and grades its own confidence.

The SigNoz documentation confirms that this MCP server exposes signoz_search_traces, signoz_search_logs, signoz_query_metrics and signoz_list_alert_rules, among others. HTTP mode is switched on with TRANSPORT_MODE=http, and the endpoint is /mcp. For the wider picture, read monitoring MCP agents with OpenTelemetry.

The grading logic lives in reasoning.py:

  • MCP server unreachable: LOW confidence, with in-memory local context only;
  • traces plus another signal (logs, metrics or alerts): HIGH;
  • a single signal: MEDIUM;
  • no usable data: NONE.

A critical reading: the rule counts signals, it does not measure correctness. One condition in the code even grades HIGH whenever the MCP server answers with at least two signals of any kind. The badge says « I found material to back my answer », not « my answer is right ».

Testing observability with injected faults

The chaos engine (chaos/injector.py) draws faults at random, at an intensity between 0 and 1: timeouts, 429 or 500 errors, tool failures, a 2 to 8 second slowdown, agent crashes, and corrupted output. The README lists five, while the code has six, adding invalid output. Each fault creates an error span carrying chaos.mode and chaos.injected.

For an SRE the value is clear: check that your alerts and dashboards see the degradation before a real one arrives. The chaos.injected marker also stops you mistaking a drill for an incident. For the matching safeguards, read AI agent guardrails in production.

What is real and what is scripted

Here is the main limit. The workflow’s incidents are staged. In graph.py, depending on words found in the scenario text, the code emits Redis, SQL and third-party API spans with hard-coded values: 48 connections out of 50, a 429 status from a payment service, and waits faked with asyncio.sleep.

Excerpt of graph.py: redis\_span, db\_span and third\_party\_api\_span emitted from scenario keywords, with active\_connections=48 and status\_code=429 written in the code

That is fine for a demonstration, but it changes how to read the README screenshots. They prove the telemetry pipeline works, not that the tool detects real outages. Model pricing is also a table inside the code (four models, a default price otherwise). For genuine cost tracking, see AI agent cost per token with OpenTelemetry.

The README itself admits further gaps in its improvement list: no authentication or multi-tenancy, no built-in alerting, no long-term retention. I did not reuse the README screenshots here, because the repository has no licence file.

Reusing the pattern: step by step

  1. Define your attribute keys: a workflow identifier, an agent name, a tool name, and the gen_ai.* keys for models.
  2. Wrap every model call and every tool in a span, setting the error status and recording the exception.
  3. Send everything over OTLP to a Collector, with memory_limiter and batching.
  4. To run it locally, the README says to copy .env.example, supply a Groq or OpenAI key, then run docker compose up -d (nine containers are announced).
  5. Check the services: localhost:3000 (UI), 8000 (API) and 8080 (SigNoz), according to the README.

To see how a graph of agents turns into a trace in code, our article on LangChain and LangGraph tracing covers the instrumentation.

Best practices

  • Put a workflow identifier on every span, and in the logs too.
  • Use standard keys for what you compare and documented house keys for the rest.
  • Sample in production: 100 % suits a demo, not a fleet of agents.
  • Never put full prompts or secrets in attributes.
  • Mark injected faults so they cannot be confused with real incidents.

Frequently asked questions

Does AgentOps Center work with Dynatrace?

It is written for SigNoz: the Collector exports to ClickHouse and the copilot queries the SigNoz MCP server. The spans themselves are standard OpenTelemetry and could go elsewhere, but I have not tested that route.

Does the confidence badge guarantee a good diagnosis?

No. It measures how much telemetry was found, not whether the conclusion is correct. Always read the traces it cites.

Is it production ready?

The README presents it as a hackathon demonstration and lists authentication, alerting and retention among future work. Treat it as a source of ideas.

Conclusion

AgentOps Center answers a real operations question with a clear pattern: well-named spans, an OTLP pipeline, and an assistant forced to cite its sources. Its incidents are staged and the code is hackathon code, but the architecture reads in an hour and transposes well. Take the ideas, not the code as it stands.

span.observer is an independent community blog, not affiliated with the AgentOps Center project, its authors, SigNoz or Dynatrace LLC. Trademarks belong to their respective owners. Tools change quickly: check commands and options in their official documentation.

Originally published on span.observer. French version: https://span.observer/fr/agentops-center-multi-agents-signoz/.