Most demo applications stop at the happy path: click a button, get a response, and maybe store a row in a database. Real systems are less predictable. Requests become slow, services fail, and useful information gets scattered across logs, traces, and metrics. Developers eventually face the same question: What actually happened?
I wanted to explore this through a practical example, so I built The Daily Grind, a coffee shop app that doubles as an observability lab.
Customers can order coffee and use the Where is my coffee? feature to track their orders. Behind this simple interaction, three Python services communicate over HTTP and use PostgreSQL to manage application state.
The goal isn't to build another CRUD application. I want to demonstrate how four observability signals work together when a request moves through a distributed system:
- Traces: Where did the request spend its time?
- Metrics: Is the problem isolated or widespread?
- Logs: What happened during the operation?
- Profiles: Which parts of the code consume CPU and memory?
In my earlier articles, I explored how OpenTelemetry generates telemetry, how the Collector processes it, and how open-source tools store and analyze it. This project brings those concepts together in a working application. You can find the entire project at Coffee shop GitHub repository..
The Daily Grind: Where Is My Coffee?
The app offers eight coffee options and three cup sizes. When a customer places an order, the system reserves the required ingredients, prepares the drink, and updates its status so the customer can track its progress.
The backend consists of three independent Python services:
- Order API: Accepts orders, maintains status history, and coordinates the workflow
- Inventory: Checks and reserves the ingredients required for each drink
- Barista: Simulates drink preparation and controlled performance scenarios
The services share a PostgreSQL instance, which remains the source of truth for order status. The customer interface retrieves updates through the Order API, while Jaeger, Prometheus, OpenSearch, and Pyroscope provide the information developers need to investigate application behavior.
How to place a coffee order
When a customer places an order, the Order API:
- Validates the request and creates an order in PostgreSQL.
- Records the initial received status.
- Returns an HTTP 202 Accepted response with the order ID.
- Schedules the order-processing workflow using API BackgroundTasks.
Order creation and background processing across the Order API, PostgreSQL, Inventory, and Barista services.
The background workflow calls Inventory to reserve ingredients, then calls Barista to prepare the drink. The Order API records each status transition, including ready or failed. Customers can retrieve the latest status and order history using their order ID.
Architecture: How the Components Work Together
In my earlier article, Beyond the Collector: How the Open-Source Observability Stack Fits Together, I explained that OpenTelemetry provides the foundation for generating and transporting telemetry, but it doesn't replace observability backends.
The app uses OpenTelemetry for traces, metrics, and logs. The services export telemetry through the OpenTelemetry Protocol (OTLP) to the OpenTelemetry Collector, which routes each signal to the appropriate backend. Profiling follows a separate path in this implementation. The Pyroscope Python SDK sends CPU and memory profiles directly to Pyroscope.
The local observability stack includes:
| Component | Responsibility |
|---|---|
| OpenTelemetry Python | Provides instrumentation and telemetry SDKs |
| OpenTelemetry Collector | Receives and routes traces, metrics, and logs |
| Jaeger | Stores and provides an interface for exploring distributed traces |
| Prometheus | Scrapes and stores app metrics |
| OpenSearch | Stores Jaeger traces and indexed app logs |
| Data Prepper | Processes and indexes app logs |
| Pyroscope | Stores and explores CPU and memory profiles |
| Perses | Provides dashboards across observability data sources |
| PostgreSQL | Stores app state, orders, and inventory |
In Your Telemetry Doesn't Go Straight to the Backend: Understanding the OpenTelemetry Collector, I explored how the Collector receives, processes, and exports telemetry. The same three concepts apply here:
- Receivers accept telemetry from the application services.
- Processors enrich, filter, batch, or transform telemetry.
- Exporters send telemetry to the appropriate backends.
The Collector separates app instrumentation from backend configuration. The other signals follow different paths.
| Signal | Telemetry path |
|---|---|
| Traces | OpenTelemetry SDK → Collector → Jaeger → OpenSearch |
| Metrics | OpenTelemetry SDK → Collector → Prometheus exporter → Prometheus |
| Logs | Python logging → OpenTelemetry Logs SDK → Collector → Data Prepper → OpenSearch |
| Profiles | Pyroscope Python SDK → Pyroscope |
Note: The Collector's Prometheus exporter exposes metrics through an HTTP endpoint. Prometheus periodically scrapes that endpoint. The Collector doesn't push metrics directly into the Prometheus time-series database.
Profiling is the exception. The Pyroscope SDK uses its native ingestion protocol, so profile data bypasses the Collector.
Controlled Failure and Performance Scenarios
To investigate different performance problems, I added five scenarios that reproduce predictable application behavior. They let us compare signals without introducing an unrelated infrastructure failure.
| Scenario | What it demonstrates | Useful signals |
|---|---|---|
| normal | Baseline order workflow | All four signals |
| slow_io | Simulated asynchronous wait of approximately 1.5 seconds | Traces and metrics |
| slow_cpu | CPU-intensive recipe calculation | Traces, metrics, and CPU profiles |
| slow_db | PostgreSQL pg_sleep() delay | Traces and metrics |
| slow_memory | Temporary memory allocations | Memory profiles |
Instrumenting the Services
Generating telemetry is only the first step. We also need enough context to understand where it came from and what it represents.
Your Telemetry Has Data. Does It Have Context?, I explored resource attributes, semantic conventions, and context propagation. The Daily Grind applies those ideas with two types of instrumentation:
- Automatic: FastAPI captures incoming HTTP requests. HTTPX captures outgoing service calls and propagates trace context.
- Manual: I add spans around order processing, inventory reservations, database operations, coffee preparation, and status updates.
Note: Automatic instrumentation captures HTTP activity, but it doesn't understand our business operations.
Since the Order API processes orders in the background, I also distinguish between the initial order creation time and the time required to complete the entire workflow. I use Jaeger to verify that the instrumentation captures the service calls and application operations involved in processing an order. This allows us to measure two parts of the application:
- Order creation duration: How long the API takes to accept an order.
- Order processing duration: How long the background workflow takes to complete it.
Following an Order Through Traces
Let's follow a cappuccino order in Jaeger. The trace should show the incoming POST /orders request, order creation, background processing, the Inventory reservation, the Barista preparation call, and the final status update.
This helps answer the first debugging question: Where did the order spend its time? A slow Inventory operation points to ingredient reservation or its database work. A long preparation operation points to Barista. A delayed HTTP call shows where the application is waiting for another service.
A trace shows the sequence and duration of operations in a request. It can identify a slow operation, but it does not necessarily explain which functions consumed CPU or allocated memory. Use profiles to investigate resource consumption over time.
Metrics: Understanding Behavior Over Time
Traces help us investigate individual requests. Metrics help us identify patterns across many requests. The application uses the OpenTelemetry Metrics API to record business-level measurements such as:
- coffee.orders measures orders received
- coffee.order.duration measures complete order-processing time
- coffee.inventory.reservations measures ingredient reservation attempts
- coffee.preparation.duration measures drink preparation time
It also records status changes and reservation duration. The Collector's Prometheus exporter exposes the metrics at /metrics, which Prometheus periodically scrapes and stores as time series.
OpenTelemetry metric names and units can be translated into Prometheus naming conventions. For example, coffee.orders can become coffee_orders_total, while a duration histogram with the unit s can become a name such as coffee_preparation_duration_seconds.
Distinguishing request duration from order duration
The Order API processes orders in the background, so HTTP request duration doesn't represent the complete order-processing time. I track three separate measurements:
| Measurement | What it tells us |
|---|---|
| HTTP request duration | How quickly the API accepts and responds to an order request |
| Order processing duration | How long the complete background workflow takes |
| Coffee preparation duration | How long the Barista service takes to prepare the drink |
This distinction helps us separate API responsiveness from actual order completion. A successful 202 Accepted response means the application accepted the order, not that the coffee is ready.
You can use these metrics to investigate questions such as:
- Are orders completing successfully?
- Has coffee preparation latency increased?
One important design decision is that I don't use order IDs as metric labels. They belong in traces and logs, where we investigate individual orders. Adding them to metrics would create a high-cardinality time series. Metrics should describe aggregated behavior.
Logs: Connecting Events to Traces
Logs describe application events, but an individual entry often lacks enough context to explain its place in a workflow. The Daily Grind uses Python's standard logging module for events such as order receipt, inventory checks, reservations, preparation start and completion, and order readiness.
The OpenTelemetry logging bridge connects these records to the Logs SDK. With trace-context injection enabled, records emitted inside an active span can include its trace and span IDs. The Collector sends the records over OTLP to Data Prepper, which indexes them in OpenSearch. To investigate an order:
- Find a slow or failed order in Jaeger.
- Copy its trace ID.
- Search OpenSearch for log records with the same trace ID.
- Examine the events and attributes associated with the operation.
For example, an indexed log record might look like this:
A log entry by itself might say that an inventory reservation failed. A trace-correlated log gives us more context: which request caused the failure, which service handled it, and which operation was active.
Protecting telemetry data
Logs and traces can contain sensitive information, including access tokens, email addresses, customer identifiers, and internal implementation details. We should control what the application collects, where it stores telemetry, and who can access it.
This becomes especially important when engineers use AI assistants to investigate incidents.
In my article, Before You Paste Telemetry Data into an AI Agent, I explored how telemetry can expose sensitive information and how attacker-controlled input can introduce malicious instructions into logs.
Profiling: Finding Resource-Intensive Code
Traces and metrics help us identify slow operations, but they don't always explain which parts of the application consume resources.
The Daily Grind uses the Pyroscope Python SDK to collect CPU and memory profiles from its services. The SDK sends profiles directly to the Pyroscope server.
- The slow_cpu scenario runs a CPU-intensive recipe calculation in Barista. Jaeger shows that preparation takes longer than usual, while Pyroscope helps identify the function consuming CPU time.
- The slow_memory scenario creates approximately 8 MiB of temporary allocations and holds them briefly. A memory profile helps us examine where the application allocates memory and, depending on the profile type, which allocations remain in use.
Traces show where time is spent. Profiles help us understand how the application uses resources.
Investigating a Slow Coffee Order
Let's bring the four telemetry signals together.
A customer orders a cappuccino, but the Barista service takes longer than expected. The customer opens Where is my coffee? and sees that the order is still being prepared.
How do we find out what's causing the delay?
- Detect the problem with metrics: Start with Prometheus and inspect the coffee preparation-duration metric.
- If preparation latency increases across several orders, this suggests a broader issue rather than a single slow request. Metrics help us identify when the problem started and how widely it affects the application.
- Locate the slow operation with traces: Open a slow order trace in Jaeger. Check whether Inventory or Barista accounts for most of the processing time.
- For this scenario, the Barista preparation span should reveal where the order spends most of its time.
- Add context with logs: Search OpenSearch using the trace ID from Jaeger to find the related application events.
- The logs show when preparation started and completed, while the trace shows how long the operation took and where it fits into the order workflow.
- Identify the code path with profiles: Inspect Barista's CPU profile in Pyroscope. The slow_cpu scenario adds an expensive recipe calculation that the profile should help expose.
This takes the investigation from a customer-visible symptom to the slow operation and the code path contributing to it. Each signal contributes a different piece of information. Together, they help us move from detecting a performance problem to investigating its cause.
Dashboards: Bringing Telemetry Together with Perses
Each observability backend provides its own interface for exploring telemetry. Perses provides a common dashboard layer without replacing those backends.
For The Daily Grind, I use Perses to create an operational view with:
- Order rate and completion counts.
- Preparation and inventory reservation duration.
- Failed orders and service health.
- Links to related traces, logs, and profiles.
Perses queries each backend through its datasource plugins. Prometheus continues to store metrics, Jaeger provides trace exploration, OpenSearch indexes application logs, and Pyroscope stores profiles. The datasource configuration and links between signals still need to be maintained.
Perses provides an operational dashboard with links to related telemetry where the datasource integrations support them. In this implementation, we use Jaeger's native interface for detailed trace exploration.
Operating the Open-Source Stack Takes Work
The open-source stack gives us flexibility. We can choose the tools we need, configure their integrations, and control how we store and query telemetry. However, that flexibility comes with operational responsibility.
The Daily Grind app uses several independent components. Each component has its own configuration, storage requirements, upgrade cycle, and operational considerations.
| Component | Operational responsibility |
|---|---|
| OpenTelemetry Collector | Maintain receivers, processors, exporters, and pipelines |
| Jaeger | Configure trace ingestion, storage, and retention |
| Prometheus | Manage scraping, storage, retention, and queries |
| OpenSearch | Manage indices, mappings, storage, and retention |
| Data Prepper | Maintain log ingestion pipelines and troubleshoot processing failures |
| Pyroscope | Configure profiling, sampling, and profile storage |
| Perses | Maintain datasources, plugins, dashboards, and access |
The main challenge is maintaining the integrations. The Collector must export to compatible backends, Data Prepper must preserve log context, and Perses must connect to working datasources. Every backend also needs suitable capacity, access controls, upgrades, and retention policies.
As telemetry volume grows, these tasks affect storage costs, reliability, and the engineering time required to maintain the stack. This leads to another question:
Can we keep the flexibility of OpenTelemetry without managing every observability component ourselves?
Taking the Daily Grind to Dash0
The open-source implementation gave us control over the observability stack, but it also required us to operate multiple backends.
We don't need to rewrite The Daily Grind or replace its OpenTelemetry instrumentation. The existing services continue to generate telemetry, while we configure the OpenTelemetry Collector to export supported signals to Dash0 through OTLP.
The Daily Grind sending supported OpenTelemetry signals to Dash0.
Explore services and dependencies
Dash0 provides a Service Catalog that identifies application services through OpenTelemetry resource attributes such as service.name and service.namespace.
For The Daily Grind, we should see the Order API, Inventory, and Barista services, along with information such as request rate, error percentage, and latency percentiles.
The Service Map builds on this information by showing relationships between services based on distributed traces. For example, if the Barista service starts reporting higher latency, we can use the map to understand its dependencies and open the related service telemetry.
Refer to the Dash0 Service Catalog documentation and Service Map documentation.
Explore telemetry and dashboards
Dash0 provides a unified environment for exploring logs, metrics, and traces. We can investigate an order and move between related telemetry without managing separate storage and query interfaces.
It also supports dashboards based on the Perses open dashboard specification. This creates an interesting connection to our previous implementation: we used Perses as a self-hosted dashboard layer, while Dash0 provides an integrated dashboard experience using the same open specification.
Dash0 dashboard with multiple telemetry views.
See the Dash0 dashboards documentation.
Monitor reliability with checks and SLOs
Dash0 also provides check rules and Service Level Objectives (SLOs).
For The Daily Grind, we could configure checks for increased coffee preparation latency or a higher percentage of failed orders. We could also define an SLO that measures the percentage of accepted orders that reach the ready state within a specified time.
This extends our investigation workflow from identifying performance problems to monitoring whether the application consistently meets its reliability targets.
See the Dash0 alerting documentation.
What about profiling?
The Pyroscope Python SDK sends profiles through Pyroscope's native ingestion protocol. Changing the Collector's trace, metric, and log exporters does not automatically redirect profiles to Dash0.
Dash0 accepts profiles through OTLP. To use its profiling capabilities, we need a compatible profiler or exporter that produces OpenTelemetry Profiles, or a supported conversion path.
Dash0's continuous profiling capability reached general availability in November 2026, but the OpenTelemetry Profiles signal remains Alpha. We must verify compatibility between the profiler, Collector, and backend before choosing an implementation.
Refer to the Dash0 OTLP integration and profile ingestion documentation.
Conclusion
Building The Daily Grind made observability feel less like a collection of tools and more like an investigation workflow. The biggest lessons were:
- Context and propagation connect operations across service boundaries.
- Metrics, traces, logs, and profiles answer different questions during an investigation.
- Background workflows require separate duration measurements and careful trace-context handling.
- Managing an open-source observability stack requires more than installing the tools. We must integrate, configure, secure, upgrade, and maintain each component.
- An integrated platform can reduce that operational effort without requiring us to abandon OpenTelemetry.
The final result is an interactive coffee shop and a local observability lab. It is small enough to understand, but complete enough to demonstrate how open-source tools work together to investigate a distributed application.
The customer tracks the coffee. The developer tracks the system.
Project Resources
The README contains the project setup instructions, dependency versions, Docker Compose configuration, and validation steps.
Related Articles
This project builds on the concepts explored in the earlier articles:
- Native OpenTelemetry — How OpenTelemetry generates and transports telemetry, and why visibility into telemetry processing matters.
- Your Telemetry Doesn't Go Straight to the Backend: Understanding the OpenTelemetry Collector — How the Collector receives, processes, and exports telemetry through different pipelines.
- Beyond the Collector: How the Open-Source Observability Stack Fits Together — How Prometheus, Jaeger, OpenSearch, and Perses fit into an observability architecture.
- Your Telemetry Has Data. Does It Have Context? — Semantic conventions, resource attributes, and context propagation.
- Before You Paste Telemetry Data into an AI Agent — The security and privacy considerations when sharing telemetry with AI tools.











