In this post, we discuss how Rakbank retail digital banking platform ingests approximately 150 GB a day of telemetry across a cross-account, Multi-Availability Zone (Multi-AZ) Amazon OpenSearch Service design, with hot/warm/cold tiering, 7-year compliance retention backed by Amazon Simple Storage Service (Amazon S3), and approximately 55 percent cost savings over an all-hot cluster.

Context and workload profile

Rakbank’s retail digital banking platform runs as containerized microservices on Amazon Elastic Kubernetes Service (Amazon EKS), and its observability requirements are driven by a few hard numbers. The platform now generates roughly 150 GB of logs per day on average, up from about 5 GB/day before the migration. That produces around 9 TB retained in OpenSearch Service over a 60-day window (150 GB/day × 60 days), with the most recent 30 days directly searchable. Retention runs to seven years for compliance, aligned with Central Bank of the UAE (CBUAE) requirements and Rakbank’s internal data-retention policy. At an average of 150 GB per day, that archives approximately 54 TB of log data per year, or approximately 383 TB over seven years. On the consumption side, 25 dashboards serve about 25 concurrent users. The data itself has an important property: logs are append-only. After written, a log document is never updated or deleted, so there is no update or delete-by-query churn. (Where regulatory write-once-read-many immutability is required, the S3 archive bucket can enforce it with Amazon S3 Object Lock.)

The append-only pattern is architecturally significant: because indices are no longer written to once they roll over, force-merging them to a single segment per shard before warm migration is safe, and there is no update overhead as indices roll down the tiers.

Telemetry pipeline: Fluent Bit

Instrumentation follows an instrument once, standardize everywhere model on a Fluent Bit standard across all banking domains (retail, business, wholesale). Concretely:

  • Fluent Bit runs as a DaemonSet on the EKS cluster, tailing container stdout/stderr on each node and enriching records with Kubernetes metadata (pod, namespace, labels) before routing.
  • Fluent Bit performs a dual-write/parallel archive: one path ships to the OpenSearch Service ingest endpoint (hot), a parallel path lands raw log objects directly in Amazon Simple Storage Service (Amazon S3) for durable backup independent of the search tier.

Cross-account topology and private connectivity

The design separates the workload from its observability plane using two accounts:

Application account

  • Amazon EKS cluster running the app pods in the Application virtual private cloud (VPC).
  • Fluent Bit DaemonSet for container-log collection and routing.
  • A parallel archive path to Amazon S3 (Log Archive & Backup).

Observability account

  • A dedicated OpenSearch Service centralized logging domain in the OpenSearch Service VPC, spanning three Availability Zones.
  • OpenSearch Service Dashboards as the access/visualization layer, routed to the hot data nodes.

Connectivity. Cross-account log flow traverses an AWS Transit Gateway, keeping all communication private within AWS. No traffic egresses to the public internet. In practice, Transit Gateway route tables provide reachability between the application and OpenSearch Service VPCs. Access is scoped at the VPC boundary by security groups and network ACLs that permit only the required ingest traffic (HTTPS/443) to the domain.

The following diagram shows the full cross-account path: application pods write logs through a Fluent Bit DaemonSet, which both forwards them privately over AWS Transit Gateway to the OpenSearch Service domain and archives a parallel copy to Amazon S3.

Cross-account architecture showing Amazon EKS pods sending logs through Fluent Bit over AWS Transit Gateway to a Multi-AZ OpenSearch Service domain with Amazon S3 archival

Figure 1: Centralized observability platform showing cross-account log flow from the EKS application account across AWS Transit Gateway into the Multi-AZ OpenSearch Service domain with hot, UltraWarm, and cold tiering and Amazon S3 archival

In the Application account, three application pods on the Amazon EKS cluster write logs that a Fluent Bit DaemonSet collects. Fluent Bit ships one copy over AWS Transit Gateway to the Observability account and archives a parallel copy to the Amazon S3 Log Archive and Backup bucket. In the Observability account, the OpenSearch Service centralized logging domain spans three Availability Zones. Each zone has a dedicated cluster-manager node (m8g.large.search) and a hot data node (m8g.2xlarge.search) that ages out to an UltraWarm node (ultrawarm1.medium.search) and then to a shared, node-free S3-backed cold storage tier. OpenSearch Service Dashboards reads from the hot data nodes for visualization.

Identity and cross-account authorization flow

The platform spans two accounts, with access governed by the layered security model of OpenSearch Service: network isolation (VPC and private Transit Gateway connectivity), the domain’s resource-based access policy, and fine-grained access control (FGAC) on the VPC domain. End users reach OpenSearch Service Dashboards through Security Assertion Markup Language (SAML) 2.0 single sign-on federated to Rakbank’s corporate Active Directory. Access follows existing enterprise identities and group memberships, with no separate OpenSearch Service credentials. Log ingestion from the application account is authorized separately from user access, keeping the write path and the read path on distinct, least-privilege roles. Together these layers keep all telemetry flowing privately within AWS while access stays centrally governed in Active Directory.

Cluster topology (per-AZ node roles)

The domain is a 3-AZ deployment with dedicated cluster-manager nodes separated from data nodes:

Role Per-AZ instance Count Purpose
Cluster manager m8g.large.search 3 (1/AZ) Cluster state, shard allocation, no data
Hot data node m8g.2xlarge.search 3 (1/AZ) Active indexing + real-time query
UltraWarm node ultrawarm1.medium.search 3 (1/AZ) Read-only, S3-backed warm segments
Cold tier No nodes — Detached S3-backed storage, managed by OpenSearch Service

The hot data and cluster-manager nodes are Graviton (m8g) for better price/performance. Dedicated cluster-manager nodes isolate cluster-state management from indexing and query load, which matters after you’re pushing 150 GB/day through the hot tier.

Storage tiering: right data · right store · right cost

The core mechanism is OpenSearch Service multi-tier storage, with indices aging through tiers using Index State Management (ISM) policies keyed off index age:

Tier Backing Retention Access pattern Instance/store
Hot Amazon Elastic Block Store (Amazon EBS) (gp3) volumes on data nodes 0–15 days Real-time dashboards, active troubleshooting, alerting m8g.2xlarge.search
Warm (UltraWarm) Amazon S3 + local cache 15–30 days Read-only historical analysis. Cost-optimized, still fully searchable ultrawarm1.medium.search
Cold Amazon S3 (detached, no nodes) 30–60 days Older logs for on-demand investigation. Attach to UltraWarm when needed S3-backed, 0 compute at rest

How the lifecycle behaves:

  • Hot → UltraWarm: ISM transitions indices at approximately 15 days. UltraWarm keeps primary shard data in S3 while caching hot blocks locally, so queries still run, only at higher latency and far lower cost per GB. Because rolled-over indices are no longer written to, they are force-merged before migration for faster warm-tier read performance.
  • UltraWarm → Cold: At approximately 30 days, indices detach to the cold tier, which is pure S3 storage with no dedicated compute. Cold indices are not directly searchable: to query them, you selectively attach them back to UltraWarm nodes on demand. (Querying the raw S3 archive without reattaching is a separate path that uses Amazon Athena.)
  • Compliance archive (S3 + Athena): Everything also lands in a separate Amazon S3 bucket for the full 7-year window, and is queried with Amazon Athena, at the lowest cost per GB. This decouples how long data must be kept from how much hot compute is paid for.

Durability details called out in the design: The durability of Amazon S3 means the design can run 0 replicas on the warm/cold tiers and remain durable (S3 handles redundancy), which further cuts cost. UltraWarm is a read-only warm tier, and Graviton (m8g) powers the hot and cluster-manager nodes.

Alerting. Hot-tier data drives OpenSearch Service alerting with email and Microsoft Teams destinations for real-time operational alarms.

Production sizing and cost model

Sizing target: 150 GB/day average, 60-day retention (30 days directly searchable) (the cluster is provisioned with headroom for an approximately 200 GB/day peak, which is the basis for the node sizing that follows). Rakbank evaluated two topologies:

Option 1 — Hot + UltraWarm (recommended)

  • 3 × m8g.2xlarge hot data nodes.
  • 3 × ultrawarm1.medium.search + cold tier.
  • 3 × m8g.large dedicated cluster-manager nodes.
  • Approximately 55 percent savings compared to all-hot.

Option 2 — All Hot

  • 3 × r8g.4xlarge, approximately 8,320 GB each.
  • 3 × m8g.large admin, no warm tier.
  • Higher cost, all data resident hot.

Pricing is On-Demand. Reserved Instances reduce it further. The takeaway: for a 9 TB, 60-day window, the bulk of data is rarely queried after the first couple of weeks. Pushing older indices to an S3-backed warm/cold tier removes the need to keep the full working set on hot r8g nodes, which is where the approximately 55 percent delta comes from.

Scaling the observability architecture with OpenSearch Service

As the platform grew from approximately 5 GB/day to approximately 150 GB/day with 25 dashboards and approximately 25 concurrent users, the observability architecture had to meet three requirements:

Predictable cost at high ingestion volume, repeated dashboard queries without per-query cost coupling, and multi-tier retention spanning real-time access through to 7-year archival. OpenSearch Service meets these requirements with an instance-based model and hot/warm/cold tiering: ingestion and queries run on provisioned capacity, and older indices age into S3-backed tiers for long-term retention at a low cost per GB. This gives Rakbank a predictable, capacity-based cost model as data volume and dashboard usage grow.

Resilience: Multi-AZ and the Standby trade-off

Deployment option SLA Failover Nodes
Multi-AZ (no Standby) 99.9% out-of-the-box automatic 3 AZ · 3 data nodes + 3 dedicated cluster-manager nodes
Multi-AZ with Standby 99.99% < 1 min automatic on AZ failure 3 AZ · 2 data active + 1 standby

The trade-off to understand: with Multi-AZ with Standby enabled, only two of three data nodes serve search requests. The third is reserved for failover. That is a 33 percent reduction in active search capacity in exchange for the higher service level agreement (SLA). Standby also requires at least two replicas per index, which increases storage, and is supported only on specific instance families, so check the current list before planning it. If a 99.99 percent SLA is not a hard requirement, running without Standby keeps all data nodes serving search. Rakbank runs its production domain without Standby, with data nodes across all three Availability Zones and replica shards for redundancy.

AI-powered observability capabilities

Rakbank is evaluating newer, machine learning (ML) and agent-driven features of OpenSearch Service that can layer on top of the tiered store:

  • AI Assistant – Natural-language → query generation (for example, “Show error spikes in the last 24h”), backed by Amazon Bedrock.
  • MCP Server + Agent investigator – DevOps agents autonomously execute PPL queries for root-cause analysis. A Memory Store provides persistent context across agent conversations.
  • Direct Query – Query Amazon S3 data without ingesting it into OpenSearch Service, an option alongside Amazon Athena for the S3 archive.
  • Trace Correlation – Available as trace collection is added to the pipeline: unify logs, traces, and metrics in one view, with automatic trace/log correlation and service maps.
  • Query surface – PPL, DSL, SQL, PromQL. Integrates with Amazon Managed Service for Prometheus and Amazon OpenSearch Ingestion for high-performance ingest.

Outcomes

Running on the Fluent Bit standard, the platform now serves three audiences from a single pane of glass. Support operations get faster triage and guided troubleshooting, engineering teams use it for root-cause analysis and debugging across the request path, and the business gains journey trends and actionable insights.

Net results: lower mean time to resolution (MTTR), higher reliability, data-driven insights, and improved operational efficiency. In their own environment, Rakbank measured faster performance after upgrading across OpenSearch Service versions (1.3 → current), with single-digit-millisecond latencies for lexical and vector queries.

Engineering lessons learned

  1. Multi-tier from day one – Design hot/warm/cold up front. Retrofitting tiering onto a live 150 GB/day cluster is significantly harder than starting with ISM policies in place.
  2. Size for peak, not today – Ingestion grows as more microservices come online. Provision headroom in the hot tier and shard count.
  3. Model the Standby trade-off explicitly – 99.99 percent costs you 33 percent of active search capacity and requires two replicas per index. Decide per workload whether that SLA is worth it.
  4. Exploit the append-only advantage – Append-only logs are the recommended candidate for multi-tier storage: no update overhead, clean force-merge before warm migration, and predictable rollover.


About the authors

Georges Hamieh

Georges Hamieh

Georges is a Senior Technical Account Manager at AWS. He supports FSI customers across the Middle East and North Africa in building scalable and intelligent cloud solutions.

Tapas Kumar Mondal

Tapas Kumar Mondal

Tapas is the Executive Vice President & Chief Technology Officer in Digital & AI at Rakbank. He heads Rakbank architecture, design, and technology strategy, and is the CTO for digital channels and AI platforms.

Amit Pareek

Amit Pareek

Amit is a Senior Vice President and Enterprise architect in Archtecture, Design & Strategy at Rakbank. He leads the solution architecture practice at Rakbank.

Chandramouli Pitchaiyappan

Chandramouli is a Cloud Solution Architect – Architecture, Design and Strategy at Rakbank. He designs and runs Rakbank’s AWS platforms, focusing on Amazon EKS, observability at scale and cloud cost optimization.