Beyond Green Cluster Health: What I Learned Designing Amazon OpenSearch for Production

Building a reliable OpenSearch environment with dedicated master nodes, VPC networking, Terraform, and operational monitoring.

When you set up Amazon OpenSearch Service, it's easy to focus on one thing. Get the domain deployed and see a green cluster status.

The infrastructure provisions fine. The endpoint comes up. The cluster says everything is healthy.

That's a good start. But it doesn't tell you if the environment is ready for production.

A green cluster can still have badly spread shards, too little storage headroom, expensive queries, or security settings that make future maintenance hard. I had to think through all of this while working on an Amazon OpenSearch deployment for a production environment.

The job wasn't just to create another AWS resource. I needed an architecture that could carry production workloads and fit into an existing deployment process. It also had to stay manageable as the environment grew.

In this article, I'll go through the architecture, the decisions behind it, the deployment points and the operational checks I think matter most.

1. Starting With the Architecture

The OpenSearch environment was built around this setup:

The six data nodes handle indexing, storage and search. The three dedicated master nodes do something else. They manage the cluster state and keep the cluster coordinated.

This split matters. When data nodes are busy with queries or indexing, you don't want that work fighting with cluster management. Dedicated master nodes lower that risk.

I also had to think about Availability Zone spread, storage capacity and network access. On top of that, I had to plan how the configuration would be maintained after the first deployment.

Node count alone doesn't make a cluster highly available. Where the nodes sit, how shards are allocated and how the cluster recovers matter just as much.

2. Why I Used Dedicated Master Nodes

One of my first decisions was to keep cluster management apart from data processing.

OpenSearch keeps track of its nodes, indexes, shards and routing. Whenever a node joins or leaves, or an index is created or changed, the cluster state has to be updated. If the cluster can't elect a master node, the service can be affected.

For production, three dedicated master nodes give a good balance between resilience and cost. With three master-eligible nodes, a quorum of two is needed. So the cluster can lose one master node and still elect a leader, as long as the remaining nodes can talk to each other.

There's one more thing. Dedicated master nodes don't handle normal search and indexing requests. They still need enough memory and CPU to manage the cluster state.

A cluster with many indexes, mappings and shards can put real pressure on the master nodes, even when the data nodes look fine. Because of this, master node sizing shouldn't be based on the data node count alone.

I would watch these closely:

• Master JVM memory pressure
• Master CPU utilisation
• Cluster state changes
• Total shard count
• Master reachability

These signals tell you if the master nodes have enough capacity.

3. Availability Zones: A Deployment Detail That Matters

Availability Zone configuration needed attention during the planning.

It's tempting to turn on zone awareness and assume AWS handles the rest. It doesn't. The subnet setup must support the number of Availability Zones you pick. If you configure a domain across two zones, the VPC needs suitable subnets in two separate zones.

This gets tricky inside an existing enterprise VPC, where network policies may already limit which subnets you can use.

For this deployment, I adjusted the configuration around a two-zone data-node layout. With six data nodes, that gives three data nodes per zone.

However, spreading nodes across zones is one thing. Guaranteeing that every workload survives a zone failure is another. You also have to look at replica placement, the capacity that's left and how the cluster recovers after losing nodes.

If I were designing this from scratch today, I would look at Multi-AZ with Standby, which AWS recommends for production workloads. It uses three Availability Zones and gives a more structured way to handle availability and recovery.

The bigger lesson is simple. Availability should be a clear architectural decision. It shouldn't be a checkbox in the deployment config.

4. Storage Is More Than Choosing gp3

The data nodes use Amazon EBS gp3 volumes. I like to treat storage capacity and storage performance as two separate decisions.

The amount of data you expect to store is only part of the sum. OpenSearch also needs room for replica shards, segment merges, indexing and recovery. If you size storage only around today's document volume, the cluster can hit trouble much sooner than expected.

Here's an example. Say a workload has 1 TiB of primary index data and one replica for each primary shard. That's already about 2 TiB of shard data, before any overhead or operational headroom. The real storage need will be higher.

I would work out storage around four questions:

1. How much primary data will the cluster hold?
2. How many replicas will be kept?
3. How fast will the data grow?
4. How much free space must stay available for normal operations and recovery?

For gp3, I would also check the required IOPS and throughput against the expected indexing and search patterns. A big volume doesn't mean every storage-heavy operation will run well. The workload still has to be measured.

Why free storage matters

OpenSearch uses disk watermarks to manage shard allocation and protect the cluster when storage runs low. As free space drops, shard allocation can be restricted. At the more severe thresholds, indexes can become write-blocked.

This is why I wouldn't wait for an application error before looking at storage pressure. I would monitor FreeStorageSpace, disk utilisation, shard distribution and indexing failures together. The goal is to spot capacity problems before users feel them.

5. Keeping OpenSearch Inside the VPC

Network isolation was part of the design here. The OpenSearch domain was set up for VPC access, with no public endpoint.

That means clients need a proper network path to reach the domain. It also changes how you handle administration and troubleshooting. You can't assume an engineer can fire a request from a laptop and hit the endpoint. The request has to come from an authorised network location, such as an approved workload, a connected private network or a controlled admin environment.

I would build the access model in three layers.

Network access: Security groups and routing decide which systems can connect to the domain.

IAM and domain access policies: These decide which AWS principals and requests are allowed.

Fine-grained access control: Where it's enabled, OpenSearch roles can limit access to certain indexes, documents and operations.

Each layer solves a different problem. A security group that allows TCP 443 doesn't mean a caller should be able to delete indexes. And a valid IAM identity doesn't remove the need for the right network access.

For production, I also treat encryption at rest, node-to-node encryption and HTTPS enforcement as the baseline.

What matters is making access intentional and auditable. Relying on the endpoint not being public is not enough.

6. Integrating the Deployment With an Existing Azure DevOps Pipeline

Something interesting about this work was the deployment route. The AWS infrastructure went out through an existing Azure DevOps process.

The pipeline already used a repository structure to pick the target environment and the right AWS deployment context. I didn't want to redesign the whole pipeline just to add one OpenSearch resource. That would widen the change and could affect other deployments that use the same workflow.

So the aim was to make the new infrastructure fit the existing deployment conventions. I've learned to watch for this in enterprise environments. A technically correct change can still add risk if it needs wide changes to shared deployment automation.

For this environment, I had to think about:

• Keeping the existing environment selection behaviour
• Using the correct AWS account and region
• Keeping production configuration separate from other environments
• Handling sensitive credentials through an approved secret store
• Keeping deployment and rollback procedures predictable

The infrastructure definition and the pipeline have different jobs. The definition describes what should exist. The pipeline controls how changes are validated, approved and applied. Keeping them apart makes infrastructure changes easier to review, and you don't change deployment behaviour for unrelated services by accident.

7. Why a Green Cluster Isn't Enough

Once a domain is available, the next question is whether it's running properly. This is where monitoring comes in.

OpenSearch exposes many CloudWatch metrics. I wouldn't treat them all as equally useful. For production, I would start with these:

Each one tells a different part of the story. A cluster can report green while JVM memory pressure stays high. That doesn't mean the cluster is down. It could point to a capacity problem on the way.

The same goes for storage. A cluster might have enough total space while one node has far less free space than the others. If you only look at the combined number, you'll miss that gap.

I would also set up CloudWatch alarms with thresholds that fit the workload. Relying on dashboards that someone has to remember to check is a weak plan. A dashboard helps with investigation. An alarm is what tells you a problem is building. You need both.

8. Shard Allocation Can Become a Hidden Problem

Shard management is one area I would never ignore in OpenSearch.

Every index has primary shards, and those shards can have replicas. Shards let you spread data and search work across nodes. But too many shards has a cost. Each shard uses resources, including heap memory and file handles. A large number of small shards adds cluster management overhead and can hurt performance.

With six data nodes, I wouldn't automatically create six primary shards for every index. So how many is right? It depends on data volume, query patterns, expected growth and recovery needs. A small index may need far fewer shards. A larger one may gain from more spread. The decision should come from the workload, not from matching shard count to node count.

Two useful diagnostic commands are:

GET /_cluster/health?pretty GET /_cat/shards?v 

I would also check allocation and node-level stats:

GET /_cat/allocation?v GET /_nodes/stats/jvm,fs,indices 

These help answer a few questions:

• Are shards spread reasonably across nodes?
• Are there unassigned or relocating shards?
• Is one node carrying a lot more data?
• Are JVM or storage limits affecting the cluster?

A green cluster tells you primary and replica shards are assigned. That's all. It doesn't promise the spread is efficient, or that the cluster can handle the next workload spike.

9. Think About Recovery Before You Need It

One of the most important questions I ask when designing production infrastructure is this. What happens when part of the system becomes unavailable?

For OpenSearch, that could mean losing a data node, an Availability Zone disruption, running out of disk space or an unhealthy index. A recovery plan has to cover more than restarting a service. I would want to know:

• Can the remaining nodes handle the workload after a failure?
• Are replica shards placed properly?
• How long could shard recovery take?
• Are snapshots available and recoverable?
• What happens if an infrastructure update fails?
• Can we restore the service without making the original problem worse?

Amazon OpenSearch Service provides automated snapshots. Even so, I would check the retention and recovery options that apply. For workloads that need longer retention or more recovery flexibility, manual snapshots to a properly configured S3 repository may also be needed.

Having a snapshot is not the same as having a tested recovery process. For a production system, restore steps should be written down and practised in a suitable non-production environment.

10. What I Would Improve in a Future Deployment

Looking back at this architecture, a few areas would get extra attention in a new build.

First, look at three Availability Zones from day one. Two zones give you redundancy. But a three-zone Multi-AZ with Standby design may handle failures better for supported production workloads.

Second, size storage from evidence. I would use real ingestion rates, index sizes, replica needs and retention policies to set capacity.

Third, manage index lifecycles on purpose. Index State Management policies can automate rollover and retention where it makes sense. That lowers the risk of old indexes eating up storage forever.

Fourth, make monitoring part of provisioning. CloudWatch alarms, log publishing and operational dashboards should ship with the infrastructure. They shouldn't be added after an incident.

Finally, test failure scenarios. A production-ready design should have written expectations for node loss, recovery, capacity pressure and restoration. These exercises often show assumptions that you can't see in the infrastructure config.

Final Thoughts

Working on this OpenSearch architecture reinforced something I've seen across AWS infrastructure projects. Provisioning the service is usually the easy part. The harder work starts when you think about how the service behaves after deployment.

For OpenSearch, that means understanding cluster management, node placement, shard allocation, storage headroom, access controls and recovery.

The architecture I worked with used six data nodes, three dedicated master nodes, gp3 storage and VPC networking, all fitted into an existing enterprise deployment workflow. Those pieces gave the environment its foundation.

But production readiness isn't set by how many resources you've provisioned. It comes down to how well you understand the workload, how you catch problems and how confidently you can recover when something breaks.

A green cluster is a useful health signal. It isn't a production-readiness certificate.

That's the point I'd want engineers to keep in mind when they design Amazon OpenSearch environments.