While everything works, backing up an observability cluster feels like a side issue: after all, it is « only » metrics. Then the incident hits: a lost datacentre, a corrupted storage array, an operating system upgrade that goes wrong on several nodes at once. That is when you discover that what you miss most is not the metric history, but years of configuration: alerting rules, dashboards, management zones, integrations, tokens, users and environment settings. A Dynatrace Managed backup strategy is therefore something to prepare before the incident, not during it.

With Dynatrace Managed, this responsibility is entirely yours. The cluster runs on your servers, its data lives on your disks, and the only copy outside the nodes is the one you have configured. The built-in backup mechanism exists, but it relies on shared storage that you provide, and restoring follows a precise procedure that you would do well to rehearse before you need it.

This article explains what a Managed cluster backup covers, how to configure it properly and how to prepare for a full restore.

What does a Dynatrace Managed backup contain?

A Managed cluster stores its data in several layers, and the backup does not treat them all the same way:

Data Storage Backed up?
Cluster and environment configuration Internal files and databases Yes, always
Metrics and timeseries data Cassandra Yes, unless excluded: daily snapshot, without 1- and 5-minute data
User data, sessions, events Elasticsearch Yes: incremental snapshots every 2 hours by default; user sessions can be excluded
Distributed traces (PurePaths) and code-level data Transaction storage No: short-retention data

Three options set the scope: exclude user sessions (useful for GDPR), exclude metrics if only configuration history matters, and include Log Monitoring events. As Elasticsearch backups are incremental, never delete their files by hand.

Shared storage requirements

Each node writes the backup to a common directory, typically an NFS share. The key constraints:

  • the share must be mounted at the same path on every node in the cluster;
  • the system user running the Dynatrace services must be able to read and write to it;
  • capacity must cover several backup generations: the published estimation method uses about 20% of the metrics storage across all nodes, doubled because the old backup is kept until the new one completes, plus the total Elasticsearch volume;
  • the share should ideally sit outside the nodes’ datacentre, or be replicated itself, otherwise it disappears with them;
  • latency and throughput must allow the daily backup to finish well before the next one starts.

Step-by-step tutorial

Step 1 — Mount the NFS share on each node 

sudo mkdir -p /mnt/dynatrace-backup echo "{nfs-server}:/exports/dynatrace /mnt/dynatrace-backup nfs4 defaults,_netdev,hard 0 0" \ | sudo tee -a /etc/fstab sudo mount -a df -h /mnt/dynatrace-backup

Repeat exactly the same steps on every node. The path must be strictly identical everywhere. Beware: if the mount is missing at boot, Dynatrace will not start on that node; NFSv4 is recommended and CIFS discouraged.

Step 2 — Check write permissions

Identify the user the Dynatrace services run as (often dynatrace, depending on your installation) and test writing:

ps -eo user,comm | grep -i dynatrace | sort -u sudo chown {dynatrace-user}:{dynatrace-group} /mnt/dynatrace-backup sudo -u {dynatrace-user} touch /mnt/dynatrace-backup/.test && \ sudo -u {dynatrace-user} rm /mnt/dynatrace-backup/.test && echo "Write OK"

Watch out for NFS root squashing and for UID/GID mismatches between the nodes and the NFS server.

Step 3 — Enable backup in the CMC

In the Cluster Management Console, open the cluster backup settings, then:

  1. enable automatic backup;
  2. enter the share path (/mnt/dynatrace-backup);
  3. choose the scope: exclude user sessions or metrics, include log events;
  4. set the start time, preferably during a low-load period;
  5. save, then check the status of the first run.

Step 4 — Check the backup 

du -sh /mnt/dynatrace-backup/* find /mnt/dynatrace-backup -maxdepth 4 -type d -newermt "-1 day" | head df -h /mnt/dynatrace-backup

Make sure each node has produced its share of the backup and that free space remains sufficient for the following runs. Also keep an eye on the backup status shown in the CMC; ideally, build this check into your cluster health monitoring (alert if the last successful backup is more than 24 to 48 hours old).

Step 5 — Keep an off-site copy

The Managed backup is only a first copy. Apply the 3-2-1 rule: replicate the share to a second site or to object storage with your enterprise backup tool, outside Dynatrace’s write window.

Step 6 — Prepare the restore

A cluster is restored by reinstalling the nodes from the backup. Key points:

  • use the dynatrace-managed-{version}.sh installer of the same version as the backed-up cluster; a copy is kept in each node’s backup tree (files/{backup-number}/), to be copied to a local disk;
  • stop the old cluster if it is still running, so that two clusters with the same ID do not contact Mission Control;
  • prepare machines with the same number of nodes (at most two fewer, in exceptional cases), the same UID:GID for the system user, at least equivalent resources and the share mounted at the same path;
  • note the node IDs and the cluster ID found in the backup directory tree;
  • plan the DNS switch of {your-domain} to the new addresses if they change.

Step 7 — Restore the nodes

The installer has a restore mode that takes, among other parameters, the node’s IP address, the list of cluster nodes, the seed node and the backup file to use. The general form looks like this:

sudo /bin/sh dynatrace-managed-{version}.sh --restore \ --cluster-ip {node-ip} \ --cluster-nodes {node-id-1}:{ip-1},{node-id-2}:{ip-2},{node-id-3}:{ip-3} \ --seed-ip {seed-node-ip} \ --backup-file /mnt/dynatrace-backup/{path-to-the-node-backup-file}

Always refer to the documentation for your version for the exact list of parameters and the order in which to run it on the nodes. Next, on each node, start the firewall, Cassandra and Elasticsearch, restore Elasticsearch from the seed node, then metrics and configuration node by node, before starting Dynatrace:

/opt/dynatrace-managed/launcher/firewall.sh start /opt/dynatrace-managed/launcher/cassandra.sh start /opt/dynatrace-managed/launcher/elasticsearch.sh start /opt/dynatrace-managed/utils/cassandra-nodetool.sh status /opt/dynatrace-managed/utils/restore-elasticsearch-data.sh {backup-path}/{cluster-uuid} /opt/dynatrace-managed/utils/restore-cassandra-data.sh {node-backup-file} /opt/dynatrace-managed/launcher/dynatrace.sh start

Then confirm in the CMC the state of the nodes, followed by the presence of the environments, users and configuration. If agents reach the cluster through a DNS name, updating the records is enough; otherwise, reconfigure the target address of the ActiveGates (or of OneAgents without ActiveGates) and restart them.

Backup and high availability

Backup does not replace high availability, and vice versa. A multi-node cluster tolerates the loss of one node; Premium High Availability replicates the cluster across two datacentres and survives the loss of a site. But no replication protects you from a configuration error propagated everywhere or from logical corruption: only a backup does.

Best practices

  • Place the NFS share outside the nodes’ datacentre, or replicate it.
  • Monitor the date of the last successful backup and the share’s free space.
  • Keep a copy of the installer for the current version next to the backups.
  • Document the topology (node IDs, IPs, DNS, certificates) in a runbook.
  • Test a full restore at least once a year on isolated infrastructure.
  • Recheck the backup configuration after every node addition or major update.

Frequently asked questions

Are distributed traces included in the backup?

No. PurePaths and code-level data are short-retention data and are not backed up. Configuration is always included, Cassandra metrics unless excluded, and Elasticsearch data as incremental snapshots.

Can I restore with a newer installer?

No: the restore must use the dynatrace-managed-{version}.sh installer of the same version as the backed-up cluster. Keep a copy of that installer next to the backups.

Does a highly available cluster still need a backup?

Yes. Replication protects against losing a node or a site, not against a configuration error propagated everywhere or logical corruption: only a backup lets you go back.

Conclusion

Backing up a Dynatrace Managed cluster comes down to a few settings in the CMC, but the reliability of the whole depends on the shared storage, on monitoring the runs and above all on restores that have actually been tested. Take the time to mount a robust share, keep copies off-site and write a restore runbook: on the day you need it, years of accumulated configuration will be at stake.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Always check values (ports, versions, sizes) against the official documentation for your Managed version.

Originally published on span.observer. French version: https://span.observer/fr/sauvegarde-restauration-cluster/.