Our purchasing server runs the jobs that turn the morning's replenishment into purchase orders and send them to suppliers, some by email and some as files to their own systems. In May we needed to test an upgrade to the purchasing software, and the quickest way to get a realistic test machine was to clone the production server.
An engineer made the copy on a Friday afternoon and attached it to our test network. Our test network can reach the internet, because testers need to download things and reach suppliers' test portals. He planned to start the upgrade on Monday.
At six on Saturday morning, the clone woke up and did exactly what the original did. It read the same purchasing database, built the same purchase orders and sent them to the same suppliers, from the same addresses and under the same order numbers. Every supplier on that morning's run received each order twice.
Most of their systems rejected the repeats or someone noticed. Three did not. On Monday we received a second lorry of cement, a double delivery of roof battens and twice the copper fittings we had asked for. Returning them cost us collection charges and an awkward call to each supplier.
Nothing about the clone was faulty. A copy of a server includes its scheduled jobs, its credentials and its idea of who it is. We had taken great care that test systems never touch production data, and none that a copy of production never behaves as production.
Clones are now created switched off and started on an isolated network with no route out. The engineer disables every scheduled job and checks the list with a colleague before the clone is moved anywhere it can reach a supplier, a customer or a mail server. Every job that sends anything outside the company first asks a central setting whether the server it runs on is the named production server, and stops if it is not. We also asked our twenty largest suppliers to reject a purchase order number they have already received, and most of them now do.
Automation does not know it has been copied. Make sure the copy can tell.
– Serguey Shinder
