Test failover in your CI/CD pipeline with AWS Fault Injection Service

Introduction

When a task in an ECS service stops, ECS starts a new one by itself. You do not have to do anything. The load balancer then sends traffic to the new task when it is healthy.

That is the design. But a design is a claim until you test it. These things can still go wrong:

• The new task cannot start. For example, the image cannot be pulled or a setting is missing.
• The new task starts but never passes the load balancer health check.
• The service recovers, but it takes longer than you can accept.
• The service runs only one task, so users see an outage while the replacement starts.

Most teams find these problems during a real outage. In this guide, you find them on every release instead.

You build a pipeline that does this:

1. It deploys the service.
2. It uses AWS Fault Injection Service (FIS) to stop one running task on purpose.
3. It waits, then checks that the service is back to its desired task count and that the load balancer shows all targets healthy.
4. If recovery takes longer than the time you set, the stage fails.

You will build it twice. First by hand in the AWS console, so you see every part. Then with two CloudFormation stacks.

The demo service runs plain nginx, so the test will pass. The demo shows how the pieces work. The test becomes useful when you point it at a real service.

What this test proves: the service recovers from a lost task within a time limit that you choose. What it does not prove: zero failed requests during the failure. It does not send traffic while the task is down.

Architecture

• An S3 bucket holds source.zip. A change to this file starts the pipeline.
• CodePipeline has three stages: Source, Deploy, FailoverTest.
• The Deploy stage updates the ECS service.
• The FailoverTest stage runs a CodeBuild project. It starts the FIS experiment and checks recovery.
• The FIS experiment stops one task of fis-demo-service.
• A CloudWatch alarm watches the number of healthy targets. It is the experiment's stop condition.

The service runs 2 tasks. The experiment stops 1 of them, so one task stays up while ECS replaces the other.

How the pieces fit

Deploy and FailoverTest are two different jobs. The Deploy stage is a normal release. It puts the new version on ECS. The FailoverTest stage runs after it and asks one question: does the service still recover if a task dies? It works like an integration test that runs after a deploy.

What FIS does. FIS is the tool that breaks the task. It stops one task and only one, because the target is set to select a count of 1. It uses its own IAM role, which can only stop tasks in this cluster, so the pipeline does not need broad ECS permissions to break things. It also keeps a record of each experiment.

You could stop a task with aws ecs stop-task and get the same failure. FIS adds the limited role, the record, and the stop condition. FIS is worth more with faults that run over time, such as CPU stress or network delay, because a stop condition can halt those. For this one action, the stop-condition alarm cannot undo a stopped task. The real safety control is the count of 1.

Why CodeBuild is in the pipeline. At the time of writing, I found no built-in CodePipeline action that starts a FIS experiment. So the FailoverTest stage runs a CodeBuild project, and CodeBuild runs a script. CodeBuild does not break anything itself. It gives the script a place to run, and it turns the result into pass or fail for the pipeline.

What the script does. CodeBuild reads buildspec/run-fis-experiment.yml from source.zip. That file runs scripts/run-fis-experiment.sh. The script has four steps. It exits with an error if any step fails.

1. Check before. It asks ECS how many tasks are running and asks the load balancer how many targets are healthy. Both must equal the desired count. If not, it stops. A service that was already broken should not be blamed on the test.
2. Start. It runs aws fis start-experiment with the experiment template ID. FIS stops one task.
3. Wait for the experiment. It checks the status every 10 seconds until the experiment is completed. If the status is stopped, failed or cancelled, or it takes more than 300 seconds, the script fails.
4. Wait for recovery. It keeps checking running tasks and healthy targets until both match the desired count again. If this takes longer than RECOVERY_TIMEOUT (300 seconds by default), the script fails. Otherwise it prints The service recovered. Test passed.

The script gets the template ID, cluster name, service name and target group ARN from environment variables on the CodeBuild project.

Why S3 is used. The project uses two S3 buckets, for two different jobs.

• The source bucket holds source.zip. A pipeline needs a place to read its source from. S3 keeps the demo simple, because it needs no GitHub connection and no extra setup step. CodePipeline reads the source as one zip file, and the bucket must have versioning turned on. A new upload of source.zip creates a new version and starts a pipeline run. In a real project, you would replace this with a Git repository.
• The artifacts bucket belongs to the pipeline. CodePipeline uses it to pass files from one stage to the next. It is a separate bucket from the source bucket, and every pipeline needs one.

When it runs. The test runs on every pipeline run, after the Deploy stage succeeds. If Deploy fails, the test does not run. In this demo, a new source.zip in the S3 bucket starts the pipeline. In real use, point the test at a staging environment first. Stopping a task in production on every release is a decision your team should make on purpose.

Before you start

You need:

• An AWS account and permission to create the resources below.
• AWS CLI v2 on your computer, signed in to your account.
• zip on your computer.
• A Region where FIS, ECS Fargate, CodePipeline and CodeBuild are available. Use the same Region for every step.

Cost: FIS charges per action-minute. At the time of writing, the price is $0.10 per action-minute. You also pay for the Fargate tasks, the load balancer and CodeBuild while they run. Delete everything when you finish (see Teardown).

Use these names exactly. The scripts and templates depend on them.

Get the code:

git clone https://github.com/Gbengard/aws-fis-failover-testing.git cd aws-fis-failover-testing 

Where to watch what happens

Open these pages in separate tabs before you start the test. Refresh them as the pipeline runs.

Manual approach in the AWS console

The manual path uses the default VPC. The CloudFormation path creates its own VPC.

Part 1. Build the service

Step 1. Create the security groups.

1. Open the EC2 console. In the left menu, choose Security Groups, then choose Create security group.
2. Security group name: fis-demo-alb-sg. Description: ALB for the FIS demo. VPC: the default VPC.
3. Under Inbound rules, choose Add rule. Type: HTTP. Source: Anywhere-IPv4.
4. Choose Create security group.

1. Choose Create security group again. Name: fis-demo-task-sg. Description: Tasks for the FIS demo. VPC: the default VPC.
2. Under Inbound rules, choose Add rule. Type: HTTP. Source: Custom, then pick fis-demo-alb-sg.
3. Choose Create security group.

Step 2. Create the target group.

1. In the EC2 console, choose Target Groups, then Create target group.
2. Target type: IP addresses. Target group name: fis-demo-tg. Protocol: HTTP. Port: 80. VPC: the default VPC.
3. Under Health checks, keep the path as /.
4. Choose Next. Do not register any targets. Choose Next. Choose Create target group.

Step 3. Create the load balancer.

1. In the EC2 console, choose Load Balancers, then Create load balancer.
2. Under Application Load Balancer, choose Create.
3. Load balancer name: fis-demo-alb. Scheme: Internet-facing. VPC: the default VPC.
4. Under Network Mappings, select at least two Availability Zones.
5. Under Security groups, remove the default group and select fis-demo-alb-sg.
6. Under Listeners and routing, keep HTTP and port 80. For Default action, select fis-demo-tg.

1. Choose Create load balancer.

Step 4. Create the cluster.

1. Open the ECS console. Choose Clusters, then Create cluster.
2. Cluster name: fis-demo-cluster. Keep AWS Fargate (serverless) selected.
3. Choose Create.

Step 5. Create the task execution role.

ECS uses this role to pull the image and write logs.

1. Open the IAM console. Choose Roles, then Create role.
2. Trusted entity type: Custom trust policy. Paste this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } 

1. Choose Next. In the permissions list, search for AmazonECSTaskExecutionRolePolicy and select it. Choose Next.
2. Role name: fis-demo-task-execution-role. Choose Create role.

Step 6. Create the task definition.

1. In the ECS console, choose Task definitions, then Create new task definition.
2. Task definition family: fis-demo-task. Launch type: AWS Fargate. CPU: .25 vCPU. Memory: .5 GB.
3. Task execution role: choose fis-demo-task-execution-role.
4. Under Container - 1, set Name: web. Image URI: public.ecr.aws/docker/library/nginx:latest. Container port: 80. Protocol: TCP.
5. Choose Create.

Step 7. Create the service.

1. Open fis-demo-cluster. On the Services tab, choose Create.
2. Set Family: fis-demo-task (latest revision). Service name: fis-demo-service.
3. Under Compute options, choose Launch type, then FARGATE.
4. Under Deployment configuration, set Desired tasks: 2.
5. Under Networking, keep the default VPC and subnets. Remove the default security group and select fis-demo-task-sg. Turn Public IP on.
6. Under Load balancing, check use load balancing, choose Application Load Balancer. Choose Use an existing load balancer and select fis-demo-alb.
7. For the container, choose web 80:80. Choose Use an existing listener (HTTP:80) and Use an existing target group (fis-demo-tg).
8. Choose Create.

Step 8. Check the service.

1. Wait until the service shows 2 running tasks.

1. In the EC2 console, open fis-demo-tg. On the Targets tab, wait until both targets are healthy.

1. Open the ALB DNS name in your browser, make sure you open with http. You should see the nginx welcome page.

Part 2. Add the FIS experiment

Step 9. Create the CloudWatch alarm.

1. Open the CloudWatch console. Choose Alarms, then Create alarm.
2. Choose Select metric. Choose ApplicationELB, then Per AppELB, per TG Metrics.
3. Select HealthyHostCount for fis-demo-tg. Choose Select metric.
4. Statistic: Minimum. Period: 1 minute.
5. Conditions: Static, Lower, than 1. Choose Next.
6. Remove any notification action. Choose Next, click on the prompt message proceed without actions.

1. Alarm name: fis-demo-no-healthy-hosts. Choose Next, then Create alarm.

Step 10. Create the IAM role for FIS.

1. Open the IAM console. Choose Roles, then Create role.
2. Trusted entity type: Custom trust policy. Paste this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "fis.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } 

1. Choose Next. Do not attach any policy. Choose Next.
2. Role name: fis-demo-experiment-role. Choose Create role.
3. Open the role. Choose Add permissions, then Create inline policy. Choose the JSON tab and paste this. Replace <ACCOUNT_ID> and <REGION>.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["ecs:DescribeTasks", "ecs:ListTasks", "tag:GetResources"], "Resource": "*" }, { "Effect": "Allow", "Action": "ecs:StopTask", "Resource": "*", "Condition": { "ArnEquals": { "ecs:cluster": "arn:aws:ecs:<REGION>:<ACCOUNT_ID>:cluster/fis-demo-cluster" } } } ] } 

1. Choose Next. Policy name: fis-demo-stop-task. Choose Create policy.

Step 11. Create the experiment template.

1. Open the FIS console. Choose Experiment templates, then Create experiment template.
2. Description: Stop one running task of the demo ECS service. Name: fis-demo-stop-one-task.
3. Under Actions, choose Add action. Name: StopOneTask. Action type: aws:ecs:stop-task. Choose Save.

1. Under Targets, edit the target that FIS created for the action. Name: OneTask. Resource type: aws:ecs:task.
2. Set the target to use resource parameters: cluster = fis-demo-cluster and service = fis-demo-service.
3. Selection mode: Count. Number of resources: 1. Choose Save.

1. Click on next. Under Service access, choose Use an existing IAM role and select fis-demo-experiment-role and click next.
2. Under Stop conditions, select the alarm fis-demo-no-healthy-hostsand click next.
3. Choose Create experiment template.

Step 12. Run the experiment by hand.

1. Open the template. Choose Start experiment. Type start when asked, then choose Start experiment.
2. Watch the experiment state change until it shows Completed.

1. In the ECS console, open fis-demo-service. On the Tasks tab, you see one task stopped and a new task starting.

1. In the target group, wait until both targets are healthy again.

The service recovered. Now put this check into a pipeline.

Part 3. Add the pipeline

Step 13. Create the buckets.

1. Open the S3 console. Choose Create bucket.
2. Bucket name: fis-demo-source-<ACCOUNT_ID>-<REGION>.
3. Under Bucket Versioning, choose Enable.
4. Choose Create bucket.
5. Choose Create bucket again. Bucket name: fis-demo-artifacts-<ACCOUNT_ID>-<REGION>. Leave versioning off. Choose Create bucket.

The source bucket holds source.zip. The pipeline keeps its working files in the artifacts bucket.

Step 14. Upload the source.

From the repo folder, run:

bash tools/package-source.sh fis-demo-source-<ACCOUNT_ID>-<REGION> 

This builds source.zip and uploads it. The zip has three files: imagedefinitions.json, buildspec/run-fis-experiment.yml and scripts/run-fis-experiment.sh.

Step 15. Create the CodeBuild role.

1. Open the IAM console. Choose Roles, then Create role.
2. Trusted entity type: Custom trust policy. Paste this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "codebuild.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } 

1. Choose Next. Do not attach any policy. Choose Next.
2. Role name: fis-demo-codebuild-role. Choose Create role.
3. Open the role. Choose Add permissions, then Create inline policy. Choose the JSON tab and paste this. Replace <ACCOUNT_ID>, <REGION> and <TEMPLATE_ID>. You get the template ID from the FIS console, on the experiment template page.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:<REGION>:<ACCOUNT_ID>:log-group:/aws/codebuild/fis-demo-failover-test*" }, { "Effect": "Allow", "Action": ["s3:GetObject", "s3:GetObjectVersion", "s3:PutObject"], "Resource": "arn:aws:s3:::fis-demo-artifacts-<ACCOUNT_ID>-<REGION>/*" }, { "Effect": "Allow", "Action": "ecs:DescribeServices", "Resource": "*" }, { "Effect": "Allow", "Action": "elasticloadbalancing:DescribeTargetHealth", "Resource": "*" }, { "Effect": "Allow", "Action": "fis:StartExperiment", "Resource": [ "arn:aws:fis:<REGION>:<ACCOUNT_ID>:experiment-template/<TEMPLATE_ID>", "arn:aws:fis:<REGION>:<ACCOUNT_ID>:experiment/*" ] }, { "Effect": "Allow", "Action": "fis:GetExperiment", "Resource": "arn:aws:fis:<REGION>:<ACCOUNT_ID>:experiment/*" }, { "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/fis-demo-experiment-role", "Condition": { "StringEquals": { "iam:PassedToService": "fis.amazonaws.com" } } }, { "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "*", "Condition": { "StringLike": { "iam:AWSServiceName": "fis.amazonaws.com" } } } ] } 

1. Choose Next. Policy name: fis-demo-codebuild-policy. Choose Create policy.

Step 16. Create the pipeline role.

1. Open the IAM console. Choose Roles, then Create role.
2. Trusted entity type: Custom trust policy. Paste this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "codepipeline.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } 

1. Choose Next. Do not attach any policy. Choose Next.
2. Role name: fis-demo-pipeline-role. Choose Create role.
3. Open the role. Choose Add permissions, then Create inline policy. Choose the JSON tab and paste this. Replace <ACCOUNT_ID> and <REGION>.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:GetBucketVersioning", "s3:GetBucketLocation", "s3:ListBucket", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::fis-demo-source-<ACCOUNT_ID>-<REGION>", "arn:aws:s3:::fis-demo-source-<ACCOUNT_ID>-<REGION>/*", "arn:aws:s3:::fis-demo-artifacts-<ACCOUNT_ID>-<REGION>", "arn:aws:s3:::fis-demo-artifacts-<ACCOUNT_ID>-<REGION>/*" ] }, { "Effect": "Allow", "Action": ["codebuild:StartBuild", "codebuild:BatchGetBuilds"], "Resource": "arn:aws:codebuild:<REGION>:<ACCOUNT_ID>:project/fis-demo-failover-test" }, { "Effect": "Allow", "Action": [ "ecs:DescribeServices", "ecs:DescribeTaskDefinition", "ecs:DescribeTasks", "ecs:ListTasks", "ecs:RegisterTaskDefinition", "ecs:UpdateService" ], "Resource": Source：https://dev.to/gbengard/test-failover-in-your-cicd-pipeline-with-aws-fault-injection-service-588n
My favorites

Related posts

interview questions and answers based on real interview for Jibek

TELL ME ABOUT YOURSELF My name is Zhibek. I am a DevOps Engineer with more than 5 years of experience in AWS cloud. I st

0dev.to DevOps

THRIVE MARKET SRE – TECHNICAL INTERVIEW PREP, questions and answers for Aisel for the next step

PART 1. WHAT YOU KNOW ABOUT THRIVE (use this in your answers) Company is 10+ years old, everything runs on AWS. Core app

0dev.to DevOps

Canary Historian vs TDengine: two historians that publish prices by tag count

Canary and TDengine are two of the few historians that publish their prices, and both price by tag count. Canary is a cl

0dev.to Database

Show HN: CloudSlasher – Multi-Cloud FinOps for OCI, AWS, Azure, GCP and K8s

Article URL: https://cloudslasher.com/ Comments URL: https://news.ycombinator.com/item?id=50343391 Points: 1 # Comments:

0Hacker News Show

The new November update for the aerynOS operating system has been released

aerynOS has released version 2026.11, bringing updated desktop environments including Budgie 10.10.4, KDE Plasma 6.8.2,

0dev.to Linux

How to Set Up Wubi98 with IBus + Rime on Linux

🐧 How to Set Up Wubi98 with IBus + Rime on Linux This guide walks you through installing the Wubi98 input method using R

0dev.to Linux