Introduction

Around 23:00 JST on November 19, 2026, the PHP team shipped security releases for 8.2 through 8.5. There were 17 GitHub Security Advisories (GHSAs), 7 of them critical. Thirty-eight minutes after the GHSAs went public, we put production back behind the load balancer (LB), running patched PHP.

This post shows how to get patched on release day when you run PHP from Ubuntu packages on an LB → Auto Scaling group (ASG) → EC2 stack.

  • Some PHP versions have reached end of life on php.net, so upstream ships no fixes for them
  • Neither Ubuntu nor the ondrej PPA had fixed packages that day, so we backported the upstream fixes onto the Ubuntu source and built the packages ourselves
  • Instead of patching ASG instances one by one, we built a patched AMI and replaced every instance

Assumed setup

This post assumes a mix of environments that use an ASG and environments that don't.

With ASG Without ASG
Topology LB → ASG → EC2. Some instances are registered directly with the LB ALB → EC2
How AMIs are built From a dedicated golden instance. Creating an AMI automatically adds a new launch template (LT) version, and the ASG launches instances from the LT's $Latest With Packer, which installs PHP from apt at build time
  • Instances are managed with AWS Systems Manager (SSM) Run Command, so no SSH logins are needed
  • PHP comes from Ubuntu packages, and security updates normally arrive through apt

Why the usual approach is slow

As long as you wait for the distribution's fix, you don't control when you're done. With an ASG, AMIs add another layer of problems.

  • You don't know when fixed packages will land. This time the PPA (ondrej) published at 05:38 the next morning and Ubuntu published to Launchpad at 09:46, 6 to 11 hours after upstream. The packages reached the apt repositories even later
  • Patching running instances isn't enough. Instances launched by scale-out or replacement boot from the old AMI
  • You can't bake a fresh AMI yet. If Packer installs PHP from apt, you can't bake until the fix is in apt
  • Instances get replaced while you're out of the LB. If health checks or AZ rebalancing make the ASG replace instances, unpatched ones come up

Approach

Build the fix yourself instead of waiting, rebuild the AMI, and replace every instance in the ASG. We followed five rules.

  1. Prepare as soon as the release is pre-announced. Launch an EC2 build host and build the Ubuntu source once without patches. That warms ccache and gives you a test baseline (the tests that fail even without patches). Rehearse the distribution steps with this build too
  2. Take production out of the LB before the release. You only learn the details after publication, so take it out whether or not you turn out to be affected, and patch the same day
  3. Backport the upstream fixes onto the Ubuntu source and build debs. Append a suffix such as +local1 to the Ubuntu version. Ubuntu's next version sorts higher, so moving back to the official package later is easy
  4. Distribute via S3 and SSM. Depend on neither apt mirrors nor SSH
  5. Never patch ASG instances directly. Patch the golden instance, create an AMI, terminate every ASG instance, and let them come back on the new AMI. Running instances and future ones end up on the same AMI

Flow: build host → S3 → SSM → golden instance → AMI → launch template → replace all ASG instances

ASG instances get the fix in this order: golden instance → AMI → LT → replace all instances. Instances outside the ASG are patched directly.

Timeline

Identifying, backporting, and building the fix took 13 minutes, the AMI and LT 6 minutes, and replacing the ASG instances 11 minutes. These three account for most of the 38 minutes.

Timeline: back behind the LB 38 minutes after GHSA publication

Applying the packages to an instance takes only 1 to 2 minutes. Environments without an ASG were patched directly, without waiting for the AMI.

Key steps

Taking production out of the LB

Before taking it out, stop the ASG from replacing instances automatically. Otherwise unpatched instances boot from the old AMI while you're out of the LB.

aws autoscaling suspend-processes --auto-scaling-group-name <ASG> \ --scaling-processes ReplaceUnhealthy AZRebalance AlarmNotification aws autoscaling detach-load-balancers --auto-scaling-group-name <ASG> --load-balancer-names <CLB> aws elb deregister-instances-from-load-balancer --load-balancer-name <CLB> --instances <instance IDs outside the ASG> 
  • A CLB can't return a fixed response, so detach the instances to stop traffic. Closing the security group doesn't cut established connections
  • Once the instances are detached, users get 503. Turn on the app's maintenance mode before detaching, and after reattaching, verify behavior while still in maintenance before turning it off
  • With an ALB, switch the listener's default action to a fixed 503 response
  • Record which processes were already suspended before you start, so that you don't resume them when you put production back
  • If you use scheduled scaling (scheduled actions), also suspend ScheduledActions or make sure nothing is scheduled while you're out. A scale-out would boot unpatched instances from the old AMI

Backporting and building the fix

  1. Pick the fix commits from the diff between the previous and new tags of the oldest supported branch in php-src (for example git log php-8.2.35..php-8.2.36). Skip commits that only bump versions or touch CI or tests
  2. For each GHSA, decide whether your version is affected. If you run an end-of-life version, treat it as affected when the fix landed on every branch from 8.2 up. If the function or extension being fixed doesn't exist in your version, it's out of scope
  3. Apply the commits with git am -3 onto a tree with Ubuntu's patches applied, and resolve conflicts by hand. Export the result as a single quilt patch into debian/patches
  4. Add a +local1 version to debian/changelog and build. Bump to +local2 on every rebuild so you never ship different debs under the same version
  5. Run the test suite and confirm there are no new failures compared to the unpatched baseline. Run the phpt tests that came with each fix against both the official deb and yours, and confirm that only yours passes

With ccache warm, a build with the patch added finishes in 4 to 5 minutes. However, Ubuntu's PHP build puts the version into -fdebug-prefix-map, so changing the version misses the cache entirely. Exclude that option from the ccache hash with CCACHE_IGNOREOPTIONS.

Distributing to instances

Bundle the debs into a tarball, upload it to a temporary S3 bucket, and pass a presigned URL valid for one hour to the instances via SSM Run Command. Each instance downloads from the URL, verifies against SHA256SUMS, and then swaps the packages.

On each instance:

  • Replace only the PHP packages that are already installed, with dpkg -i. Older apt can't install a deb whose version contains + with apt-get install ./x.deb
  • apt-mark hold the replaced packages, so automatic updates or configuration management don't roll them back
  • During the swap, use policy-rc.d to keep maintainer scripts from reloading or restarting services, then restart the web server once at the end
  • Afterwards, check versions and holds, and make sure no process is still running the old PHP

Rebuilding the AMI and replacing ASG instances

After patching the golden instance, create an AMI and wait for the LT to point to it (6 minutes this time). Then terminate every ASG instance without decrementing the desired capacity.

aws autoscaling terminate-instance-in-auto-scaling-group --instance-id <ID> \ --no-should-decrement-desired-capacity 

The ASG launches replacements from the new AMI. Since production is out of the LB, terminating everything at once doesn't affect users. Replacement took 11 minutes this time. Before replacing, confirm that the ASG is out of the LB, that ReplaceUnhealthy is suspended, and that the LT has the new AMI.

Check versions, holds, and health checks on the new instances. Then attach the ASG to the LB, and resume the suspended processes once all instances are InService.

After the official fix ships

The self-built packages are a stopgap: switch back once the official fix ships. We set things up so that removing the hold is all it takes.

  1. Compare your fix with the official one. Run debdiff against the previous version and compare the new debian/patches with your patch. Look beyond the changelog at the patch contents and series
  2. If they match, unhold. The official version sorts above +local1, so apt upgrade or the next AMI rebuild picks it up naturally
  3. Unhold the golden instance too. AMIs baked from it carry the hold, so if you forget, PHP stays frozen in every later AMI
  4. Rebake Packer-built AMIs only after the fix reaches the apt repositories. Baking earlier installs the unpatched version. Add a check to the build script that the PHP version is at least the fixed one
dpkg --compare-versions "$(dpkg-query -W -f='${Version}' <PHP package name>)" ge <fixed version> 

Pitfalls and lessons

  • Publication comes before the php.net announcement. The php-src tags and GHSAs went public at 23:06 and the php.net announcement came at 23:12. The php.net Atom feed can lag, so watch the GitHub tags and GHSAs directly
  • apt mirrors go down. On the day, the Ubuntu mirrors didn't respond from 21:20 to 21:48. Distributing via S3 kept us unaffected
  • Long-running processes keep the old PHP. Restart CLI workers run by supervisor and the like after the swap. Look for processes still mapping deleted shared libraries (marked (deleted) in /proc/<pid>/maps)
  • Holds silently block updates. Neither apt upgrade nor configuration management reports an error; PHP just stays old. Always record unholding as a follow-up task
  • Fixes can include behavior changes. This time a fix changed the default of an ini setting. Read the diff before taking it in

Summary

Even on an LB → ASG → EC2 stack, patched production was back behind the LB within 40 minutes of publication. Three things made it work:

  • Don't wait for the fix: backport it onto the distribution source and build it yourself. Use a version suffix and a hold so you can go back to the official package
  • Prepare the build host, the test baseline, and the distribution steps between the announcement and the release
  • Don't patch ASG instances one by one. Bring them all in line at once through golden instance → AMI → LT → replace all instances, and stop the ASG's automatic replacement before taking production out of the LB