The 2 a.m. problem

A production incident hits at 2 a.m. Your on-call engineer needs roles/owner — or at least broad project admin — to restart a stuck Cloud Run service, read a Secret Manager secret, or roll back a bad deploy. In most small teams, the answer is a standing grant: someone has had roles/editor or roles/owner since the project was created, and nobody has ever removed it.

That standing grant is the whole problem. It is an always-on credential that an attacker, a compromised laptop, or a leaked service-account key can use at any hour, for any purpose, with no incident attached to it. The fix is not "give nobody admin access." The fix is access that exists only when it is needed, only for as long as it is needed, and only for the person who asked for it. On GCP, that is IAM Conditions — and it is one of the most underused security features on the platform.

This article walks through a break-glass pattern I would actually run: a permanent, near-zero-privilege identity; a just-in-time elevation path with a time-bound condition; an audit trail that shows exactly who elevated, when, and why; and a test that proves the grant really expires. Everything here uses standard IAM — no third-party PAM tool, no extra bill.

What IAM Conditions actually are

IAM Conditions attach a Common Expression Language (CEL) expression to a role binding. The binding only applies when the expression evaluates to true. The most useful variables are:

  • request.time — the time of the request, so you can write request.time < timestamp("2026-11-11T18:00:00Z").
  • request.auth.access_levels — Access Context Manager access levels, for device or network conditions.
  • resource.name, resource.type, resource.service — scope the binding to specific resources.
  • request.auth.principal — the identity making the call.

The condition is evaluated at request time, on every request. When the time window passes, the binding stops applying — no cleanup job, no ticket to revoke the role, no forgotten grant. That property — expiry enforced by the platform, not by human memory — is the entire point.

There are limits worth knowing up front. Conditions are supported on most IAM roles, but not on every Google Cloud service's data-level permissions, and some legacy roles behave oddly with conditions attached. Conditions also add evaluation latency that is negligible for human-speed operations and irrelevant for break-glass use. And a condition is not a substitute for MFA or for phishing-resistant authentication — it narrows the window, it does not authenticate the person. Treat it as one layer.

The design: three identities, not one

The pattern has three parts:

  1. The everyday identity. Each engineer has their normal account with the least privilege they need daily: read logs, view Cloud Run services, read dashboards. Nothing that can change production state.
  2. The break-glass group. A Google group, e.g. break-glass@your-domain.com, with almost no members most of the time. Membership is the trigger, not the privilege itself.
  3. The time-bound binding. A role binding on the project (or folder) that grants the elevated role to the break-glass group, but only while a time condition holds.

The key decision: the elevated role should be the narrowest role that can fix the incident, not roles/owner. For a Cloud Run + Cloud SQL + Secret Manager stack, that is usually a combination like roles/run.admin, roles/cloudsql.client, and roles/secretmanager.secretAccessor — not owner, not editor. Owner can change IAM itself, which means a compromised break-glass session can make itself permanent. If your incident genuinely needs owner-level action, that is a signal to redesign the runbook, not to hand out owner.

Step 1: Create the time-bound binding

Here is the binding as a Terraform resource, because break-glass access should be in version control, reviewed like code, and never clicked into existence at 2 a.m.:

variable "break_glass_group" { type = string default = "group:break-glass@your-domain.com" } variable "elevation_end" { type = string description = "RFC 3339 timestamp when the elevation expires, e.g. 2026-11-11T18:00:00Z" } resource "google_project_iam_member" "break_glass_run_admin" { project = var.project_id role = "roles/run.admin" member = var.break_glass_group condition { title = "break-glass window" description = "Time-bound elevation for incident response only" expression = "request.time < timestamp(\"${var.elevation_end}\")" } } resource "google_project_iam_member" "break_glass_secret_accessor" { project = var.project_id role = "roles/secretmanager.secretAccessor" member = var.break_glass_group condition { title = "break-glass window" description = "Time-bound elevation for incident response only" expression = "request.time < timestamp(\"${var.elevation_end}\")" } }

Two details matter. First, each role gets its own binding with the same condition, so you can audit and revoke them independently. Second, the expiry timestamp is a variable, not hardcoded — the same module grants a one-hour window or a four-hour window depending on the incident.

If you prefer gcloud, the equivalent is:

gcloud projects add-iam-policy-binding "$PROJECT_ID" \ --member="group:break-glass@your-domain.com" \ --role="roles/run.admin" \ --condition="title=break-glass window,description=Time-bound elevation for incident response only,expression=request.time < timestamp('2026-11-11T18:00:00Z')"

One trap: add-iam-policy-binding with a condition will fail if a binding for the same role and member already exists without a condition. Check the current policy first with gcloud projects get-iam-policy, and remove the standing grant before adding the conditional one — otherwise you have built an expiry that never applies because the unconditional binding still exists.

Step 2: Make the elevation requestable, not self-service

A time-bound grant that anyone can create for themselves is just a standing grant with extra steps. The elevation flow should be:

  1. The engineer opens an incident channel or ticket and states what they need and why. Even a one-line message counts — the point is a record, not bureaucracy.
  2. A second person — a team lead, or for a solo builder, a documented self-approval step with a reason field — approves and sets the expiry. For a one-person team, the approval is you writing down the reason and the window before you run the Terraform apply. That sounds performative; it is the difference between "I elevated because production was down" and "I cannot explain why I had admin access for six hours."
  3. The Terraform apply (or gcloud command) runs, adding the member to the group or applying the conditional binding with the expiry.
  4. A calendar reminder or a Cloud Scheduler job fires at expiry to confirm the access is gone — not to revoke it (the condition does that), but to verify it.

For teams with more than one person, the approval step should be a different human from the requester. For a solo builder, the honest version is: the elevation command lives in a script that requires a --reason flag and a --duration flag, and writes both to a log file. Here is the shape of that script:

#!/bin/bash # elevate.sh --reason "INC-123: Cloud Run deploy stuck" --duration 2h set -euo pipefail # Parse flags, compute expiry in UTC, run terraform apply with # -var="elevation_end=$EXPIRY", and append a line to break-glass.log: # timestamp, requester, reason, expiry

The script is not the security control. The condition is. The script is the audit habit that makes the condition meaningful.

Step 3: Prove the expiry works

A control you have never tested is a hope. Test the expiry in a non-production project first:

  1. Grant a conditional binding with a five-minute window.
  2. Confirm the elevated action works inside the window — for example, gcloud run services describe succeeds where it previously failed, or a deploy goes through.
  3. Wait six minutes. Repeat the same command. It must fail with a permission error.
  4. Check Cloud Audit Logs for both the successful call and the denied call. You should see the allow, then the deny, with the same identity.

That deny entry is the proof. If the action still works after expiry, you have a second, unconditional binding somewhere — usually a group membership or a folder-level grant you forgot about. IAM is additive: a single unconditional grant anywhere in the hierarchy defeats every condition you wrote. Audit the folder and organization policies too, not just the project.

A second test worth running quarterly: list every binding in the project that has no condition and grants a privileged role, and justify each one out loud.

gcloud projects get-iam-policy "$PROJECT_ID" \ --format="json" | python3 -c " import json, sys policy = json.load(sys.stdin) for b in policy.get('bindings', []): if 'condition' not in b and b['role'] in ( 'roles/owner', 'roles/editor', 'roles/iam.securityAdmin', 'roles/run.admin'): print(b['role'], '->', b['members']) "

If that list is longer than you expect, that is the finding. Fix it before the next incident, not during it.

Step 4: Audit everything, and read the audit

Break-glass access without audit logging is a backdoor with good branding. Three things to enable and actually read:

  • Data Access audit logs for IAM and Secret Manager in the project. Admin Activity logs are on by default; Data Access logs for iam.googleapis.com and secretmanager.googleapis.com need to be enabled explicitly, and they are the ones that show who read which secret and who changed which binding.
  • A log-based alert for any SetIamPolicy call on the project. If someone changes IAM outside the elevation flow, you want to know within minutes, not at the next quarterly review. A simple log-based metric on protoPayload.methodName="SetIamPolicy" with an alerting policy is enough.
  • A weekly review of elevation events. How many elevations happened, who requested them, how long each lasted, and whether any lasted longer than the incident. A pattern of frequent, long elevations is a signal that everyday permissions are too narrow — fix the everyday role instead of normalizing break-glass.

One more honest limitation: audit logs tell you what happened after the fact. They do not stop a malicious elevation in real time. If your threat model includes a compromised admin account, pair time-bound access with a second control — for example, an Access Context Manager access level requiring a managed device. Time-bound access shrinks the window; it does not close every path through it.

What this looks like in practice

The trade-off is real: the first time you use this in a genuine incident, the elevation step feels like friction when you most want speed. That friction is the point, and it is smaller than it looks — a scripted elevation takes under a minute, and the alternative is an always-on privileged credential waiting for the worst day of the year. Pre-write the runbook, pre-test the expiry, and the 2 a.m. version of you will be glad the 2 p.m. version of you did the work.

Takeaways

  • Standing privileged grants are the risk; time-bound conditional bindings are the fix.
  • Grant the narrowest role that can resolve the incident — never roles/owner for break-glass.
  • Put the binding in Terraform, with the expiry as a variable, and require a written reason for every elevation.
  • Test the expiry in a non-production project: allow inside the window, deny outside it, and confirm both in audit logs.
  • Audit for unconditional privileged bindings regularly — one forgotten grant defeats every condition.
  • Review elevation history weekly. Frequent break-glass use means everyday permissions need fixing.