Leaked AI API keys after a cloud platform incident: blast radius, rotation, and bill forensics

Leaked AI API keys after a cloud platform incident: blast radius, rotation, and bill forensics
A cloud platform security notice about AI API keys does not interrupt anything visible. Containers keep serving, model calls keep returning results, and the meter keeps running. The order of operations after such a notice matters more than the speed of any single step.

Revoke the old credentials in each model provider's console first. Then map where the key was deployed, which instances are still running with it, and what model access it grants. Rotate to new keys, and reconcile the platform's audit logs against provider bills for charges nobody on the team can explain.

Applications may keep responding normally and model APIs may return no errors. That is not evidence the credentials are safe. External model services bill by usage, so a key carried off the platform and used somewhere else still lands on your account.

The Northflank incident, in the order the notices were published

What Northflank runs, and why model keys end up there

Northflank is a Kubernetes-based platform as a service. Teams build and deploy microservices, managed databases, scheduled background jobs, and GPU workloads directly from a Git repository. It also offers a bring-your-own-cloud mode that connects the control plane to an AWS, Google Cloud, or Azure account, which removes the work of hand-writing Kubernetes configuration files.

Many teams run AI agents, backend API services, sandboxes, and automated data-processing scripts on that kind of platform. To reach model APIs such as OpenAI and Anthropic, those containers need credentials, and the usual storage spot is workload environment variables, Secret Groups, or global secrets. A large share of a company's model credentials sits in one place, so a platform-side security problem puts them at risk of exposure and abuse at the same time.

Three updates, three different asks

On October 31, 2026, Northflank said on its official status page that users had reported unauthorized use of AI API keys hosted on the platform, and that audit logs were available to all customers, including those on the free tier. The detailed sequence came in three steps.

Later updates clarified two things. Rotation should cover credentials in workloads, Secret Groups, and global secrets, and any add-on with public access enabled should have it turned off.

What the notices do not establish

Two misreadings are common. Asking users to act as if their environment variables had leaked is a baseline precaution in incident response, meant to contain risk quickly. It does not confirm that every user's variables were stolen. Reports from some users that their keys were abused do not mean every customer on the platform saw unauthorized usage either.

The Hacker News discussion drew heavy attention and produced useful advice on checking credentials, along with speculation about the attack method and the platform's architecture. Community discussion is not evidence of how the breach happened. The official notice mentions similar attack activity, but the attacker's identity and the full intrusion path cannot be pinned down until the final investigation report is published.

For a development team, the urgent task is not reconstructing the intrusion. It is establishing where credentials are stored, which external services they reach, and whether the old ones are actually dead.

Mapping the blast radius of a leaked AI API key

The real blast radius depends on how many places a key is deployed today, not on what it was originally created for. A key meant for local testing gets copied into production. A demo project is retired, but its scheduled jobs keep running. Several agents share one key under the same provider account to skip setup. Update only the core production service and the peripheral environments stay exposed.

1. Inventory platform configuration, including the parts nobody maintains

Start with workload environment variables, Secret Groups, and global secrets, as the official guidance says. Then work through CI/CD build scripts, automated deployment pipelines, preview environments, and third-party webhook integrations to check whether any of them hardcodes or caches the same key.

2. Match the configuration to instances that are actually running

One environment variable can be inherited by several microservices, or copied by hand into different projects. Updating a frontend web service does not refresh a background queue worker or a cron job. Old containers that nobody owns but that still run belong on the list too.

3. Keep the tracking sheet redacted

Record each key's name, the key prefix or ID the provider shows, the project it belongs to, where it is stored, which services use it, and its current status. Never paste a full key in plain text into the sheet, team chat, or a ticket. That creates a fresh leak inside your own collaboration tools.

4. Check what the key can do at the provider

Once the usage map is clear, log in to each provider console and confirm which organization and project the key belongs to. Look at which model endpoints it can call, and whether it can read or write fine-tuning data or upload files. Permission granularity varies widely between providers. An API key is not necessarily limited to text generation, and it is not necessarily an account-wide admin credential either.

5. Clean up the other credentials in the same environment

Do not stop at model APIs. If the affected runtime also holds database connection strings, object storage keys such as AWS S3 or Google Cloud Storage, or payment API tokens, rotate those as well, following the official security guidance.

Rotating keys so the old ones actually stop working

The most common mistake in a rotation is updating application configuration and forgetting to revoke the old credential with the issuer.

Deleting an environment variable from a cloud platform removes a string from that platform's database. If an attacker already copied the key to an outside server, whether it still works depends entirely on the provider's revocation state. Deleting a local .env file or scrubbing old code from Git history does the same amount of nothing against a key that has already been exfiltrated.

A rotation that closes the loop has four moves.

Revoke at the issuer

Log in to the provider's official console, find the key you suspect, revoke or delete it, and confirm its status shows it is no longer active. If suspicious calls are arriving in real time, cut the old credential immediately to cap the charges, while weighing the impact of a short service interruption and preparing a recovery plan.

Generate the replacement from a trusted device

Confirm the machine you are working on is secure and reach the console through the provider's official domain. Check the account's team member list and role permissions for unfamiliar changes. Generate the new key under the principle of least privilege. If the cloud platform has not finished isolating, remediating, or migrating, hold the new production key back from the high-risk environment to avoid a second exposure.

Redeploy, because running processes keep the old value

Many backend services and long-lived processes read environment variables once, at container start. Saving a new value in a web console does not touch the process already in memory, which keeps sending requests with the old key. Restart or redeploy before assuming the change is live.

Verify in both directions

At the end of the rotation, confirm two facts. The old key is rejected by the provider. The services make successful model calls with the new key. Do not print the new key into application logs, and do not paste a curl command containing the full key onto a shared server where shell history captures it.

Keep watching for a while afterwards. Occasional 401 Unauthorized errors in the logs point to an instance or scheduled job that never picked up the new configuration. Unexplained usage on the bill sends you back to the account for backup keys you have not dealt with yet.

Reconciling platform audit logs against provider bills

Audit logs open to all customers give a starting point for reviewing activity inside the platform. During reconciliation, platform logs and provider logs verify different things, so they have to be combined and cross-checked.

A key exfiltrated and used from a machine outside the platform leaves no trace in the original application's service logs. An audit log with nothing unusual does not rule out abuse. Before relying on it, confirm how far back the log retention goes and which fields are kept. From a forensics standpoint, an event missing from the logs is not the same as an event that never happened.

Step 1. Put every log on one time zone

Convert platform audit logs, application-internal logs, and provider billing timestamps to a single time zone such as UTC, and mark the window where unusual charges started climbing.

Step 2. Compare workload history with call patterns

When the bill spikes, check first whether the team shipped a release, ran a batch data-processing job, or changed a scheduled script in the same period. Then check whether the models billed in the provider console match the ones the code calls. A service that normally uses lightweight models and suddenly bursts on expensive flagship models at 3 a.m., or a provider that keeps billing after the application containers were shut down, points to someone outside using the key.

Step 3. Rule out your own code

A jump in charges is not proof of theft on its own. An agent stuck in an infinite loop, aggressive retries after API timeouts, misconfigured concurrency, or a batch document-parsing job without rate limiting can burn a quota in minutes. Overlay job execution records on the provider's token consumption chart so an internal bug is not mistaken for an external breach.

Step 4. Break the bill down by key and preserve evidence

If the provider lets you filter usage by individual API key or project, filter down to the affected credentials. If the console only gives an account-level summary, narrow it by elimination using release times and application logs. After confirming unauthorized usage, capture the abnormal window, Project ID, Request ID, models called, token usage, and the exact time the old key was revoked. A structured timeline moves a ticket forward far more effectively than a complaint about a drained balance. Redact business data from screenshots before submitting anything, and never attach old or new keys in plain text.

Watch for delayed charges after the rotation

Keep reviewing bills for the next two or three days. Some provider dashboards lag behind final billing by several hours or a day or more. For later charges, check whether the underlying requests landed before or after revocation. Whether a leak-related charge can be credited depends on each provider's risk policies and how they assess the incident, so budget for absorbing what has already happened.

Once the immediate response is closed, one engineering question remains. Why did a single leaked key reach so many services?

Early in a project, pointing local development, integration testing, demos, and production at one API key saves configuration work. It also leaves the team exposed. Revoke that shared key and production goes down together with testing. Keep it alive to protect production and the suspicious traffic keeps draining the balance. With every environment billed to one line, tracing the source afterwards becomes nearly impossible.

Reducing the blast radius of the next incident

Split credentials by service boundary and environment

Keep development, testing, and production credentials strictly separate. Each microservice and each standalone agent gets its own project and key. Where the provider supports scoped keys, enable only the endpoints the service uses and turn off fine-tuning, file upload, or organization management permissions it does not need. Assign an owner to each key. Revoke idle keys when temporary projects retire, remove console access when people leave or change roles, and shut down old containers after a migration so stale credentials stop accumulating in code and configuration.

Virtual keys for local projects and CLI tools

After a platform incident, many developers move debugging back to their own machines or set up several AI coding tools and agents on one laptop. Running code locally does not make credentials safe. Real keys written in plain text into several .env files or into different CLI configs get committed when a .gitignore entry is missing, or picked up by debug logs and cloud backups.

ServBay, an AI development management tool, handles this with virtual keys in its AI Gateway. The real API keys for each provider stay encrypted inside the gateway, and the team issues separate virtual keys for each project or tool, such as Claude Code, Codex, or an in-house script. A switcher tool decides which provider you point at. A gateway decides how the traffic moves and which credential each project holds.

Three things follow from that split.

• Real keys stop spreading. Project configs and third-party tools hold only virtual keys that point at the gateway endpoint, so provider keys never appear in project directories or terminal environment variables.
• Access and rate limits get granular. Each virtual key can be restricted to specific model channels, given an expiration time, and capped with per-minute or per-day request and token limits (RPM/TPM/RPD/TPD).
• Rotation happens independently. A test project whose config may have leaked can have its virtual key revoked or reset in the gateway, without touching the real credentials at the upstream provider. Per-project usage and spending are visible on the gateway side.

A local gateway still needs its environment checked. Run the full workflow with a disposable test credential you are willing to throw away, then search local log files, build output directories, and test deployment configs for spots that print request headers in plain text. Production keys stay out of that exercise.

Know where provider spending controls stop

A budget line in a provider console is not blanket coverage. Spending controls trigger in different ways and follow different billing rules, and knowing which one a team actually relies on belongs in the post-incident review. Unnecessary auto-recharge is worth turning off at the same time.

A short checklist

Four steps that close the loop

• Cut access first. Once a credential is confirmed or strongly suspected exposed, revoke it in the provider's official console. Deleting a cloud environment variable or a local config file is not revocation.
• Inventory every location. Follow the key through environment variables, Secret Groups, CI/CD pipelines, scheduled jobs, and leftover test containers. Generate the replacement from a trusted device, then restart every running instance so the change takes effect.
• Cross-check the bills. Put platform audit logs, application job records, and the provider console on one time zone, separate internal loops from external abuse, export request details for the abnormal window, and keep checking for delayed charges.
• Isolate environments and real credentials. Separate development, testing, and production keys, and turn off unnecessary auto-recharge. For local work, the virtual key feature in ServBay AI Gateway keeps real keys inside the gateway, with per-project access control, rate limiting, and separate usage tracking.

The root cause of a platform breach can take weeks to establish. Credential cleanup and bill reconciliation cannot wait for that report.