What happened
The client had created an IAM access key themselves and attached administrator permissions to it. The key was exposed, and the incident began at 11 at night, when nobody was at a desk to notice. Whoever found the key did not go after the client's data first: they went after compute, creating Amazon Bedrock resources to run AI models on the client's bill. Then they made sure that deleting the key would not lock them out, adding backdoors through Lambda and Git.
- A long-lived access key with full administrator rights, created outside any review
- Amazon Bedrock resources created by the attacker
- Persistence planted in Lambda and in Git, independent of the original key
Sweeping every region and service
The AWS console shows one region at a time, and attackers favour the regions a team never opens. We went through the account region by region and service by service, looking for anything running that the client's team could not account for. Everything was checked against what they had actually deployed. What they did not recognise was stopped and removed.
- Every region checked, not only the ones the client used
- Servers, Lambda functions, containers and AI model resources inventoried
- Attacker-created Bedrock resources and Lambda functions removed
- Changes planted through Git found and reversed
Removing every access key
Not only the leaked key. Once an administrator key has been exposed there is no reliable way to tell which other keys have been copied or newly minted, so trusting them one by one is how a backdoor survives. Every access key attached to a user was deleted. That closed the door the attacker used and removed the weakness behind it: a credential that never expires, works from anywhere and has no second factor.
Moving to multi-factor authentication
With the keys gone, people sign in as themselves, and every sign-in requires a second factor. A password or credential that turns up in the wrong place is no longer enough on its own.
Clearing out the roles
A role does not disappear when a key is deleted, which makes it the natural place to hide a way back in. We reviewed every role in the account: who can assume it, what it may do, when it was created and when it was last used.
- Suspicious roles created recently, which nobody on the client's team could explain, were removed
- Deprecated roles left over from old projects and suppliers were removed too, because each one is permission nobody is watching
Getting the charges waived
The attacker's usage had been billed to the client. We guided them through asking AWS to review it: registering for a paid support plan so the case had a proper channel, documenting what happened and when, and providing proof that the usage was unauthorised and that the account had since been secured. The thoroughness of the clean-up was part of that proof. AWS waived the charges.
- Paid AWS support plan registered so the case was handled by a support engineer
- A written account of the incident: timing, services, regions and attacker-created resources
- Evidence that the cause had been fixed: keys removed, MFA enforced, unexplained roles and resources gone
- Outcome: the unauthorised charges were waived
What we changed in our own practice
The clean-up made the account safe. Keeping it safe takes limits that apply even to administrators. Every AWS organisation we manage, our own included, now runs a baseline of single sign-on with short-lived credentials, organisation guardrails that block the services attackers reach for first, an audit trail that cannot be switched off, and spend alerts that reach a person. The full account is in our insights.
- AWS Organizations
- IAM Identity Center
- Service control policies
- CloudTrail
- AWS Budgets
- Cost Anomaly Detection