Case study · Cloud security · Incident response

An admin access key leaked. The attacker went for AI compute, then built ways back in.

A client's self-configured AWS access key with administrator rights was exposed. Attackers used it to create Amazon Bedrock resources and to plant persistence through Lambda and Git. We swept every region and service, removed every user access key, moved everyone to multi-factor sign-in and cleared out stale and attacker-created roles. Then we helped the client make its case to AWS, and the unauthorised charges were waived.

Client
A client company (name withheld)
Country
Malaysia
Services
Cloud security and incident responseCloud architecture and DevOps
cloudtrail / event-history
Illustrative timeline. Identifiers and times are not from the incident.
0
access keys left on any user after the clean-up
100%
of regions and services swept for attacker resources
MFA
now required every time a person signs in
Waived
the unauthorised charges, after a documented case to AWS

What happened

The client had created an IAM access key themselves and attached administrator permissions to it. The key was exposed, and the incident began at 11 at night, when nobody was at a desk to notice. Whoever found the key did not go after the client's data first: they went after compute, creating Amazon Bedrock resources to run AI models on the client's bill. Then they made sure that deleting the key would not lock them out, adding backdoors through Lambda and Git.

  • A long-lived access key with full administrator rights, created outside any review
  • Amazon Bedrock resources created by the attacker
  • Persistence planted in Lambda and in Git, independent of the original key

Sweeping every region and service

The AWS console shows one region at a time, and attackers favour the regions a team never opens. We went through the account region by region and service by service, looking for anything running that the client's team could not account for. Everything was checked against what they had actually deployed. What they did not recognise was stopped and removed.

  • Every region checked, not only the ones the client used
  • Servers, Lambda functions, containers and AI model resources inventoried
  • Attacker-created Bedrock resources and Lambda functions removed
  • Changes planted through Git found and reversed

Removing every access key

Not only the leaked key. Once an administrator key has been exposed there is no reliable way to tell which other keys have been copied or newly minted, so trusting them one by one is how a backdoor survives. Every access key attached to a user was deleted. That closed the door the attacker used and removed the weakness behind it: a credential that never expires, works from anywhere and has no second factor.

Moving to multi-factor authentication

With the keys gone, people sign in as themselves, and every sign-in requires a second factor. A password or credential that turns up in the wrong place is no longer enough on its own.

Clearing out the roles

A role does not disappear when a key is deleted, which makes it the natural place to hide a way back in. We reviewed every role in the account: who can assume it, what it may do, when it was created and when it was last used.

  • Suspicious roles created recently, which nobody on the client's team could explain, were removed
  • Deprecated roles left over from old projects and suppliers were removed too, because each one is permission nobody is watching

Getting the charges waived

The attacker's usage had been billed to the client. We guided them through asking AWS to review it: registering for a paid support plan so the case had a proper channel, documenting what happened and when, and providing proof that the usage was unauthorised and that the account had since been secured. The thoroughness of the clean-up was part of that proof. AWS waived the charges.

  • Paid AWS support plan registered so the case was handled by a support engineer
  • A written account of the incident: timing, services, regions and attacker-created resources
  • Evidence that the cause had been fixed: keys removed, MFA enforced, unexplained roles and resources gone
  • Outcome: the unauthorised charges were waived

What we changed in our own practice

The clean-up made the account safe. Keeping it safe takes limits that apply even to administrators. Every AWS organisation we manage, our own included, now runs a baseline of single sign-on with short-lived credentials, organisation guardrails that block the services attackers reach for first, an audit trail that cannot be switched off, and spend alerts that reach a person. The full account is in our insights.

  • AWS Organizations
  • IAM Identity Center
  • Service control policies
  • CloudTrail
  • AWS Budgets
  • Cost Anomaly Detection

More work

All work
clausework / mortgage-debtors
Legal · Banking litigation

Clausework

From a bank's spreadsheet to court-ready documents, with every step on the record.

Kek Ooi & Lee Hong
iqc / quality-control
Manufacturing · Water reticulation

IQC

Every product traced from the factory floor to the purchaser, with a certificate anyone can verify.

Soon Chuan Hong Hardware Sdn. Bhd.
question-bank / review-jobs
Education · Assessment content

Question Bank AI

Textbooks in, reviewed exam questions and learning cards out, in English and Bahasa Melayu.

Tavis

Start a conversation

Facing something similar?

Tell us what is slowing the business down. You will talk to an engineer, not a salesperson, and get a straight answer on whether software is the right fix.