The $12,000 AWS Bill That Changed Everything

The Setup

I was building a side project, a simple image processing app. Nothing fancy. I needed AWS S3 for storage and EC2 for some compute tasks. Like many developers, I was working fast and cutting corners to ship quickly.

The credentials were stored in a config file. I had added it to .gitignore, or so I thought. One typo in the ignore pattern, and the file was being tracked.

Turns out, everyone. Or at least, every automated bot scanning for exposed AWS keys.

The Discovery

I was checking email on a Saturday morning when I saw the AWS Cost Alert. The message said my account had exceeded the billing threshold I'd set.

My heart rate doubled when I logged into the AWS Console.

My average monthly bill was around $15. This was a thousand times that.

What the Attackers Did

I immediately went to the EC2 dashboard. What I found was terrifying:

• 47 EC2 instances running in regions I'd never used
• All were p3.16xlarge instances (the most expensive GPU instances)
• They were running in us-east-1, eu-west-1, ap-northeast-1, and several other regions
• Total running time: approximately 18 hours

Each p3.16xlarge instance costs about $24.48 per hour. Multiply that by 47 instances running for 18 hours, and you get a very expensive weekend.

The Timeline

The attackers found my credentials within 8 minutes of the push. That's how fast these automated systems work.

The Response

Immediate Actions

1. Terminated all EC2 instances in every region
2. Revoked the compromised access keys in IAM
3. Enabled MFA on the root account (should have done this day one)
4. Changed my AWS root password
5. Reviewed CloudTrail logs for any other suspicious activity

Contacting AWS Support

I opened a support case explaining the situation. AWS has dealt with this before (many times), and they have a process:

1. They reviewed my CloudTrail logs to verify unauthorized access
2. They confirmed the activity matched known crypto-mining patterns
3. They issued a credit for the fraudulent charges

The process took about a week. I was sweating the entire time, not knowing if I'd be stuck with a $12,000 bill.

What I Did Wrong

Looking back, I made several critical mistakes:

1. No Billing Alerts

I had set a billing alert, but at $50. By the time it triggered, the charges were already in the thousands. I should have set multiple alerts at lower thresholds ($5, $10, $25).

2. Root Credentials in Code

I was using the root account's access keys. This gave attackers unlimited access to do anything in my account. I should have created an IAM user with limited permissions.

3. No MFA

Multi-factor authentication wasn't enabled. If it had been, the attackers couldn't have created new resources even with my credentials.

4. All Regions Enabled

AWS has regions I've never used and never will use. The attackers exploited this by launching instances in regions I wouldn't think to check.

The Security Overhaul

After this incident, I implemented these changes:

• IAM users only: Never use root credentials. Create IAM users with minimum necessary permissions.
• MFA everywhere: Required on root account and all IAM users.
• Service Control Policies: Restricted which services and regions can be used.
• AWS Organizations: Created separate accounts for different projects.
• CloudWatch Alarms: Alerts for unusual EC2 activity, not just billing.
• Pre-commit hooks: Scan for AWS credentials before any commit.
• AWS Secrets Manager: No more credentials in code, ever.

The Silver Lining

This was an expensive lesson ($12,000 scary, even if most was credited back). But it fundamentally changed how I think about cloud security.

Before this, security felt optional. Something I'd "do later" when the project got serious. Now I know that attackers don't wait for your project to be serious. They're scanning constantly, and they'll exploit any credential they find within minutes.