The $12,000 AWS Bill That Changed Everything

The Setup

I was building a side project: a simple image processing app. S3 for storage, EC2 for compute, nothing fancy. I was moving fast.

The credentials were stored in a config file. I'd added it to .gitignore, or so I thought. One typo in the ignore pattern and the file was being tracked. That's it. That's the whole setup error.

Turns out, everyone. Or at least, every automated bot scanning for exposed AWS keys.

The Discovery

I was checking email on a Saturday morning when I saw the AWS Cost Alert. The message said my account had exceeded the billing threshold I'd set.

My heart rate doubled when I logged into the AWS Console.

My average monthly bill was around $15. This was a thousand times that.

What the Attackers Did

I immediately went to the EC2 dashboard. What I found was terrifying:

• 47 EC2 instances running in regions I'd never used
• All were p3.16xlarge instances (the most expensive GPU instances)
• They were running in us-east-1, eu-west-1, ap-northeast-1, and several other regions
• Total running time: approximately 18 hours

Each p3.16xlarge instance costs about $24.48 per hour. Multiply that by 47 instances running for 18 hours, and you get a very expensive weekend.

The Timeline

The attackers found my credentials within 8 minutes of the push. That's how fast these automated systems work.

The Response

Immediate Actions

1. Terminated all EC2 instances in every region
2. Revoked the compromised access keys in IAM
3. Enabled MFA on the root account (should have done this day one)
4. Changed my AWS root password
5. Reviewed CloudTrail logs for any other suspicious activity

Contacting AWS Support

I opened a support case explaining what happened. AWS has a documented process for credential theft: they pulled my CloudTrail logs, confirmed the activity matched known crypto-mining patterns, and issued a credit. Straightforward in retrospect.

The process took about a week. I was sweating the entire time.

What I Did Wrong

Looking back, I made several critical mistakes:

1. No Billing Alerts

I had set a billing alert, but at $50. By the time it triggered, the charges were already in the thousands. I should have set multiple alerts at lower thresholds ($5, $10, $25).

2. Root Credentials in Code

I was using the root account's access keys. This gave attackers unlimited access to do anything in my account. I should have created an IAM user with limited permissions.

3. No MFA

Multi-factor authentication wasn't enabled. If it had been, the attackers couldn't have created new resources even with my credentials.

4. All Regions Enabled

AWS has regions I've never used and never will use. The attackers exploited this by launching instances in regions I wouldn't think to check.

The Security Overhaul

After this incident, I implemented these changes:

• IAM users only: Never use root credentials. Create IAM users with minimum necessary permissions.
• MFA everywhere: Required on root account and all IAM users.
• Service Control Policies: Restricted which services and regions can be used.
• AWS Organizations: Created separate accounts for different projects.
• CloudWatch Alarms: Alerts for unusual EC2 activity, not just billing.
• Pre-commit hooks: Scan for AWS credentials before any commit.
• AWS Secrets Manager: No more credentials in code, ever.

The Silver Lining

Most of that $12,000 came back, but the week of waiting taught me something that a blog post probably wouldn't have.

Attackers don't care how small your project is. They run automated bots that scanned my repo within 8 minutes of the push and launched 47 GPU instances before I even finished dinner. No target-picking involved. The credential was just fuel, and any credential works.