TL;DR
We woke up to find our entire system crashed after bots hammered our API with millions of requests overnight. Without rate limiting, proper monitoring, or DDoS protection, our servers buckled under the load. We lost 8 hours of uptime and learned expensive lessons about building resilient systems.
At 3:17 AM, my phone exploded with alerts. By the time I fumbled awake and checked the dashboard, everything was red. Our servers had been down for over four hours, and I didn't even know it until customer complaints started rolling in.
The Night Everything Went Down
The attack started at 11:23 PM. A botnet had discovered our API and started hammering it relentlessly. Our system, designed to handle maybe 100 requests per second on a busy day, was suddenly getting 50,000+ requests per second.
"Why didn't the alerts wake me up sooner? Because our monitoring service was hosted on the same server that went down. We were monitoring our system with our own system. Classic mistake."
The Attack Timeline
First bot requests hit our API. Traffic starts climbing rapidly.
Connection pool exhausted. Database starts rejecting new connections.
Server process crashes. Automatic restart fails due to database state.
External customer complains. I finally wake up.
All systems stable after blocking IPs and restarting services.
Why We Were Vulnerable
- No rate limiting on any API endpoints
- No bot detection or CAPTCHA challenges
- Monitoring hosted on same infrastructure being monitored
- No DDoS protection or CDN in front of our servers
- No auto-scaling or circuit breakers
What We Implemented Immediately
# nginx rate limiting
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
location /api/ {
limit_req zone=api burst=20 nodelay;
limit_req_status 429;
}
We set up external monitoring (UptimeRobot, Better Uptime) and put Cloudflare in front of everything. Their bot detection and DDoS protection became our first line of defense.
- Always use external monitoring services - never self-host your only alerting
- Rate limiting isn't optional - it's essential for every public API
- Put a CDN/DDoS protection service in front from day one
- Design for graceful degradation - better to serve 429s than crash
- Test your disaster recovery plan before you need it
How can I tell if I'm being targeted by bots?
Look for sudden traffic spikes, requests at unusual hours, high volume from single IP ranges, requests with suspicious user agents, or many requests to specific endpoints.
::
Is rate limiting enough to prevent bot attacks?
Rate limiting is essential but not sufficient alone. You'll want a layered approach: rate limiting, bot detection, DDoS protection services, and monitoring.
Should I use a CDN even for small projects?
Yes. Services like Cloudflare offer free tiers that provide basic DDoS protection and bot filtering. There's no reason not to use one.
::
Scan your vibe coded projects for missing rate limits and exposed endpoints.
Check Your Vibe Now