TL;DR
An ed-tech platform woke up to find its entire system crashed after bots hammered the API with millions of requests overnight. Without rate limiting, proper monitoring, or DDoS protection, the servers buckled under the load. The company lost 8 hours of uptime and learned expensive lessons about building resilient systems.
At 3:17 AM, the lead engineer's phone exploded with alerts. By the time he fumbled awake and checked the dashboard, everything was red. The servers had been down for over four hours, and nobody on the team knew it until student complaints started rolling in.
The Night Everything Went Down
The attack started at 11:23 PM. A botnet had discovered the platform's API and started hammering it relentlessly. The system, designed to handle maybe 100 requests per second on a busy day, was suddenly getting 50,000+ requests per second.
"Why didn't the alerts wake anyone up sooner? Because the monitoring service was hosted on the same server that went down. The team was monitoring their system with their own system. Classic mistake."
The Attack Timeline
First bot requests hit the API. Traffic starts climbing rapidly.
Connection pool exhausted. Database starts rejecting new connections.
Server process crashes. Automatic restart fails due to database state.
External student complains via email. The lead engineer finally wakes up.
All systems stable after blocking IPs and restarting services.
- No rate limiting on any API endpoints
- No bot detection or CAPTCHA challenges
- Monitoring hosted on same infrastructure being monitored
- No DDoS protection or CDN in front of the servers
- No auto-scaling or circuit breakers
What the Team Implemented Immediately
# nginx rate limiting
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
location /api/ {
limit_req zone=api burst=20 nodelay;
limit_req_status 429;
}
The team set up external monitoring (UptimeRobot, Better Uptime) and put Cloudflare in front of everything. Cloudflare's bot detection and DDoS protection became the first line of defense.
- Always use external monitoring services - never self-host your only alerting
- Rate limiting isn't optional - it's essential for every public API
- Put a CDN/DDoS protection service in front from day one
- Design for graceful degradation - better to serve 429s than crash
- Test your disaster recovery plan before you need it
How can I tell if I'm being targeted by bots?
Look for sudden traffic spikes, requests at unusual hours, high volume from single IP ranges, requests with suspicious user agents, or many requests to specific endpoints.
Is rate limiting enough to prevent bot attacks?
Rate limiting is essential but not sufficient alone. You'll want a layered approach: rate limiting, bot detection, DDoS protection services, and monitoring.
Should I use a CDN even for small projects?
Yes. Services like Cloudflare offer free tiers that provide basic DDoS protection and bot filtering. There's no reason not to use one.
Scan your vibe coded projects for missing rate limits and exposed endpoints.