During the week of April 2nd, ZEIT became the target of frequent Denial-of-Service attacks targeting core infrastructure.
The affected datacenter was sfo1. We did not experience any downtime in our newly announced Brussels (bru1) datacenter. Deployments that were scaled to that region were therefore served uninterrupted.
First, we wanted to write to you to acknowledge the situation and describe what happened. This is what we know so far:
  • The attacks were of varying length. From as little as 1 minute to as long as 20.
  • They consisted of a massive range of unique IPs.
  • The load on our load balancers ranged from 20x-100x the regular amounts.
  • The pattern of the connections, origin of the attacking IPs and content of the requests did not remotely resemble any legitimate usage of our platform, which makes us certain the primary goal was the disruption of our service.
Second, we wanted to share the steps we have taken to answer to situations like this in the future:
  • We have dedicated full time resources within the company to the monitoring and mitigation of this class of attack.
  • We have strengthened, scaled and upgraded our load balancers.
  • We have deployed specific solutions to combat this and other kinds of attacks. We will preserve the specific details of these defenses confidential for the time being.
  • We have simulated and re-played the pattern of attacks we received, to test and verify these mitigations.
The most important takeaway is that every single one of our customers is now better protected. The attacks had a specific target, but the mitigations we have deployed benefit everyone.
We will continue to have a watchful and alert attitude, as new and diversified attacks are always possible. We are confident in our ability to overcome these challenges and make the service as a whole better for everyone.