As you may have noticed, there was another outage on our Las Vegas router
early this morning lasting approximately two hours. While unfortunate and
inexcusable, this has allowed us to pinpoint the exact cause of the recent
networking problems with our Vegas deployment.
The prior filtering provider, CNServers, completely blocked all UDP on our
filtered IPs. When we switched Vegas over to Staminus for filtering (who
mitigates and cleans UDP, but doesn't block it), the sudden influx of excess
UDP is causing severe CPU spikes and seizing up the router. When we first
started seeing these spikes, we initially attributed it to 'growing pains'
from the recent increase in orders and new services - it wasn't until we
replaced Vyatta that we were able to see the actual cause.
At this time, we've had Staminus block all UDP in our Vegas deployment (this
only affects Filtered IPs). This will keep the network stable until we are
able to get new equipment in place that will be able to sufficiently 'tank'
the extra workloads.
We apologize for all of the turbulence in the network lately - as soon as we
have a replacement for the router picked out and a time/date set for the
swap, we will send another notification out. Thank you for your patience
while we work to get these issues resolved for good.
我笑了.