Monday 13th November 2017

Routing Services Reduced availability of EU routing infrastructure

We are investigating issues with our EU routing infrastructure

Update: After investigation we found that a number of hosts were continuously connecting to our MQTT server (100s per second), causing high CPU and memory on that server. As a result, the other services running on the same VM were impacted, and traffic throughput was severely reduced. We mitigated the issue by blocking a number of IP addresses from connecting to the server.

A long-term solution is to replace our MQTT server software with an implementation that will be better at dealing with such flooding, and can alert the ops team if this happens again.