Tuesday 20th February 2018

Routing Services Major outage of EU Gateway Bridges

We're investigating a major outage of our EU gateway bridges.

Update The problems appear to have been caused by a couple of UDP gateways (possibly not real gateways) that are flooding our UDP endpoints. It looks like Docker's ingress routing couldn't keep up with this particular form of traffic, which also affected MQTT ingress routing. UDP has temporarily been disabled, MQTT routing is recovering.

Update We are slowly starting to accept UDP traffic again, which is now being handled outside Docker. As we can no longer benefit from Docker's load balancing, we expect degraded performance of UDP gateways until we come up with a more permanent fix.

Update The deployment of our UDP bridge is stable. We'll probably push the permanent fix to v3.