Outage on TTN V2 Cluster

Incident Report for The Things Network

Resolved

This incident has been resolved.

Posted May 04, 2021 - 10:22 CEST

Update

Our metrics show that traffic is flowing again. We have re-enabled the MQTT API and are monitoring the system.

Posted May 04, 2021 - 10:01 CEST

Monitoring

We discovered that the router is the failing component at the beginning of the investigation. We recovered the router quite easily, but the handler crashed (similar to a previous outage) and took a while to recover.
We have temporarily disabled MQTT ingress and are monitoring the situation.

Posted May 04, 2021 - 09:28 CEST

Investigating

We are investigating an outage in the TTN V2 Cluster

Posted May 04, 2021 - 08:41 CEST