The Things Indoor Gateway Pro connectivity issues in The Things Stack Sandbox EU1 cluster

Incident Report for The Things Network

Postmortem

Summary

On 9 Jul 2026 , it was reported that some The Things Indoor Gateway Pro (TTIG Pro) gateways lost connection to the LNS for some users in EU1 region. This was triggered by certificate renewal issue. Once the remediation was in place, the affected gateways started reconnecting.

Impact

  • Some gateways got disconnected for some users in EU1 region.
  • The number of connected gateways kept declining gradually.
  • Eventually, once the fix was in place, the affected gateways have reconnected and service has recovered without intervention.

Root Cause

The certificate serving port 8889 expired on Jul 9 09:21:27 UTC. This port is used exclusively by TTIG Pro gateways for their WebSocket connection to the LNS.

Certbot had already renewed the certificate before expiry and stored the new YR-signed certificate in the proxy_tls secret in AWS Secrets Manager. However, the secret was not synced by the GatewayProxy (Envoy) service. As a result, Envoy continued to serve the expired R12 certificate from memory, causing all TTIG Pro TLS handshakes to fail with certificate verification error.

Resolution

The fix was applied by restarting the GatewayProxy ECS task on each affected cluster, which forced Envoy to re-read the proxy_tls secret from Secrets Manager on startup and load the already-renewed YR certificate.

Prevention / Long-term improvements

Investigate why the renewed certificate was not synchronized to Envoy and implement a permanent fix to ensure certificate updates are automatically reloaded before certificate expiry. Additionally, improve monitoring and alerting to detect certificate synchronization issues before they can impact customer traffic.

Posted Jul 10, 2026 - 15:55 CEST

Resolved

This incident has been resolved.
Posted Jul 09, 2026 - 15:24 CEST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jul 09, 2026 - 14:02 CEST

Identified

The issue has been identified and a fix is being implemented.
Posted Jul 09, 2026 - 13:55 CEST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jul 09, 2026 - 13:44 CEST

Investigating

We are currently investigating an issue affecting TTIG Pro gateway connectivity in The Things Stack Sandbox EU1 region due to a certificate renewal issue. As a result, a subset of gateways in EU1 may experience intermittent connectivity disruptions.
Our engineering team is actively investigating the issue and is closely monitoring the situation. We will provide further updates as more information becomes available.
Posted Jul 09, 2026 - 13:28 CEST
This incident affected: The Things Stack Sandbox (Europe 1 (eu1.cloud.thethings.network)).