LBS Gateway Disconnections in TTS Sandbox Clusters

Summary

On June 4, 2026 at ~13:30 UTC, following an automatic Let's Encrypt certificate renewal on the TTS Sandbox clusters (eu1, au1, nam1), semtechws/lbslns (BasicStation / LNS) gateways began disconnecting and entering a persistent reconnect loop.

The renewal switched from the R-generation intermediate chain (signed directly under ISRG Root X1) to the new YR-generation intermediate chain (cross-signed: ISRG Root YR → ISRG Root X1). LoRaWAN Basics Station (LBS) gateways that do not support cross-signed certificate chains could not verify the new certificates.

Other protocol gateways were unaffected.

Impact

All tenants on the TTS Sandbox (eu1, au1, nam1) using semtechws/lbslns (BasicStation / LNS) gateways were unable to maintain a connection to the Gateway Server. Gateways entered a continuous reconnect loop that power resets could not resolve. TTN Indoor Gateways were among the affected devices, with widespread reports from the TTN community starting ~21:30 UTC on June 4.

Root Cause

A scheduled certbot renewal obtained new Sandbox TLS certificates issued under Let's Encrypt's newer YR-generation intermediates (YR1/YR2).

We had set --preferred-chain 'ISRG Root X1' for backward compatibility, but the YR chain satisfied this constraint because ISRG Root X1 sits at the end of a cross-signed path:

End-entity cert ← YR1/YR2 ← ISRG Root YR ← ISRG Root X1

Our gateways running mbedTLS don't support cross-signed root resolution. They stop at the first self-signed certificate (ISRG Root YR), which isn't in their trust store, so TLS verification failed.

Resolution

The previous R-generation Let's Encrypt certificates were re-imported on all three Sandbox clusters. The certbot renewal window was narrowed from 30 to 7 days before expiry to reduce the chance of an unwanted automatic renewal before a permanent fix is in place.

After re-import, TLS connections were restored and gateways resumed reconnecting normally.

Prevention / Action items

Code improvements — already applied

• Narrowed certbot renewal window from 30 to 7 days before expiry on Sandbox and Cloud (to be reverted once a permanent fix is in place).

Code improvements — planned

• Replace the --preferred-chain strategy in 5-8a-certs-le with a check that validates the renewed chain's actual issuing intermediate (e.g. R13), since --preferred-chain only constrains the root and let LE silently switch intermediate generations. May require switching providers.
• Add a post-renewal verification step that confirms the issued chain matches the expected intermediate before certificates are deployed.
• Add a GCS tool to bulk-update the trusted server certificate on a batch of gateways by LNS address.