SSL Certificate Expiry: Why Renewal Fails Silently
At 00:47 a certificate hit its notAfter date. Nothing in the deploy pipeline changed. Nothing in nginx config changed. The site still answered on port 443, which is exactly why the uptime monitor stayed green for another six hours while every browser showed a full-page interstitial. The health check was curl -k.
Expiry is the most predictable outage in web operations, and it still happens to companies with SRE teams. Not because people forget the concept, but because renewal is a chain of small steps and any one of them can fail silently.
How a certificate actually expires
Almost nobody "forgets to renew". The failures cluster into a few patterns.
- The renewal succeeded but the server never reloaded. certbot wrote a new certificate to
/etc/letsencrypt/live/example.com/, the web server kept the old one in memory. This is the single most common cause. A renewal cron without a deploy hook is half a renewal. - The cron ran, failed, and reported success. A wrapper like
certbot renew --quiet || trueturns an authentication failure into exit code 0. Silent for 60 days, loud at 00:47. - DNS-01 broke on a credential rotation. Wildcard certificates require a DNS TXT record. Rotate the API token in your provider, forget the renewal config, and every wildcard renewal fails from that moment.
- The host was off during the window. Let's Encrypt certificates last 90 days, and certbot renews when 30 days remain. A suspended VPS, a paused container, or a CI runner that only executes on pushes can miss the whole window. A renewal job that only runs during deploys is not a renewal job.
- Clock skew. Validation uses the client's clock, but tooling on the server uses the server's. A VM restored from a snapshot with the wrong time can decide a certificate is still valid when browsers disagree.
- The chain expired, not your certificate. September 2021, when the DST Root CA X3 cross-sign expired, broke a long list of Android and older-client setups while the leaf certificate was perfectly current. Your certificate is one link; the intermediate above it has its own dates.
Reading the real expiry date
Do not trust your panel, your ACME client's log, or the file mtime. Ask the socket what it is serving right now:
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -dates -subject -issuer
That returns notBefore and notAfter in GMT. Compare that with the file on disk. When they disagree, you have a reload problem, not a renewal problem, and the fix is systemctl reload nginx plus whatever is missing from the deploy hook.
Some tools report expiry as a Unix timestamp instead of a date. Those are seconds since 1970 in UTC, and eyeballing 1767225600 is a waste of time: paste it into a Unix timestamp converter and read the date in your own timezone. Certificate dates are always UTC, your on-call calendar is not.
The SSL certificate checker does the same handshake from outside your network, which matters: a check from inside the VPC can hit an internal load balancer with a different, longer-lived certificate and hide the problem entirely.
Monitoring that catches it before a user does
Three layers, and you need all three because they fail differently.
- External expiry check, from at least two networks. Parse the served certificate, alert on days remaining, not on connection failure. Thresholds of 21, 14, and 7 days give you three chances. An alert at 3 days is a page, not a warning.
- Local file check, on the host that terminates TLS. If the served certificate and the file on disk diverge by more than a day, you have a reload bug today and an outage in a month.
- Full chain validation without
-k. A plaincurl -sI https://example.comwithout the insecure flag is the cheapest end-to-end test there is. It fails on expiry, on an incomplete chain, and on a hostname mismatch. Every check that disables verification is decoration.
Add a fourth if you run many domains: Certificate Transparency logs. Every publicly trusted certificate is published to CT within hours. Watching crt.sh for your apex domain catches both a missed renewal and a certificate someone issued for a subdomain you did not know existed.
When it is already expired
Order matters, because the first ten minutes set the tone.
- Renew manually, now:
certbot renew --force-renewal --cert-name example.com. Do not debug the cron while the site is down. - Reload the web server and confirm with
openssl s_clientfrom outside. Confirm the date changed, not just that the command exited 0. - If renewal fails, read the challenge error before retrying. Five failed validations per account per hostname per hour will lock you out, and Let's Encrypt enforces 5 duplicate certificates per week. Retrying blind is how a 20-minute outage becomes a 6-hour one.
- Keep a fallback: a paid certificate from another CA, or a second ACME account with a different validation path (HTTP-01 alongside DNS-01). Two independent renewal paths is the cheapest insurance in this category.
After recovery, the postmortem question is never "why did it expire". It is "which step in the chain was silent, and what now fails loudly if it breaks again". Turn || true into a real failure, add the reload hook, add the expiry alert, and confirm the whole thing by revoking a staging certificate on purpose.
The DNS part people miss
HTTP-01 validation needs port 80 reachable from the internet and the right A record. When a renewal suddenly fails on a domain that worked for a year, check the record before you touch the web server: a CDN proxy in front of the origin, or a stale A record pointing at a retired IP, will break validation while the site itself looks healthy. A DNS lookup from a public resolver takes ten seconds and rules that out.
One more habit worth building: put the expiry date in the same dashboard as uptime and error rate. Certificates are the only outage cause with a known date in advance. Treating that date as an incident waiting to be scheduled, rather than a surprise, is most of the work.