Certificates always expire on a Sunday

Monday morning, a customer forwards you a screenshot. A full-page red warning: Your connection is not private. NET::ERR_CERT_DATE_INVALID. Underneath it, a button that says "Back to safety," which is exactly what most people click.

The certificate expired sometime over the weekend. Nobody was looking. And the part that stings is that it didn't break over the weekend — it broke weeks earlier, quietly, and the expiry date was just when the browser finally said so.

Renewal is automated, and that's the problem

Modern certificates are short-lived on purpose. Let's Encrypt issues for 90 days, and the whole model assumes a machine renews them without a human involved — certbot's systemd timer wakes up twice a day, checks whether anything is inside the 30-day renewal window, and acts if it is.

That works beautifully, which is why nobody watches it. You get sixty days of successful renewals training you to ignore it, then one silent failure with a 30-day fuse on it.

What actually breaks

Renewals almost never fail because of the certificate authority. They fail because something changed underneath the automation.

The vhost moved. HTTP-01 validation works by serving a token at http://example.com/.well-known/acme-challenge/<token>. certbot writes the file to a webroot path it recorded when the cert was first issued. Someone reworks the nginx config, changes the document root, adds a new server block that now matches the request first — the token is written to a directory nothing serves, and validation fails.

Something blocks the challenge path. A catch-all redirect that sends every HTTP request to /, basic auth in front of the whole site, a WAF rule, an IP allowlist, or a "coming soon" gate. Redirects themselves are fine — the ACME validator follows them — but it has to end at that token.

A hostname on the cert stopped resolving. One certificate typically covers example.com and www.example.com. If DNS for either name is changed, or a name is added to the cert for a subdomain that later goes away, validation for that one name fails and the entire renewal fails with it. The other names on the cert are collateral damage.

Rate limits. Once you're failing, a retry loop is dangerous. Let's Encrypt caps duplicate certificates — the same exact set of hostnames — at five per week. Rerun a broken renewal enough times while debugging and you can burn the budget, so even after you fix the real problem you're locked out of issuance for days.

The one that gets everybody: it renewed, and the site still broke

Here's the failure I'd want a stranger to know about. certbot renews successfully, writes a brand-new certificate to disk, logs a cheerful success, and your site keeps serving the expired one.

nginx reads certificates at startup and holds them in memory. New file on disk, old certificate in RAM. Without a --deploy-hook "systemctl reload nginx" (or the equivalent renewal hook), nothing tells the running process to pick up the new file. Everything looks correct to anyone checking the wrong way.

And that's the real trap, because the obvious check is the wrong one:

# what's in the file — says you're fine
openssl x509 -enddate -noout -in /etc/letsencrypt/live/example.com/fullchain.pem

# what visitors are actually served — the truth
echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null \
  | openssl x509 -noout -enddate

Those two commands can disagree for weeks. Only the second one matters. Any monitoring that reads the filesystem, or trusts certbot's exit code, will report green all the way to the interstitial.

Monitoring that would actually have caught it

Check over the network, from outside the server. Open a TLS connection to port 443 and read the expiry from what's presented. That's the only measurement that reflects what a browser sees.
Check every hostname separately. Apex and www, plus any subdomain with its own cert. They fail independently and one is usually forgotten.
Send SNI. A server hosting several sites picks its certificate from the -servername value. Omit it and you're testing the default vhost, not the customer's site.
Alert at 21 days, again at 14, again at 7. The renewal window opens at 30 days. If a cert is still under 21, automation has already tried and failed at least a dozen times — that's your signal, and there's a full three weeks of runway to fix it calmly.
Watch the renewal job itself. A failed timer run should page someone, not just append to a log file.

Do that and the expiry date stops being an event. It becomes a number on a dashboard that never gets near the bottom — and the certificate can go on expiring on Sundays, because you'll have dealt with it on a Tuesday three weeks earlier.

If nobody currently owns your certificate renewals, that's the kind of thing our support plans cover.