I recently wrote about the complexities of getting programs to report the TLS certificates they use , where I theorized about writing a script to scrape this information out of places like the Apache configuration files, and then today I realized the obvious specific approach for our environment :

Obvious realization is obvious: since we universally use Let's Encrypt with certbot and follow standard naming, I can just look in /etc/letsencrypt/live to find all live TLS certificates and (a) host name for them, for cross-checking against our monitoring.

Our TLS certificates usually have multiple names associated with them, only one of which is the directory name in /etc/letsencrypt/live. However, we usually monitor the TLS certificate under what we think of as the primary name, and in any case we can make this our standard Prometheus operating procedure.

In our Prometheus environment we create a standard label for the 'host' being monitored , including for metrics obtained through Blackbox. Given that Blackbox exposes TLS certificate metrics , we can use things like direct curl queries to Prometheus to verify that we have TLS certificate monitoring for everything in /etc/letsencrypt/live. The obvious thing to check is that we have a probe_ssl_earliest_cert_expiry metric with the relevant 'host' value for each Let's Encrypt primary name.

If we want to, we can go further by looking at probe_ssl_last_chain_info. This Blackbox metric directly exposes labels for the TLS 'subject' and 'subjectalternative', so we can in theory search them for either the primary name that Let's Encrypt will be using or for what we consider an important name to be covered. It appears that this wouldn't be needed to cover any additional TLS certificates for us, as we're already checking everything under its primary name.

(Well, we are after I found one omission in a manual check today.)

With the right tools ( also ), I don't need to make this a pre-written shell script that runs on each machine; instead, I can do this centrally by hand every so often. On the one hand this isn't as good as automating it, but on the other hand every bit of locally built automation is another bit of automation we have to maintain ourselves. We mostly haven't had a problem with tracking TLS certificates, and we have other things to notice failures .

(I should probably write a personal script to do this, just to capture the knowledge.)