Let’s Encrypt's Root Certificate Change and your Heroku App

Issue

On September 30th, 2021 a root certificate used by Let’s Encrypt expired. This may affect your Heroku apps in several different ways:

1. Heroku Automated Certificate Management (ACM) uses Let’s Encrypt certificates. Older external clients accessing Heroku apps that use ACM may not be able to establish TLS/HTTPS connections to your app after the root certificate expired
2. Your Heroku app may interact over HTTPS with external services (e.g. APIs) that use Let’s Encrypt certificates. If your Heroku app is not using an updated certificate bundle, those external callouts may fail.

Below we detail how Heroku customers can identify and remediate the latter two problems.

Resolution

External clients cannot access my Heroku App

This problem will manifest as browsers, API libraries, or curl being unable to establish secure connections to your Heroku app, for example with errors such as certificate verify failed (certificate has expired).

Clients experiencing this problem will typically be using very old browsers, libraries, or other code running on operating systems that have not been updated in some time (e.g. with an old ca-certificate bundle, or running a very old Node.js version with an old version of OpenSSL statically compiled in, etc.).

What to do

There are two ways to address this problem:

1. Update the client or client code accessing your Heroku app by updating the operating system that the client runs on, updating the Node.js version that an external app is built on, etc. Ensure that the update means that your client’s root certificate store is updated so that it will trust Let’s Encrypt certificates again as laid out in the Let’s Encrypt documentation, and uses a version of OpenSSL at least as recent as v1.1.0.
2. Procure a certificate (not from Let’s Encrypt) for your custom domain, turn off Heroku ACM, and upload the custom certificate to your Heroku app. By doing this, your app no longer relies on Let’s Encrypt certificates nor the expired root CA certificate.

My Heroku App cannot access external APIs and services that use Let’s Encrypt certificates

This will problem will manifest as your Heroku app not working correctly when accessing external services or APIs. You may see errors in the app’s logs with messages such as certificate verify failed (certificate has expired).

This problem is a result of your Heroku app not using an updated root certificate bundle (meaning it no longer trusts Let’s Encrypt issued certificates), or else using an outdated version of OpenSSL, and trying to access external sites or services that use Let’s Encrypt certs for HTTPS.

What to do

• If your app is using the cedar-14 stack, update immediately. The cedar-14 stack is deprecated and not receiving security updates or updates to its certificate bundle
• If you are using Private Spaces and your app is still on the heroku-16 stack you may also be affected. heroku-16 is end-of-life and you should upgrade to the heroku-20 stack. Heroku is also in the process of patching the heroku-16 stack in Private Spaces (Common Runtime has already received heroku-16 stack update)
• If your app is built with Node.js version 9 or older, update immediately. Node.js statically links the OpenSSL library and the version linked into old Node.js versions is not updated
• If your app uses a custom certificate bundle instead of the one provided in the stack image, update the bundle and re-deploy your app. Note: Some dependencies package their own certificate bundle (for example the excon and httpclient Ruby gems), and will need updating to pick up the new bundle.