Why am I getting H12 request timeouts when using Unicorn/Gunicorn?

Issue

My application is getting H12 timeout errors and the following are true:

  • Using Unicorn or Gunicorn for my web server
  • Gunicorn logs show [CRITICAL] WORKER TIMEOUT
  • Unicorn logs show SIGKILL

Other possible symptoms:

  • Increase in database connections
  • Timeouts may correlate with restarts or dyno cycling

Resolution

The below explanation applies to both Unicorn and Gunicorn web servers.

Your Unicorn master worker is timing out and killing web workers. With (temporarily) fewer web workers, your app can't handle as many requests, hence the H12 timeouts.

Sometimes this can turn into a bigger problem during restarts. Process forking done by Unicorn can be long/expensive to start up. If these workers take too long to start up or take too long on their first request, Unicorn will kill the processes and start up a new worker. This can result in a seemingly unexplained death spiral of workers being killed on startup, and when there aren't enough workers to handle requests, requests timeout, and workers are once again killed.

Since the workers are sent a SIGKILL (no SIGTERM) they aren't able to log any errors during this time, so most of the time you'll just see a SIGKILL in your logs when this happens and nothing else.

Heroku and tools like New Relic or Scout are unaware of this happening, so it can seem like you're getting H12 timeout errors for no reason. It's an unfortunately harsh behaviour of Unicorn servers.

How to mitigate the Unicorn problem

Check to ensure that all of your dependencies and initializers have been configured with timeouts, reducing your worker dynos' startup time. Also, be sure that nothing else is delaying the booting/forking process.

Using Ruby? Try using Rack timeout

Instead of having Unicorn timeout your requests, using Rack timeout will allow the errors to bubble up to the application logs instead of dying silently with a SIGKILL.

You can also check out more timeout solutions for Ruby apps here.

Ask on Stack Overflow

Engage with a community of passionate experts to get the answers you need

Ask on Stack Overflow

Heroku Support

Create a support ticket and our support experts will get back to you

Contact Heroku Support