Why am I seeing H12 request timeouts / high response times in my app?
When requests take longer than 30 seconds to complete, Heroku’s router will time them out and issue an H12 error in your logs.
Since Heroku doesn't provide detailed application performance metrics beyond what you see in the Metrics page, to diagnose which parts of your requests are taking the longest to complete, you’ll want to use an add-on like New Relic.
The data in your performance tool (like New Relic) will provide you with the details necessary to decide between the following ways to mitigate the problem:
- Optimize your code to run faster – this may look like delegating longer-running tasks as background jobs.
- Run more web workers per dyno – this will increase your memory usage, but it's free.
- Scale up your number of web dynos – this won't increase your memory usage but it will cost more.
The following three questions will help you decide which of the above solutions is best for your situation.
Question 1: Can my code run faster? The first thing you should always look for are tasks that could be optimized. Anything over 500ms to complete is a good place to improve. Look for slow DB queries, inefficient calls to other services, etc.
Question 2: Have I delegated long-running tasks as background jobs? If you can’t make your code run any faster and you simply need more time (i.e. process images, parsing documents, making API calls, etc), you can run them as background jobs on worker dynos. Worker dynos do not face the same 30 second timeout that web dynos do, making them perfect for heavy lifting.
Question 3: Am I seeing high request QUEUING? This means requests are stacking up on your dyno because all of your web workers are currently occupied. This can sometimes be caused by slow tasks, but even the fastest applications sometimes just need more hands.
If you’ve gone through the first two questions (see above) and don’t see any way to optimize your code, you need to enable your app to handle more requests at a time. You can do this by:
- Running more web workers per dyno: This won’t cost extra, but it will use more memory, so make sure your dyno types have enough RAM to spare
- Scale up your number of web dynos: This will cost extra, but is a better option if your app can’t spare the extra RAM.
There are some edge cases in which the above solutions will have seemingly no effect. If you’re still not sure what’s going on, here are some possible problems:
Using Unicorn/Gunicorn? Your master process could be killing off your slow web workers. Read more about it here.
Using non-Performance dynos? Your dynos are on shared hosts, and on rare occasions another app will be hogging the resources. This can be fixed by simply restarting your app (thus changing dyno hosts).