Issue
Im seeing H12 errors in my logs and/or high response times.
Resolution
When requests take longer than 30 seconds to complete, Heroku's router will time them out and issue an H12 error in your logs.
Since Heroku doesn't provide detailed application performance metrics beyond what you see in the Metrics page, to diagnose which parts of your requests are taking the longest to complete, you need to use an add-on like New Relic or another performance monitoring add-on.
The data in your performance tool (like New Relic) will provide you with the details necessary to decide between the following ways to mitigate the problem:
- Optimize your code to run faster: this may look like delegating longer-running tasks as background jobs.
- Run more web workers per dyno: this will increase your memory usage, but it's free.
- Scale up your number of web dynos: this will not increase your memory usage but it will cost more.
The following three questions will help you decide which of the above solutions is best for your situation.
Question 1: Can my code run faster? The first thing you should always look for are tasks that could be optimized. Anything over 500ms to complete is a good place to improve. Look for slow DB queries, inefficient calls to other services, etc.
Question 2: Have I delegated long-running tasks as background jobs? If you cant make your code run any faster and you simply need more time (i.e. process images, parsing documents, making API calls, etc), you can run them as background jobs on worker dynos. Worker dynos do not face the same 30 second timeout that web dynos do, making them perfect for heavy lifting.
Question 3: Am I seeing high request queuing? This means requests are stacking up on your dyno because all of your web workers are currently occupied. This can sometimes be caused by slow tasks, but even the fastest applications sometimes just need more hands.
If you have gone through the first two questions (see above) and do not see any way to optimize your code, you need to enable your app to handle more requests at a time. You can do this by:
- Running more web workers per dyno: This won't cost extra, but it will use more memory, so make sure your dyno types have enough RAM to spare
- Scale up your number of web dynos: This will cost extra, but is a better option if your app can't spare the extra RAM.
Edge Cases
There are some edge cases in which the above solutions will have seemingly no effect. If you are still not sure whats going on, here are some possible problems:
Using Unicorn/Gunicorn? Your master process could be killing off your slow web workers. Read more about it here.
Using Eco, Basic (formerly Hobby), 1X, or 2X dynos? Your dynos are on shared hosts, and on rare occasions other apps may be using more resources than normal, causing that dyno to run more slowly. If only a single dyno is affected, this could be the cause. This can be fixed by restarting your app or the affected dynos (thus changing dyno hosts). Consider moving to Performance dynos for more consistent performance.