I received an email about index corruption and have additional questions about this issue.
What is index corruption?
Index corruption is an issue that occurs where the data contained in the Postgres index is no longer valid due to hardware failure or software bugs. This corruption can lead to invalid results or errors being returned during query execution.
How do I fix index corruption?
When Heroku detects non-unique index corruption we will attempt to remediate the corruption manually. If you have a corrupted unique index or the automated fix fails, you may need to fix the corruption manually. We have more information detailing how to fix corrupted indexes here:
Was my index corruption caused by maintenance?
If you experienced index corruption shortly after maintenance was performed on your database, there is a high probability that your index corruption is the result of a bug when updating software on your server.
One known instance of this occurred when upgrading server images from Ubuntu Trusty to Xenial. This upgrade resulted in a fundamental system library (
glibc) change that affected string order, specifically strings with Asian glyphs, ASCII bytes and a punctuation character. In this case, we confirmed that upgrading from Trusty to Xenial server images with this example data and index resulted in the index returning this content in a new order.
Changes such as these are an essential part of operating system and security upgrades. This might happen again. As such, our focus must be on detecting and remediating the effects of any such change. To address this moving forward, we will run corruption checks (and automated remediation where possible) for all databases after maintenance.
Why did I receive an email about index corruption?
As a part of our managed Postgres offering, Heroku periodically scans all of your indexes to check for corruption. These corruption scans are performed using a Postgres module called amcheck. When index corruption is detected we notify the app owners.
If possible, we will then attempt to automatically remediate the corruption. After remediation, an email is sent to the app owners which will state whether or not the remediation was successful.