Issue
I would like to understand how multi-availability zones work within Heroku and how to architect high availability across different regions.
Resolution
Availability Zones
The Heroku Platform is spread across multiple AWS availability zones and application dynos are provisioned automatically so there is no guarantee that your dynos will be evenly balanced across availability zones (it is effectively random allocation).
Please note that an app's Dynos are locked to a single region (Common Runtime US, Common Runtime EU, or one of the Private Spaces available to Heroku Enterprise customers). In the normal course of operation, and in line with dyno cycling policy, the allocation between availability zones changes quite often.
Basically, the dynos themselves can bounce between AZ's but if an AZ becomes unavailable, the affected dynos would be started another AZ. With so many apps on our platform, there will always be some dyno hosts that need maintenance or downtime and the platform is built to cycle dynos ensure dynos are running in healthy hosts or AZ's regardless of what causes the cycling -- Causes such as: something known like a maintenance window or something unknown like host failure. Restarted dynos may move across availability zones and if the dyno failure is due to a problem with an AZ, the dyno(s) will be restarted in (moved to) a healthy AZ.
Premium Tier Database Plans
For premium tier database plans, the failover database is in a separate availability zone automatically. Likewise if you create a follower database it will be hosted in a separate availability zone.
Private Space Dynos and AZ's
In a private space, currently, an app's first three dynos for each process type are started in three different availability zones. With three or more web dynos, three nodes of the load balancers are placed in the three availability zones. Each node of the load balancer forwards requests to one of the web dynos in both of the three availability zones. Please note that we may change the details in the architecture at any time to improve the platform. Likewise, the first three dynos of other process types will also be started in different availability zones.
Discovering the AZ for a dyno
Unfortunately we don't currently expose which availability zone a dyno is in and this is by design.
High Availability Architectures
To achieve cross-region high availability you would need to run identical applications in different regions and then use a service such as Fastly, Cloudflare, or another, to distribute the load between the two regions or to treat one as a failover region. Considerations would need to be made for how you handle data across regions. For example, if you have a primary application with a Postgres database you could have a follower in the failover region, but this would be read-only unless it was promoted to master.