Issue
You have observed L13 errors and dropped logs when searching for them in the log drain configured for a Shield Private Space. Additionally, it is difficult to determine the root cause of the drops as you lack visibility into both the Heroku logging infrastructure as well as the log drain service.
Resolution
These dropped logs are of particular importance in Shield Spaces, as these logs are verified in auditing for compliance, needed for investigation, and dropped logs means unacceptable blind spots.
Background
In general, a log pipeline between a dyno and a log drain comprises four components:
- the dyno,
- a sidecar log-shuttle process (which is responsible for delivering messages from a process to log routers and drains over HTTPs),
- a named pipe (which provides interprocess communication between the dyno and the log-shuttle process), and
- the log drain itself.
Both the Common Runtime and non-Shield Private Space runtimes contain an additional Heroku log routing service called Logplex that mediates the log-shuttle process and log drains. Among other things, this service provides Heroku staff with service-level metrics about to an internal log aggregation service as well as to the application or Space's log drain. When troubleshooting a log-related incident, you can access this kind of information by opening a Support ticket. Once involved, you can also permit Heroku staff to investigate logs directly via access to an application's logging add-on.
This kind of visibility is not available with Shield Private Spaces.
Shield Private Spaces provide additional features for high-compliance applications. The same mechanisms that implement these features and satisfy high-compliance requirements also limit the access of Heroku staff and the extent to which they are able to investigate logging issues. Among these, the log-shuttle processes in Shield Private Spaces do not go through the intermediary Logplex service, but rather transmit logs directly to the drain provider. Additionally, a logging Add-on cannot be added as-is to a Shield Space because the log-shuttle component only supports HTTPs, which is incompatible with add-ons supporting only the syslog format. Nor is heroku logs -t available as this feature relies upon Logplex. Because of these limitations, Heroku staff have limited access to information required to help investigate logging issues.
When you create a Support ticket for Shield Private Space Logging issues, providing certain additional information makes it easier for Heroku Support to help investigate, monitor, and resolve the issue faster.
A good issue report includes the following information:
- Drop rate of logs with different levels of log flows or traffic patterns
- Whether there is a daily or weekly pattern in the drop rate
- Whether the drain service requires a maximum line length less than the Shield Space maximum of 10 KB.
- Whether the drain service requires a maximum total batch size, i.e. HTTP request size, less than the Shield Space maximum of 5000 KB.
- Any additional information available from the log destination or drain service provider, such as logging add-on error messages, additional constraints, or metrics from your own logging tools.