Data-driven decision making

At a macro level, overall health can be ascertained by certain telemetry and metrics. The status of deployments and updates overall and over time gives insight into release health. Volumes of alerts give a sense of a baseline of expectations, where outliers could reveal a successful emerging threat or a rule in need of tuning.

Automatically analyzing telemetry on exception lists could be used to help tune rules, where the most common exceptions in response to high FP rates are considered the most applicable. Average time to triage an alert could provide insights into the complexity of a given rule or alert. Low adoption rate rules provide feedback that could be used to deprioritize or deprecate certain rules or categories of rules.

Not all actionable feedback needs to come from direct telemetry, but could come from intermediate services or infrastructure. As an example, looking at statistics within git and GitHub reveals valuable insights that can be leveraged to determine rule maintenance costs.

The following is a sample breakdown of considerations to determine maintenance costs at a per rule bases:

• Count of total issues related

  • Average life of the issues

  • Average number of comments

• Count of total pull requests related

  • Average life of PRs

  • Average number of comments

  • Average number of pre-squashed commits

This information can be used to build an algorithm to determine the costliest rules and make adjustments automatically, such as opening an issue to review the justification of a cost or prompt an engineer to review the efficacy of the rule. It can also be used to make meta observations such as the costliest threat categories or techniques, to help with future planning.