Maintainability

Maintainability is concerned with the ability to upkeep a rule based on available resources and whether the overall cost of maintenance justifies the value of the rule.

Directly building off of the previous sections, if a rule is non-performant or has efficacy issues, then the resulting tuning and maintenance of the rule goes up, making it less maintainable. Essentially, the worth of a rule should be seen as a balance across the trifecta of performance, efficacy, and maintainability. Tuning a rule should also be broken down further to truly understand the source of the need to tune, as well as determining if it is justified. Tuning tends to fall into metadata, functional, or detection logic changes. The value of logic changes should be pretty obvious, but metadata changes, such as modifying a description is much harder to determine. Additionally, whether logic is tuned within the rule directly or via decoupled exception lists will also impact the maintenance cost.

The ability to test rules also plays a factor in determining the cost to maintain it, because a rule that can be tested easily, will allow more time to make the necessary changes, whereas a rule requiring a complicated setup to replicate or emulate the behavior necessary to trigger the rule will be harder to validate future changes. This ties in with the process of change management, which focuses on the incorporation of the new and changed rules. If the change management process is cumbersome, it will only amplify rule maintainability issues.

To get a sense of how often rules tend to be created and changed based on an industry baseline, we can explore some data from the REx project. The following graphic shows a breakdown of new and tuned rules over the course of a year.

Fig. 71 REx project: new and tuned rules over time

These next two graphics show how many unique fields within rules are being introduced over the course of the changes over time. More fields usually adds to the complexity of a rule, which also adds to the maintenance costs.

Fig. 72 REx project: new logic query fields over time

Fig. 73 REx project: new unique fields introduced over new and tuned rules

As alluded to earlier in scoping coverage, the greater the number of techniques a rule focuses on, the greater the complexity. This next graphic shows the distribution of ATT&CK techniques over time and within new and tuned rules.

Fig. 74 REx project: new techniques over time

False positives also contribute to the complexity of a rule since they basically are just identified gaps in the rule logic, where FPs are expected to hit, and so be ignored. The next graphic shows the number of newly introduced false positive cases introduced in new and tuned rules over time.

Fig. 75 REx project: false positives in new and tuned rules over time

The complexity of an entire rule set is also important to consider. Datasource diversity certainly adds to the complexity of a rule set since it will require more understanding and expertise across unique technologies and use cases. The next graphic shows the introduction of total threat categories over time, within new and tuned rules.

Fig. 76 REx project: threat categories over time

The following visualization shows the top most modified rules across all the included rule sets, which reveals the rule with the most modifications to the detection logic within a year was 18 changes.

Fig. 77 REx project: most modified rules across all rule sets

These next three graphics show a breakdown of the average modification per rule across threat category, technique, and individual detection logic field. This truly gives a sense of the required maintenance costs, across each of those attributes, which we can then extrapolate on to the rules directly.

Fig. 78 REx project: normalized average changes by category

Fig. 79 REx project: normalized average changes by technique

Fig. 80 REx project: normalized average changes by detection logic field

The purpose of exploring real-world maintenance costs and its distribution over the course of a year was to provide perspective and a baseline. Maintainability is relative to the ability to maintain based on available resources, so will vary for all.