Automatic testing of runtime enforcers with Test4Enforcers

Users regularly use apps to access services in a range of domains, such as health, productivity, entertainment, and business. The safety and correctness of the runtime behavior of these apps is thus a key concern for users. Indeed, unreliable apps may generate dissatisfaction, frustration and issues to users. Runtime enforcement techniques can be used to implement software enforcers that monitor executions and apply corrective actions when needed, potentially preventing misbehaviors and failures. However, enforcers might be faulty themselves, applying the wrong actions or missing to apply the right actions. To address this problem, this paper presents Test4Enforcers, an approach to automatically test software enforces. Test4Enforcers relies on an enforcement model describing the strategy that shall be applied at runtime to correct misbehaviors. Test4Enforcers first uses the enforcement model to derive a specification of the test cases that shall be executed to validate any software enforcer implemented from the given model. Then, it automatically turns the test specification into a set of concrete test cases that can be executed against apps augmented with the enforcers. We evaluated Test4Enforces with a set of 3,135 faults injected in the enforcers derived from 13 enforcement models. Results show that Test4Enforcers can automatically reveal 64% of the faults, while existing approaches relying on crash detection can only reveal 6% of the faults. Test4Enforcers is also practical since testing an enforcer required 9 min, in the worst case.