Introduction

Reported security breaches over the last 3 years suggest that a large number of security procedures are not currently operating at full effectiveness. Security breaches have ranged from the loss of personal details of 25 million UK citizens to the disclosure of national security information assets.

It is highly likely that the organisations involved in these security breaches performed risk assessments for their information assets and implemented a range of security controls to manage these risks, leading to the resulting residual risks being within acceptable risk appetites. But as investigations into security breaches have shown, these controls are often ignored, bypassed or incorrectly implemented [ICO07].

Organisations may not currently understand how ineffectively their security controls are being managed, resulting in higher levels of risk exposure through controls operating at below optimal effectiveness. By introducing real world effectiveness measurements into an organisation’s risk management activities, organisations can improve their understanding of their current risk exposure.


Research

We have found that a number of organisational issues exist with the use of security metrics in measuring control effectiveness, which can be summarised as follows:

* Metrics that measure effectiveness can be difficult to define.
* Resulting measurements can be difficult to interpret by non-security professionals.
* Effectiveness metrics cannot be easily compared to allow benchmarking of an organisation’s performance.

Our research has concluded that there is a gap in current IT governance models and management best practices for the definition of how to measure the effectiveness of security controls. While these standards do recognise the requirement for continual assessment of operational effectiveness, the definition of these measurements and how to interpret the results are left to the organisation.


Information Security Effectiveness Framework (ISEF)

This project introduces ISEF, a framework that assists organisations in defining, visualising and comparing security metrics.

The framework uses the concept of grouping controls based on their implementation type and temporal objectives to present common characteristics that can be measured. The framework uses the relationship between controls and risks to align security metrics against organisational risk, and visualises these to support the direction of remedial efforts.

The ISEF is designed to complement current IT governance models and standards such as COBIT and ISO27002. This is provided by its alignment with these ‘what’ should be done models and standards by providing the ‘how’.

The ISEF provides a method of comparing security metrics based on the financial stock markets indices. This allows the comparison of security control management between organisations and allows the organisations to benchmark themselves against peers without revealing specific security control information.


Conclusion

A case study using ISEF has shown that the framework provides a method for defining metrics in order to obtain real world data to modify current residual risk levels. For organisations with a risk management approach, the framework can visualise effectiveness in the context of risk allowing resources to be focused on improving security management where it will make the greatest risk reduction.