ResearchSOC Webinar Series - Using metrics to improve operational cybersecurity

Community Forums ResearchSOC Cybersecurity
1

Using metrics to improve operational cybersecurity

ResearchSOC’s deputy director, Susan Sons, gave an online webinar in March on using cybersecurity metrics to make operations more effective. You can watch the presentation here and view the slides here. Susan’s main points are summarized below; please view the entire talk for more details and helpful examples of effective metrics.

Metrics with Purpose

Many times the metrics we select for cybersecurity don’t adequately capture the behaviors we want to reinforce, worse, often times they actually create a perverse set of expectations, working against the goal of having better and more effective security. These bad metrics can build up a false sense of security, since we aren’t measuring the things that are important to our goals, and they can lead us to invest in “magic box” technologies, which provide positive feedback for our metrics, but offer little operational benefit.

Why do we care about having good metrics? Good metrics let us know that our work is accomplishing something tangible, they tell us which of our security investments are paying off, and they can help identify problems in our information security program.

Good security metrics provide a lot of useful information. They help us to understand and communicate risk internally and externally, they help us understand what our current controls and processes are doing (or not doing) and they help us to reveal weaknesses in our infrastructure. Threat intelligence is often thought of as an external resource which we consume, but good metrics can allow us to spot emerging threats in our environments before they come through our TI feeds. This feedback allows us to spend our resources where they will be most effective for our goals and they allow us to stop doing things which aren’t paying off in operational returns.

Selecting Useful Metrics

There are several characteristics of a good metric. A good metric:

  • Consistently measured
  • Cheap to gather
  • Expressed as a cardinal number or percentage
  • Expressed using at least one unit of measure
  • Actionable

There are two ways of approaching selecting a good metric.

  1. What are we measuring?

    • Process - are we doing the things we say we are going to do? Are we using best practices?
    • Outcome - did the thing we did work?
    • Impact - does it matter that it worked (or not)?

  2. What are we doing with those measurements?

    • Target - where do we want to be? Once we get there, these become…
    • Heartbeat - what are we already doing?
    • Compliance - are we managing our compliance risk?

Susan’s fictional example identifying metrics across nine target areas of operations.

Using Metrics

How to implement metrics in your organization:

  1. Start with the cheap and easy - get your organization used to using metrics.
  2. Systematically eliminate unknowns and mature your program - look for things you don’t know enough about to adequately secure them.
  3. Enlist other teams in improving security - show teams your metrics and show them how they can help improve.
  4. Find efficiencies and make vendors prove their value - utilize metrics to compare vendors to one another and get more out of magic boxes.
  5. Old metrics never die, they just move from targets to heartbeats - these let us show our effectiveness and they identify negative changes as the organization evolves.

Avoiding traps

  1. Watch for absolutism - not every metric will trend to 0 or 100, moving further in one direction may create perverse incentives.
  2. Watch out for “we’re done” mindset - heartbeat metrics let you confirm ongoing effectiveness of controls.
  3. Some things that are hard to measure are still important - beware of focusing only on things you have good metrics for.
  4. Some people will equate measurement with blame - demonstrate that you are measuring outcomes and concerned with the future, not past performance of individuals.

Communicating about metrics is crucial. Everytime you communicate about a metric you should include three pieces of information:

  1. What are we shooting for?
  2. Why are we doing this ?
  3. What’s the context?

Providing this information with every metric communication helps to break down resistance within organizations to security controls, and ensures that your metrics serve a purpose of letting your people know what they should be doing.

Sharing your metrics with peer organizations and asking for their metrics helps us identify threats, identify areas of improvement, and coordinate information which strengthens both entities.

Keeping things simple - Metrics over time

The foundation of effective metrics is getting moving. Pick an aspect of your security operations and mature it, implement baseline controls, establish foundational policies, but start building momentum in the right direction. Once you have some momentum, start to identify things that aren’t effective, and fix or get rid of them. Focus effort on areas that work and systematically improve those to a set of targets.

Once targets start being achieved, migrate those metrics to heartbeats, and carefully monitor that baseline so you don’t regress. Finally, ask “what’s next?” Identify new targets and new threats, find things that are unkown as begin to measure and improve those, beware of being “done,” new threats appear constantly.

ResearchSOC Webinars

Our webinars are held monthly, open to all, and free. You can sign up for them by going to researchsoc.iu.edu/webinars where you’ll find information about upcoming training, and can view past webinars and slide decks.

Our next ResearchSOC webinar is coming up on May 21st, at 3pm Eastern. The topic is Stakeholder Management in a Crisis: Lessons from a Crisis Communicator

Incidents happen. The real test of a cybersecurity program is how those incidents are managed when they do happen. Executing well on technical incident response is important, but if stakeholders start to panic, or try to run the process themselves, even a straightforward incident can balloon into a complex crisis. Preventing this takes solid communication and the ability to constantly nudge people with different motivation and points of view toward a common and reasonable direction. Join this ResearchSOC webinar to learn a time-tested stakeholder management method taken from hostage negotiation and how to apply it to incident response. ResearchSOC Deputy Director, cybersecurity incident responder, and volunteer crisis communicator Susan Sons will lead the training.

You can register to attend live here