SIEM – One Year Later
Sunday, July 26, 2015
We are overwhelmed with data and are not sure what to look at or collect? I came across this paraphrased comment in a SIEM forum and it echoes a sentiment I have seen about SIEM. Deploying the technology results in a ton of noise and alerts making it hard to make sense of. Some organizations struggle using SIEM effectively and at times their staff are drowning in a sea of logs and alerts. The comment is also one foreign to me. I’ve read about it and seen others say this but I never witnessed it for myself. My path, my journey was a different one. This post reflects on my SIEM journey for the past year in hopes that it can help others who are either taking their first step or are already on their SIEM journeys.
Disclosure: jIIr is my personal blog and is not affiliated with my employer. This post only covers my personal experience and does not go in to details related to SIEM implementation in my employer’s environment. Any questions along these lines will unfortunately go unanswered. Some lines are not meant to be crossed and this is one of them.
My journey didn’t start when the SIEM was acquired but it occurred long before then when my perspective about security strategies changed. The security strategy is critical to explore since the strategy is the force pushing organizations down the SIEM path in the first place. Let’s go back in time to when I was in a unit performing vulnerability assessments against other public sector organizations. Over time I began to see fundamental problems with various security strategies I encountered.
Some strategies were completely centered on prevention. Almost every security resource – including funding, staffing, etc. – was applied to tasks and projects related to preventing attacks. In these organizations we always found something; in every organization we always found something. With each finding came the task of explaining to auditors on my side the cause of the finding. Auditors see things in black and white but security findings are not clear cut. The truth is we probe the target’s environment and security program moving from well protected areas to areas not well protected. These were the areas our findings came from. Organizations cannot protect everything in a manner to prevent everything; it’s an impossible task and even with an unlimited budget it is not achievable.
Some strategies were centered on compliance. Security resources and priorities are focused on the findings in the latest audit. As time goes by the security strategy is to address and reduce those findings without taking in to consideration areas posing the highest risks to the organization. At times during our engagements we were welcomed. The expectation was we would highlight areas they should focus on and help the security folks convince management to allocate the appropriate resources to address those areas. For a while I thought the work we were doing accomplished this as well. In time I became to see things differently. No matter how effective an audit is, this security strategy will never work since there is a fundamental problem. Audits only confirm if something complies with criteria outlined in a regulation, policy, or standard. If something has no criteria then it is very difficult for an audit to list it as a finding since each finding needs to be tied back to a criteria. If the criteria (regulation or policy) doesn’t address something that is a high risk to the organization then the audit/assessment will miss it as well. Making matters worse is it is very difficult to determine how effective something is. If something exists, has supporting documentation, and passes some tests in most cases this satisfies the audit requirement. Audits don’t truly evaluate how effective processes and controls are. They check a box as something being present then move on to other areas not well protected (remember it’s impossible to protect everything). This results in significant gaps in security strategies driven by compliance and the news headlines of breaches for organizations compliant with regulation X highlights this.
Then some security strategies were reactive. This is when the security strategy is based on the latest event; both inside the organization and outside of it. With each new event the organization switches focus and resources to address it even if it is not the highest risk to the organization. This leads down a path of random controls put in place to combat random threats and what little security resources are available is used in an ad-hoc manner. Reactive security strategies in my opinion are not even a strategy and are doomed for failure.
Over time, the fundamental problems with various security strategies I encountered made me ask myself a single question. If I ever found myself in a position to lead an organization’s immature security program. How should I approach building their program from scratch? Exploring this question brought me to various information security resources. It even lead me to obtaining my Masters of Science in Information Assurance. In time I came to the following conclusion:
1. There are fundamental problems with security strategies based on prevention, compliance, and reactive.
2. Most information security decisions I witnessed in my entire career were not based on actual data to support the decisions. Decisions were based on experience, intuition, what someone else recommended, or shiny new objects. At times, decisions not based on actual data resulted in bad choices, wasted what little security resources are available, and didn’t address the actual threats the organizations faced.
3. What security strategies need is “an intelligence-driven approach, whereby decisions are made based on real-time knowledge regarding the cyber adversaries and their attack methods, and the organization’s security posture against them”.
The fundamental problems with security strategies based on prevention, compliance, and reactive would be addressed in an intelligence-driven approach. Security decisions would not only address the most significant risks an organization faces but the decisions would be influenced by facts and information. The path to intelligence-driven security needs to start with the foundation mentioned in the quote below.
“To be ready to take on an intelligence program, the organization needs to have a foundation in place for monitoring the network for intrusions and a workflow process for responding to incidents.”
This became my perspective on how security strategies needed to be and I ended up in a security office who agreed with the strategy. This strategy started me down the SIEM path as part of laying a foundation for security monitoring. The driving force behind SIEM was security monitoring and it influenced what logs were collected and how they were analyzed.
I knew there was going to be a significant time requirement at my $dayjob but I didn’t know about the impact on my personal time. I had a well-rounded background to take on a SIEM project but I never built the equivalent of a SOC. I read the often quoted percentage that “somewhere between 20% and 30% of SIEM deployments among his client base fail, meaning not only do they not meet predefined goals, but also that many organizations don't even bother using the product”. I also read the articles and comments about how difficult SIEM deployments are, how complicated SIEM management is, and how companies frequently end up in a sea of alerts with no clue what to do about them. I guessed what the impact would be on an organization for a failed security investment. How after getting buy-in, making a sizable investment in technology, and allocating staff to only end up with something that doesn’t meet any goals would be devastating. Not only would this not provide the needed foundation for intelligence driven security but the failure would linger for a long time for the organization. Any other request for security resources would be even more difficult because it will be looked upon as another wasted investment since the investment in SIEM failed. Any other security initiatives may be looked at with doubt and wonder if they can even be successful since the security office failed with the SIEM initiative. Needless to say, failure was possible but it wasn’t an option in my opinion.
I put most of my personal time I allocate for research, writing, and reading on hold to allow me to focus on building the SOC. I spent my time instead researching and learning from others about how to build an effective security monitoring capability. A small portion of what I explored was mentioned in the posts: Linkz for SIEM, Linkz for Detection and Response, Making Incident Response a Security Program Enabler, and Linkz for Intelligence Driven Security and Threat Intelligence. In essence, my time at the $dayJob was spent implementing SIEM capabilities, building processes, training staff, and managing security monitoring while my personal time was spent exploring all aspects related to security monitoring and intelligence driven security.
It was a personal sacrifice I made for my employer to ensure our SIEM project would be successful but in the past year my knowledge and skills have grown by leaps and bounds. There’s a personal sacrifice leading a SIEM implementation but making that sacrifice is worth it in the end.
In John Boyd’s autobiography Boyd: The Fighter Pilot Who Changed the Art of War one lesson we all can learn is how he approached things. Take the following quote about someone who Boyd influenced and was a member on his team:
“His attitude was “Maybe so. But if not me, who?” He was the right man in the right place at the right time. He had done his homework and knew his briefing was rock solid.”
The lesson we can all learn is for us to do our homework; to prepare for each possibility and the things that can and will go wrong. This is probably the best advice I could give anyone traveling down this path. You might be the right man in the right place at the right time to lead an organization as they deploy SIEM technology but you need to do your homework. You need to research and explore the issues trying to be solved, developing a detailed project plan with phases, getting buy-in by explaining the issues and why SIEM is the technology to solve it, identifying the exact logs required and what in those logs is needed, etc.. The list goes on but the point of this advice is to prepare. Prepare, and then prepare some more since the effort and time spent doing this will ensure a smooth deployment. I spent a considerable amount of time preparing upfront and during the deployment and this helped me avoid pitfalls that could had impacted the project.
As I started this journey one of the first things I did was to learn from others who took this journey before me. The main person who influenced my thoughts and thus my approach was Anton Chuvakin. In most articles he advocates to leverage use cases when deploying a SIEM and hands down this is the best advice for a successful SIEM project. He authored a lot of posts on the subject but the best one as it relates to a SIEM project is the one hyperlinked in my quote below:
“The best reference I found to help architect a SIEM solution is a slide deck by Anton Chuvakin. The presentation is Five Best and Five Worst Practices for SIEM and it outlines the major areas to include in your SIEM project (16 to be exact). It may not cover everything -such as building rules, alarms, and establishing triage processes - but it does an outstanding job outlining how to avoid the pitfalls others have fallen in. If anyone is considering a SIEM deployment or in the midst of a SIEM deployment then this is the one link they will want to read.”
His slide deck influenced my SIEM project plan from the point of getting buy-in to the actual implementation. To implement a use case there wasn’t a lot of information on it so I put together the SIEM Use Case Implementation Mind Map to have a repeatable process for each use case.
The beauty in leveraging use cases. Not only does it make building a SOC more manageable by focusing on detecting certain activity in smaller chunks and building the processes around those chucks but it makes it very easy to show others the value SIEM adds. If the SIEM deployment takes one year to complete then using multiple use case can show progress and what was accomplished throughout the year. The value added is shown throughout the year instead of waiting until the end. That is if the proper preparation was done in advanced.
Thinking back over the past year and what I found to be the most challenging with SIEM technology or any detection technology for that matter is how events/alerts need to be handled. I found bringing in logs, creating custom detection rules, and tuning rules to be easy compared to developing the triage processes surrounding each category of events/alerts. My previous thoughts on the subject still ring true today:
In my opinion establishing triage processes is the second most critical step (detection rules are the first.) Triage is what determines what is accepted as "good" behavior, what needs to be addressed, and what needs to be escalated. After the rules are implemented take some time reviewing the rules that fired. Evaluate the activity that triggered the rule and try out different triage techniques. This is repeated until there is a repeatable triage process for the rules. Continue testing the repeatable triage process to make it more efficient and faster. Look at the false positives and determine if there is a way to identify them sooner in the process? Look at the techniques that require more in-depth analysis and move them to later in the process? The triage process walks a fine line between being as fast as possible and using resources as efficient as possible. Remember, the more time spent on one alarm the less time is spent on others; the less time on one alarm increases malicious activity being missed.
The triage processes are critical and can help prevent encountering the paraphrased comment I opened this post with. We are overwhelmed with data and are not sure what to look at or collect? Unfortunately, when it comes to triage for the most part you are on your own. This was one area I found to be truly lacking in documentation and the few cheat sheets I found didn’t account for the majority of stuff one needs to consider. Determining what triage processes were needed followed by developing those processes and then training others was definitely a tough challenge. It was tougher since it also involved honing those triaging processes to improve their efficiency and speed. Despite the difficulty, focusing on triage enables you to see through the noise and know what to look for.
If someone would had told me two years ago I be reading about security metrics I would had laughed out loud. Two years ago I didn’t see the value metrics offer until my eyes were opened exploring intelligence driven security. Another comment I saw in the SIEM forum was the following quote: “what information should be provided to the management on the daily basis which can justify the purchase of SIEM”. This is where metrics can come in to play. By recording certain information it makes it easier to communicate what one is detecting and responding to. Communicating metrics and trends shows value to management, highlights weak areas in the security program, or uncovers patterns in attacks. During the past year I explored various information security metrics. As it relates to the SIEM, the VERIS schema is probably the best one I found for recording the information from detecting and responding to security events. As soon as you have the documented information then the fun part is finding creative ways to communicate these to others.
As I embarked on this journey I learned from others who have either dealt with SIEM technology or security monitoring. Initially, I read online what they published and made available to the public. I reached out to a few people about the additional questions I had. I looked around locally and reached out to others who were managing their own security monitoring capabilities. Needless to say, my SIEM journey was successful since I was standing on the shoulders of those who came before me.
However, at the end of the day each organization is different. Others can provide you with advice and share their experience but what is actually encountered is unique. The environment is unique and certain aspects of the use cases are unique as well. Due to this uniqueness, most of the time throughout the year we (my team and myself) were on our own. I mention this as a word of caution to others who may expect a vendor or others to solve their SIEM challenges for them. Others and vendors can help to a certain extent but you and your team are the ones who need to solve the challenges you face. At that point you will find yourself alone standing on the shoulders of others.
Reflecting back over the past year it has been a demanding challenge. The journey hasn't ended since work still needs to be done. There will always be work to do and new challenges to work through. Hopefully, the experienced I shared will avert someone from stepping on a SIEM land mind resulting in them yelling out for help as they are drowning in the sea of logs and alerts.
Disclosure: jIIr is my personal blog and is not affiliated with my employer. This post only covers my personal experience and does not go in to details related to SIEM implementation in my employer’s environment. Any questions along these lines will unfortunately go unanswered. Some lines are not meant to be crossed and this is one of them.
Start with Why It Is Needed
My journey didn’t start when the SIEM was acquired but it occurred long before then when my perspective about security strategies changed. The security strategy is critical to explore since the strategy is the force pushing organizations down the SIEM path in the first place. Let’s go back in time to when I was in a unit performing vulnerability assessments against other public sector organizations. Over time I began to see fundamental problems with various security strategies I encountered.
Some strategies were completely centered on prevention. Almost every security resource – including funding, staffing, etc. – was applied to tasks and projects related to preventing attacks. In these organizations we always found something; in every organization we always found something. With each finding came the task of explaining to auditors on my side the cause of the finding. Auditors see things in black and white but security findings are not clear cut. The truth is we probe the target’s environment and security program moving from well protected areas to areas not well protected. These were the areas our findings came from. Organizations cannot protect everything in a manner to prevent everything; it’s an impossible task and even with an unlimited budget it is not achievable.
Some strategies were centered on compliance. Security resources and priorities are focused on the findings in the latest audit. As time goes by the security strategy is to address and reduce those findings without taking in to consideration areas posing the highest risks to the organization. At times during our engagements we were welcomed. The expectation was we would highlight areas they should focus on and help the security folks convince management to allocate the appropriate resources to address those areas. For a while I thought the work we were doing accomplished this as well. In time I became to see things differently. No matter how effective an audit is, this security strategy will never work since there is a fundamental problem. Audits only confirm if something complies with criteria outlined in a regulation, policy, or standard. If something has no criteria then it is very difficult for an audit to list it as a finding since each finding needs to be tied back to a criteria. If the criteria (regulation or policy) doesn’t address something that is a high risk to the organization then the audit/assessment will miss it as well. Making matters worse is it is very difficult to determine how effective something is. If something exists, has supporting documentation, and passes some tests in most cases this satisfies the audit requirement. Audits don’t truly evaluate how effective processes and controls are. They check a box as something being present then move on to other areas not well protected (remember it’s impossible to protect everything). This results in significant gaps in security strategies driven by compliance and the news headlines of breaches for organizations compliant with regulation X highlights this.
Then some security strategies were reactive. This is when the security strategy is based on the latest event; both inside the organization and outside of it. With each new event the organization switches focus and resources to address it even if it is not the highest risk to the organization. This leads down a path of random controls put in place to combat random threats and what little security resources are available is used in an ad-hoc manner. Reactive security strategies in my opinion are not even a strategy and are doomed for failure.
Over time, the fundamental problems with various security strategies I encountered made me ask myself a single question. If I ever found myself in a position to lead an organization’s immature security program. How should I approach building their program from scratch? Exploring this question brought me to various information security resources. It even lead me to obtaining my Masters of Science in Information Assurance. In time I came to the following conclusion:
1. There are fundamental problems with security strategies based on prevention, compliance, and reactive.
2. Most information security decisions I witnessed in my entire career were not based on actual data to support the decisions. Decisions were based on experience, intuition, what someone else recommended, or shiny new objects. At times, decisions not based on actual data resulted in bad choices, wasted what little security resources are available, and didn’t address the actual threats the organizations faced.
3. What security strategies need is “an intelligence-driven approach, whereby decisions are made based on real-time knowledge regarding the cyber adversaries and their attack methods, and the organization’s security posture against them”.
The fundamental problems with security strategies based on prevention, compliance, and reactive would be addressed in an intelligence-driven approach. Security decisions would not only address the most significant risks an organization faces but the decisions would be influenced by facts and information. The path to intelligence-driven security needs to start with the foundation mentioned in the quote below.
“To be ready to take on an intelligence program, the organization needs to have a foundation in place for monitoring the network for intrusions and a workflow process for responding to incidents.”
This became my perspective on how security strategies needed to be and I ended up in a security office who agreed with the strategy. This strategy started me down the SIEM path as part of laying a foundation for security monitoring. The driving force behind SIEM was security monitoring and it influenced what logs were collected and how they were analyzed.
Expect a Significant Time Commitment
I knew there was going to be a significant time requirement at my $dayjob but I didn’t know about the impact on my personal time. I had a well-rounded background to take on a SIEM project but I never built the equivalent of a SOC. I read the often quoted percentage that “somewhere between 20% and 30% of SIEM deployments among his client base fail, meaning not only do they not meet predefined goals, but also that many organizations don't even bother using the product”. I also read the articles and comments about how difficult SIEM deployments are, how complicated SIEM management is, and how companies frequently end up in a sea of alerts with no clue what to do about them. I guessed what the impact would be on an organization for a failed security investment. How after getting buy-in, making a sizable investment in technology, and allocating staff to only end up with something that doesn’t meet any goals would be devastating. Not only would this not provide the needed foundation for intelligence driven security but the failure would linger for a long time for the organization. Any other request for security resources would be even more difficult because it will be looked upon as another wasted investment since the investment in SIEM failed. Any other security initiatives may be looked at with doubt and wonder if they can even be successful since the security office failed with the SIEM initiative. Needless to say, failure was possible but it wasn’t an option in my opinion.
I put most of my personal time I allocate for research, writing, and reading on hold to allow me to focus on building the SOC. I spent my time instead researching and learning from others about how to build an effective security monitoring capability. A small portion of what I explored was mentioned in the posts: Linkz for SIEM, Linkz for Detection and Response, Making Incident Response a Security Program Enabler, and Linkz for Intelligence Driven Security and Threat Intelligence. In essence, my time at the $dayJob was spent implementing SIEM capabilities, building processes, training staff, and managing security monitoring while my personal time was spent exploring all aspects related to security monitoring and intelligence driven security.
It was a personal sacrifice I made for my employer to ensure our SIEM project would be successful but in the past year my knowledge and skills have grown by leaps and bounds. There’s a personal sacrifice leading a SIEM implementation but making that sacrifice is worth it in the end.
Prepare, Prepare, and Prepare Some More
In John Boyd’s autobiography Boyd: The Fighter Pilot Who Changed the Art of War one lesson we all can learn is how he approached things. Take the following quote about someone who Boyd influenced and was a member on his team:
“His attitude was “Maybe so. But if not me, who?” He was the right man in the right place at the right time. He had done his homework and knew his briefing was rock solid.”
The lesson we can all learn is for us to do our homework; to prepare for each possibility and the things that can and will go wrong. This is probably the best advice I could give anyone traveling down this path. You might be the right man in the right place at the right time to lead an organization as they deploy SIEM technology but you need to do your homework. You need to research and explore the issues trying to be solved, developing a detailed project plan with phases, getting buy-in by explaining the issues and why SIEM is the technology to solve it, identifying the exact logs required and what in those logs is needed, etc.. The list goes on but the point of this advice is to prepare. Prepare, and then prepare some more since the effort and time spent doing this will ensure a smooth deployment. I spent a considerable amount of time preparing upfront and during the deployment and this helped me avoid pitfalls that could had impacted the project.
Leverage Use Cases
As I started this journey one of the first things I did was to learn from others who took this journey before me. The main person who influenced my thoughts and thus my approach was Anton Chuvakin. In most articles he advocates to leverage use cases when deploying a SIEM and hands down this is the best advice for a successful SIEM project. He authored a lot of posts on the subject but the best one as it relates to a SIEM project is the one hyperlinked in my quote below:
“The best reference I found to help architect a SIEM solution is a slide deck by Anton Chuvakin. The presentation is Five Best and Five Worst Practices for SIEM and it outlines the major areas to include in your SIEM project (16 to be exact). It may not cover everything -such as building rules, alarms, and establishing triage processes - but it does an outstanding job outlining how to avoid the pitfalls others have fallen in. If anyone is considering a SIEM deployment or in the midst of a SIEM deployment then this is the one link they will want to read.”
His slide deck influenced my SIEM project plan from the point of getting buy-in to the actual implementation. To implement a use case there wasn’t a lot of information on it so I put together the SIEM Use Case Implementation Mind Map to have a repeatable process for each use case.
The beauty in leveraging use cases. Not only does it make building a SOC more manageable by focusing on detecting certain activity in smaller chunks and building the processes around those chucks but it makes it very easy to show others the value SIEM adds. If the SIEM deployment takes one year to complete then using multiple use case can show progress and what was accomplished throughout the year. The value added is shown throughout the year instead of waiting until the end. That is if the proper preparation was done in advanced.
Focus on Triage
Thinking back over the past year and what I found to be the most challenging with SIEM technology or any detection technology for that matter is how events/alerts need to be handled. I found bringing in logs, creating custom detection rules, and tuning rules to be easy compared to developing the triage processes surrounding each category of events/alerts. My previous thoughts on the subject still ring true today:
In my opinion establishing triage processes is the second most critical step (detection rules are the first.) Triage is what determines what is accepted as "good" behavior, what needs to be addressed, and what needs to be escalated. After the rules are implemented take some time reviewing the rules that fired. Evaluate the activity that triggered the rule and try out different triage techniques. This is repeated until there is a repeatable triage process for the rules. Continue testing the repeatable triage process to make it more efficient and faster. Look at the false positives and determine if there is a way to identify them sooner in the process? Look at the techniques that require more in-depth analysis and move them to later in the process? The triage process walks a fine line between being as fast as possible and using resources as efficient as possible. Remember, the more time spent on one alarm the less time is spent on others; the less time on one alarm increases malicious activity being missed.
The triage processes are critical and can help prevent encountering the paraphrased comment I opened this post with. We are overwhelmed with data and are not sure what to look at or collect? Unfortunately, when it comes to triage for the most part you are on your own. This was one area I found to be truly lacking in documentation and the few cheat sheets I found didn’t account for the majority of stuff one needs to consider. Determining what triage processes were needed followed by developing those processes and then training others was definitely a tough challenge. It was tougher since it also involved honing those triaging processes to improve their efficiency and speed. Despite the difficulty, focusing on triage enables you to see through the noise and know what to look for.
Metrics – These Are Needed
If someone would had told me two years ago I be reading about security metrics I would had laughed out loud. Two years ago I didn’t see the value metrics offer until my eyes were opened exploring intelligence driven security. Another comment I saw in the SIEM forum was the following quote: “what information should be provided to the management on the daily basis which can justify the purchase of
You Are Alone Standing on the Shoulders of Others
As I embarked on this journey I learned from others who have either dealt with SIEM technology or security monitoring. Initially, I read online what they published and made available to the public. I reached out to a few people about the additional questions I had. I looked around locally and reached out to others who were managing their own security monitoring capabilities. Needless to say, my SIEM journey was successful since I was standing on the shoulders of those who came before me.
However, at the end of the day each organization is different. Others can provide you with advice and share their experience but what is actually encountered is unique. The environment is unique and certain aspects of the use cases are unique as well. Due to this uniqueness, most of the time throughout the year we (my team and myself) were on our own. I mention this as a word of caution to others who may expect a vendor or others to solve their SIEM challenges for them. Others and vendors can help to a certain extent but you and your team are the ones who need to solve the challenges you face. At that point you will find yourself alone standing on the shoulders of others.
Looking Forward to SIEM Year 2
Reflecting back over the past year it has been a demanding challenge. The journey hasn't ended since work still needs to be done. There will always be work to do and new challenges to work through. Hopefully, the experienced I shared will avert someone from stepping on a SIEM land mind resulting in them yelling out for help as they are drowning in the sea of logs and alerts.
Posts
excellent article! I hope to draw on many of your insights for our implementation.
I started reading up on VERIS.. Are you able to share how you capture your incidents in VERIS format? are there tools to do that? VERIS just look like a schema to capture info. I think it will be challenging to get SOC folks writing JSON to record what they are doing!