Where's the IR in DFIR Training?
Sunday, August 10, 2014
I'm writing this post to voice a concern about trainings for incident response. I am painting this picture with a broad stroke. The picture does not apply to every $vendor nor does it apply to every training course. I'm not trying to lump everyone into the same boat. I'm painting with a broad stroke to not single out any specific entity or course but to highlight areas for improvements as well as opportunities for future training offerings. I started seeing this issue a while ago when I was looking at various incident response trainings for people brand new to our field. However, a recent event prompted to me to paint this picture. A picture showing: traditional digital forensic training does not equal incident response training.
As I start to sketch I hope the picture becomes more clear. Our field is one referred to as the Digital Forensic and Incident Response field. Digital forensics and incident response are closely related; some say one is a subset of the other. Both use common tools, common techniques, and are interested in the same artifacts on systems. Despite the similarities, the two have drastically different objectives and these objectives is what impacts how different the training needs to be for both incident response and digital forensics.
Digital forensics has numerous definitions depending on who is the one defining it. The NIST Guide to Integrating Forensic Techniques into Incident Response states the following:
"it is considered the application of science to the identification, collection, examination, and analysis of data while preserving the integrity of the information and maintaining a strict chain of custody for the data"
The guide further explains digital forensics by laying out the process it follows as shown below. The collection includes: identifying, recording, and acquiring data from possible sources while ensuring data preservation. The examination includes: forensically processing the collected data to identify data of interest while ensuring data preservation. The analysis includes: "analyzing the results of the examination, using legally justifiable methods and techniques, to derive useful information that addresses the questions that were impetus for performing the collection and examination." Lastly, the reporting includes: reporting the results of the analysis.
The types of cases where I've personally seen the above digital forensic process used varies from: acceptable use policy violations (internal investigations), financial fraud investigations, civil proceedings (one entity suing another), and divorce proceedings. The types of cases where I heard this process is used but never participated in is criminal proceedings such as murder, robberies, white collar, and child pornography.
Digital forensic techniques are leveraged in the incident response process but the processes are not the same. The phrase incident response is typically used to describe two separate activities organizations perform. The two activities (incident handling and incident response) are defined in the document Defining Incident Management Processes for CSIRTs: A Work in Process. Incident Handling is the activity "that involves all the processes or tasks associated with “handling” events and incidents." Incident Response are the "actions taken to resolve or mitigate an incident, coordinate and disseminate information, and implement follow-up strategies to prevent the incident from happening again."
There are different incident response workflows -such as those listed in Enisa's Good Practice Guide for Incident Management. The NIST Computer Security Incident Handling Guide also outlines an incident response process workflow as shown below. The preparation includes: establishing an incident response capability and preventing incidents. The detection and analysis includes: accurately detecting possible security events, triaging those events to confirm if they are an incident, analyzing the incident, determining its scope, determining how it occurred, and what originated the incident. The containment, eradication, and recovery includes: developing remediation strategy, carrying out the containment strategy, eliminating the components of the incident, and restoring systems to normal operations.
The types of cases where the incident response process is used varies from: malicious network behavior, malicious code, website compromise, unauthorized access, denial of service, or account compromise.
The digital forensic process differs greatly from the incident response process. Digital forensics is the "application of science to the identification, collection, examination, and analysis of data" typically in support of investigations. Incident response on the other hand is to effectively detect, investigate, contain, and remediate security incidents. The training needs of each process is significantly different even though forensic techniques are used in both. To illustrate this point it's necessary to explore some of the concepts in DFIR trainings and show how they are not sufficient for incident response.
The topics listed below are the ones I noted from some entry level DFIR training courses:
- Recover deleted partitions
- Introduction to NTFS
- Deleted data recovery
- Web browser history
- Print spooler recovery
- Collection techniques
- Windows registry analysis to include: USB device analysis, file/folder access, and program execution
- Email forensics
- Windows artifacts to include: jump lists, VSCs, and link files
- Windows event log analysis
Those topics are all outstanding for a DFIR training. As it relates to digital forensics, these definitely need to be covered due to examiners frequently encountering them on all cases. As it relates to incident response, these techniques and artifacts may be relevant to the security event at hand but there are even more relevant incident response topics that are not covered. In my opinion, these trainings are more meant for those doing digital forensics instead of those doing incident response. This is because the curriculum in these trainings are not sufficient for training people on how to do incident response. I'll elaborate with two examples.
The incident response work flow consists of: detecting security event, triaging the security event, analyzing the security incident, containing the incident, eradicating the incident and recovering from the incident. Now let's say someone attended a training that covered all of the digital forensic topics listed above. As soon as they return to their organization they are faced with a potential web server compromise. That analyst will not had learned the skills to do the following:
- Detecting the web server attacks in the first place. Entry level DFIR trainings barely mention detection, how to improve detection, and how to leverage different detection techniques.
- Triaging the potential security event. Entry level DFIR trainings are mostly focused on the digital forensic case examples I listed previously. The little incident response cases exposed in the trainings are slated towards malware or advanced threats with very little mention about compromised webservers.
- Analyzing the web server compromise. Entry level DFIR barely cover web server compromises and almost all are focused on the Windows OS. A good percentage of web servers are Linux based so Windows focused trainings don't add much value in this case.
- Scoping the incident. Practically no DFIR trainings discusses how to identify and extract indicators and how they can be used to scope an incident in an environment.
- Containing the incident. This is not addressed at all in most trainings.
- Eradicating the incident. Again a topic not even addressed.
In essence, that analyst would be incapable of handling the web server compromise even if they attended the DFIR training. Let's explore another type of common security event; multiple systems hit with a piece of malware. That same analyst would be incapable of dealing with this event since they wouldn't learn the skills to do the following:
- Detecting all the systems compromised with malware. Most DFIR trainings are single system focus and don't cover methods to detect all of the systems involved with a security event
- Triaging the event. DFIR trainings lack how one should do forensics remotely over the wire (with both free and paid options) to triage an event. Plus, the trainings don't go into detail about how to extract indicators and use them to detect other compromised systems.
- Analyzing the system(s) impacted with malware. Entry level DFIR trainings don't go into detail about how to perform malware root cause analysis or how to explore the malware to determine its capabilities.
- Scoping, containing, and eradicating the incident. Again, topics not covered
The shortcomings of the available DFIR trainings is not limited to web server compromises or malware incidents. The same could be said for the other types of security events: account compromise, malicious network behavior, and unauthorized access. The reason - in my opinion - is because those DFIR trainings are more geared towards traditional digital forensics than they are for incident response. Case in point, that same analyst could be successful in the following digital forensic cases: acceptable use policy violations (internal investigations), financial fraud investigations, civil proceedings (one entity suing another), and divorce proceedings. This shows that the current DFIR trainings are actually digital forensic trainings with very little incident response.
The picture has been painted. Digital forensics and incident response are two different processes with different objectives and different trainings needs. The current entry level DFIR trainings are more satisfying the digital forensic needs without even addressing the incident response needs. At this point there is still one outstanding option. Most $vendors have multiple training courses available so that analyst needs to take multiple DFIR courses. Before I pick apart this argument I suggest to the reader to take a hard look at the DFIR trainings available and not even the entry level ones. Again, this does not apply to every $vendor nor does it apply to every training course but how many truly address the needs of incident response. How many really instruct on how to: detect, triage, analyze, and contain security events.
The reflection about the available DFIR trainings should had shed some light on the lack of choices for those looking for incident response focused trainings. For the sake of an argument, let's say the needs of incident response was addressed but one would have to take numerous courses. To me, this is not a feasible option for most places due to the costs involved.
It's been a well reported fact that in most organizations information security is a very small percentage of the organization's overall budget. Incident response typically falls within information security in organizations so the place where we are starting is already underfunded with very little money available. Now, it has also been widely reported that within information security most resources are dedicated to prevention with small percentages applied towards detection and response. The small slice of the pie is now even smaller.
That analyst already has the odds stacked against them with most organizations applying very little resources towards incident response. Most of the DFIR trainings range from $3,000 to $5,000 dollars per course. On top of that an organization has to pay travel, lodging, and per diem. Let's say the trainings are on the lower end of $3,000 per course. The travel includes plane ticket and transportation to and from the hotel; let's say this is $1,000. The hotels vary based on location but most DFIR trainings last for five days; let's say the room costs $200 per night for a total of another $1,000. The per diem rate varies on location; let's use the federal per diem rate for upstate New York even though trainings never come up this way. The per diem rate is $110 per day for a total of $660 (six days with the extra day for travel). The true cost for this analyst to attend a single training is $5,660.
Remember, the slice of the budget pie is already small to begin with. The analyst could justify to the organization to get sent to training for $5,660 to improve their capability to perform incident response. However, for that same analyst to say "for me to have all the skills I need to do incident response then I'll need to attend these three or four trainings at a cost of about $17,000 to $22,000." That's a very very hard sell in most organizations especially if their first investment of $5,660 does not even enable their staff to handle the commonly faced security events. Now this organization may not want to send just one person but multiple to build out their incident response capability. The costs go from about $17,000/$22,000 for one person to $34,000/$44,000 for two people to $68,000/$88,000 for three people. As can be seen, the costs add up quickly and this cost is only for the training. It doesn't include the other required resources such as equipment. The multiple training courses option is not a feasible option for most organizations so they are left with the current training offerings, which don't address incident response.
Don't get me wrong, there are some companies who can afford to follow this training model by sending multiple people to multiple trainings per year. I think these companies are the exception though since most organizations only have a small piece of a small slice of the budget pie allocated for detection and response.
The picture I painted is not a master piece but it does show that the current traditional digital forensic training does not equal incident response training. This is not a picture I want on my wall nor is it a picture I want to show to those who are brand new to the incident response field. I would rather have a new picture; a picture where an investment of $5,660 provides instant results to an organization wanting to improve their incident response capability. By instantly showing results will actually encourage organizations to invest even more resources into their detection and response capability. A picture where a single training addresses numerous commonly faced security events such as: malware incidents, web server compromises, and an advanced threat compromise. A training that covers how to perform the incident response process (at least detection, triage, analysis, and containment) for each one of those security events. A training that does not regurgitate the high level incident response process stuff - which can be read online - but jumps right in into the practical content showing how to do this work within an enterprise. This is the picture I would prefer; this is the picture I want to show to those new to our field.
I wrote this post to voice a concern I see with the various DFIR trainings for people brand new to our field. A good portion of the current trainings are geared towards digital forensics and they are not incident response trainings. This is an issue organizations are faced with and one I even see within my lab. The way I worked around this issue is also not a suitable option for most organizations who lack having access to a person with the DFIR skillset. We developed an extensive in-house training to ensure our incident response needs are meet. However, at some point we do incorporate third party training but there are few options I see that will add instant value. The trainings don't address the incident response process. For other organizations, the digital forensics focused training is the only option left on the table. To send people new to the field to a DFIR training and have them return to their organization capable of doing digital forensics but not the incident response. The capability the organization was trying to improve in the first place.
Sketching The Picture
As I start to sketch I hope the picture becomes more clear. Our field is one referred to as the Digital Forensic and Incident Response field. Digital forensics and incident response are closely related; some say one is a subset of the other. Both use common tools, common techniques, and are interested in the same artifacts on systems. Despite the similarities, the two have drastically different objectives and these objectives is what impacts how different the training needs to be for both incident response and digital forensics.
Defining Digital Forensics
Digital forensics has numerous definitions depending on who is the one defining it. The NIST Guide to Integrating Forensic Techniques into Incident Response states the following:
"it is considered the application of science to the identification, collection, examination, and analysis of data while preserving the integrity of the information and maintaining a strict chain of custody for the data"
The guide further explains digital forensics by laying out the process it follows as shown below. The collection includes: identifying, recording, and acquiring data from possible sources while ensuring data preservation. The examination includes: forensically processing the collected data to identify data of interest while ensuring data preservation. The analysis includes: "analyzing the results of the examination, using legally justifiable methods and techniques, to derive useful information that addresses the questions that were impetus for performing the collection and examination." Lastly, the reporting includes: reporting the results of the analysis.
The types of cases where I've personally seen the above digital forensic process used varies from: acceptable use policy violations (internal investigations), financial fraud investigations, civil proceedings (one entity suing another), and divorce proceedings. The types of cases where I heard this process is used but never participated in is criminal proceedings such as murder, robberies, white collar, and child pornography.
Defining Incident Response
Digital forensic techniques are leveraged in the incident response process but the processes are not the same. The phrase incident response is typically used to describe two separate activities organizations perform. The two activities (incident handling and incident response) are defined in the document Defining Incident Management Processes for CSIRTs: A Work in Process. Incident Handling is the activity "that involves all the processes or tasks associated with “handling” events and incidents." Incident Response are the "actions taken to resolve or mitigate an incident, coordinate and disseminate information, and implement follow-up strategies to prevent the incident from happening again."
There are different incident response workflows -such as those listed in Enisa's Good Practice Guide for Incident Management. The NIST Computer Security Incident Handling Guide also outlines an incident response process workflow as shown below. The preparation includes: establishing an incident response capability and preventing incidents. The detection and analysis includes: accurately detecting possible security events, triaging those events to confirm if they are an incident, analyzing the incident, determining its scope, determining how it occurred, and what originated the incident. The containment, eradication, and recovery includes: developing remediation strategy, carrying out the containment strategy, eliminating the components of the incident, and restoring systems to normal operations.
The types of cases where the incident response process is used varies from: malicious network behavior, malicious code, website compromise, unauthorized access, denial of service, or account compromise.
Painting the Sketch with Color
The digital forensic process differs greatly from the incident response process. Digital forensics is the "application of science to the identification, collection, examination, and analysis of data" typically in support of investigations. Incident response on the other hand is to effectively detect, investigate, contain, and remediate security incidents. The training needs of each process is significantly different even though forensic techniques are used in both. To illustrate this point it's necessary to explore some of the concepts in DFIR trainings and show how they are not sufficient for incident response.
The topics listed below are the ones I noted from some entry level DFIR training courses:
- Recover deleted partitions
- Introduction to NTFS
- Deleted data recovery
- Web browser history
- Print spooler recovery
- Collection techniques
- Windows registry analysis to include: USB device analysis, file/folder access, and program execution
- Email forensics
- Windows artifacts to include: jump lists, VSCs, and link files
- Windows event log analysis
Those topics are all outstanding for a DFIR training. As it relates to digital forensics, these definitely need to be covered due to examiners frequently encountering them on all cases. As it relates to incident response, these techniques and artifacts may be relevant to the security event at hand but there are even more relevant incident response topics that are not covered. In my opinion, these trainings are more meant for those doing digital forensics instead of those doing incident response. This is because the curriculum in these trainings are not sufficient for training people on how to do incident response. I'll elaborate with two examples.
The incident response work flow consists of: detecting security event, triaging the security event, analyzing the security incident, containing the incident, eradicating the incident and recovering from the incident. Now let's say someone attended a training that covered all of the digital forensic topics listed above. As soon as they return to their organization they are faced with a potential web server compromise. That analyst will not had learned the skills to do the following:
- Detecting the web server attacks in the first place. Entry level DFIR trainings barely mention detection, how to improve detection, and how to leverage different detection techniques.
- Triaging the potential security event. Entry level DFIR trainings are mostly focused on the digital forensic case examples I listed previously. The little incident response cases exposed in the trainings are slated towards malware or advanced threats with very little mention about compromised webservers.
- Analyzing the web server compromise. Entry level DFIR barely cover web server compromises and almost all are focused on the Windows OS. A good percentage of web servers are Linux based so Windows focused trainings don't add much value in this case.
- Scoping the incident. Practically no DFIR trainings discusses how to identify and extract indicators and how they can be used to scope an incident in an environment.
- Containing the incident. This is not addressed at all in most trainings.
- Eradicating the incident. Again a topic not even addressed.
In essence, that analyst would be incapable of handling the web server compromise even if they attended the DFIR training. Let's explore another type of common security event; multiple systems hit with a piece of malware. That same analyst would be incapable of dealing with this event since they wouldn't learn the skills to do the following:
- Detecting all the systems compromised with malware. Most DFIR trainings are single system focus and don't cover methods to detect all of the systems involved with a security event
- Triaging the event. DFIR trainings lack how one should do forensics remotely over the wire (with both free and paid options) to triage an event. Plus, the trainings don't go into detail about how to extract indicators and use them to detect other compromised systems.
- Analyzing the system(s) impacted with malware. Entry level DFIR trainings don't go into detail about how to perform malware root cause analysis or how to explore the malware to determine its capabilities.
- Scoping, containing, and eradicating the incident. Again, topics not covered
The shortcomings of the available DFIR trainings is not limited to web server compromises or malware incidents. The same could be said for the other types of security events: account compromise, malicious network behavior, and unauthorized access. The reason - in my opinion - is because those DFIR trainings are more geared towards traditional digital forensics than they are for incident response. Case in point, that same analyst could be successful in the following digital forensic cases: acceptable use policy violations (internal investigations), financial fraud investigations, civil proceedings (one entity suing another), and divorce proceedings. This shows that the current DFIR trainings are actually digital forensic trainings with very little incident response.
Framing the picture
The picture has been painted. Digital forensics and incident response are two different processes with different objectives and different trainings needs. The current entry level DFIR trainings are more satisfying the digital forensic needs without even addressing the incident response needs. At this point there is still one outstanding option. Most $vendors have multiple training courses available so that analyst needs to take multiple DFIR courses. Before I pick apart this argument I suggest to the reader to take a hard look at the DFIR trainings available and not even the entry level ones. Again, this does not apply to every $vendor nor does it apply to every training course but how many truly address the needs of incident response. How many really instruct on how to: detect, triage, analyze, and contain security events.
Economics of Incident Response
The reflection about the available DFIR trainings should had shed some light on the lack of choices for those looking for incident response focused trainings. For the sake of an argument, let's say the needs of incident response was addressed but one would have to take numerous courses. To me, this is not a feasible option for most places due to the costs involved.
It's been a well reported fact that in most organizations information security is a very small percentage of the organization's overall budget. Incident response typically falls within information security in organizations so the place where we are starting is already underfunded with very little money available. Now, it has also been widely reported that within information security most resources are dedicated to prevention with small percentages applied towards detection and response. The small slice of the pie is now even smaller.
That analyst already has the odds stacked against them with most organizations applying very little resources towards incident response. Most of the DFIR trainings range from $3,000 to $5,000 dollars per course. On top of that an organization has to pay travel, lodging, and per diem. Let's say the trainings are on the lower end of $3,000 per course. The travel includes plane ticket and transportation to and from the hotel; let's say this is $1,000. The hotels vary based on location but most DFIR trainings last for five days; let's say the room costs $200 per night for a total of another $1,000. The per diem rate varies on location; let's use the federal per diem rate for upstate New York even though trainings never come up this way. The per diem rate is $110 per day for a total of $660 (six days with the extra day for travel). The true cost for this analyst to attend a single training is $5,660.
Remember, the slice of the budget pie is already small to begin with. The analyst could justify to the organization to get sent to training for $5,660 to improve their capability to perform incident response. However, for that same analyst to say "for me to have all the skills I need to do incident response then I'll need to attend these three or four trainings at a cost of about $17,000 to $22,000." That's a very very hard sell in most organizations especially if their first investment of $5,660 does not even enable their staff to handle the commonly faced security events. Now this organization may not want to send just one person but multiple to build out their incident response capability. The costs go from about $17,000/$22,000 for one person to $34,000/$44,000 for two people to $68,000/$88,000 for three people. As can be seen, the costs add up quickly and this cost is only for the training. It doesn't include the other required resources such as equipment. The multiple training courses option is not a feasible option for most organizations so they are left with the current training offerings, which don't address incident response.
Don't get me wrong, there are some companies who can afford to follow this training model by sending multiple people to multiple trainings per year. I think these companies are the exception though since most organizations only have a small piece of a small slice of the budget pie allocated for detection and response.
Wanting a New Incident Response Picture
The picture I painted is not a master piece but it does show that the current traditional digital forensic training does not equal incident response training. This is not a picture I want on my wall nor is it a picture I want to show to those who are brand new to the incident response field. I would rather have a new picture; a picture where an investment of $5,660 provides instant results to an organization wanting to improve their incident response capability. By instantly showing results will actually encourage organizations to invest even more resources into their detection and response capability. A picture where a single training addresses numerous commonly faced security events such as: malware incidents, web server compromises, and an advanced threat compromise. A training that covers how to perform the incident response process (at least detection, triage, analysis, and containment) for each one of those security events. A training that does not regurgitate the high level incident response process stuff - which can be read online - but jumps right in into the practical content showing how to do this work within an enterprise. This is the picture I would prefer; this is the picture I want to show to those new to our field.
Where's the IR in DFIR Training?
I wrote this post to voice a concern I see with the various DFIR trainings for people brand new to our field. A good portion of the current trainings are geared towards digital forensics and they are not incident response trainings. This is an issue organizations are faced with and one I even see within my lab. The way I worked around this issue is also not a suitable option for most organizations who lack having access to a person with the DFIR skillset. We developed an extensive in-house training to ensure our incident response needs are meet. However, at some point we do incorporate third party training but there are few options I see that will add instant value. The trainings don't address the incident response process. For other organizations, the digital forensics focused training is the only option left on the table. To send people new to the field to a DFIR training and have them return to their organization capable of doing digital forensics but not the incident response. The capability the organization was trying to improve in the first place.
Labels:
IR
Posts
First, thanks for the time writing such a very informative post, I really enjoyed reading it.
What I know about DF and IR, is based on "If you have a fire in an apartment". Then DF will be to investigate what caused the fire, were IR is to contain and eliminate the fire in that apartment. Correct me if I'm wrong.
Now regarding the courses and why they are more DF than IR; could be categorized to the following (IMHO):
1- Budget
2- Time limitation (course duration)
3- Student Skillset
You've already mentioned #1, but I think the other two are important too. For example, teaching a DF course at a University means that we have time limitations, and we have to gear the course towards a couple of topics to cover and do our best to add some take home labs or labs to be done after the official lecture time. If I do both "lecture and labs" then the time will end quickly without covering lots of info! And the same applies when doing the course for a business cooperate!
Now, the students skillset. If most of the students or those who attend a course never read the requirements, and just jump into your class, either because their company paid for it, or it was a mandatory course to attend. Their skillset is very limited if any found (for some)! There will be a huge difference between a course and another for the same instructor, and the problem was the lack of the student's knowledge about different stuff. I see this on a semester basis, and its very hard to balance!
Hope my English was good to be understood, and correct me if anywhere there I was wrong.
Thanks again...
@B!n@ry
Thanks for leaving a well thought out comment and your english was pretty good.
> If you have a fire in an apartment
This is a great analogy and I agree with your explanation about DF doing the deep dive to see what caused it and IR responding to it. However, I would add a few additional points to the IR activity. IR has to identify where the fire is coming from in order to know how to contain it. Is it the back bedroom, kitchen, or the neighbor? Also, IR needs to identify certain indicators about the fire in order to respond properly and ensure containment. Is the fire just in the kitchen or both the kitchen and living room? Is the fire really contained or after we contained it how can we be sure it stays contained?
> Other two are important
I agree with both of those points and I even have one more. Time limitation is a big one and this is why I suggested a course only covering two or three security events. For a five/six day course I think covering how to do IR for three different events is doable. By covering detection, triage, analysis, and some containment strategies would point people in the right direction. The course material could go into more depth to be a reference after the fact. This type of training would be enable a person to handle a good portion of the security events and demonstrate value to get more resources for IR. I guess if someone wants a IR training spending the entire time on DF training is not meeting the need for IR training.
> I see this on a semester basis
I can see the challenges in an academic environment but my post was more geared towards training to obtain a skillset quickly. What options are there for a person in IT or InfoSec to attain basic IR skills? Going through most DFIR training will give them DF skills but not IR skills.
> I even have one more
This one I didn't put into the post on purpose; the economics of developing and offering a training. The larger the audience the more revenue it can generate. There are more people doing DF than IR. A DFIR course focused on DF will get more attendees wanting to take it and thus more revenue. A IR focused training in theory could generate less revenue due to a smaller audience. However, there are very few IR focused trainings so this is an opportunity for those in providing training to capitalize on it.
Great freaking article, I had been thinking the same especially after attending my first DFIR Summit in Austin. Glad you posted this!
@MrShowtyme
Corey,
Very interesting post.
In a way, this kind of reminds me of the Tweet I saw not so long ago about how Volatility training was "disappointing" to one individual, because it didn't include MacOSX and Linux info...even when the main page for the training said "Windows" over and over again.
My point is two-fold...the first is that most training offerings include a description of what they cover, so it shouldn't be surprising that a lot of the things you mention are not covered in training.
My second point is...how does someone provide all of the training possible? Let's look at your web server example...investigating a Windows system is a popular training topic because there are so many Windows web servers out there.
Maybe a better way to put it is, why does someone need to go to training for EVERYTHING? Is it at all possible that someone develops skills without having to go away to another environment and sit in a classroom for 5 days?
What about extrapolating skill sets? Let's look at the web server example again...if training says to look at the web server logs on a Windows system, wouldn't it be a good idea to also look at the web server logs on a Linux-based web server?
Some other thoughts:
> Detecting the web server attacks in the first place.
Okay, this is where the training becomes very infrastructure-dependent. How does one detect an attack on a web server? One might start by reviewing the web server logs. Well...I've actually been to infrastructures where _THE_ compromised web server had logging disabled. So, how do you review the web server logs? Again, that depends upon your infrastructure...what sort of mechanisms are in-place? How many web servers and what overall volume of logs are we looking at? Is this a web server farm in your organization's DMZ, or is it a single hosted web server in AWS?
> Triaging the potential security event.
This can't be extrapolated from previous training?
> Analyzing the web server compromise.
2/2
Again...isn't this something that can be extrapolated? If the DFIR training you attended for Windows systems suggest reviewing the web server logs, why not extrapolate that to (a) Linux, and (b) the web server that you're using? I don't think that it's economically feasible to teach someone how to analyze a web server compromise, with specific training for each variant...Apache on Windows would be one course, and Apache on Linux another?
- Scoping, containing, and eradicating the incident
This is where the activity can become very infrastructure-dependent.
First, scoping an incident becomes an extrapolation of the DF training that has already taken place. How was the web server compromised? SQL injection? Ah, okay...now, did the intruder move laterally? Well, let's go to the logs....don't have logs because the proper auditing wasn't configured? Well, the training was based on the fact that you DID have the proper auditing configured. Okay, what else can we look at?
Containing and eradicating the incident...what to do about that? Oh, you found a RAT on a couple of systems, so let's block the C2 domain in your proxy. Oh...you don't _HAVE_ a proxy. Okay, the training was based on the fact that you did, because it's just good sense. Okay, so how about enabling DNS logging within in the infrastructure and monitoring that? No, I can't tell you how to enable DNS logging because you can't tell me the version of DNS server that you're employing within your infrastructure...but the documentation that you should have does. Can you block outbound connections to IP addresses in your firewall? How about this...can you block outbound connections for all servers? No? Wow.
My point is that every infrastructure is different...not only technically, but culturally, as well. Windows Event Logs are just that...Windows Event Logs. They follow a fairly consistent format, and you can go online and look up information about them. However, you can give two people the same equipment and goal, and end up with two entirely different network infrastructures...and that doesn't even take into account the culture of the people and organization using that infrastructure. This is why you don't see a lot of this stuff in IR training...someone's going to come to the training and not get from it what _they_ need for _their_ infrastructure. What this means is that you need to develop a base of knowledge and learn how to apply it yourself, to your own infrastructure.
Hi Corey.
I have a different take on this, and it comes from a different background. I have many decades of experience as a sys admin, but I am relatively new to DF. After taking one IR course, I found it relatively easy to set up an effective IR program for my business. It has taken me much longer to set up a half ways decent DF program though. Not as much of my knowledge was transferable.
Looking at most of the attendees at a DFIR conference, your comment about not many IR people makes sense. If you took that comment to a Cisco or Palo Alto conference you would find the crowd saying "We need more DF people. Most of us know IR". For most network and sys admins, IR is just another form of troubleshooting.
Its relatively easy to teach a network admin packet analysis. It's a natural extension of what they already know. Similarly, your server admins are used to going though logs. Let them do it! They may not know exactly what you would be looking for, but they are much more likely to see something and realize "that's not normal".
If you send all four of your staff on the same DFIR training course, you will have great coverage for vacations, but for a very narrow field of expertise. If you send them all on different DFIR courses, you will have a much better chance of detecting, containing, eradicating and recovering from an incident. If they are doing that properly, digital forensics will often be involved.
My best advice is build a team. Very few people can be great at everything. With a good team, though, you get a great variety of strengths and skills.
@Harlan,
Thanks for leaving a great comment with so many things to consider.
> are not covered in training
I was trying to highlight what I thought was missing in the available trainings that are typically recommended for entry level DFIR. I think the topics I mentioned would make for a solid training, whether if it is a brand new course or updated trainings
> how does someone provide all of the training possible
I know everything is improssible; but one or two IR focused training covering detection, triage, and analysis for different types of security events are doable.
> why does someone need to go to training for EVERYTHING
I agree with you that people don't need to go to training to learn anything new. People need to be able to develop their own skills. However, I also find it beneficial to leverage training to fill in any weak areas or gaps. Personal, I found the mix of formal education, training, and self development to be beneifical for my own development so this is the approach I use with others (minus the formal education piece)
> training becomes very infrastructure-dependent
I agree about the infrastructure-dependent and this part will be tricky. On the one hand it can't be too vendor dependent while on the other it has to cover what may be used. I know most data sources in the course may not be available in people's organizations but exposing people to the benefits of what could be available could make them want to change things in their own organizations. In my org I had to figure this out on my own but my networking background helped me know ahead of time what data sources I need.
> how do you review the web server logs?
I keep using web as an example is because over the years web attacks has been mentioned as a common vector organizations are compromised. Thousands of websites are compromsied every year and leveraged by attackers for various reasons. Despite this, there are very little sources available about how to respond to compormised web servers.
> This can't be extrapolated from previous training?
To extrapolate the digital forensic training to IR the person needs context about what they should and should not be doing. A person with zero background in DFIR will have trouble doing this. Plus, triaging a potential security event is a lot different than examining a confirmed incident. The event could be a false positive or a real event; this type of triaging is not covered a lot in training including doing it remotely.
> Windows systems suggest reviewing the web server logs
A web attack looks like a web attack in the logs. However, there are very few trainings that cover how to respond to this type of security event including confirming the vulnerability used.
> with specific training for each variant
It wouldn't need to be for every combination. Maybe just two different scenarios based on two popular recent attacks. If you know how to respond to this event on one type of platform then the same method can be used for others.
> How was the web server compromised? SQL injection ....
What you described and how you walked through your thought process is what I would like to see in a IR focused training. This would expose people knew to IR how to work there way through an event
> This is why you don't see a lot of this stuff in IR training
That makes sense. I just wish there were a few more options that did try to address this; especially to expose those new to IR work.
> What this means is that you need to develop a base of knowledge and learn how to apply it yourself, to your own infrastructure.
This was the path I took but I'm looking for an easier approach for those coming behind me.
@Terry
Thanks for reading and taking the time to leave a comment.
> Not as much of my knowledge was transferable
Similiar to you, I came from a sys admin and networking background. I found that this background applied to both DF and IR. On the DF side, my understanding of Windows networks enabled me to know where data my be located both on the system in question and within a domain. It would had been very difficult for me to know this without this experience. On the IR side, my understanding of networking made things a lot easier about the various logs available, what those logs mean (as it relates to the network services), and how TCP/IP works. I do agree that the background is more transferable to IR than DF; depending on the casework you face that is.
> Looking at most of the attendees at a DFIR conference
Conferences are different and I wouldn't apply this post to that. Even in DFIR conferences the attendees really depend on the conference. For example, most of the SANs DFIR summit attendees know or do IR as well as know DF (most are very good at it). Most of the CEIC attendees know DF but not as many know IR. I think conferences tend to draw certain crowds based on previous year's content
> For most network and sys admins, IR is just another form of troubleshooting.
This is where I've seen problems in the past. Treating a security event like troubleshooting resulted in the security event being treated like a technical issue. The focus was on identfying the issue, getting rid of the issue, and then putting systems back into production. There wasn't much work for doing root cause analysis, understanding the risk, and identifying the vulnerability. The end result is the systems are back up and running but nothing was really understood about what exactly happened. A good example of this is malware infections where systems are re-imaged and then redeployed. Again, this is just what I've seen so it may not apply everywhere.
> Similarly, your server admins are used to going though logs. Let them do it!
This does depend on the type of team you are building. The training I'm looking for is for the core members of the CSIRT who will be doing incident response and handling. How to take a server admin with zero DFIR experience and turn then into somebody who knows IR but has DF skills. Someone who will know detection as well as response; triaging as well as deep dive analysis.
> but for a very narrow field of expertise
I think it depends on how and what type of team you are trying to build. The analogy I'll use is the Marines. In the Marines everyone is trained as a basic rifleman before they are sent off to their speciality school. This way every Marine could stop their jobs, pick up a rifle, and go off to fight. I see building a CSIRT in the same light. Everyone goes through a base incident handling and response training prior to focusing on a certain area. This way if needed every core member can do what's needed to be done to work through an incident. The training I'm referring to in the post is to be used as a part of the base training prior to people focusing on other areas. The hope is to have a versatile team that can adapt to any situation they face; similar to the Marines.
Corey,
My overall point...perhaps not well made...was that based on doing IR for 14+ yrs, the single best way that I've seen to teach someone IR is to NOT seek out trainings, but to provide it through various means within the infrastructure...'brown bag" lunches, OJT, mentoring, etc.
Corey - very relevant post. There are many in this industry that are in similar situations as you described; 'we need IR training, we will send you to DFIR and you WILL come back with everything you need'. This is oftentimes a set up for failure.
I attended the DFIR Summit in June and while I learned a lot about the tools and techniques to use when responding to an event, there was much less in the way of how to run an Incident Response/Incident Management program which I think is just as important. I did not leave disappointed, rather still unprepared for handling incidents in my industry.
I believe the answer lies with a split in the two disiplines. One track for strictlly 'forensicators and responders' (finding the who, what, where and how) and the other track for 'handlers and managers'. (triage, communication, business impact).
Great discussion that should be explored throughout the DFIR community.