Journey Into Incident Response

This post is a quick update about a few things I've been working on over the years.

auto_rip updates

auto_rip is a wrapper script for Harlan Carvey's RegRipper and the script has a few updates. For those unfamiliar with the program please refer to my post Unleashing auto_rip. The script's home has always been on the RegRipper Google Code site but Google dropped support for adding new downloads. As a result, I thought it might be helpful to make newer versions available at different places since Google Code can no longer be used. One of the download locations where the script will be available is Google Drive. The link to the download folder is located here. In addition, along the right side of this blog is a link to the script's download location.

Harlan has put in a ton of work on Regripper and in the spring he released some updates to the program. Inside the downloadable archive he made available a file named updates.txt that outlines all of the work he did. New plug-ins, combined plug-ins, retiring plug-ins, etc.. Needless to say, an outstanding tool is now even better. After Harlan released the updates others asked if I was going to update auto_rip to match. Things have been busy so it took longer than I wanted. However, I finally updated auto_rip to account for the new RegRipper updates.

The latest auto_rip download works with the RegRipper rr_20140414.zip download. All changes are noted at the top of the code. The changes include: adding plug-ins to parse, removing plug-ins no longer supported, adding the malware category (not all malware plug-ins run), and parsing the AMcache.hve registry hive with a new switch (Harlan, awesome job making this plug-in). I also renamed the executed to reflect it is 64bit and won't work on 32bit systems. Harlan, again thanks for all the work you put into maintaining the project.

tr3secure_collection_script

Another script I released is the tr3secure_collection_script. This script automates the collection of volatile and non-volatile data from systems to support incident response activities. For more information about the script refer to my posts: Dual Purpose Volatile Data Collection Script and Tr3Secure Data Collection Script Reloaded. This script was another Google Code causality and had to find a new home (Google Drive again.) The link to the download folder is located here. In addition, along the right side of this blog is a link to the script's download location.

Besides getting a new home there is only one small change in this version. I dropped support for pv.exe since it is no longer downloadable. At some point in the future there may be another update on this front.

Digital Forensic Search Update

I have been keeping track of the various blogs and websites to add to the Digital Forensic Search over the past six months. I finally added the new sites to the index. To access the custom Google you can use this link directly. To see the full list of what is included in the index refer to the post: Introducing the Digital Forensics Search.

I'm writing this post to voice a concern about trainings for incident response. I am painting this picture with a broad stroke. The picture does not apply to every $vendor nor does it apply to every training course. I'm not trying to lump everyone into the same boat. I'm painting with a broad stroke to not single out any specific entity or course but to highlight areas for improvements as well as opportunities for future training offerings. I started seeing this issue a while ago when I was looking at various incident response trainings for people brand new to our field. However, a recent event prompted to me to paint this picture. A picture showing: traditional digital forensic training does not equal incident response training.

Sketching The Picture

As I start to sketch I hope the picture becomes more clear. Our field is one referred to as the Digital Forensic and Incident Response field. Digital forensics and incident response are closely related; some say one is a subset of the other. Both use common tools,  common techniques, and are interested in the same artifacts on systems. Despite the similarities, the two have drastically different objectives and these objectives is what impacts how different the training needs to be for both incident response and digital forensics.

Defining Digital Forensics

Digital forensics has numerous definitions depending on who is the one defining it. The NIST Guide to Integrating Forensic Techniques into Incident Response states the following:

"it is considered the application of science to the identification, collection, examination, and analysis of data while preserving the integrity of the information and maintaining a strict chain of custody for the data"

The guide further explains digital forensics by laying out the process it follows as shown below. The collection includes: identifying, recording, and acquiring data from possible sources while ensuring data preservation. The examination includes: forensically processing the collected data to identify data of interest while ensuring data preservation. The analysis includes: "analyzing the results of the examination, using legally justifiable methods and techniques, to derive useful information that addresses the questions that were impetus for performing the collection and examination." Lastly, the reporting includes: reporting the results of the analysis.


The types of cases where I've personally seen the above digital forensic process used varies from: acceptable use policy violations (internal investigations), financial fraud investigations, civil proceedings (one entity suing another), and divorce proceedings. The types of cases where I heard this process is used but never participated in is criminal proceedings such as murder, robberies, white collar, and child pornography.

Defining Incident Response

Digital forensic techniques are leveraged in the incident response process but the processes are not the same. The phrase incident response is typically used to describe two separate activities organizations perform. The two activities (incident handling and incident response) are defined in the document Defining Incident Management Processes for CSIRTs: A Work in Process. Incident Handling is the activity "that involves all the processes or tasks associated with “handling” events and incidents." Incident Response are the "actions taken to resolve or mitigate an incident, coordinate and disseminate information, and implement follow-up strategies to prevent the incident from happening again."

There are different incident response workflows -such as those listed in Enisa's Good Practice Guide for Incident Management. The NIST Computer Security Incident Handling Guide also outlines an incident response process workflow as shown below. The preparation includes: establishing an incident response capability and preventing incidents. The detection and analysis includes: accurately detecting possible security events, triaging those events to confirm if they are an incident, analyzing the incident, determining its scope, determining how it occurred, and what originated the incident. The containment, eradication, and recovery includes: developing remediation strategy, carrying out the containment strategy, eliminating the components of the incident, and restoring systems to normal operations.


The types of cases where the incident response process is used varies from: malicious network behavior, malicious code, website compromise, unauthorized access, denial of service, or account compromise.

Painting the Sketch with Color

The digital forensic process differs greatly from the incident response process. Digital forensics is the "application of science to the identification, collection, examination, and analysis of data" typically in support of investigations. Incident response on the other hand is to effectively detect, investigate, contain, and remediate security incidents. The training needs of each process is significantly different even though forensic techniques are used in both. To illustrate this point it's necessary to explore some of the concepts in DFIR trainings and show how they are not sufficient for incident response.

The topics listed below are the ones I noted from some entry level DFIR training courses:

     - Recover deleted partitions
     - Introduction to NTFS
     - Deleted data recovery
     - Web browser history
     - Print spooler recovery
     - Collection techniques
     - Windows registry analysis to include: USB device analysis, file/folder access, and program execution
     - Email forensics
     - Windows artifacts to include: jump lists, VSCs, and link files
     - Windows event log analysis

Those topics are all outstanding for a DFIR training. As it relates to digital forensics, these definitely need to be covered due to examiners frequently encountering them on all cases. As it relates to incident response, these techniques and artifacts may be relevant to the security event at hand but there are even more relevant incident response topics that are not covered. In my opinion, these trainings are more meant for those doing digital forensics instead of those doing incident response. This is because the curriculum in these trainings are not sufficient for training people on how to do incident response.  I'll elaborate with two examples.

The incident response work flow consists of: detecting security event, triaging the security event, analyzing the security incident, containing the incident, eradicating the incident and recovering from the incident. Now let's say someone attended a training that covered all of the digital forensic topics listed above. As soon as they return to their organization they are faced with a potential web server compromise. That analyst will not had learned the skills to do the following:

- Detecting the web server attacks in the first place. Entry level DFIR trainings barely mention detection, how to improve detection, and how to leverage different detection techniques.

- Triaging the potential security event. Entry level DFIR trainings are mostly focused on the digital forensic case examples I listed previously. The little incident response cases exposed in the trainings are slated towards malware or advanced threats with very little mention about compromised webservers. 

- Analyzing the web server compromise. Entry level DFIR barely cover web server compromises and almost all are focused on the Windows OS. A good percentage of web servers are Linux based so Windows focused trainings don't add much value in this case.

- Scoping the incident. Practically no DFIR trainings discusses how to identify and extract indicators and how they can be used to scope an incident in an environment.

- Containing the incident. This is not addressed at all in most trainings.

- Eradicating the incident. Again a topic not even addressed.

In essence, that analyst would be incapable of handling the web server compromise even if they attended the DFIR training. Let's explore another type of common security event; multiple systems hit with a piece of malware. That same analyst would be incapable of dealing with this event since they wouldn't learn the skills to do the following:

- Detecting all the systems compromised with malware. Most DFIR trainings are single system focus and don't cover methods to detect all of the systems involved with a security event

- Triaging the event. DFIR trainings lack how one should do forensics remotely over the wire (with both free and paid options) to triage an event. Plus, the trainings don't go into detail about how to extract indicators and use them to detect other compromised systems.

- Analyzing the system(s) impacted with malware. Entry level DFIR trainings don't go into detail about how to perform malware root cause analysis or how to explore the malware to determine its capabilities.

- Scoping, containing, and eradicating the incident. Again, topics not covered

The shortcomings of the available DFIR trainings is not limited to web server compromises or malware incidents. The same could be said for the other types of security events: account compromise, malicious network behavior, and unauthorized access. The reason - in my opinion - is because those DFIR trainings are more geared towards traditional digital forensics than they are for incident response. Case in point, that same analyst could be successful in the following digital forensic cases: acceptable use policy violations (internal investigations), financial fraud investigations, civil proceedings (one entity suing another), and divorce proceedings. This shows that the current DFIR trainings are actually digital forensic trainings with very little incident response.

Framing the picture

The picture has been painted. Digital forensics and incident response are two different processes with different objectives and different trainings needs. The current entry level DFIR trainings are more satisfying the digital forensic needs without even addressing the incident response needs. At this point there is still one outstanding option. Most $vendors have multiple training courses available so that analyst needs to take multiple DFIR courses. Before I pick apart this argument I suggest to the reader to take a hard look at the DFIR trainings available and not even the entry level ones. Again, this does not apply to every $vendor nor does it apply to every training course but how many truly address the needs of incident response. How many really instruct on how to: detect, triage, analyze, and contain security events.

Economics of Incident Response

The reflection about the available DFIR trainings should had shed some light on the lack of choices for those looking for incident response focused trainings. For the sake of an argument, let's say the needs of incident response was addressed but one would have to take numerous courses. To me, this is not a feasible option for most places due to the costs involved.

It's been a well reported fact that in most organizations information security is a very small percentage of the organization's overall budget. Incident response typically falls within information security in organizations so the place where we are starting  is already underfunded with very little money available. Now, it has also been widely reported that within information security most resources are dedicated to prevention with small percentages applied towards detection and response. The small slice of the pie is now even smaller.

That analyst already has the odds stacked against them with most organizations applying very little resources towards incident response. Most of the DFIR trainings range from $3,000 to $5,000 dollars per course. On top of that an organization has to pay travel, lodging, and per diem. Let's say the trainings are on the lower end of $3,000 per course. The travel includes plane ticket and transportation to and from the hotel; let's say this is $1,000. The hotels vary based on location but most DFIR trainings last for five days; let's say the room costs $200 per night for a total of another $1,000. The per diem rate varies on location; let's use the federal per diem rate for upstate New York even though trainings never come up this way. The per diem rate is $110 per day for a total of  $660 (six days with the extra day for travel). The true cost for this analyst to attend a single training is $5,660.

Remember, the slice of the budget pie is already small to begin with. The analyst could justify to the organization to get sent to training for $5,660 to improve their capability to perform incident response. However, for that same analyst to say "for me to have all the skills I need to do incident response then I'll need to attend these three or four trainings at a cost of about $17,000 to $22,000." That's a very very hard sell in most organizations especially if their first investment of $5,660 does not even enable their staff to handle the commonly faced security events. Now this organization may not want to send just one person but multiple to build out their incident response capability. The costs go from about $17,000/$22,000 for one person to $34,000/$44,000 for two people to $68,000/$88,000 for three people. As can be seen, the costs add up quickly and this cost is only for the training. It doesn't include the other required resources such as equipment. The multiple training courses option is not a feasible option for most organizations so they are left with the current training offerings, which don't address incident response.

Don't get me wrong, there are some companies who can afford to follow this training model by sending multiple people to multiple trainings per year. I think these companies are the exception though since most organizations  only have a small piece of a small slice of the budget pie allocated for detection and response.

Wanting a New Incident Response Picture

The picture I painted is not a master piece but it does show that the current traditional digital forensic training does not equal incident response training. This is not a picture I want on my wall nor is it a picture I want to show to those who are brand new to the incident response field. I would rather have a new picture; a picture where an investment of $5,660 provides instant results to an organization wanting to improve their incident response capability. By instantly showing results will actually encourage organizations to invest even more resources into their detection and response capability. A picture where a single training addresses numerous commonly faced security events such as: malware incidents, web server compromises, and an advanced threat compromise. A training that covers how to perform the incident response process (at least detection, triage, analysis, and containment) for each one of those security events. A training that does not regurgitate the high level incident response process stuff - which can be read online - but jumps right in into the practical content showing how to do this work within an enterprise. This is the picture I would prefer; this is the picture I want to show to those new to our field.

Where's the IR in DFIR Training?

I wrote this post to voice a concern I see with the various DFIR trainings for people brand new to our field. A good portion of the current trainings are geared towards digital forensics and they are not incident response trainings. This is an issue organizations are faced with and one I even see within my lab. The way I worked around this issue is also not a suitable option for most organizations who lack having access to a person with the DFIR skillset. We developed an extensive in-house training  to ensure our incident response needs are meet. However, at some point we do incorporate third party training but there are few options I see that will add instant value. The trainings don't address the incident response process.  For other organizations, the digital forensics focused training is the only option left on the table. To send people new to the field to a DFIR training and have them return to their organization capable of doing digital forensics but not the incident response. The capability the organization was trying to improve in the first place.