Journey Into Incident Response

The Overall Digital Forensic Investigation Process post outlined the various phases involved with the investigation process. These phases were: preparation, identification, collection, analysis, reporting, and archiving. There was one issue with this basic overall process that comes to light when an investigation involves a network.

If an investigation is to determine if a person accessed a file on their computer the overall process matches up pretty well. The identification only involves one person, the collection is only a single computer, and the analysis only involves data from one data source which is the computer. The only thing missing is how to organize your evidence so your conclusion can be tested. The previous example is a pretty straight forward investigation but now let's see what happens when a network becomes involved. What if the investigation is to determine if a person accessed a file on a server within an organization? How about if the investigation is to determine if a person accessed a file on a Internet web server? The addition of a network results in the addition of data sources which could potentially store evidence. The data sources can now include servers (proxy servers, Windows domain controllers, or file and print servers), network logs (web server logs, router logs, or firewall logs), or captured network traffic. The basic overall process doesn't cover how to identify these data sources, examine these data sources, how to combine the evidence from different data sources, or how to organize your evidence for analysis.

I noticed the limitation of the basic overall process on my cases which involved multiple subjects, multiple computers, multiple servers, etc. The scenario of a malware infection I have been using made the limitation even worse because this could involve one, 10, 20, or 500 computers across a network. I needed a scalable and repeated process that I could use for the identification, collection, and analysis of information in a networked environment regardless of the type of case (security incident, audit, policy violation, etc.). This is where Dr. Stephenson's End to End Digital Investigation (EEDI) framework comes into play. Not only does EEDI meet the needs I was looking for but it didn't require any significant changes from what I was currently doing because it is just a different way to approach the investigation.

I know I can't do an adequate job of explaining EEDI compared to Dr. Stephenson which is why I am quoting his articles when discussing the framework. If you want to know more about EEDI then I would recommend you read one of his articles. Currently, there is limited information about EEDI on the Internet but I was able to locate a brief article which can be found here.

What is the EEDI

The premise of the framework takes into count that "every digital crime has a source point, a destination point and a path between those two points" (Stephenson, Getting the Whole Picture, 2002, 2003). This means EEDI takes into account the source of the incident, destination of the incident, and all the intermediate devices along the path through the network. EEDI is a "structured method of collecting evidence along the entire path from source to target, using each piece of evidence in that chain to corroborate other evidence (either digitally or traditionally developed) and an approach to presenting the completed chain effectively in court"(Stephenson, Getting the Whole Picture, 2002, 2003).

The EEDI process consists of the following nine activities:

* Collecting evidence
* Analysis of individual events
* Preliminary correlation
* Event normalization
* Event deconfliction
* Second level correlation
* Timeline analysis
* Chain of evidence construction
* Corroboration

Collecting Evidence

EEDI helped me with the issue I was having during the Identification and Collection phases. This issue was scoping an investigation to determine the systems involved and the data sources with potential evidentiary items. I have found the approach of viewing each case as having a source point, destination point, and a path between them to be effective when identifying the scope of an investigation. Take the previous example of a person accessing a file. The source point is the computer the person is using, the destination point is the computer storing the file being accessed, and the path is the network between those two computers. Following this path can help you identify the data sources with potential evidentiary items which needs to be collected.

Analysis of Individual Events

"This analysis step examines isolated events and assesses what value they may have to the overall investigation and how they may tie into each other" (Stephenson, Cyber Investigation, 2009). EEDI is framework to investigate security incidents so this activity's focus is on the examination of each event in a security incident. I had to modify this activity so it would accommodate any type of investigation from security incidents to policy violations to financial audits.

The first change I made was to change the purpose of the activity to be the analysis of individual events or individual case. This slight change enables an investigation to be performed into other types of cases which don't fall into the security incident category. The second change was to incorporate the examination of digital data to locate evidence which may be relevant to the investigation. I decided to organize the examination based on the data sources being examined. Two potential data sources are network logs and a computer's hard drive. This organization allowed me to examine each data source individually. The last change was to incorporate the various examination steps under each data source. The example below shows a few examination steps for examining a computer's hard drive:

* Analysis of individual events (or individual case)
     * System examination
         * Examination of volatile data
         * Hash the files on the system
         * Search for known malware

Basically, the changes I made were to incorporate all of my existing examination steps into this EEDI activity. This means there was hardly any change from the way I was currently performing examinations and the only difference is how I organized any found evidence.

Preliminary correlation

The "first correlation step is to examine the individual events and see how they may correlate into a chain of evidence"(Stephenson, Getting the Whole Picture, 2002, 2003). The "main purpose here is to understand in broad terms what happened, what systems or devices were involved and when the events occurred"(Stephenson, Getting the Whole Picture, 2002, 2003).

The slight change I made in the analysis of individual events trickles down into this activity. All of the evidence located through the examination of the various data sources is correlated into a chain of evidence. The chain of evidence provides an overview of the evidence in your investigation.

Event Normalizing

The definition of normalization is the "combining evidentiary data of the same type from different sources with different vocabularies into a single, integrated terminology that can be used effectively in the correlation process" (Stephenson, Cyber Investigation, 2009). One example of normalization is adjusting the times in order to take into account the time differences between data sources. All of the times should be normalized into a single time. For example, if there were two computers with different times then the time stamps of the evidence from one computer should be adjusted to the time of the other computer.

Event Deconfliction

The definition of deconfliction is the "combining of multiple reportings of the same evidentiary event by the same or different reporting sources, into a single, reported, normalized evidentiary event" (Stephenson, Cyber Investigation, 2009). This activity is required when an item is reported multiple times from the same source. The one example I have come across when this was required involved emails. During an email examination, I will review emails located on the email server, the person's email file, and any backup copies of the person's email file on their computer. Sometimes this results in multiple copies of the same email being found. All of the copies of the email doesn't have to be in the chain of evidence since only one email is required.

Second-Level Correlation

"Second-level correlation is an extension of earlier correlation efforts. However, at this point, views of various events have been refined through normalization or deconfliction" (Stephenson, Cyber Investigation, 2009).

Timeline Analysis

"In this step, normalized and deconflicted events are used to build a timeline using an iterative process that should be updated constantly as the investigation continues to develop new evidence" (Stephenson, Cyber Investigation, 2009).

Chain of Evidence Construction

The evidence in the timeline should be used to form a chain of evidence. "Ideally, each link in the chain, supported by one or more pieces of evidence, will lead to the next link" (Stephenson, Cyber Investigation, 2009). When it is not possible to establish a direct link between evidence a lead can be used to point to the next piece of evidence. "Leads can point us to valid evidence and that valid evidence can, at some point, become the evidence link" (Stephenson, Cyber Investigation, 2009). I briefly touched on this topic in an earlier post titled Broken Chain.

Corroboration

In this step, "we attempt to corroborate each piece of evidence and each event in our chain with other, independent evidence or events" (Stephenson, Cyber Investigation, 2009). This final "evidence chain consists of primary evidence corroborated by additional secondary evidence" (Stephenson, Cyber Investigation, 2009).

To date, the majority of my digital forensic examinations are in support of an investigation being conducted by a group outside of the forensic unit. For example, the human resource department may be conducting an investigation of an employee violating company policy and asks for a forensic analysis to help their investigation. This results in the majority of the corroboration of primary evidence with the secondary evidence being conducted by the persons performing the investigation. However, there is still some secondary evidence which can be collaborated such as information obtained through research.

Conclusion

The EEDI framework combined with the overall digital forensic investigation process provided me with a flexible, scalable, and repeatable investigation process. This process could be used regardless if someone drops off a hard drive asking you to find how it became infected, if an audit team needs assistance investigating a suspected fraud involving multiple people with multiple data sources, or if a malware outbreak is affecting 50 computers on a network. All of these scenarios would require a cyber investigation and I wanted a process which could be used for any type investigation. At this point in my journey I feel that I have found this process.

To see how the cyber investigation methodology looks with the steps for a system examination can be located on the Journey into IR methodology page.



References

As I mentioned previously, one of Dr. Stephenson's articles titled A Comprehensive Approach to Digital Incident Investigation can be located here.

Stephenson, P. (2009). Cyber Investigation. In S. Bosworth, M. Kabay, & E. Whyne, Computer Security Handbook (pp. 55.1 - 55.27). Hoboken: John Wiley & Sons, Inc.

Stephenson, P. (2002, 2003). Getting the Whole Picture A Series of 12 Columns End to End Digital Investigation Appearing in Elsevier Advanced Technology's "Computer Fraud and Security" Publication in 2002 and 2003. Retrieved from Web Site of Peter Stephenson: Document is no longer hosted on Dr. Stephenson's webpage

In order to paint an accurate picture of how I started my journey, my next two posts will be about overall digital forensic (DF) investigation process. Technically these should have been the first two posts but I decided to discuss the examination steps and the testing I did last spring first in order to share information which was relevant to a few discussions at the time. After these two posts are completed then I will finally be caught up to where I currently am in my journey.

The majority of the time when you encounter something new one of the first things you should try to do is understand the overall process. If you want to plant a garden you don't just dig a hole in your yard, throw in some seeds, and hope for the best. If you want to learn how to fish you don't buy fishing equipment at a local store then go to the closest body of water to toss the equipment in. These approaches may result in some of the plants growing or catching a fish after the fishing pole knocked it unconscious but most likely the majority of the time these approaches will fail. The reason for this is because both approaches just tried to wing it instead of first trying to understand the overall process.

What does this have to do with investigating a security incident? When I approached this topic I started by trying to understand the overall DF investigation process prior to the complexities of the investigation such as the various examination steps, tools, techniques, or test systems. My goal was to have a repeatable investigation process which would provide consistent results instead of occasionally being lucky by winging it. To accomplish this goal I started with understanding the different phases, including their purposes, of the DF investigation process. i think the various activities within these phases are critical to understand but my focus in this post is just on the phases.

There are different models outlining the phases of the DF process with three of them being the DFRWS Framework, NIST Guide to Integrating Forensic Techniques into Incident Response, and Building a Digital Forensic Laboratory book. These models also discuss the various activities which can occur within these phases such as case management, evidence management, chain of custody, and documentation.

In 2001, the Digital Forensic Research Workshop (DFRWS) released the Investigation Process for Digital Forensic Science (A Road Map for Digital Forensic Research, 2001). The image below outlines the phases of this investigation process.


In 2006, the National Institute of Standards and Technology (NIST) released the special publication 800-86 Guide to Integrating Forensic Techniques into Incident Response (Kent, Chevalier, Grance, & Dang, 2006). The image below outlines the phases of this investigation process.


In 2009, Elsevier, Inc released the book Building a Digital Forensic Laboratory. This book discussed the phases of the investigation process which is shown below (Jones & Valli, 2009).


As can be seen in the pictures above, there are similarities between all three investigation processes (actually two of the processes are similar to the DFRWS process). The picture below shows the phases of the investigation process I decided use (note: the phases below were created using a combination of the references used in this post, my past experience processing cases, and conversations with a colleague who helped me understand the overall process when I first started in this field).


As you can see the phases above are nothing new and basically just a reorganization of the phases in the models I briefly discussed. The following are brief descriptions about these phases:

The Preparation phase covers all of the activities which would occur before you are working on a case. This would include the activities for preserving evidence and to establish guidelines on how to manage evidence (Jones & Valli, 2009). These guidelines can help ensure evidence is preserved throughout the entire investigation process.  This phase would also cover other activities such as staff training, staff recruitment, tool validation, and quality assurance measures.

The Identification phase is when there's a request for a DF investigation. In my past experience, DF has been more of a service which supports other business processes. This means a request by a customer starts the investigation process. This phase involves understanding the purpose of the request and the scope of the investigation such as type of case, subjects involved, and systems involved.

The Collection phase is when the identification and collection of any items that could be of evidential value occurs (Jones & Valli, 2009). This could include digital content such as hard drives and removable media but it can also include other types of information such as interviews and observations.

The Analysis phase includes the examination and analysis of the information. The examination is to identify evidence in the data which may be relevant to the case while the analysis is to analyze the evidence collected, identified, and extracted to develop a set of conclusions (Stephenson, 2009). The analysis would also include testing those conclusions to ensure they are valid.

The Reporting phase is when the evidence and your conclusion are presented to the person or group requesting the DF investigation.

The Archival phase is the management of the long term storage of the case materials including the evidence once the case has been closed.

My journey has initially focused on the Identification, Collection, and Analysis phases. The scenario I decided to use was a malware infection then I realized the potential complexity of this scenario in a networked environment. It could be one system or 100 systems. Potential sources of evidence could be servers, clients, network logs, or removable media. The infection vector could be email, network shares, or the Internet. I think you can see the picture of this complexity and I wanted to know how to approach this type of investigation during the Identification, Collection, and Analysis phases.

This is where Dr. Stephenson's End to End Digital Investigation (EEDI) framework comes into the picture. My next post will explain why I needed EEDI, how EEDI works, how EEDI can help test your conclusions, and the benefits of EEDI for the investigation process.


References

A Road Map for Digital Forensic Research. (2001, August 7-8). Retrieved from DFRWS 2011[banner]: http://www.dfrws.org/2001/dfrws-rm-final.pdf

Jones, A., & Valli, C. (2009). Building a Digital Forensic Laboratory Establishing and Managing a Successful Facility. Burlington: Elsevier, Inc.

Kent, K., Chevalier, S., Grance, T., & Dang, H. (2006, August). Guide to Integrating Forensic Techniques into Incident Response. Retrieved from National Institute of Standards and Technology: http://csrc.nist.gov/publications/nistpubs/800-86/SP800-86.pdf

Stephenson, P. (2009). Cyber Investigation. In S. Bosworth, M. Kabay, & E. Whyne, Computer Security Handbook (pp. 55.1 - 55.27). Hoboken: John Wiley & Sons, Inc.

The examination of the Infected 2 system didn't complete one of the initial examination steps which was examining the executables of interest. I wanted to explain how I didn't completely answer the question of how the system became infected because this step wasn't completed.

The picture below shows the evidence located from the examination and I concluded the malware identified in memory was the related to the website hosting a malicious PDF, which answered to a certain extent both of my questions 1) is the system infected and 2) how did the system become infected.


I have incorporated Dr. Peter Stephenson's End to End Digital Investigation (EEDI) framework into my investigation process. I will be explaining how I have been using EEDI in my next few posts but for this post I wanted to discuss one of the activities, which is the chain of evidence construction. The evidence located throughout the examination is correlated into a chain of evidence. "The term chain of evidence refers to the chain of events related in some consistent way that describes the incident" (Stephenson, 2009). The description of the incident could explain what happened including how the incident occurred. Each piece of evidence (event) in the chain should link to the next piece of evidence, and when the link does not exist then it should be able to be explained.

The importance of creating the chain of evidence is that the chain can be used to test your theories and/or conclusions. A theory can be tested against the evidence in the chain to see if the theory is based on fact instead of an assumption. Also, this testing can help highlight links between evidence that may require additional collaboration in order to support how the evidence ties together.

To explain how the chain of evidence can be used to test a theory I will use a scenario that every parent will encounter at some point. You walk into a room where your child is and see a disaster. One of the first questions you ask is what happened. The answer you may encounter is probably similar to some of the answers my teenager and toddler say to me when I ask them a question about some disaster. The answer will have some truth to it but will not completely answer the question because it doesn't explain how all of the evidence in the room fits together. Let’s say a window is broken and in the corner of the room is a baseball. Naturally, you ask what happened and the answer given is the window just broke. This starts the “question and answer process” of pointing out how all of the evidence in the room does not link together. The purpose of the questions is to get additional information that helps support your theory of what happened. You may point out the baseball in the corner of the room, the baseball bat in your front yard, or how all of the kids who were playing outside suddenly disappeared. Your goal is to get the answer which explains how all of the evidence links into a chain thereby describing how the disaster occurred.

Now back to why the question (how the Infected 2 system became infected) was not completely answered. Last spring, I based my conclusion that the system became infected as a result of the administrator account visiting a malicious website on the various pieces of evidence. This evidence included: the malware existing in the administrator account's temporary folder, the activity on the system only involved Internet usage, the malicious PDF was downloaded from a website, malware appeared on the system within one second of the malicious PDF appearing on the system, and the malicious PDF targeted a vulnerability that was present on the system.

However, reviewing the evidence in the picture shows there are a few unanswered questions about how the system became infected because not all of the evidence is linked together (or I haven't explained how I think the evidence links together). For example, the examination did not explain where all of the malware came from. The payload of the malicious PDF made calls to additional URLs and does not directly show a call to download any malware.

When my conclusion is tested against the evidence in the picture this test highlights the evidence which is not linked to the next piece of evidence. Take for example the first piece of malware on the system which was ~TM4.tmp. The evidence does not show directly where this malware came from so I cannot be certain this malware was the payload of the malicious PDF or came from the hxxp://googlecounter.cn website. The link between ~TM4.tmp, googlecounter.cn, and the malicious PDF needs additional collaboration. The examination step of examining executables of interest could help explain how the malware fits into the chain of evidence.

The brief examination I am going to describe is for illustration purposes only and by no means am I suggesting this is how executables should be analyzed. I didn't complete this activity last spring but I wanted to show how this step could provide additional information to help answer your questions.

The ~TM4.tmp's Virus Total report was reviewed in order to locate a detection that could be used for further analysis. I decied to use Microsoft's detection of TrojanDownloader:Win32/Bredolab.X. Microsoft’s Malware Protection Center states this malware connects to a remote server to download and execute additional files.

The installation information section on the webpage states the malware “is installed to the Startup folder using a variable file name. It injects itself in the 'svchost.exe' and 'explorer.exe' processes”. The examination of the Infected 2 system located a copy of the ~TM4.tmp file (mgjwin32.exe) in the Startup folder while the examination of the strings shows the explorer.exe and svchost.exe processes made references to the mgjwin32.exe program located in the Startup folder (this isn't reflected in my earlier post). This is useful information that could help explain how the other pieces of malware ended up on the system. However, this information doesn't explain how ~TM4.tmp ended up on the system, which is the link we are trying to establish.

Continuing with the examination of the ~TM4.tmp file, a Google search of the file's MD5 hash was performed. The search result only had one hit and this was an entry in the Malc0de database which is shown below.
 
     * date of entry was 2010-04-08
     * domain was googlecounter.cn/web/load.php
     * IP address was 109.196.143.33
     * Autonomous System Name was VLTELECOM-AS VLineTelecom LLC  
        Moscow, Russia
     * file's MD5 hash is e40b4170d3f9252a4339bb2c2d0db7f2
 
The Malc0de database entry was on 04-08-2010 which was one day after the system was infected and the domain involved was the same one the Administrator user account visited (hxxp://googlecounter.cn). I think the most significant piece of information is that the domain listed (googlecounter.cn/web/load.php) is the same domain in the URL requests in the malicious PDF. This additional collaborating evidence establishes the link between the first piece of malware on the system, malicious PDF, and the suspected website; thereby completing this portion of the chain of evidence. The examination of the executables of interest would continue with the remaining malware in order to complete the chain of evidence.
 
Infected 2 System Examination Conclusion
 
The examination of the Infected 2 system did not completely link all of the evidence together. (Eventually, I want to be able to completely link all of the evidence together) Despite this, the examination was able to answer both of my initial questions to a certain extent. These questions were is the system infected and how did the system become infected. Memory analysis was able to locate suspicious files while uploading those files to Virus Total confirmed them as being malicious. The remaining examination steps combined with timeline analysis was able to determine the system was infected by visiting a website hosting a malicious PDF.
 
The examination was performed on a test system so I didn't have to determine what made the administrator user account access the malicious website. Something would have to deliver the website's URL to the administrator and this delivery mechanism could have been a link in an email, link in instant messaging, or a browser redirect from a compromised website. I think this would have been my next step in answering the question how did the system become infected. This line of thinking is what spawned my next area to explore which will be trying to gain a better understanding of the common attack vectors.
 
 
References
Stephenson, P. (2009). Cyber Investigation. In S. Bosworth, M. Kabay, & E. Whyne, Computer Security Handbook (pp. 55.1 - 55.27). Hoboken: John Wiley & Sons, Inc.