Journey Into Incident Response

The previous blog post discussed the examination of volatile data to locate malware running in memory. This type of examination will only find the programs or files that were loaded in memory at the time the image was acquired, which means there is a possibility of not identifying all of the malicious files present on the system. Keeping this in mind, I would complete the remaining examination steps even if malware was found running on the system. This post will continue with the examination of Infected 2 by using the remaining examination steps.

Working through the remaining steps provided me with a better understanding of the artifacts left on the system by an infection. As I was working through the examination, I found myself doing two things. First, I was manually creating a timeline of the evidence located in order to establish a chain of events. For example, the examination of Infected 2's memory image located two pieces of malware, which were 75622830.exe and aaclientt.exe. The examination of these files on the hard disk involved reviewing the files' metadata including their timestamps. These timestamps would start the creation of the timeline and the timeline would be updated when I came across anything else. The second thing I found myself doing was reviewing the other activity on the computer around the same time as the evidence I found. Continuing with the example, I would review the activity on the system around the time the 75622830.exe and aaclient.exe files were created on the system. Not only could reviewing this activity help identify additional malicious software but it could help determine how the files ended up on the computer.

Around this point in my testing I was following numerous discussions about using timelines in forensic investigations. Windows Incident Response was discussing it here, here, and here (these are just three posts but Windows IR has a lot more). Security Ripcord was discussing it here and here. The SANs forensic blog was discussing it here and here. The Win4n6 yahoo group had a few discussions about timelines as well. Needless to say, I used all of the sites I hyperlinked to in order to gain a better understanding of timelines and how to generate them. I tested a range of tools for generating timelines including Harlan Carvey's tools (post has the link to the tools), Security Ripcord's tools, log2timeline, and the Sift workstation with log2timeline installed.

One of the first things I realized was timelines could accomplish both of the things I found myself doing, which was creating a timeline and reviewing the other activity on the system. In addition to this, I also found out timelines can include the majority of the data found using the examination steps. For example, one of the examination steps is to examine the programs ran on the system, and two areas that can be examined are the prefetch files and the userassist key (for a good post about examining these areas check out this post). These are just two of the artifacts that can be included in a timeline in addition to the timestamps of all of the files and folders located on the hard drive. This further opened my eyes to how useful timelines can be to an investigation.

The second part of this post will discuss how timeline analysis was incorporated into the initial examination steps in order to help answer the question of how the system became infected.

The next two blog posts will be answering two questions: 1) is the system infected and 2) how did the system become infected. I prefer to include all of the information at once instead of highlighting a few areas. Therefore, both posts will be long. I promise to try to avoid making this a habit unless if there is good feedback. ;)


My previous post provided a little background information about the images I am referencing. One of the earlier steps to identify an infection would be to search for known malicious code by scanning the computer or image of the computer with various anti-virus products. However, I wanted to find the infection by examining the system’s volatile data so I wouldn’t have to be solely dependent on anti-virus products. I thought about the time a test system was accidently infected with Trojan and the anti-virus product did not have a signature for it. Volatile data examination helped me locate the malware even though my anti-virus was ineffective. Plus I wanted to learn how to examine volatile data, which is also one of the earlier steps.

The books listed in my references section outline a methodology to collect volatile data and some of the tools that can be used. One of the first steps during the collection of volatile data is to acquire an image of the memory before any additional tools are run on the live system to collect additional information. I started off examining only the memory image since this is the first item collected. To date, I have found limited information on the process to use to examine a memory image but there is plenty of documentation about examining volatile data on a live system. I figured I would just use the steps of examining volatile data against the memory image. The following are the examination steps (again the books do an outstanding job of explaining these steps in detail):

     * Review the open ports and network connections
     * Review the running processes
     * Review the system hooks
     * Review the loaded dynamic-link libraries (DLLs)
     * Review the open files
     * Review the loaded drivers
     * Review the strings associated with a process or driver of interest.

Basically, the examination is looking for what does not belong. Unusual ports, processes, files, DLLs, and filenames. I found having a baseline system to reference was useful for some of the steps, such as the steps for reviewing the DLLs and drivers. The tools I used for the collection and examination was Mandiant’s Memoryze and AuditViewer. If you want to read a few blog posts on how to use these tools check out The Digital Standard here then here then here.


The review of the open ports and network connections involve looking for unusual or unfamiliar Internet Protocol addresses or ports as well as taking into consideration the processes involved.
 
The picture below displays a section of the port’s section for Infected 1.

Figure 1 Infected 1 Ports and Network Connections

The activity in the image looks pretty normal with one open port on 135 for Microsoft's RPC and Internet Explorer connecting to the default HTTP port (80) on a remote system. The activity may look pretty normal but Internet Explorer was not visible on the system when the memory image was acquired. This was the first red flag in Infected 1 that needs to be examined further.
 
The second image referenced in this post (Infected 2) appeared to be infected with rogue security software which prevented any new processes from running. This rogue software prevented the memory imaging software from executing so to bypass the rogue software the memory was imaged as soon as the user logged on. This enabled Memoryze to execute before the rogue security software but Memoryze also executed before the network services fully loaded. This resulted in the memory image not containing any information related to network connections or open ports.
 
 
Review the Running Processes
The review of the running processes involve looking for which ones do not belong. For each process the following is some of the information that should be reviewed: process’s executable path, command argument to launch the process, the security context of the process, and the enumerated handle information such as the open files and registry keys.
 
For Infected 1, the review of the open ports and network connections showed an unusual network connection involving Internet Explorer. This will be the first process examined as illustrated below.

Figure 2 Infected 1 Internet Explorer Process

The command argument shows the Internet Explorer executable path is correct but the argument used to launch Internet Explorer included an URL in order which makes Internet Explorer connect to the website when it launches. The last bit of information from the picture shows that Internet Explorer is running in the context of the local administrator account (Security ID ending in 500).
 
The review of the rest of the running processes in Infected 1 identified another questionable process, which is shown in Figure 3. Asr64_ldm.exe is questionable because the program was launched from a temporary folder, which is not a standard location for programs to be run from. This process is also running in the context of the local administrator account similar to the Internet Explorer process.

Figure 3 Infected 1 Asr64_ldm.exe Process

The review of the processes in Infected 2 identified three unusual processes. The first process is illustrated in Figure 4. The svchost.exe process is highlighted in red because the process has an injected DLL, and the rating next to the process was done by the MRI feature. Despite this process being highlighted by Auditviewer, the other characteristic that makes it unusual is the command argument launched a program called aaclienttt.exe instead of svchost.exe. This action is trying to hide the presence of aaclientt.exe by running the program under a different name. If someone were to use the task manager to inspect this process they would only see the name svchost.exe listed in the processes tab.

Figure 4 Infected 2 Aaclientt.exe Process

The review of the open file handles for aaclientt.exe shows the process has a few files open including a .dat file. This is shown below.

Figure 5 Infected 2 Aaclientt.exe Open Files

The second unusual process in Infected 2 is another svchost.exe process. This process also had an injected DLL and a MRI rating. The characteristic of this process making it unusual is the lack of switches in the command argument used to launch the svchost.exe program. The other svchost.exe processes' command arguments use a switches as illustrated in Figure 6.

Figure 6 Infected 2 Svchost.exe Process

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
The last unusual process in Infected 2 is shown in Figure 7. Similar to Figure 3, the unusual characteristic about this program is the path it was started from, which was the applications folder in the All Users profile.

Figure 7 Infected 2 75622830.exe Process

The enumerated handles for this process, 75622830.exe, also revealed something interesting. The task manager was a process as shown in Figure 8. As mentioned previously, Infected 2 was obtained from a system that appeared to be infected with a rogue security application that prevented any new programs from running. Not only did the rogue security application prevent the memory imaging software from running, but at the time of imaging the application also prevented the task manager from starting. The process 75622830.exe appears to be the rogue security application since the task manager is listed as a child process.

Figure 8 Infected 2 75622830.exe with Task Manager Child Process

 
 
Review the System Hooks
I thought a brief definition of system hooking would be helpful before I explain this step. The Wikipedia webpage defines hooking as a range of techniques used to alter or augment the behavior of an operating system or other software components by intercepting function calls, messages or events passed between software components. There are several methods and techniques for hooking, and I found the Hacking Exposed Malware & Rootkit book provided a good explanation of the different types of hooking. Also, openrce.org has a good article about a malware sample that uses hooking.
 
Similar to the previous steps, the review of the hooking involves looking for what doesn’t belong. The picture below shows the System Service Descriptor Table Hooks for the Infected 1 image. The picture shows that two functions are being hooked but the name of the hooking module isn’t displayed. I still have to follow up on what would prevent a name from not displaying in this area.

Figure 9 Infected 1 SSDT Hooks

The first function being hooked is NtEnumeratekey. This function returns information about a subkey of an open registry key. Hooking this function enables the information to be filtered, and this could be used to hide the presence of something in the registry. The second function being hooked is NtFlushinstructioncache. This function appears to be undocumented and it empties the execution cache for specified region of code. I came across an interesting analysis describing this function, which allows a driver to execute additional commands such as hide an injected DLL, obtain list of running processes, and to terminate a current process. (note: this analysis was released in August so I did not have access to this information when I first examined this system but I wanted to include the information)
 
The Infected 2 image did not have any hooks in the three hooking techniques displayed by AuditViewer.
 
 
Review the Loaded Dynamic-Link Libraries
Another definition before I explain this step. Microsoft’s MSDN defines a DLL as an executable file that acts as a shared library of functions, which provides a way for a process to call a function that is not part of its executable code. The review of the loaded DLLs is to determine if any are rogue. This was one of the steps I thought was helpful to compare the infected memory images to a memory image from a known good system. There are a lot of DLLs loaded on a system and I found it hard to determine which ones actually belonged on the system. If I do eventually become a cyber investigator then I will be keeping this tip in mind because I would want a baseline of all of the standard configurations for the computers on the network.
 
I was unable to locate the DLLs that were present in the Infected 1 image during this step but I was able to find them once I completed the review of the strings step. As for the Infected 2 image, there was no rogue DLLs.
 
 
Review the Open Files
The review of the open files on the system can not only show unauthorized files on the system but the review can also show what files a user and/or process was accessing at the time of the memory image. I reviewed the open files in both images but I wasn’t able to identify anything else in addition to what I found using the other examination steps. Plus there wasn’t much time available for additional files to be opened because once the system became infected it was disconnected from the network and imaged.
 
 
Review the Loaded Drivers
The review of the drivers involve looking for unusual or unfamiliar drivers. The review of the memory images found that Infected 2 did not have any unfamiliar drivers while Infected 1 did. If you remember Infected 1 was also the image that had hooked functions (refer to Figure 9). The picture below shows the driver in question, which is the _VOIDd.sys driver. This driver appeared in the list of drivers where the memory image was scanned for driver objects but didn’t appear in the list of drivers from traversing the PsLoadedModuleList. The AuditViewer documentation explains the significance of this but basically the driver is hiding from the operating system because the PsLoadedModuleList contains the list of drivers the operating system can see.

Figure 10 Infected 1 Loaded Drivers

 
 
Review the Strings Associated with a Process or Driver of Interest
Finally we are at the last step, which is to review the strings associated with a process or driver of interest. The strings of the _VOIDd.sys driver found in Infected 1 were examined. The picture below shows an unfamiliar DLL that has a similar name as the driver.

Figure 11 Infected 1 _VOIDd.sys Driver Strings

At this point I decided to try to search the memory image for DLLs starting with the name of “_void”. Using the search feature in AuditViewer (shown below) I searched all of the DLLs on the system and there were about five processes that had a loaded DLL with name starting with _void.

Figure 12 Mandiant's AuditViewer Search Feature

Infected 1 had the instance of Internet Explorer connecting to a remote host so I reviewed the search hits for Internet explorer. There were two unfamiliar DLLs and neither of them was the DLL first found in the driver strings.

Figure 13 Infected 1 Internet Explorer Unfamiliar DLLs

The strings of the 75622830.exe process found in Infected 2 were also examined. The first interesting set of strings is shown in the picture below since this was the error message that appeared when the task manger was blocked from running.

Figure 14 Infected 2 75622830.exe Strings

The next string of interest is probably what would have appeared if I selected the option to remove all of the infections.

Figure 15 Infected 2 75622830.exe Strings

 
 
Conclusion
The examination of the memory images from the infected systems provided me with a few files to examine first. The Infected 1 image had the following: suspicious Internet Explorer process, rogue asr_64_ldm.exe process, hooked functions, rogue _VOIDd.sys driver, and rogue DLLs with name starting with _void. The Infected 2 image had the following: rogue aaclientt.exe process, one file of interest opened by aaclientt.exe, svchost process with an injected DLL, and rogue 75622830.exe process.
 
The initial question of is the system infected requires an additional examination step in order to determine if the files are malicious. This step is to confirm if the file is malicious. The files can be acquired directly from memory or the files can be accessed from the hard drive. I opted for the latter option and submitted the files to VirusTotal for scanning. All of the files were confirmed as being malicious with a detection rate less than 20 percent when I uploaded them in the Spring.

The examination of a memory image is one method that can be used to find an infection on a system, and one additional step can be used to confirm the infection. This was the process I used to answer the question of is the system infected. However, I think the examination of memory is just one step out many that is required to examine an infected system. I am interested in how others approach examining a memory image to locate an infection. Are you doing anything different than what I posted?

My next blog post will explain a few more examination steps to answer the question of how the Infected 2 system became infected. *Hint* I visited a malicious website listed in Contagiodump's March links.



References
Aquilina, J. M., Casey, E., & Malin, C. H. (2008). Malware Forensics Investigating and Analyzing Malicious Code. Burlington: Syngress Publishing, Inc.

Carvey, H. (2009). Windows Forensic Analysis. Burlington: Syngress Publishing, Inc.

Hidden Registry Detection. (2009). Retrieved August 16, 2010, from Rootkit Analytics: http://www.rootkitanalytics.com/userland/Hidden-Registry-Detection.php

Malware links March 2010 . (2010, March 29). Retrieved April 06, 2010, from contagiodump.blogspot.com: http://contagiodump.blogspot.com/2010/03/malware-links-march-2010.html

Mandia, K., Prosise, C., & Pepe, M. (2003). Incident Response & Computer Forensics. Emeryville: McGraw-Hill/Osborne.

NtEnumerateKey. (2010). Retrieved August 16, 2010, from msdn.microsoft.com: http://msdn.microsoft.com/en-us/library/ff556510(v=VS.85).aspx

NtFlushInstructionCache. (2001). Retrieved August 16, 2010, from undocumented.ntinternals.net: http://undocumented.ntinternals.net/UserMode/Undocumented%20Functions/NT%20Objects/Process/NtFlushInstructionCache.html

Initial Examination Steps

Starting out I decided to investigate a single system infected with malware. The books I referenced helped identify the data that can be examined to answer my two questions, which for a single computer involved volatile data, the hard drive, and the various locations in the both of them. Next I was interested in the examination steps I could use to help collect and examine the data. I was already involved in performing digital forensic investigations therefore my focus was on identifying the additional examination steps required to investigate a system infected with malware. For example, I already perform keyword searches, hash analysis, and file signature analysis on my cases so I wanted to identify the additional required steps. The following were the initial steps I used to investigate a single system:

     * Examine the volatile data
     * Hash the files on the system
     * Perform a file signature analysis against the files on the system
     * Examine the files on the system that were identified in volatile data
     * Search for known malware on the system
     * Examine the programs ran on the system
     * Examine the host-based logs
     * Examine the auto-start locations
     * Perform a keyword search
     * Examine any executables of interest

Windows Forensic Analysis and Malware Forensics Investigating and Analyzing Malicious Code do an outstanding job of explaining the majority of those steps. As a result, I am not going to explain the steps in detail but I wanted to post the examination steps that I used to for my initial investigation. Plus, I thought it would help explain why I decided to do certain things like use memory analysis to find the infection. The steps I mentioned above were the initial steps I used last Spring. I have not listed any additional steps because I wanted to present how I approached the examination at the time.

First Challenge

The first issue I encountered on my journey was when I wanted to test the process and examination steps I had learned from researching. This is when I noticed the lack of available test images of compromised systems.

This made it difficult to understand the investigation process because tests could not be conducted against a known image to see if I could duplicate the results. I was hoping to find images of compromised systems similar to challenge files available for the forensic challenges on the Internet. For example, the Honeynet Project has an archive of challenges along with the winners’ solutions. I think these challenges are great learning tools because the challenges can be attempted then the winners’ solutions can be referenced to see what was done correct or wrong. However, I was unable to locate an equivalent for images of compromised systems including images of memory along with the image of the hard disk.

To get around this issue, I resorted to creating my own images of compromised systems. At first I was compromising systems using random malware samples from Offensive Computing. This worked well for trying to find the infection on a system but the infection vector was always an executable being launched by a user account. To find the infection vector I had to use a different method to infect a system in order to simulate how a real attack might look like. This evolved into creating compromising systems by opening suspected malicious emails and/or visiting suspected malicious websites. There was a lot of trial and error but I was able to get enough compromised systems for the testing of answering the question of how did the system become infected.

All of the systems were restarted before imaging the memory in order to remove any potential artifacts of the attack in the volatile data. The following blogs about answering my two initial questions will be referencing images from two systems infected by visiting malicious websites. The images will be referenced as Infected 1 and Infected 2.


References

Aquilina, J. M., Casey, E., & Malin, C. H. (2008). Malware Forensics Investigating and Analyzing Malicious Code. Burlington: Syngress Publishing, Inc.

Carvey, H. (2009). Windows Forensic Analysis. Burlington: Syngress Publishing, Inc.

Mandia, K., Prosise, C., & Pepe, M. (2003). Incident Response & Computer Forensics. Emeryville: McGraw-Hill/Osborne.

I have been following a few discussions in the Yahoo win4n6 group that relate to the research and testing I have been doing. This made me realize I did not explain where I am currently at in my journey. My journey began last winter when I first started to do research on this topic. My sole intention, then and now, is to be able to understand the cyber investigation process. As a way to give back to the community, I thought I was going to write an article or two about my experiences. However, I began questioning if two articles would be sufficient when I read Harlan Carvey's thread in the win4n6 group about sharing information with others in the community.

Last month is when I decided to share the information in blog format. I did not want to start this blog where I am currently at in my journey because it wouldn't include a lot of background information. I decided my initial posts will document how I reached the current point in my journey. This will cover a period from the end of last winter to the present time.

I wanted to present the exact approach I took instead of modifying my blog posts to reflect new information. I am writing these posts in advance to make it easier for me to write them because my life is pretty busy between work, school, three kids (one month old, 3 y/o and 13 y/o), wife... I think you can see the picture. I will be trying to post every four to seven days until I am caught up (it will be about seven to nine posts from my second post). Anyway, I will be placing updates at the top of a few future posts to cover any new information or changes that have occurred on the topic being discussed since I completed the research and/or testing. I will only be using the updates until my blog posts catch up to where I currently am in my journey.

If I had known I was going to be using the blog format then I would have started this blog when I started my research. As they say, hindsight is always 20/20.

I was wondering how does someone go about trying to learn the process of investigating a security incident. Security incidents include a range of different scenarios. For example, a security incident could be an employee violating a company policy, unauthorized access to a database, or a server being breached by an attacker.

Complicating matters, each scenario can have numerous sources of data that can assist with your investigation. To illustrate, take the example of a server being breached by an attacker from the Internet. The devices that could contain potential evidence are all of the devices between the attacker to the breached server. Not only does this cover the attacker’s system and the breached server but it also includes the intermediate devices such as routers, switches, other servers, and firewalls. Plus, each device can have numerous data sources where evidence could be located; the breached server could have evidence stored in memory, hard drive, or on a backup tape somewhere. Lastly, each data source can have various locations where evidence can be located, such as the application logs, registry, or user profiles stored on the server’s hard drive.

Thinking about all of this I needed to figure out where was the best place to start so I turned to research using books and the Internet. My initial focus was to understand the process for investigating security incidents. This included trying to understand the overall forensic process and the data that can help an investigation. Right off the bat I knew I had to use baby steps so I decided to learn how to investigate a single system since I already had an understanding of the forensic process. (I plan on discussing the overall forensic process in future posts). There are different scenarios for a single system such as a hacked server or a compromised client; both of these scenarios can involve numerous ways of how the system was breached. I opted to start with learning how to investigate a single system infected with malware. I thought what I learned in this scenario could be applied to other scenarios. Plus, this is most likely the first scenario I will encounter once I understand the process. However, my eventually goal is to understand how to investigate a network wide incident that includes clients, servers, routers, and firewalls.

An investigation for a breached system would include various questions that have to be answered such as: what occurred, when did the incident occur, how did the incident happen, where did the incident occur on the network, and what can be done to prevent the incident from occurring in the future. Continuing with the baby steps approach I decided to only focus on answering two questions. Is the system infected and how did it become infected? Having established the two questions I needed to answer, the next step was to determine what data can be used to answer those questions and how that data can be collected and examined. As I mentioned previously this is a new area for me so I heavily referenced the books Windows Forensic Analysis, Incident Response and Computer Forensics, and Malware Forensics Investigating and Analyzing Malicious Code.

These books helped identify the data that can be examined to answer the questions, which for a single computer involved volatile data, the hard drive, and the various locations in the both of them. The next few blog posts will cover how I was able to answer both of these questions.

Welcome to Journey into Incident Response.

The purpose of this blog is to share my experiences about learning the process of investigating security incidents from both the management and technical perspectives.

My journey into incident response started with my desire to combine my background in performing security assessments with digital forensics. If you can find and sometimes exploit weaknesses in an information security program; can’t that same knowledge be leverage in investigating security incidents? I thought it could and this blog will discuss the approach I am taking by sharing the research and testing I have been doing on this subject.

A little background about myself; I am currently an IT Specialist employed in the New York State public sector. I have over 9 years experience in the information technology field involving desktop support, Microsoft Windows server administration, and networking. I have over 5 years of experience in information security involving security assessments, vulnerability management, and configuration management. For the past three years I have had the opportunity to perform security assessments and audits against enterprise-wide information security programs. In addition to performing security assessments, I have been involved with conducting digital forensic investigations in support of financial audits and acceptable use policy violations.

The digital forensic community has a wealth of knowledge and is willing to share that information for the benefit of everyone. The sharing includes techniques, tools, findings, and advice. This blog is my effort to share back to the community that is helping me on my journey into the digital forensics and incident response fields.

Enjoy, thank you for visiting, and stay tuned …...