Journey Into Incident Response

Jesse Kornblum’s paper “Exploiting the Rootkit Paradox with Windows Memory Analysis” explains the predicament Rootkits find themselves in. The predicament is:

        1. They want to remain hidden.

        2. They need to run.

Jesse’s paper goes into detail about this paradox so I won’t. The main point is Rootkits want to remain hidden on a system but they need to run on an operating system. If an operating system can find the Rootkit then so can an examiner. The paradox applies to all kinds of malware from data stealers to remote access Trojans to droppers.

I’ve been thinking about the Rootkit Paradox and there is another aspect to it that is important for examiners to understand. The second predicament is:

        1. They need to run.

        2. They want to remain hidden

In this case the “they” in the need to run is a piece of malware while the “they” in the want to remain hidden are Rootkits. Basically, a program needs to run in order to hide a Rootkit on a system. The program could be the Rootkit itself or a different piece of malware such as a dropper or installer. When a program runs on the system it creates artifacts showing it executed. An examiner could then look at those artifacts along with changes made to the file system (files/folders creations and modifications) to determine where the Rootkit or its components are hiding. Again this predicament applies to all kinds of malware.

The Rootkit Paradox is alive and well, and can be leveraged to find malware hiding on a system.

This is an open letter to a person I will never get to meet. We will never exchange greetings nor will I ever know their name or identity. I had to settle for an open letter since I’m unable to give them a proper thank you. Please don’t misunderstand my attempt at thanking them since I’m truly sincere. This is not an attempt at humor nor is it an attempt to condescend them. This is my thank you; thanking them for infecting my system so many years ago.


To Whom It May Concern,

As the holidays are approaching it’s a time of reflection. I’ve been reflecting on my “Journey into Incident Response.” There have been numerous people who helped me along the way; people who influenced me and helped make me into the Digital Forensic and Incident Response (DFIR) practitioner I am today. Reflecting on all these people I can’t help but think how you impacted my journey. You may not even be aware of the impact you had on me, but your actions are what opened my eyes to a whole new world and made me take the first step of my journey.

At the time I was in a position doing two different responsibilities. I was performing vulnerability assessments against organizations, testing networks to find vulnerabilities. At other times I was performing digital forensic examinations to determine if someone was committing fraud. I saw my two responsibilities as separate; as duties that didn’t blend together. My perspective changed when I saw your handy work appear on my computer, which was a rogue security program holding my computer hostage. That one single event is what started me on my journey; my journey of leveraging my offensive security background to improve my DFIR knowledge and skills when investigating security incidents.

I never attempted malware forensics before this, but I wanted to know if I could figure out how you were able to install your malware on my box. I successfully traced your malware to a PDF targeting an Adobe Reader vulnerability that was delivered through a malicious advertisement served up by Yahoo’s website. Some may say it was luck, but I beg to differ. I knew that in order to get a remote code execution, an exploit had to be used. The answer was clear as day once I found the PDF that was created on the system around the same time as the malware.

I know I was not even a blip on the radar screen; maybe just an IP address showing a successful install. You opened my eyes to a whole new area; an area where the offensive side of security meets the investigative side. Where knowing how to attack systems is as beneficial as knowing what artifacts are created by those attacks. How this combination of skills and knowledge can be used to obtain intelligence to help better secure organizations and people. I know I would have come to this conclusion eventually, but your actions helped me see it sooner rather than later.

My passion for information security, my motivation to learn, and my drive to share my knowledge with others took care of the rest. Little did you know you were awakening a sleeping giant. I don’t consider myself a giant but it’s the only words that come to mind when looking at the aftermath of the infection. I dedicated myself to learning and researching how to investigate security incidents. Eventually it evolved into me using my knowledge and skills to help others investigate security incidents. I started my blog as one avenue to share what I have learned. I try to help my colleagues become better DFIR investigators through presentations and sharing my thoughts. I use my skills to help home users become better equipped to secure their computers through education about what holes to close after I remove malware from their systems. I help organizations by providing them with intelligence to improve their security by analyzing the systems impacted by attacks. I am developing a graduate course to help strengthen the ranks of DFIR practitioners capable of investigating malware incidents. I’m even considering authoring a book to reach a larger audience about malware forensics. I truly wanted to show the chain of events to illustrate why it’s as if a sleeping giant was disturbed.

I bet you have never heard a user you infected say thank. I will be the first; the first to say thank you for infecting my system so many years ago. Thank you for opening my eyes to a whole new world, sooner rather than later. Thank you for showing me the first step I needed to take to become your adversary.

Almost every “CSI” episode begins the same way. An attack took someone’s life, first responders secured the crime scene, and then the main characters show up to start processing a well preserved scene. The technological equivalent of this is the following: an attack compromises a system, first responders image the system’s memory before immediately powering the system down, and then a forensicator shows up to process the scene. This is an ideal situation but life is not always ideal. Actions taken by users and first responders do not always leave a well preserved scene. Anti-virus scans are ran, files are deleted, temporary folders are emptied, programs are installed/removed, system restore points/volume shadow copies are deleted, or the system is left running resulting in data being altered. Only after all these actions is the forensicator consulted and asked to figure out how the compromise occurred. Needless to say conducting the analysis is more difficult and time consuming but it is not impossible. There still may be artifacts left on the system providing clues to what happened.

In order to demonstrate this, I setup a simulation where I purposely infected a Windows XP system and then I mimicked actions someone would take to remove the infection. Afterwards I examined the system to see what artifacts I could find. I wanted to simulate a real life scenario so I performed the following common troubleshooting techniques administrators do when trying to “clean” a system and get it back into working order as quickly as possible.

        • McAfee antivirus scans were conducted and any detections were cleaned
        • All temporary folders’ contents were deleted
        • All prefetch files were deleted
        • All recycle bins were emptied
        • All system restore points were deleted
        • Avast! was installed and a scan was conducted

The actions may be common troubleshooting techniques but I picked them for a different reason. All of those actions potentially destroy vital information that could help determine how an infection occurred. Antivirus software deleting malware not only removes the suspicious files from the system but it also destroys their filesystem metadata. This metadata can narrow down the timeframe to when the infection occurred. Deleting the files in the Windows\Prefetch folder makes it harder to determine what programs executed on the system. Deleting files in the Windows temporary, user profiles’ temporary, and Internet browser folders removes any secondary artifacts –such as exploit artifacts – that can provide additional information about an event. Emptying the recycle bin ensures the files are actually deleted from the system. Windows System Restore Points are an interesting artifact because they contain backup copies of registry hives and Windows system files. Deleting the System Restore Points destroys the historical information they contain. Lastly, installing and running additional anti-virus software makes more changes to the system and potentially alters or deletes information beneficial to an analysis.

Potential Artifacts

The common troubleshooting techniques destroyed some of the information I look for when faced with a malware infected system. Despite those actions there is still data left on a system providing clues about how the infection occurred. Based on the actions taken, four potential areas containing information are:

        1. Host Based Logs
        2. NTFS Artifact
        3. Registry Artifacts
        4. System Timeline

Host Based Logs

Antivirus software typically records installation and other system activities in text logs. The location of these text log files is specific to each antivirus product. One method used to identify the installed antivirus products is the Uninstall Registry key in the Software hive[1]. Afterwards, Google searches on the products can show the where the folders containing logs are located. I reviewed the Uninstall Registry key on the infected system using RegRipper and found the following:

Sun Apr 8 04:49:07 2012 (EDT)
      avast! Free Antivirus v.7.0.1426.0

Sat Apr 7 10:52:52 2012 (EDT)
      McAfee VirusScan Enterprise v.8.6.0

The above information showed there were two installed antivirus products; they were McAfee antivirus and Avast!. The Google searches I performed showed me where both products store their logs on a Windows XP system. In addition to these logs, antivirus products activity may also be recorded in the Windows events logs. I examined both the antivirus and Windows events logs looking for any information about the infection.

The McAfee logs are located in the DesktopProtection folder located under “Documents and Settings\All Users\Application Data\McAfee\”[2]. Two specific logs of interest were the OnAccessScanLog and OnDemandScanLog. The OnAccessScanLog log records information regarding McAfee continuously monitoring a system. The second log – OnDemandScanLog – records information regarding scans conducted against a system. The OnAccessScanLog did not contain any entries for the malware. However, the OnDemandScanLog contained entries outlining what had been deleted, cleaned, or quarantined by McAfee. Below were some entries found:

• 4/8/2012 4:06:46 PM (EDT) Deleted corey C:\SYSTEM VOLUME INFORMATION\_RESTORE{3F806DB1-464B-46B0-B724-4376EC868222}\RP6\A0003833.EXE BackDoor-AMQ(Trojan)

• 4/8/2012 4:06:46 PM (EDT) Deleted corey C:\SYSTEM VOLUME INFORMATION\_RESTORE{3F806DB1-464B-46B0-B724-4376EC868222}\RP6\A0003834.COM BackDoor-AMQ(Trojan)

• 4/8/2012 4:06:49 PM (EDT) Deleted corey C:\WINDOWS\SVCHOST.EXE BackDoor-AMQ(Trojan)

• 4/8/2012 4:06:50 PM (EDT) Deleted corey C:\WINDOWS\MSAGENT\MSQGIX.COM BackDoor-AMQ(Trojan)

• 4/8/2012 4:06:50 PM (EDT) Deleted corey C:\WINDOWS\SYSTEM32\MSMHXY.COM BackDoor-AMQ(Trojan)

The Windows Application Event log - AppEvent.Evt – collaborated the information in the OnDemandScanLog log. There were numerous Event IDs 258 showing McAfee deleted numerous files detected as BackDoor-AMQ (Trojan) on 4/8/2012 4:06:50 PM (EDT). The OnDemandScanLog and Windows Application Event logs revealed that the “corey” user account conducted a McAfee On-demand scan around 4/8/12 4:06 PM (EDT). The scan deleted the files svchost.exe, msqgix.com, and msmhxy.com from different locations in the Windows folder while files with the same extensions were deleted from a System Restore Point. The last piece of information indicated that McAfee detected the files as the BackDoor-AMQ Trojan. I searched McAfee’s website for additional information about BackDoor-AMQ[3] but it wasn’t useful without having access to the actual malware or their hashes.

I proceeded to examine the Avast! logs located in the Log folder under “Documents and Settings\All Users\Application Data\AVAST Software\Avast”. I was looking at the names of the files in the folder when I noticed one named Setup.log. The Setup.log shows the information related to installation of Avast! and the log’s first entry stated “Started: 08.04.2012, 16:46:02”. Later on in the Setup.log I found an entry showing Avast! antivirus was installed at 4/08/12 4:49:06 PM (EDT). I wanted to correlate the information with the Windows event log. The System Event log - SysEvent.Evt – Event ID 7035 showed at 4/08/12 4:48 PM (EDT) the Windows installer started. The first entries for the Avast! in the System Event log occurred one minute later (4:49 PM (EDT)) when numerous Avast! services were started (Event IDs 7035). Something I found interesting was that the Avast! services were started under the context of the local administrator user account.

I continued looking at the other Avast! text log files in the folder and noticed that the logs starting with asw were the ones containing the results of any scans conducted against the system. The Avast! logs did not contain any entries related to files being deleted, cleaned, or quarantined on the test system.

The examination of the host based logs provided some indicators about what occurred on the system. There were numerous suspicious files (svchost.exe, msqgix.com, and msmhxy.com) located within the Windows folder. These files were detected as BackDoor-AMQ Trojans followed by being deleted when the “corey” user account conducted a McAfee antivirus scan. I knew my best bet at finding artifacts showing how the system was infected was to identify when those files first appeared or executed on the system. I turned my attention to the NTFS artifacts to see what information was available.

NTFS Artifact

When antivirus products delete or quarantine files the filesystem may still maintain metadata about those files. One piece of information that can be pulled from this metadata is timestamps; this can be valuable to the analyst, as it helps identify the timeframe of when the system first became infected. One NTFS artifact[4] that may shed light about an infection after a system is cleaned is INDX files.

INDX files contain records for each file within a directory. Each record contains information about the file including: filename, file size, and timestamps[5]. The records in INDX files may remain after files are deleted from a system[6]. This behavior makes INDX files an excellent artifact to provide information about files deleted from the system in attempts to remove malware. An analyst can use Willi Ballenthin’s INDXParse python script to parse these artifacts. On the test system the McAfee antivirus logs indicated the following files were deleted:

        • A0003833.EXE and A0003834.COM from RP6 (System Restore Point)
        • SVCHOST.EXE from the C:\Windows folder
        • MSMHXY.COM from the C:\Windows\System32 folder
        • MSQGIX.COM from the C:\Windows\MSAGENT folder

I was interested to see if either the Windows, System32, or Msagent folders’ INDX files contained any timestamps for the svchost.exe, msmhxy.com, and msqgix.com files. I examined the Windows and System32 folders’ INDX files but it didn’t provide any information about the timestamps I was looking for. However, the Msagent folder’s INDX file contained timestamps for the msqgix.com file.

        • Filename: msqgix.com (slack at 0x5c0)
        • Modified Time: 2012-04-08 4:06:50.390625 PM (EDT)
        • Accessed Time: 2012-04-08 4:06:50.328125 PM (EDT)
        • Changed Time: 2012-04-08 4:06:50.390625 PM (EDT)
        • Created Time: 2004-08-04 12:00:00

Correlating the above timestamps to the information I learned from examining the host based logs shows the timestamps are around the time when McAfee deleted the file. I also checked the Prefetch folder’s INDX file hoping to find information about program execution but it did not provide any useful information about the attack. This time around the INDX files did not provide any new information about how the attack occurred but it is an NTFS artifact to keep in mind with respect to files that may have been deleted.

I still needed to identify the timeframe of when the infection occurred since I didn’t get the information from the host based logs and INDX files. I know when malware executes it may change system configuration settings in the Registry which is where I looked next.

Registry Artifacts

When an administrator takes actions to remove malware from a system not only are important files deleted but many times, those files’ metadata is lost as well. Not having access to the metadata makes it tougher to identify the timeframe of when the system was initially infected. However, if the malware makes changes to the Registry then the Last Write times associated with modified registry keys modify can help fill in some of these gaps. Different methods can be used to locate the information stored in the Registry, from checking common keys used by malware to performing keyword searches. The McAfee log file provided names of three files of interest that were used in a keyword search. The following list provides the results of these searches.

1. C:\Windows\SVCHOST.EXE Search Hit 1
     a. Last Write Time: 2012/04/08 3:06:31 PM (EDT)
     b. Key: HKLM\System\ControlSet001\Services\SharedAccess\Parameters\FirewallPolicy\StandardProfile\AuthorizedApplications\List
     c. Data: C:\Windows\svchost.exe


2. C:\Windows\System32\MSMHXY.COM Search Hit 1
     a. Last Write Time: 2012/04/08 3:06:33 PM (EDT)
     b. Key: HKLM\Software\Microsoft\Active Setup\Installed Components\{42CE4021-DE03-E3CC-EA32-40BB12E6015D}
     c. Name: Stubpath
     d. Data: C:\Windows\System32\MSMHXY.COM


3. C:\Windows\MSAGENT\MSQGIX.COM Search Hit 1
     a. Last Write Time: 2012/04/08 3:06:33 PM (EDT)
     b. Key: HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer\Run
     c. Name: COM Service
     d. Data: C:\Windows\MSAGENT\MSQGIX.COM


4. C:\Windows\MSAGENT\MSQGIX.COM Search Hit 2
     a. Last Write Time: 2012/04/08 3:06:33 PM (EDT)
     b. Key: HCU\ntuser-corey\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer\Run
     c. Name: COM Service
     d. Data: C:\Windows\MSAGENT\MSQGIX.COM

I found the Registry keyword searches enlighten because it showed the changes the malware made when it executed. The earliest timestamp - 2012/04/08 3:06:31 PM (EDT) – is when the firewall configuration was changed to make the svchost.exe file an authorized application. The search hits also identified how the malware remains persistent; the msqgix.com file was added to the Run key in the Software and "corey" user account hives. In the past, I’ve seen when malware infects a system it adds a persistence mechanisms to both the logged on user account NTUSER.DAT and to either the System or Software hives. It appears the malware did this by adding entries to two different Run keys. I thought the “corey” user account was not only involved with running the McAfee antivirus scan but might have been logged on when the infection occurred. Lastly, all the Registry keys Last Write times were 04/08/2012 3:06 PM (EDT) and this provided me with the first indicator to when the infection may have occurred.

I wanted to see if any programs were run by the “corey” user account around the 3:06 PM (EDT). I ran RegRipper with the userassist plug-in against the “corey” user account’s NTUSER.DAT hive which showed the following activity (I adjusted the time from Coordinated Universal Time to Eastern Daylight Time).

Sun Apr 8 3:06:21 2012 PM (EDT)
     UEME_RUNPATH:\\XP-SP3-SHARES\Main_Share\Software\npp.6.0.0.Installer.exe (1)

The Userassist registry key[7] showed that the “corey” user account ran a program called “npp.6.0.0.Installer.exe” at 4/08/12 3:06:21 PM (EDT) from the “Main_Share” network share. 10 seconds later the svchost.exe file was added to the firewalls authorized applications list which was immediately followed by persistence mechanisms being set for the msqgix.com file. I identified the timeframe of interest as 4/08/12 3:06 PM (EDT). Next I needed to see what other activity was occurring on the system so I decided to create a timeline.

System Timeline

System timelines can provide a good deal of context about data found on a system. Timelines can reveal what other activity was occurring on the system around the time when an event (i.e. malware infection) occurred on the system. This other activity can help answer questions with respect to “how did the compromise happen?” This type of question can still be answered even if someone took actions to clean the system and in the process destroyed the best artifacts available. The registry artifacts provided indications that the “corey” user account might have been responsible for the malware infection. As a result, I wanted my timeline to include the Last Write times of every Registry key in the “corey” user account NTUSER.DAT hive to help see what the account was doing and what changes the malware made to the hive. The only other timeline information I wanted initially was the filesystem metadata since it shows activity on the system. My timeline consisting of just filesystem metadata and the “corey” NTUSER.DAT Registry hive metadata showed the following activity occurring around 4/08/2012 3:06 PM (EDT). The table below illustrates some entries from the timeline.


The timeline revealed a great deal of information about the activity on the system around the time the Registry was modified to include references to the malware. Specifically, the “corey” user account accessed a network share named Main_Share on the XP-SP3-Shares computer followed by executing a program named “npp.6.0.0.Installer.exe”.

How the Compromise Occurred

The information I obtained from the host based logs, Windows Registry, and a system timeline was enough to determine how the how the infection occurred.

The “corey” user account launched a program named "npp.6.0.0.Installer.exe" from the Main_Share on the computer XP-SP3-Shares. Within 15 seconds there were multiple indicators in the Registry showing when malware first executed on the system. Those indicators involved three files names (svchost.exe, msqgix.com, and msmhxy.com) that were eventually detected by McAfee antivirus as BackDoor-AMQ (Trojan) prior to being deleted.

The information I obtained from the artifacts left on the system confirmed how I purposely infected the system. I Trojanized a Notepad ++ installer[8] which drops malware when executed[9].

Summary

There are some instances when the “CSI” main characters show up to a crime scene that appears to be spotless. There is no victim, blood spatter, knocked over furniture, or anything else to indicate a crime took place. What happened was someone cleaned the crime scene before the characters arrived. When faced with a clean crime scene they do not throw their hands up in the air and say the cleaning prevents them from investigating what happened. Instead they search the scene looking for clues the cleaning might have missed. Areas get sprayed with Fluorescein and then a UV light reveals where blood existed, cracks in objects are checked for blood, or the drains in showers are checked for any evidence. Digital forensics is no different; when a “cleaned” system is encountered then the examination needs to find any information in the artifacts left on a system.

Sometimes there might not be the original malware samples, their metadata, Prefetch files, temporary files, or historical information. It is still incumbent upon us to find any remaining artifacts whether they are host based log files, NTFS artifacts, registry artifacts, or timelines. There are even additional remaining artifacts I didn’t discuss such as ones showing program execution and file access. The Registry contains more information about program execution including the MUICache[10] and the Shim Cache[11] Registry keys. There are artifacts showing what folders and files were accessed by a user account including: Shellbags Registry keys[12], Windows LNK files[13], or Internet Explorer browser history.

The common troubleshooting techniques administrators do when trying to "clean" a system and get it back in working order makes it more challenging for us to perform our analysis. What artifacts the administrators destroy will be different from system to system. However, it’s up to us to find what articles remain and how they can be pieced together to explain what happened.

References

1. The full path to the registry key is HKLM\Software\ Microsoft\Windows\CurrentVersion\Uninstall

2. McAfee outlines the log file locations in their knowledge base https://kc.mcafee.com/corporate/index?page=content&id=KB51377

3. McAfee write-up for BackDoor-AMQ http://vil.nai.com/vil/content/v_100037.htm

4. Refer to Brian Carrier’s File System Forensic Analysis book for additional information about NTFS and the artifacts available

5. Willi Ballenthin INDXParse script webpage discussing the INDX artifact http://www.williballenthin.com/forensics/indx/index.html

6. Chad Tilbury wrote an excellent article discussing this behavior http://computer-forensics.sans.org/blog/2011/09/20/ntfs-i30-index-attributes-evidence-of-deleted-and-overwritten-files

7. For additional information about the Userassist Registry key see http://forensicartifacts.com/2010/07/userassist/

8. Virustotal scan showed 32/42 antivirus detected as malicious https://www.virustotal.com/file/84824963c0989568674f329e8dd92eb735f27609ef27e6352347288f62bf1d16/analysis/1334516889/

9. The Malwr dynamic analysis http://malwr.com/analysis/0916af77efc95add28e0abdd17b8a64c/

10. Harlan Carvey’s blog post about the MUICache http://windowsir.blogspot.com/2005/12/mystery-of-muicachesolved.html

11. Whitepaper about the forensic significance of the Shim Cache https://blog.mandiant.com/archives/2459

12. Chad Tilbury article about Windows 7 Shellbags http://computer-forensics.sans.org/blog/2011/07/05/shellbags

13. Article "Evidentiary Value of Link Files" http://www.forensicfocus.com/link-file-evidentiary-value



Article Edited By:

Harlan Carvey
Don C. Weber