Finding An Infection Vector After IT Cleaned the System

Wednesday, November 7, 2012 Posted by Corey Harrell
Almost every “CSI” episode begins the same way. An attack took someone’s life, first responders secured the crime scene, and then the main characters show up to start processing a well preserved scene. The technological equivalent of this is the following: an attack compromises a system, first responders image the system’s memory before immediately powering the system down, and then a forensicator shows up to process the scene. This is an ideal situation but life is not always ideal. Actions taken by users and first responders do not always leave a well preserved scene. Anti-virus scans are ran, files are deleted, temporary folders are emptied, programs are installed/removed, system restore points/volume shadow copies are deleted, or the system is left running resulting in data being altered. Only after all these actions is the forensicator consulted and asked to figure out how the compromise occurred. Needless to say conducting the analysis is more difficult and time consuming but it is not impossible. There still may be artifacts left on the system providing clues to what happened.

In order to demonstrate this, I setup a simulation where I purposely infected a Windows XP system and then I mimicked actions someone would take to remove the infection. Afterwards I examined the system to see what artifacts I could find. I wanted to simulate a real life scenario so I performed the following common troubleshooting techniques administrators do when trying to “clean” a system and get it back into working order as quickly as possible.

        • McAfee antivirus scans were conducted and any detections were cleaned
        • All temporary folders’ contents were deleted
        • All prefetch files were deleted
        • All recycle bins were emptied
        • All system restore points were deleted
        • Avast! was installed and a scan was conducted

The actions may be common troubleshooting techniques but I picked them for a different reason. All of those actions potentially destroy vital information that could help determine how an infection occurred. Antivirus software deleting malware not only removes the suspicious files from the system but it also destroys their filesystem metadata. This metadata can narrow down the timeframe to when the infection occurred. Deleting the files in the Windows\Prefetch folder makes it harder to determine what programs executed on the system. Deleting files in the Windows temporary, user profiles’ temporary, and Internet browser folders removes any secondary artifacts –such as exploit artifacts – that can provide additional information about an event. Emptying the recycle bin ensures the files are actually deleted from the system. Windows System Restore Points are an interesting artifact because they contain backup copies of registry hives and Windows system files. Deleting the System Restore Points destroys the historical information they contain. Lastly, installing and running additional anti-virus software makes more changes to the system and potentially alters or deletes information beneficial to an analysis.

Potential Artifacts


The common troubleshooting techniques destroyed some of the information I look for when faced with a malware infected system. Despite those actions there is still data left on a system providing clues about how the infection occurred. Based on the actions taken, four potential areas containing information are:

        1. Host Based Logs
        2. NTFS Artifact
        3. Registry Artifacts
        4. System Timeline

Host Based Logs


Antivirus software typically records installation and other system activities in text logs. The location of these text log files is specific to each antivirus product. One method used to identify the installed antivirus products is the Uninstall Registry key in the Software hive[1]. Afterwards, Google searches on the products can show the where the folders containing logs are located. I reviewed the Uninstall Registry key on the infected system using RegRipper and found the following:

Sun Apr 8 04:49:07 2012 (EDT)
      avast! Free Antivirus v.7.0.1426.0

Sat Apr 7 10:52:52 2012 (EDT)
      McAfee VirusScan Enterprise v.8.6.0

The above information showed there were two installed antivirus products; they were McAfee antivirus and Avast!. The Google searches I performed showed me where both products store their logs on a Windows XP system. In addition to these logs, antivirus products activity may also be recorded in the Windows events logs. I examined both the antivirus and Windows events logs looking for any information about the infection.

The McAfee logs are located in the DesktopProtection folder located under “Documents and Settings\All Users\Application Data\McAfee\”[2]. Two specific logs of interest were the OnAccessScanLog and OnDemandScanLog. The OnAccessScanLog log records information regarding McAfee continuously monitoring a system. The second log – OnDemandScanLog – records information regarding scans conducted against a system. The OnAccessScanLog did not contain any entries for the malware. However, the OnDemandScanLog contained entries outlining what had been deleted, cleaned, or quarantined by McAfee. Below were some entries found:

• 4/8/2012 4:06:46 PM (EDT) Deleted corey C:\SYSTEM VOLUME INFORMATION\_RESTORE{3F806DB1-464B-46B0-B724-4376EC868222}\RP6\A0003833.EXE BackDoor-AMQ(Trojan)

• 4/8/2012 4:06:46 PM (EDT) Deleted corey C:\SYSTEM VOLUME INFORMATION\_RESTORE{3F806DB1-464B-46B0-B724-4376EC868222}\RP6\A0003834.COM BackDoor-AMQ(Trojan)

• 4/8/2012 4:06:49 PM (EDT) Deleted corey C:\WINDOWS\SVCHOST.EXE BackDoor-AMQ(Trojan)

• 4/8/2012 4:06:50 PM (EDT) Deleted corey C:\WINDOWS\MSAGENT\MSQGIX.COM BackDoor-AMQ(Trojan)

• 4/8/2012 4:06:50 PM (EDT) Deleted corey C:\WINDOWS\SYSTEM32\MSMHXY.COM BackDoor-AMQ(Trojan)

The Windows Application Event log - AppEvent.Evt – collaborated the information in the OnDemandScanLog log. There were numerous Event IDs 258 showing McAfee deleted numerous files detected as BackDoor-AMQ (Trojan) on 4/8/2012 4:06:50 PM (EDT). The OnDemandScanLog and Windows Application Event logs revealed that the “corey” user account conducted a McAfee On-demand scan around 4/8/12 4:06 PM (EDT). The scan deleted the files svchost.exe, msqgix.com, and msmhxy.com from different locations in the Windows folder while files with the same extensions were deleted from a System Restore Point. The last piece of information indicated that McAfee detected the files as the BackDoor-AMQ Trojan. I searched McAfee’s website for additional information about BackDoor-AMQ[3] but it wasn’t useful without having access to the actual malware or their hashes.

I proceeded to examine the Avast! logs located in the Log folder under “Documents and Settings\All Users\Application Data\AVAST Software\Avast”. I was looking at the names of the files in the folder when I noticed one named Setup.log. The Setup.log shows the information related to installation of Avast! and the log’s first entry stated “Started: 08.04.2012, 16:46:02”. Later on in the Setup.log I found an entry showing Avast! antivirus was installed at 4/08/12 4:49:06 PM (EDT). I wanted to correlate the information with the Windows event log. The System Event log - SysEvent.Evt – Event ID 7035 showed at 4/08/12 4:48 PM (EDT) the Windows installer started. The first entries for the Avast! in the System Event log occurred one minute later (4:49 PM (EDT)) when numerous Avast! services were started (Event IDs 7035). Something I found interesting was that the Avast! services were started under the context of the local administrator user account.

I continued looking at the other Avast! text log files in the folder and noticed that the logs starting with asw were the ones containing the results of any scans conducted against the system. The Avast! logs did not contain any entries related to files being deleted, cleaned, or quarantined on the test system.

The examination of the host based logs provided some indicators about what occurred on the system. There were numerous suspicious files (svchost.exe, msqgix.com, and msmhxy.com) located within the Windows folder. These files were detected as BackDoor-AMQ Trojans followed by being deleted when the “corey” user account conducted a McAfee antivirus scan. I knew my best bet at finding artifacts showing how the system was infected was to identify when those files first appeared or executed on the system. I turned my attention to the NTFS artifacts to see what information was available.

NTFS Artifact


When antivirus products delete or quarantine files the filesystem may still maintain metadata about those files. One piece of information that can be pulled from this metadata is timestamps; this can be valuable to the analyst, as it helps identify the timeframe of when the system first became infected. One NTFS artifact[4] that may shed light about an infection after a system is cleaned is INDX files.

INDX files contain records for each file within a directory. Each record contains information about the file including: filename, file size, and timestamps[5]. The records in INDX files may remain after files are deleted from a system[6]. This behavior makes INDX files an excellent artifact to provide information about files deleted from the system in attempts to remove malware. An analyst can use Willi Ballenthin’s INDXParse python script to parse these artifacts. On the test system the McAfee antivirus logs indicated the following files were deleted:

        • A0003833.EXE and A0003834.COM from RP6 (System Restore Point)
        • SVCHOST.EXE from the C:\Windows folder
        • MSMHXY.COM from the C:\Windows\System32 folder
        • MSQGIX.COM from the C:\Windows\MSAGENT folder

I was interested to see if either the Windows, System32, or Msagent folders’ INDX files contained any timestamps for the svchost.exe, msmhxy.com, and msqgix.com files. I examined the Windows and System32 folders’ INDX files but it didn’t provide any information about the timestamps I was looking for. However, the Msagent folder’s INDX file contained timestamps for the msqgix.com file.

        • Filename: msqgix.com (slack at 0x5c0)
        • Modified Time: 2012-04-08 4:06:50.390625 PM (EDT)
        • Accessed Time: 2012-04-08 4:06:50.328125 PM (EDT)
        • Changed Time: 2012-04-08 4:06:50.390625 PM (EDT)
        • Created Time: 2004-08-04 12:00:00

Correlating the above timestamps to the information I learned from examining the host based logs shows the timestamps are around the time when McAfee deleted the file. I also checked the Prefetch folder’s INDX file hoping to find information about program execution but it did not provide any useful information about the attack. This time around the INDX files did not provide any new information about how the attack occurred but it is an NTFS artifact to keep in mind with respect to files that may have been deleted.

I still needed to identify the timeframe of when the infection occurred since I didn’t get the information from the host based logs and INDX files. I know when malware executes it may change system configuration settings in the Registry which is where I looked next.

Registry Artifacts


When an administrator takes actions to remove malware from a system not only are important files deleted but many times, those files’ metadata is lost as well. Not having access to the metadata makes it tougher to identify the timeframe of when the system was initially infected. However, if the malware makes changes to the Registry then the Last Write times associated with modified registry keys modify can help fill in some of these gaps. Different methods can be used to locate the information stored in the Registry, from checking common keys used by malware to performing keyword searches. The McAfee log file provided names of three files of interest that were used in a keyword search. The following list provides the results of these searches.

1. C:\Windows\SVCHOST.EXE Search Hit 1
     a. Last Write Time: 2012/04/08 3:06:31 PM (EDT)
     b. Key: HKLM\System\ControlSet001\Services\SharedAccess\Parameters\FirewallPolicy\StandardProfile\AuthorizedApplications\List
     c. Data: C:\Windows\svchost.exe


2. C:\Windows\System32\MSMHXY.COM Search Hit 1
     a. Last Write Time: 2012/04/08 3:06:33 PM (EDT)
     b. Key: HKLM\Software\Microsoft\Active Setup\Installed Components\{42CE4021-DE03-E3CC-EA32-40BB12E6015D}
     c. Name: Stubpath
     d. Data: C:\Windows\System32\MSMHXY.COM


3. C:\Windows\MSAGENT\MSQGIX.COM Search Hit 1
     a. Last Write Time: 2012/04/08 3:06:33 PM (EDT)
     b. Key: HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer\Run
     c. Name: COM Service
     d. Data: C:\Windows\MSAGENT\MSQGIX.COM


4. C:\Windows\MSAGENT\MSQGIX.COM Search Hit 2
     a. Last Write Time: 2012/04/08 3:06:33 PM (EDT)
     b. Key: HCU\ntuser-corey\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer\Run
     c. Name: COM Service
     d. Data: C:\Windows\MSAGENT\MSQGIX.COM

I found the Registry keyword searches enlighten because it showed the changes the malware made when it executed. The earliest timestamp - 2012/04/08 3:06:31 PM (EDT) – is when the firewall configuration was changed to make the svchost.exe file an authorized application. The search hits also identified how the malware remains persistent; the msqgix.com file was added to the Run key in the Software and "corey" user account hives. In the past, I’ve seen when malware infects a system it adds a persistence mechanisms to both the logged on user account NTUSER.DAT and to either the System or Software hives. It appears the malware did this by adding entries to two different Run keys. I thought the “corey” user account was not only involved with running the McAfee antivirus scan but might have been logged on when the infection occurred. Lastly, all the Registry keys Last Write times were 04/08/2012 3:06 PM (EDT) and this provided me with the first indicator to when the infection may have occurred.

I wanted to see if any programs were run by the “corey” user account around the 3:06 PM (EDT). I ran RegRipper with the userassist plug-in against the “corey” user account’s NTUSER.DAT hive which showed the following activity (I adjusted the time from Coordinated Universal Time to Eastern Daylight Time).

Sun Apr 8 3:06:21 2012 PM (EDT)
     UEME_RUNPATH:\\XP-SP3-SHARES\Main_Share\Software\npp.6.0.0.Installer.exe (1)

The Userassist registry key[7] showed that the “corey” user account ran a program called “npp.6.0.0.Installer.exe” at 4/08/12 3:06:21 PM (EDT) from the “Main_Share” network share. 10 seconds later the svchost.exe file was added to the firewalls authorized applications list which was immediately followed by persistence mechanisms being set for the msqgix.com file. I identified the timeframe of interest as 4/08/12 3:06 PM (EDT). Next I needed to see what other activity was occurring on the system so I decided to create a timeline.

System Timeline


System timelines can provide a good deal of context about data found on a system. Timelines can reveal what other activity was occurring on the system around the time when an event (i.e. malware infection) occurred on the system. This other activity can help answer questions with respect to “how did the compromise happen?” This type of question can still be answered even if someone took actions to clean the system and in the process destroyed the best artifacts available. The registry artifacts provided indications that the “corey” user account might have been responsible for the malware infection. As a result, I wanted my timeline to include the Last Write times of every Registry key in the “corey” user account NTUSER.DAT hive to help see what the account was doing and what changes the malware made to the hive. The only other timeline information I wanted initially was the filesystem metadata since it shows activity on the system. My timeline consisting of just filesystem metadata and the “corey” NTUSER.DAT Registry hive metadata showed the following activity occurring around 4/08/2012 3:06 PM (EDT). The table below illustrates some entries from the timeline.


The timeline revealed a great deal of information about the activity on the system around the time the Registry was modified to include references to the malware. Specifically, the “corey” user account accessed a network share named Main_Share on the XP-SP3-Shares computer followed by executing a program named “npp.6.0.0.Installer.exe”.

How the Compromise Occurred


The information I obtained from the host based logs, Windows Registry, and a system timeline was enough to determine how the how the infection occurred.

The “corey” user account launched a program named "npp.6.0.0.Installer.exe" from the Main_Share on the computer XP-SP3-Shares. Within 15 seconds there were multiple indicators in the Registry showing when malware first executed on the system. Those indicators involved three files names (svchost.exe, msqgix.com, and msmhxy.com) that were eventually detected by McAfee antivirus as BackDoor-AMQ (Trojan) prior to being deleted.

The information I obtained from the artifacts left on the system confirmed how I purposely infected the system. I Trojanized a Notepad ++ installer[8] which drops malware when executed[9].

Summary


There are some instances when the “CSI” main characters show up to a crime scene that appears to be spotless. There is no victim, blood spatter, knocked over furniture, or anything else to indicate a crime took place. What happened was someone cleaned the crime scene before the characters arrived. When faced with a clean crime scene they do not throw their hands up in the air and say the cleaning prevents them from investigating what happened. Instead they search the scene looking for clues the cleaning might have missed. Areas get sprayed with Fluorescein and then a UV light reveals where blood existed, cracks in objects are checked for blood, or the drains in showers are checked for any evidence. Digital forensics is no different; when a “cleaned” system is encountered then the examination needs to find any information in the artifacts left on a system.

Sometimes there might not be the original malware samples, their metadata, Prefetch files, temporary files, or historical information. It is still incumbent upon us to find any remaining artifacts whether they are host based log files, NTFS artifacts, registry artifacts, or timelines. There are even additional remaining artifacts I didn’t discuss such as ones showing program execution and file access. The Registry contains more information about program execution including the MUICache[10] and the Shim Cache[11] Registry keys. There are artifacts showing what folders and files were accessed by a user account including: Shellbags Registry keys[12], Windows LNK files[13], or Internet Explorer browser history.

The common troubleshooting techniques administrators do when trying to "clean" a system and get it back in working order makes it more challenging for us to perform our analysis. What artifacts the administrators destroy will be different from system to system. However, it’s up to us to find what articles remain and how they can be pieced together to explain what happened.


References


1. The full path to the registry key is HKLM\Software\ Microsoft\Windows\CurrentVersion\Uninstall

2. McAfee outlines the log file locations in their knowledge base https://kc.mcafee.com/corporate/index?page=content&id=KB51377

3. McAfee write-up for BackDoor-AMQ http://vil.nai.com/vil/content/v_100037.htm

4. Refer to Brian Carrier’s File System Forensic Analysis book for additional information about NTFS and the artifacts available

5. Willi Ballenthin INDXParse script webpage discussing the INDX artifact http://www.williballenthin.com/forensics/indx/index.html

6. Chad Tilbury wrote an excellent article discussing this behavior http://computer-forensics.sans.org/blog/2011/09/20/ntfs-i30-index-attributes-evidence-of-deleted-and-overwritten-files

7. For additional information about the Userassist Registry key see http://forensicartifacts.com/2010/07/userassist/

8. Virustotal scan showed 32/42 antivirus detected as malicious https://www.virustotal.com/file/84824963c0989568674f329e8dd92eb735f27609ef27e6352347288f62bf1d16/analysis/1334516889/

9. The Malwr dynamic analysis http://malwr.com/analysis/0916af77efc95add28e0abdd17b8a64c/

10. Harlan Carvey’s blog post about the MUICache http://windowsir.blogspot.com/2005/12/mystery-of-muicachesolved.html

11. Whitepaper about the forensic significance of the Shim Cache https://blog.mandiant.com/archives/2459

12. Chad Tilbury article about Windows 7 Shellbags http://computer-forensics.sans.org/blog/2011/07/05/shellbags

13. Article "Evidentiary Value of Link Files" http://www.forensicfocus.com/link-file-evidentiary-value



Article Edited By:

Harlan Carvey
Don C. Weber

  1. Cory,

    Great job.

    Anything in the appcompatcache? How about an entry in UserAssist for the npp.6.0.0.Installer.exe?

  2. Harlan,

    Thanks for the comment. Typically looking at the program execution artifacts is one of the first steps I do but I wanted to highlight other artifacts that can contain useful info. Here is some of the information from program execution which falls within the same time frame. I extracted the info with Regrigger so the times are in UTC. To make it into EDT use the -4 offset. Also, I modified some output to make it easier for others to read

    Appcompatcache from System hive

    UNC\XP-SP3-SHARES\Main_Share\Software\npp.6.0.0.Installer.exe
    UpdTime: Sun Apr 8 19:06:21 2012 Z
    Size : 5574272 bytes

    C:\WINDOWS\system32\msmhxy.com
    UpdTime: Sun Apr 8 19:33:59 2012 Z
    Size : 30869 bytes

    C:\WINDOWS\svchost.exe
    UpdTime: Sun Apr 8 19:34:02 2012 Z
    Size : 30869 bytes

    C:\WINDOWS\msagent\msqgix.com
    UpdTime: Sun Apr 8 19:34:10 2012 Z
    Size : 30869 bytes


    MUIcache from corey's ntuser.dat hive

    C:\DOCUME~1\corey\LOCALS~1\Temp\npp.5.9.3.Installer.exe (npp.5.9.3.Installer)
    C:\DOCUME~1\corey\LOCALS~1\Temp\server.exe (server)
    C:\WINDOWS\svchost.exe (svchost)
    C:\WINDOWS\system32\msmhxy.com (msmhxy)
    C:\WINDOWS\msagent\msqgix.com (msqgix)


    userassist from corey's ntuser.dat hive

    Sun Apr 8 19:06:21 2012 Z
    UEME_RUNPATH:\\XP-SP3-SHARES\Main_Share\Software\npp.6.0.0.Installer.exe (1)

  3. Corey,

    All this is very cool. One of the things I try to express to analysts when this sort of thing comes up is that cleaners typically don't clean everything, and knowing where to look for indicators or artifacts can really mean the difference between an exam coming to a complete halt, and finding an answer.

  4. Corey,

    Another excellent blog post, my friend. I really learned a lot from it.

    I just watched a Mandiant webinar from August last night about INDX records and they occurred to me as I started reading this. I was interested to see what you found, and didn't find with regards to them.

    Did you use Log2Timeline or some other means to create your timeline of the ntuser and file system?

    Well done!
    KP

  5. @Ken

    I made this timeline last Spring so I do things differently now. At the time I used Sleuthkit's fls.exe to get the filesystem metadata and regtime.pl to dump the ntuser.dat. Afterwards, I used log2timelime against the regtime.pl and fls.exe output files to convert them into csv. I still do things similar but I'm now using RegRipper to get all timeline data from registry hives

  6. Great, thanks for the info!
    KP

  7. Corey,

    In a lot of ways, I really think that this post goes along with your root cause analysis posts, in that analysts don't go after this sort of thing enough. I don't know why...perhaps they feel it will take too long, or that the amount of effort it takes to determine the root cause/IIV is too much.

    Regardless, it's something that *NEEDS* to be done, as without it, how does an organization defend themselves? Where do you focus your efforts in defense if you don't have the intel that lets you make those decisions effectively?

  8. Excellent post Corey. Not only does this type of thing happen in IR, but across the board of computer forensics. Loved the analogies and summary....

  9. Great write up Corey. Another possible artifact could be found in the network logs. You could use the times you extrapolated in your analysis to cross reference the network logs to find the command and control.

  10. Anonymous

    Hi Corey,

    This is my first time reading your blog and I'd just like to say that your post was awesome.

    I never realized how much information still remains on a system even after AV "cleans" the infection.

    This is also a great real-world scenario and analysis that anyone can immediately incorporate these steps into their own incident response procedures.

    I am new to computer forensics, so out of curiosity, how much time did it take you to perform this analysis?

    Thanks for your post.

  11. @Alexander

    Great point about the network and it's something to keep in mind for when this type of thing occurs for real

  12. @anon

    Thanks for stopping by to check out the blog and leaving a comment. As for how much time, the examination I did for the post is a bad example since I set the scenario up. I recently worked a case similar to this scenario. "Cleaning" attempts destroyed a lot of artifacts but over a span of a week there were reoccurring infections with reoccurring cleaning attempts. That examine took my 3 hours from start to finish. On most cases it takes between 2 to 3 hours; depending on the drive size.

    The 3 hour time is when nothing is automated in my process and I am not using baselines. I am working on automation so I should be a lot faster; my goal is to make it 90 minutes or less.

Post a Comment