Journey Into Incident Response

I normally wait until my blog’s anniversary to post about the direction I want to take in the upcoming year. However, there has been a perfect storm brewing over at jIIr and the eye of the storm would have passed if I waited until my anniversary. As the New Year approaches I’m looking ahead to see the direction this blog will take. jIIr has always been a platform for me to share my thoughts and research, and this will not change. In the upcoming year I foresee some changes to the content I post due to the perfect storm that is occurring. The storm I keep referring to is the convergence of different areas in my life revolving around malware. In the upcoming year I’ll be involved in: authoring a Malware Forensic book, developing a Malware Analysis course, and taking on additional malware responsibilities with my employer.

People and organizations from all walks of life are combating malware on a daily basis. The majority of the security defensive controls are ineffective at stopping the malware since attackers are bypassing them with ease. Throwing additional security controls at the issue is not the answer. What needs to be improved is the response to the malware issue; where the systems impacted are actually examined. Examining systems is one way to obtain intelligence that could be used to improve security. The question then becomes what resources are available to those wanting to take the step to start examining malware infected systems. Do those resources outline not only the process to follow but the tools to use, artifacts to look for, artifacts meaning, and provide sample data to practice on. Well, I have not been able to locate a resource to my liking addressing malware examinations. So I decided to create one and what I just describe is a glimpse of the book I’m starting at some point next year.

My book is on hold because my current focus is on another endeavor. I am developing the Malware Analysis course for Champlain College’s Master of Science in Digital Forensic Science program. The course will cover a range of topics from malware fundamentals to memory forensics to malware forensics to malware analysis (reversing). It is an honor to be a part of Champlain College and I’m looking forward to do everything I can to put together a great course for an outstanding program.

Lastly, I am starting to take on additional malware responsibilities with my employer. jIIr has always been a personal blog and I never discuss the work I do for my employer. I will continue to have this boundary so I won’t elaborate on what my responsibilities are besides hinting at the fact malware will take up more of my time at my job.

This malware convergence means I will be living and breathing malware everyday and all day for the upcoming year. The little personal time I may have for research is going to be spent on different aspects of the malware issue. As a result, jIIr is going to be more focused on topics related to malware. I will try my best to mix in other content about DFIR or security but to be honest I don’t know if I will have the time. The perfect storm has arrived and I hope you stick around to ride it out.


Merry Christmas and Happy New Year.

This past week I was reading a paper about the ZeroAccess Trojan when a section about a clever data hiding technique caught my eye. The paper was Sophos’s The ZeroAccess Botnet: Mining and Fraud for Massive Financial Gain and I’m referring to the Services.exe section which stated (on page 6):

"If installed on Windows Vista or higher ZeroAccess will attempt to patch the Windows file services.exe. A subroutine inside services.exe is overwritten with shellcode that is carried inside the ZeroAccess dropper.

A large amount of binary data is also written to the NTFS Extended Attributes of services.exe. Extended Attributes are a feature of NTFS similar in nature to Alternate Data Streams where extra information about the file can be stored on the file system. ZeroAccess uses this feature to hide a whole PE file as well as shellcode that loads the PE file. The overwritten subroutine in services.exe reads in all the data from the Extended Attributes and executes it. The shellcode then loads and executes the embedded PE file. This file is a DLL that has equivalent functionality to the main component, so the services.exe modifications provide a backup means for the bot to function if the two main components are discovered and removed."

I was already fascinated about the Trojan’s use of two separate DLLs to maintain persistence but what I read opened my eyes. ZeroAccess leveraged a third DLL for persistence that was hidden inside the NTFS Extended Attributes. A Google search lead me to a third article by Symantec - Trojan.Zeroaccess.C Hidden in NTFS EA. I then remembered a few months back someone from the Yahoo Win4n6 group posted the write-up ZeroAccess – From Rootkit to Nasty Infection – which also discussed ZeroAccess hiding data inside NTFS Extended Attribute. At the time, I read the HitmanPro article and bookmarked it amongst my malware references but I didn’t give it too much thought. Now my eyes are open and the use of the NTFS Extended Attribute had my full attention. A few thoughts were running through my head (yes in this order)

     1. This is a really cool technique

     2. The articles don’t mention how to extract the data from the Extended Attribute. I wonder how you can do it.

     3. Hiding data in the Extended Attribute is brilliant since antivirus software might not check there. Come to think about it most digital forensic tools don’t parse the data inside Extended Attributes either.

     4. Wait a second; I never look at the NTFS Extended Attributes during my examinations. I need to come up with a way to do it and add it to my examination process. It could be a check underneath my Malware Searches examination step.

     5. Alight, this is definitely cool.

In this post I’m providing answers to all the questions that were racing through my mind. The post is broken up into the sections:

     - What Are NTFS Extended Attributes

     - My Testing Environment

     - Extracting Data from NTFS Extended Attributes

     - Extracting ZeroAccess from NTFS Extended Attributes

     - Locating Other ZeroAccess Files on System

What Are NTFS Extended Attributes

The NTFS filesystem is the default for the Microsoft Windows server operating systems as well as Windows 2000, XP, Vista, and 7. The Master File Table ($MFT) is the component in NTFS containing information about all the files and folders on the volume. “Every file and directory has at least one entry in the table, and the entries by themselves are very simple.” The entry basically consists of a header and attributes. Two attributes are the $EA index and $EA_INFORMATION which are the NTFS Extended Attributes.

The File System Forensic Analysis book shows the $EA_INFORMATION type identifier as 208 and $EA identifier as 224 (credit goes to Harlan for this reference). The book further stated both attributes are “used for backward compatibility with OS/2 applications (HPFS).” In addition, the book Windows Internals, Part 2: Covering Windows Server 2008 R2 and Windows 7 stated “extended attributes are name/value pairs and aren’t normally used but are provided for backward compatibility with OS/2 applications”. The references provide a little more clarity about what the author of ZeroAccess is doing. They are using hardly used $MFT attributes meant for backward compatibility as a place to hide data. The cool factor with this technique went up a few notches.

Testing Environment

Before diving into the answers to all my questions I wanted to provide information about my testing environment. Knowing the testing environment makes it easier for others to see exactly what I did and how to replicate my testing. For those looking for the “good stuff” skip ahead to the section Extracting Data from NTFS Extended Attributes.

Finding the ZeroAccess Sample

In all the articles about ZeroAccess using the NTFS Extended Attributes I noticed one commonality; this technique was used in an updated version of the Trojan. Check out the commonality: HitmanPro posted their write up on June 25, 2012, Symantec did theirs on August 14, 2012, and Sophos report was released in September 2012. Furthermore, Sophos stated in their opening sentence “since our last paper on ZeroAccess, The ZeroAccess Rootkit, its authors have made significant changes.” The point was crystal clear, I needed to find a recent ZeroAccess sample in order to find one leveraging the $EA hiding technique.

I searched for samples in the Virus Share malware repository as you will see why shortly. A generic search for “zeroaccess” wasn’t to helpful since it resulted in 6,056 results. I needed a more focused search so I turned to Symantec’s detection Trojan.ZeroAccess.C. It reduced my search results to 9 samples.

A nice feature about Virus-Share is it provides links to each sample’s VirusTotal’s report. I used the VT report to further narrow down my samples. On VirusTotal’s Additional Information section, I first looked to see when each sample was first seen.


Remember the commonality with the articles? I only wanted to focus on samples first seen within the past few months. The second check I did on the Additional Information section was to look at the TrID output to determine what samples were executables as opposed to DLLs. Working with an exe makes it a little easier for fast execution.


I narrowed my samples down to five based on the above checks. Not every ZeroAccess Trojan will modify the services.exe and store binary data in the NTFS Extended Attribute. So I started executing one sample after another to see if any modified the services.exe MFT record. I didn’t do any dynamic analysis; I pretty much just ran a sample and then looked at the services.exe MFT record to see if the Extended Attributes were added. On the third sample I hit pay dirt. The sample MD5 hash b437274492e215c25f96711d04b44050 (SHA256 658817f5f7722506868d9f717ee19b276fcab0d0ecac071d5d92a4178fdeb5b3) leveraged the NTFS Extended Attributes.

Side note: I mentioned in the Win4n6 group I was looking for ZeroAccess samples that used the NTFS Extended Attributes. Stefan replied with not only the MD5 hash c6e73a75284507a41da8bef0db342400 but links to the article ZeroAccess - new steps in evolution and the ThreatExpert Report. This sample is the next one I’m going to check out.

Configuring my Test System

The Sophos report indicated the services.exe is only patched if ZeroAccess is installed on a system with Windows Vista or higher. I selected for my test system Windows 7 Ultimate 32 bit which I was running inside VMware. I used a fresh install and the only configuration change I made was to disable the User Account Control. I wanted to avoid making ZeroAccess elevate its privileges. I also installed the program Capture.bat as a quick way to see changes made to the file system.

Extracting Data from NTFS Extended Attributes

I had to answer my first question before I could go forward with any testing. How do I extract the binary data from the NTFS Extended Attributes? I reached out to the Win4n6 group for suggestions and Brian Carrier pointed me to the promise land. He suggested I used the Sleuthkit; specifically istat to see what attributes a MFT record has and icat to dump the data in an attribute. The process needed to extract data stored inside NTFS Extended Attributes with TSK is as follows:

Note: the TSK tools can be ran against a forensic image or a mounted drive. My examples throughout the post are against a mounted drive (\\.\PHYSICALDRIVE#) but the path to a forensic image could be used instead.

1. Identify the starting sector of the partition where services.exe is located. Typically, the first sector for Vista and 7 is 2048 (XP is 63). The TSK tool to use is mmls and the command is:

mmls \\.\PHYSICALDRIVE#


2. Identify the inode number for the C:\Windows\System32\services.exe file. The TSK tool to use is ifind and the command is:

ifind -o 2048 -n "Windows/System32/services.exe" \\.\PHYSICALDRIVE#

Command explanation: the –o switch is to specify the sector offset that was discovered with mmls and the –n switch is the file name to get the inode for.


3. Review the attributes associated with the MFT record for C:\Windows\System32\services.exe. The TSK tool to use is istat and the command is:

istat -o 2048 \\.\PHYSICALDRIVE3 12345

Command explanation: the –o switch is the sector offset and the 12345 is the place where the inode number goes. The attributes of interest are $EA_INFORMATION (208) and $EA (224).


4. If present, make note of the NTFS Extended Attributes. In istat’s output, next to the attribute name will be a value inside parentheses. i.e. the value in my test appeared as $EA (224-4)


5. Dump the contents from the $EA and $EA_INFORMATION and redirect it to a file. The TSK tool to use is icat and the commands are:

icat -o 2048 \\.\PHYSICALDRIVE# 12345-208-# > EA_INFO.txt
icat -o 2048 \\.\PHYSICALDRIVE# 12345-224-# > EA.txt

Command explanation: the –o switch is the sector offset and the number at the end is the inode number with the attribute value added. The 208-# dumps $EA_INFORMATION and 224-# dumps $EA.

The above process can be used to either read a MFT’s record attributes or dump the data stored in those attributes.

Extracting ZeroAccess from NTFS Extended Attributes

To completely answer my first question about how to extract the ZeroAccess Trojan from NTFS Extended Attributes I actually had to do it. This is what I ended up doing:

     - Determined what a clean MFT services record looks like
     - Located an infected MFT services record
     - Extracted ZeroAccess from the MFT services record
     - Created the ZeroAccess binary

What Does a Clean MFT Services.exe Record Look like?

It’s always good to know what normal looks like so it’s easier to see what is abnormal. This was my thinking and why I first looked at a clean MFT services.exe record. Here’s what I did and what it looked like. I highlighted in blue values of interest.

1. Confirmed the starting sector for the partition.

mmls \\.\PHYSICALDRIVE2


DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors

Slot Start End Length Description
00: Meta 0000000000 0000000000 0000000001 Primary Table (#0)
01: ----- 0000000000 0000002047 0000002048 Unallocated
02: 00:00 0000002048 0020969471 0020967424 NTFS (0x07)
03: ----- 0020969472 0020971519 0000002048 Unallocated


2. Identified the inode number for the C:\Windows\System32\services.exe file.

ifind -o 2048 -n "Windows/System32/services.exe" \\.\PHYSICALDRIVE2

19211


3. Reviewed the attributes associated with the C:\Windows\System32\services.exe MFT record. Notice there are no $EA or $EA_INFORMATION attributes listed under the Attributes section.

istat -o 2048 \\.\PHYSICALDRIVE2 19211

MFT Record Header Values:
Record: 19211 Sequence: 1
$LogFile Sequence Number: 34894400
Allocated File
Links: 2

$STANDARD_INFORMATION Attribute Values:
Flags: Archive
Owner ID: 0
Security ID: 450 ()
Created: 2009-07-13 19:11:26 (Eastern Daylight Time)
File Modified: 2009-07-13 21:14:36 (Eastern Daylight Time)
MFT Modified: 2012-12-06 11:54:31 (Eastern Standard Time)
Accessed: 2009-07-13 19:11:26 (Eastern Daylight Time)

$FILE_NAME Attribute Values:
Flags: Archive
Name: services.exe, services.exe
Parent MFT Record: 7150 Sequence: 1
Allocated Size: 0 Actual Size: 0
Created: 2012-12-06 11:54:31 (Eastern Standard Time)
File Modified: 2012-12-06 11:54:31 (Eastern Standard Time)
MFT Modified: 2012-12-06 11:54:31 (Eastern Standard Time)
Accessed: 2012-12-06 11:54:31 (Eastern Standard Time)

Attributes:
Type: $STANDARD_INFORMATION (16-0) Name: N/A Resident size: 72
Type: $FILE_NAME (48-4) Name: N/A Resident size: 90
Type: $FILE_NAME (48-2) Name: N/A Resident size: 90
Type: $DATA (128-3) Name: N/A Non-Resident size: 259072 init_size: 259072
750372 750373 750374 750375 750376 750377 750378 750379
750380 750381 750382 750383 750384 750385 750386 750387
750388 750389 750390 750391 750392 750393 750394 750395
750396 750397 750398 750399 750400 750401 750402 750403
750404 750405 750406 750407 750408 750409 750410 750411
750412 750413 750414 750415 750416 750417 750418 750419
750420 750421 750422 750423 750424 750425 750426 750427
750428 750429 750430 750431 750432 750433 750434 750435

Located an Infected MFT Services.exe Record

The clean C:\Windows\System32\services.exe file only had four attributes listed. The $STANDARD_INFORMATION (16-0), $FILE_NAME (48-4), $FILE_NAME (48-2), and $DATA (128-3) attributes. Here’s what an infected services.exe MFT record looks like and what I did. I highlighted in blue values of interest and items related to the infection in red.

1. Confirmed the starting sector for the partition.

mmls \\.\PHYSICALDRIVE2


DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors

Slot Start End Length Description
00: Meta 0000000000 0000000000 0000000001 Primary Table (#0)
01: ----- 0000000000 0000002047 0000002048 Unallocated
02: 00:00 0000002048 0020969471 0020967424 NTFS (0x07)
03: ----- 0020969472 0020971519 0000002048 Unallocated


2. Identified the inode number for the C:\Windows\System32\services.exe file.

ifind -o 2048 -n "Windows/System32/services.exe" \\.\PHYSICALDRIVE2


42756


3. Reviewed the attributes associated with the C:\Windows\System32\services.exe MFT record.

istat -o 2048 \\.\PHYSICALDRIVE2 42756


MFT Record Header Values:
Record: 42756 Sequence: 1
$LogFile Sequence Number: 98696063
Allocated File
Links: 1

$STANDARD_INFORMATION Attribute Values:
Flags: Archive
Owner ID: 0
Security ID: 483 ()
Last User Journal Update Sequence Number: 17321736
Created: 2009-07-13 19:11:26 (Eastern Daylight Time)
File Modified: 2009-07-13 21:14:36 (Eastern Daylight Time)
MFT Modified: 2012-12-06 17:18:05 (Eastern Standard Time)
Accessed: 2009-07-13 19:11:26 (Eastern Daylight Time)

$FILE_NAME Attribute Values:
Flags: Archive
Name: services.exe
Parent MFT Record: 1802 Sequence: 1
Allocated Size: 0 Actual Size: 0
Created: 2009-07-13 19:11:26 (Eastern Daylight Time)
File Modified: 2012-12-06 17:18:05 (Eastern Standard Time)
MFT Modified: 2012-12-06 17:18:05 (Eastern Standard Time)
Accessed: 2012-12-06 17:18:05 (Eastern Standard Time)

Attributes:
Type: $STANDARD_INFORMATION (16-0) Name: N/A Resident size: 72
Type: $FILE_NAME (48-2) Name: N/A Resident size: 90
Type: $DATA (128-5) Name: N/A Non-Resident size: 259072 init_size: 259072
618 619 620 621 622 623 624 625
626 627 628 629 630 631 632 633
634 635 636 637 638 639 640 641
642 643 644 645 646 647 648 649
650 651 652 653 654 655 656 657
658 659 660 661 662 663 664 665
666 667 668 669 670 671 672 673
674 675 676 677 678 679 680 681
Type: $EA_INFORMATION (208-3) Name: N/A Resident size: 8
Type: $EA (224-4) Name: N/A Non-Resident size: 23404 init_size: 23404
346 347 348 349 350 351

Extract ZeroAccess from MFT Services Record

The infected MFT services.exe record had a few different changes. The inode number for C:\Windows\System32\services.exe changed from 19211 to 42756. In addition, the attributes now listed the NTFS Extended Attributes: $EA_INFORMATION (208-3) and $EA (224-4). The ZeroAccess binary data can now be extracted using the attribute values shown with istat. Here’s how I extracted the data.

4. Extracted the binary data from the MFT record for C:\Windows\System32\services.exe for both the $EA_INFORMATION and $EA attributes.

icat -o 2048 \\.\PHYSICALDRIVE2 42756-208-3> icat_services_$EA-INFORMATION.bin
icat -o 2048 \\.\PHYSICALDRIVE2 42756-224-4 > icat_services_$EA_binary.bin

Create the ZeroAccess Binary

The Sophos report indicated the ZeroAccess binary data stored in the NTFS Extended Attribute is shellcode and a DLL. This means I had to separate the two in the dumped data; my focus was only on the DLL. There was very little data stored in the $EA_INFORMATION attribute while the majority of the data – including the DLL – was in the $EA attribute. To rebuild the DLL I opened the extracted data in a Hex editor and looked for for a PE file’s characteristics. I was looking for the MZ file signature and the MS-DOS message.


I located the MZ signature at offset 1684 so I copied out the binary data from offset 1684 to the end of the file. The resulting file was SHA256 ee14dcd20b2ee118618e3a28db72cce92ccd9e85dd8410e02479d9d624934b13 which was detected as the ZeroAccess Trojan.

Locating Other ZeroAccess Files on System

Locating the infected services.exe file and extracting the binary data from the NTFS Extended Attributes only addresses one part of the infection. ZeroAccess drops other files on a system and I wanted to find these as well. However, I didn’t want to cheat by looking at my Capture.bat log file. I wanted to verify how a system timeline worked by using the services.exe as a starting point to find the other malicious files.

Timelines are my go to tool when I’m trying find files associated with an infection. A timeline can reveal other activity occurring on a system around a certain time of interest. In this case, I was interested in seeing the activity around the time when the services.exe file was last modified as shown below. (the timeline was created with AnalyzeMFT).




The timeline showed the services.exe file was modified on 12/06/12 at 18:05:09. The activity beforehand included the creation of files and folders in both the C:\Windows\Installer and C:\Users\Lab\AppData\Local folders. The file named n in both locations is the ZeroAcess Trojan.

Summary

Storing data inside the NTFS Extended Attributes is a clever way to hide data. Based on the reports I read, ZeroAccess started to use this technique within the last year. So far it appears as if this is not a commonly used technique to hide data among malware. The only reports I could find only talked about ZeroAccess. However, in time I would not be surprised to see other malware families trying to use NTFS Extended Attributes. To see why just think about these two questions: how many tools are checking the NTFS Extended Attributes for every file and how many DFIR practitioners and IT staff are actually aware that malware can be hidden there? If I was a betting man I’d put my money on both answers being “very few”. My eyes are opened to this hiding technique and I added a check underneath my Malware Searches examination step to account for it. If I ever need to determine if something is leveraging NTFS Extended Attributes then this is the process I will use.

1. Parse the MFT so that NTFS Extended Attributes are shown for every file. One tool to use is AnalyzeMFT with the command

analyzeMFT.py -f $MFT -o parsed_mft.txt


Import analyzeMFT’s output into Excel or Calc (output is comma delimited format). Apply a sort on the $EA attribute being true to show every file with a NTFS Extended Attribute.


2. Locate the files’ of interest MFT records. AnalyzeMFT’s output contains this information.

3. Identify the partition’s starting sector. One tool to use is TSK’s mmls with the command:

mmls \\.\PHYSICALDRIVE#

4. Review the attributes associated with any files of interest. One tool to use is TSK’s istat with the command

istat -o 2048 \\.\PHYSICALDRIVE3 12345

5. Make note of the NTFS Extended Attributes $EA_INFORMATION and $EA. In istat’s output, next to the attribute name will be a value inside parentheses.

6. Dump the data from the $EA and $EA_INFORMATION for closer inspection. Use TSK’s icat with the commands:

icat -o 2048 \\.\PHYSICALDRIVE# 12345-208-# > EA_INFO.txt
icat -o 2048 \\.\PHYSICALDRIVE# 12345-224-# > EA.txt


Happy Hunting

Knowing the programs that executed on a system can answer numerous questions. The answers can help on a range of cases from acceptable use policy violations to investigations to intrusions to malware. One program execution artifact is prefetch files which have been well documented including in the article Decoding Prefetch Files for Forensic Purposes and on the Prefetch Wiki page. The purpose of this post is to explore the NTOSBOOT prefetch file and how it is relevant for malware investigations.

What is the NTOSBOOT Prefetch File

The Windows operating system enables prefetching to make system boots or applications startups faster. Windows Internals, Part 2: Covering Windows 2008 Server R2 and Windows 7 states “the prefetcher tries to speed the boot process and application startup by monitoring the data and code accessed by boot and application startups and using that information at the beginning of a subsequent boot or application startup to read in the code and data”. Prefetch files are what store the information required for faster startups.

The NTOSBOOT prefetch file stores the information related to the boot process. The information in this prefetch file reflects the files accessed during system startup. Specifically, the files accessed, according to the Windows Internals, “from system start through the 30 seconds following the start of the user’s shell (typically Explorer) or, failing that, up through 60 seconds following Windows service initialization or through 120 seconds, whichever comes first”.

Relevance for Malware Investigations

The NTOSBOOT prefetch file stores information about the files accessed during the boot process and in some cases 30 seconds following a user logging onto a system. Remember the Rootkit Paradox, malware wants to hide but it must run. If a piece of malware runs during the boot process then there will be a reference for the malware in the NTOSBOOT prefetch file. To illustrate this point I examined a NTOSBOOT prefetch file to show how it contains traces of malware execution. (side note: to view the file handles in prefetch files one could use Strings or Harlan’s pref.pl)

The NTOSBOOT prefetch file in question came from a system I infected last year by opening the attachment in a NYS Department of Motor Vehicles Uniform Traffic Ticket email. Looking through the file handles for any unusual file paths (files in temporary folders, unusual file names, etc) I came across three suspicious files as shown below.




The first file 17F7FFF4.COM was suspicious since the file was located in the All Users temporary folder. The second file KB961710.exe was suspicious because Windows updates are not stored in the application data folders. The last file SVCHOST.EXE was suspicious since not only was it located in a temp folder but the svchost.exe file is normally located in the Windows\System32 folder. I located these files by solely reviewing the file handles referenced in the NTOSBOOT prefetch file and the screenshot below confirms my suspicions.

Summary

Program execution artifacts are vital when examining a system infected with malware. Despite malware’s best efforts to hide on a system at some point for them to do their jobs they have to run. When malware eventual runs there will be traces of its execution left on a system which can be used to find its hiding spot. The NTOSBOOT prefetch file is one such artifact and the file handle information will reveal what’s hiding on a system.