Tr3Secure Collection Script Updated
Tuesday, October 28, 2014
On my to-do list for some time has been to add support back into the Tr3Secure collection script to obtain the NTFS Change Journal ($UsnJrnl). This is a quick post about this functionality being added back to the collection script.
The issue I faced was the following. There are very few tools capable of collecting NTFS artifacts from live systems; even fewer for collecting the $UsnJrnl. The Tr3Secure script uses Joakim Schicht's tool RawCopy to collect files off of live systems. It is one of the few and - as far as I know - the only open source option. Rawcopy pulls files either by their $MFT record number or the file path. Pulling NTFS artifacts requires the $MFT record number. The challenge is the $UsnJrnl does not have a consistent $MFT record number like the other NTFS artifacts. For most scripting languages this wouldn't be an issue but Tr3Secure is a batch script. Batch scripting doesn't support storing a command's output into a variable. Translation: there is not an easy way in batch scripting to query the $UsnJrnl's $MFT record number, store it into a variable, and then use that variable with RawCopy to collect it. This is why adding the functionality back into the script has been and remained on my to-do list until now.
Joakim Schicht does outstanding work producing DFIR tools and releasing them open source. His Github site contains a wealth of tools. He even has a collection of tools for the collection and parsing of NTFS artifacts. For those who aren't familiar with his work then I highly advise you take the time to explore them (also his Google Code wiki page.) He recently released a new tool called ExtractUsnJrnl. The tool - in Joakim's words - does the following:
"$J may be sparse, which would mean parts of the data is just 00's. This may be a significant portion of the total data, and most tools will extract this data stream to its full size (which is annoying and a huge waste of disk space). This is where this tools comes in, as it only extract the actual data for the change journal. That way extraction obviously also goes faster. Why extract 20 GB when you might only need 200 MB?"
The tool not only collects the $UsnJrnl $J alternate data stream but it only extracts the portion containing data. This not only saves space but it makes the collection faster; especially if pulling it over the wire. The tool is command-line making it easy to script with. I updated the Tr3Secure collection script to use the ExtractUsnJrnl tool for grabbing the $UsnJrnl.
ExtractUsnJrnl is really a cool tool so I wanted to take the time to highlight it. I performed a simple test. Collect the $UsnJrnl $J file with one tool (FTK Imager) to see how long it takes and what the file size is then use ExtractUsnJrnl.
The image below shows the $UsnJrnl from a 1TB solid state drive. The file size difference is significant; one file is 4.6GB while the other is 36MB. Both tools were ran locally but ExtractUsnJrnl completed within seconds.
The image below shows the $UsnJrnl from a 300GB removable drive. Again, notice the difference between the file sizes.
Some may be wondering why am I so focused on the resulting file size. The reason is trying to pull a 4.6GB file over the wire from a remote system takes time. A lot of time if that remote system is in a location with a slow network link (think VPN users). By reducing the file size (i.e. 36MB) makes it easier to collect the $UsnJrnl both remotely and locally to an attached storage device.
The next test I ran was to parse both $UsnJrnl $J files to see if the both contain around the same number of records. I said approximate because the hard drives were not write protected and changes may had been made between the collections. Due to this I evaluated the removable store device's NTFS Change Journal since the drive had less activity than the solid state drive.
The image below shows UsnJrnl2Csv successfully parsing the $UsnJrnl $J extracted with FTK Imager.
The image below shows UsnJrnl2Csv successfully parsing the $UsnJrnl $J extracted with ExtractUsnJrnl. Notice how this $J file had significantly less records.
Lastly, the image below shows the comparison of the two parsed $UsnJrnl $J files from the removable media. Both outputs start at the same time with the same file and end at the same time with the same file.
Adding support to collect the $UsnJrnl is not the only update. The change log lists out all of them but I did want to highlight another one. An additional menu option was added to only collect the NTFS artifacts. There are times where I want to create a quick timeline with the NTFS artifacts to get more information about something. For example, an antivirus alert may had flagged a file but I'm interested in if anything else was dropped onto the system. In an instance like this, creating a timeline with both the $MFT and $UsnJrnl can quickly answer this question. I've been using a different collection script to grab just the NTFS artifacts but I decided to incorporate the functionality into the Tr3secure script. The menu option now appears as the following:
Selecting option 5 will only preserve select files then collect the $MFT, $Logfile, and $UsnJrnl.
You can download the TR3Secure Data Collection Script from the following download site. The link is also posted along the right hand side of this blog towards the top.
In the future I plan on doing a post or two illustrating how targeted collections using scripts - such as the Tr3secure collection script - can significantly speed up the time it takes to triage an alert or system.
The issue I faced was the following. There are very few tools capable of collecting NTFS artifacts from live systems; even fewer for collecting the $UsnJrnl. The Tr3Secure script uses Joakim Schicht's tool RawCopy to collect files off of live systems. It is one of the few and - as far as I know - the only open source option. Rawcopy pulls files either by their $MFT record number or the file path. Pulling NTFS artifacts requires the $MFT record number. The challenge is the $UsnJrnl does not have a consistent $MFT record number like the other NTFS artifacts. For most scripting languages this wouldn't be an issue but Tr3Secure is a batch script. Batch scripting doesn't support storing a command's output into a variable. Translation: there is not an easy way in batch scripting to query the $UsnJrnl's $MFT record number, store it into a variable, and then use that variable with RawCopy to collect it. This is why adding the functionality back into the script has been and remained on my to-do list until now.
Joakim Schicht's ExtractUsnJrnl
Joakim Schicht does outstanding work producing DFIR tools and releasing them open source. His Github site contains a wealth of tools. He even has a collection of tools for the collection and parsing of NTFS artifacts. For those who aren't familiar with his work then I highly advise you take the time to explore them (also his Google Code wiki page.) He recently released a new tool called ExtractUsnJrnl. The tool - in Joakim's words - does the following:
"$J may be sparse, which would mean parts of the data is just 00's. This may be a significant portion of the total data, and most tools will extract this data stream to its full size (which is annoying and a huge waste of disk space). This is where this tools comes in, as it only extract the actual data for the change journal. That way extraction obviously also goes faster. Why extract 20 GB when you might only need 200 MB?"
The tool not only collects the $UsnJrnl $J alternate data stream but it only extracts the portion containing data. This not only saves space but it makes the collection faster; especially if pulling it over the wire. The tool is command-line making it easy to script with. I updated the Tr3Secure collection script to use the ExtractUsnJrnl tool for grabbing the $UsnJrnl.
ExtractUsnJrnl in Action
ExtractUsnJrnl is really a cool tool so I wanted to take the time to highlight it. I performed a simple test. Collect the $UsnJrnl $J file with one tool (FTK Imager) to see how long it takes and what the file size is then use ExtractUsnJrnl.
The image below shows the $UsnJrnl from a 1TB solid state drive. The file size difference is significant; one file is 4.6GB while the other is 36MB. Both tools were ran locally but ExtractUsnJrnl completed within seconds.
The image below shows the $UsnJrnl from a 300GB removable drive. Again, notice the difference between the file sizes.
Some may be wondering why am I so focused on the resulting file size. The reason is trying to pull a 4.6GB file over the wire from a remote system takes time. A lot of time if that remote system is in a location with a slow network link (think VPN users). By reducing the file size (i.e. 36MB) makes it easier to collect the $UsnJrnl both remotely and locally to an attached storage device.
The next test I ran was to parse both $UsnJrnl $J files to see if the both contain around the same number of records. I said approximate because the hard drives were not write protected and changes may had been made between the collections. Due to this I evaluated the removable store device's NTFS Change Journal since the drive had less activity than the solid state drive.
The image below shows UsnJrnl2Csv successfully parsing the $UsnJrnl $J extracted with FTK Imager.
The image below shows UsnJrnl2Csv successfully parsing the $UsnJrnl $J extracted with ExtractUsnJrnl. Notice how this $J file had significantly less records.
Lastly, the image below shows the comparison of the two parsed $UsnJrnl $J files from the removable media. Both outputs start at the same time with the same file and end at the same time with the same file.
Another Tr3Secure Collection Update
Adding support to collect the $UsnJrnl is not the only update. The change log lists out all of them but I did want to highlight another one. An additional menu option was added to only collect the NTFS artifacts. There are times where I want to create a quick timeline with the NTFS artifacts to get more information about something. For example, an antivirus alert may had flagged a file but I'm interested in if anything else was dropped onto the system. In an instance like this, creating a timeline with both the $MFT and $UsnJrnl can quickly answer this question. I've been using a different collection script to grab just the NTFS artifacts but I decided to incorporate the functionality into the Tr3secure script. The menu option now appears as the following:
Selecting option 5 will only preserve select files then collect the $MFT, $Logfile, and $UsnJrnl.
You can download the TR3Secure Data Collection Script from the following download site. The link is also posted along the right hand side of this blog towards the top.
In the future I plan on doing a post or two illustrating how targeted collections using scripts - such as the Tr3secure collection script - can significantly speed up the time it takes to triage an alert or system.
Posts
Great update with the NTFS Change Journal !
Thanks for sharing.
great work
so great. I used TZworks ntfs tools for obtaining usnjrnl but it is good to know of free software alternative.
Corey,
First of all, thanks for contributing your script to the community. Testing it today I ran into issues with portions of the script running RawCopy. I'm getting the 16 bit MS-DOS Subsystem error that goes something like " The NTVDM CPU has encountered an illegal instruction." Have you run into this before and do you have any idea what's causing it?
Thanks in advance!
It would be great to have a help menu with the batch script so you know what options can be chosen when running it.
@anon,
I never came across that error before. I have used the script and RawCopy on numerous systems and have never seem the error.
@anon,
I'll add putting in a menu to my to do list. It never crossed my mind since I figured people would just look at the script with a text editor. Adding a menu would make things a lot easier.
@anon, @corey
I added this portion of code (after line "set selection=%3") to add options "menu":
----- CODE START
IF "%1" == "" goto :noargs
IF "%2" == "" (
:noargs
echo USAGE %0 case_id output_drive_letter collection_type
echo:
echo Collection type 1 = Acquire RAM Image only
echo Collection type 2 = Acquire RAM Image, pagefile.sys, hiberfil.sys
echo Collection type 3 = Acquire Volatile and Non-Volatile Data
echo Collection type 4 = Acquire RAM Image, Volatile and Non-Volatile Data
echo Collection type 5 = Acquire All of Above
echo DEFAULT Collection type 4 = Acquire RAM Image, Volatile and Non-Volatile Data
echo:
echo NOTE Start the batch script as ADMINISTRATOR directly from its folder.
exit /b
)
----- CODE END
Just keep in mind that I modified the script for my own needs so the numbering for the original script has to be adjusted.
Hi Everyone, quick question. I'm seeing a ton of entries in the $UsnJrnl that appear to have the "file deleted/closed" attribute occurring in rapid succession of one another; only problem is the date/time is when the search warrant would have occurred. Question: would the pulling of an external hard drive connected to the host machine cause the $UsnJrnl entries to be captured as such? Thanks!
@anon,
That is something I haven't tested or heard about. You could determine the answer by running a test. Pull the cable from a computer and then check the change journal for deleted files.