Revealing the RecentFileCache.bcf File
Monday, December 2, 2013
The Application Experience and Compatibility feature is considered one of the pillars in the in Microsoft Windows operating systems. Microsoft states in reference to the Microsoft Application Compatibility Infrastructure (Shim Infrastructure) "as the Windows operating system evolves from version to version changes to the implementation of some functions may affect applications that depend on them." This feature is to ensure compatibility of existing software between different versions of the Windows operating system. The implementation of this feature results in some interesting program execution artifacts that are relevant to Digital Forensic and Incident Response (DFIR). Mandiant highlighted one artifact in their Leveraging the Application Compatibility Cache in Forensic Investigations article. However, there are more artifacts associated with this feature and the RecentFileCache.bcf file is one of them. This post provides some clarification about the file and the relevance of the data stored within it.
Microsoft further states why the Application Experience and Compatibility feature is needed:
"Because of the nature of software, modifying the function again to resolve this compatibility issue could break additional applications or require Windows to remain the same regardless of the improvement that the alternative implementation could offer."
The Shim Infrastructure allows for those broken applications to be fixed and ensures they are compatible with newer versions of Windows. The Shim Infrastructure leverages API hooking to accomplish this. When there are calls to external binary files the calls get redirected to the Shim Infrastructure as illustrated below:
Alex Ionescu's goes into more detail about what happens when the external binary calls get redirected to the Shim Infrastructure in his post Secrets of the Application Compatilibity Database (SDB) – Part 1. "As the binary file loads, the loader will run the Shim Engine, which will perform lookups in the system compatibility database, recovering various information." The compatibility database is named sysmain.sdb and is located in the C:\Windows\AppPatch directory. sysmain.sdb contains information "that details how the target operating system should handle the application" and covers more than 5,000 applications. Furthermore, Alex states " on top of the default database, individual, custom databases can be created, which are registered and installed through the registry."
There is very little information available about what is the Windows process that the loader calls to perform the compatibility database checks. However, monitoring the Windows operating system with Process Monitor identified one of the Shim Infrastructure processes being the Application Experience.
Application Experience is a Windows service. It's service display name is Application Experience, service name is AeLookupSvc and its default description states it "processes application compatibility cache requests for applications as they are launched." Another reference mentions "this service checks a Microsoft maintained database for known problems with popular programs and automatically enables workarounds, either at first installation (using UAC) or at application launch." The service performs these checks as applications are launched (on Windows XP this was handled by the Application Experience Lookup service). The screenshot below shows the Application Experience service's properties including the path to the executable which is "svchost.exe -k netsvcs."
In addition to the Application Experience service, there are two Windows tasks involved with the Application Experience and Compatibility feature. The two default tasks for Application Experience are named AitAgent and ProgramDataUpdater. The description for the AitAgent task states it "aggregates and uploads Application Telemetry information if opted-in to the Microsoft Customer Experience Improvement Program." The second and more important task as it relates to the RecentFilecache.bcf file is the ProgramDataUpdater. It's description states it "collects program telemetry information if opted-in to the Microsoft Customer Experience Improvement Program." The other item to note about this task is its action which is to start the following program: %windir%\system32\rundll32.exe aepdu.dll,AePduRunUpdate. As with every Windows task there is information about when the task is scheduled to run and when it last ran. The screen below shows this information for the Application Experience tasks.
The one thing to keep in mind is since it impacts the RecentFileCache.bcf file is that the ProgramDataUpdater task is scheduled to run every day at 12:30 AM.
The Application Experience and Compatibility feature comes into play as a program starts. To fully explore how this feature works it is necessary to review how processes are created on Windows and what occurs when the loader runs the Shim Engine. The Windows Internals, Part 1: Covering Windows Server 2008 R2 and Windows 7 outlines the main stages of process creation:
1. Validate parameters; convert Windows subsystem flags and options to their native counterparts; parse, validate, and convert the attribute list to its native counterpart.
2. Open the image file (. exe) to be executed inside the process.
3. Create the Windows executive process object.
4. Create the initial thread (stack, context, and Windows executive thread object).
5. Perform post-creation, Windows-subsystem-specific process initialization.
6. Start execution of the initial thread (unless the CREATE_ SUSPENDED flag was specified).
7. In the context of the new process and thread, complete the initialization of the address space (such as load required DLLs) and begin execution of the program.
The picture below illustrates the process creation main stages:
The loader runs the Shim Infrastructure in stage 5 of the process creation which is where performing Windows subsystem-specific-post-initialization occurs. This is where specific operations are performed to finish initializing the process. According to the Windows internal book, one of those operations is "the application-compatibility database is queried to see whether an entry exists in either the registry or system application database for the process."
Exploring this portion of the process creation with Process Monitor reveals the files, registry keys, and processes involved with the database being queried. To demonstrate I'll follow the lead of the Windows Internal book by using the built-in Windows program notepad.exe. Notepad.exe was invoked by double clicking it; this means Explorer.exe is process that loads notepad.exe.
Explorer.exe performs other activity as illustrated below:
The Windows Internal book states the following about this activity:
"The first is a simple check for application-compatibility flags, which will let the user-mode process creation code know if checks inside the application-compatibility database are required through the shim engine."
In addition the book goes on to say
"you might see I/O [input/output] to one or more .sdb files, which are the application-compatibility databases on the system. This I/O is where additional checks are done to see if the shim engine needs to be invoked for this application."
The Application Experience and Compatibility feature is implemented through the Shim Infrastructure. This Shim Infrastructure is invoked during process creation when an application is launched. However, there was no mention about the RecentFileCache.bcf file located in the C:\Windows\AppCompat\Programs directory on Windows 7 systems. I first became aware about this file when processing malware cases. I kept seeing references (full path names to executables) to malware in the RecentFileCache.bcf file. The file was different than the Shim Cache Mandiant spoke about. The Shim Cache contains references to numerous programs over an extended period of time. The RecentFileCache.bcf file on the other hand only contained references to programs that recently executed. The reason for this is because the RecentFileCache.bcf file is a temporary storage location used during the process creation. It appears this storage location is not used during all process creation; it's mostly used for those processes that spawned from executables which were recently copied or downloaded to the system.
Extensive testing was performed to shed light on the actions which lead to executables names being listed in the RecentFileCache.bcf file. The testing included executing the same program the following ways:
- executing program already on the system
- renaming and then executing the program already on the system
- executing the program from a removable media
- copying the program to the system and then executing it
- downloading the program with a web browser and then executing it
- copying a different program that has an installation process and then executing it
The process creation process varied from what was described previously when it involved an executable that was recently created on the system (either through copying or downloading). The best way to illustrate this difference is by demonstrating it. The picture below shows part of the copying process moving E:\Tcopy.exe to C:\Tcopy.exe.
The next picture shows the part of the process creation where the application compatibility database is checked after C:\Tcopy.exe was executed from the command prompt.
Next there are checks for the application compatibility flags.
To this point the Shim Infrastructure being invoked during process creation resembles what was described previously. However, the next picture illustrates where the process creation starts to deviate. Shortly after the sysmain.sdb database is closed a scvhost.exe process becomes more active.
The properties of the svchost.exe shows it is the Application Experience service since it's command line is "svchost.exe -k netsvcs."
The Application Experience service performs various operations. One of its operations records the executable's full path in the RecentFileCache.bcf file as shown below:
The structure of the RecentFileCache.bcf file is fairly basic. Each entry only has the file path length and file path itself.
Harlan Carvey wrote a Perl script to parse the RecentFileCache.bcf file; his script makes it easier to read its contents than using a hex editor. Below is the script's output parsing a file from a test system:
c:\windows\devcon\64bits\devconx64.exe
c:\tcopy.exe
c:\users\ anon\appdata\local\microsoft\windows\temporary internet files\content.ie5\hjhf58cm\tweb.exe
The Application Experience service records the file names of executables in the RecentFileCache.bcf file. These executables were newly created on the system whether if it was copied, downloaded, or extracted from another executable. The storage of this information in the RecentFileCache.bcf file is only temporary since it is cleared when the Application Experience ProgramDataUpdater task is ran.
side note: I have seen references to executables that executed from removable media (i.e. my collection tools) as well as executables that were present on the system but never executed referenced in this artifact. However, I was unable to reproduce either with my testing.
Continuing on with the demonstration, the system was monitored to see what else interacted with the RecentFileCache.bcf file. The services.exe started the Application Experience ProgramDataUpdater task as shown in the two screenshots below.
Shortly thereafter, the Application Experience service started interacting with the ProgramDataUpdater task as shown below.
Further monitoring of the ProgramDataUpdater task showed it accessed the RecentFileCache.bcf file and cleared its contents.
The ProgramDataUpdater task runs at least once a day and knowing it clears the cache means the programs listed in the RecentFileCache.bcf file executed fairly recently. The ProgramDataUpdater task also interacted with the xml file stored inside the AppCompat\Programs folder. This xml file appears to contain the program's publisher information.
The RecentFileCache.bcf file is another artifact that shows program execution. I have found this artifact helpful when investigating systems shortly after they became infected. The artifact is a quick way to locate malware - such as droppers and downloaders - on the system as I briefly mentioned in the post Triaging Malware Incidents. The relevance of the executables listed in this artifact mean the following:
1. The program is probably new to the system.
2. The program executed on the system.
3. The program executed on the system some time after the ProgramDataUpdater task was last ran.
*** Update ***
In the Windows 8 operating system the RecentFilceCache.bcf has been replaced by a registry hive named Amcache.hve. Yogesh Khatri digs into this new artifact and provided an excellent overview about what it contains in the post Amcache.hve in Windows 8 - Goldmine for malware hunters. The Amcache.hve stores a ton of information compared to the RcentFileCache.bcf; one of which are executables SHA-1 hashes.
Exploring the Application Experience and Compatibility Feature
Microsoft further states why the Application Experience and Compatibility feature is needed:
"Because of the nature of software, modifying the function again to resolve this compatibility issue could break additional applications or require Windows to remain the same regardless of the improvement that the alternative implementation could offer."
The Shim Infrastructure allows for those broken applications to be fixed and ensures they are compatible with newer versions of Windows. The Shim Infrastructure leverages API hooking to accomplish this. When there are calls to external binary files the calls get redirected to the Shim Infrastructure as illustrated below:
Alex Ionescu's goes into more detail about what happens when the external binary calls get redirected to the Shim Infrastructure in his post Secrets of the Application Compatilibity Database (SDB) – Part 1. "As the binary file loads, the loader will run the Shim Engine, which will perform lookups in the system compatibility database, recovering various information." The compatibility database is named sysmain.sdb and is located in the C:\Windows\AppPatch directory. sysmain.sdb contains information "that details how the target operating system should handle the application" and covers more than 5,000 applications. Furthermore, Alex states " on top of the default database, individual, custom databases can be created, which are registered and installed through the registry."
There is very little information available about what is the Windows process that the loader calls to perform the compatibility database checks. However, monitoring the Windows operating system with Process Monitor identified one of the Shim Infrastructure processes being the Application Experience.
Application Experience is a Windows service. It's service display name is Application Experience, service name is AeLookupSvc and its default description states it "processes application compatibility cache requests for applications as they are launched." Another reference mentions "this service checks a Microsoft maintained database for known problems with popular programs and automatically enables workarounds, either at first installation (using UAC) or at application launch." The service performs these checks as applications are launched (on Windows XP this was handled by the Application Experience Lookup service). The screenshot below shows the Application Experience service's properties including the path to the executable which is "svchost.exe -k netsvcs."
In addition to the Application Experience service, there are two Windows tasks involved with the Application Experience and Compatibility feature. The two default tasks for Application Experience are named AitAgent and ProgramDataUpdater. The description for the AitAgent task states it "aggregates and uploads Application Telemetry information if opted-in to the Microsoft Customer Experience Improvement Program." The second and more important task as it relates to the RecentFilecache.bcf file is the ProgramDataUpdater. It's description states it "collects program telemetry information if opted-in to the Microsoft Customer Experience Improvement Program." The other item to note about this task is its action which is to start the following program: %windir%\system32\rundll32.exe aepdu.dll,AePduRunUpdate. As with every Windows task there is information about when the task is scheduled to run and when it last ran. The screen below shows this information for the Application Experience tasks.
The one thing to keep in mind is since it impacts the RecentFileCache.bcf file is that the ProgramDataUpdater task is scheduled to run every day at 12:30 AM.
Exploring Process Creation on Windows
The Application Experience and Compatibility feature comes into play as a program starts. To fully explore how this feature works it is necessary to review how processes are created on Windows and what occurs when the loader runs the Shim Engine. The Windows Internals, Part 1: Covering Windows Server 2008 R2 and Windows 7 outlines the main stages of process creation:
1. Validate parameters; convert Windows subsystem flags and options to their native counterparts; parse, validate, and convert the attribute list to its native counterpart.
2. Open the image file (. exe) to be executed inside the process.
3. Create the Windows executive process object.
4. Create the initial thread (stack, context, and Windows executive thread object).
5. Perform post-creation, Windows-subsystem-specific process initialization.
6. Start execution of the initial thread (unless the CREATE_ SUSPENDED flag was specified).
7. In the context of the new process and thread, complete the initialization of the address space (such as load required DLLs) and begin execution of the program.
The picture below illustrates the process creation main stages:
The loader runs the Shim Infrastructure in stage 5 of the process creation which is where performing Windows subsystem-specific-post-initialization occurs. This is where specific operations are performed to finish initializing the process. According to the Windows internal book, one of those operations is "the application-compatibility database is queried to see whether an entry exists in either the registry or system application database for the process."
Exploring this portion of the process creation with Process Monitor reveals the files, registry keys, and processes involved with the database being queried. To demonstrate I'll follow the lead of the Windows Internal book by using the built-in Windows program notepad.exe. Notepad.exe was invoked by double clicking it; this means Explorer.exe is process that loads notepad.exe.
Explorer.exe performs other activity as illustrated below:
The Windows Internal book states the following about this activity:
"The first is a simple check for application-compatibility flags, which will let the user-mode process creation code know if checks inside the application-compatibility database are required through the shim engine."
In addition the book goes on to say
"you might see I/O [input/output] to one or more .sdb files, which are the application-compatibility databases on the system. This I/O is where additional checks are done to see if the shim engine needs to be invoked for this application."
Exploring the RecentFileCache.bcf File
The Application Experience and Compatibility feature is implemented through the Shim Infrastructure. This Shim Infrastructure is invoked during process creation when an application is launched. However, there was no mention about the RecentFileCache.bcf file located in the C:\Windows\AppCompat\Programs directory on Windows 7 systems. I first became aware about this file when processing malware cases. I kept seeing references (full path names to executables) to malware in the RecentFileCache.bcf file. The file was different than the Shim Cache Mandiant spoke about. The Shim Cache contains references to numerous programs over an extended period of time. The RecentFileCache.bcf file on the other hand only contained references to programs that recently executed. The reason for this is because the RecentFileCache.bcf file is a temporary storage location used during the process creation. It appears this storage location is not used during all process creation; it's mostly used for those processes that spawned from executables which were recently copied or downloaded to the system.
Extensive testing was performed to shed light on the actions which lead to executables names being listed in the RecentFileCache.bcf file. The testing included executing the same program the following ways:
- executing program already on the system
- renaming and then executing the program already on the system
- executing the program from a removable media
- copying the program to the system and then executing it
- downloading the program with a web browser and then executing it
- copying a different program that has an installation process and then executing it
The process creation process varied from what was described previously when it involved an executable that was recently created on the system (either through copying or downloading). The best way to illustrate this difference is by demonstrating it. The picture below shows part of the copying process moving E:\Tcopy.exe to C:\Tcopy.exe.
The next picture shows the part of the process creation where the application compatibility database is checked after C:\Tcopy.exe was executed from the command prompt.
Next there are checks for the application compatibility flags.
To this point the Shim Infrastructure being invoked during process creation resembles what was described previously. However, the next picture illustrates where the process creation starts to deviate. Shortly after the sysmain.sdb database is closed a scvhost.exe process becomes more active.
The properties of the svchost.exe shows it is the Application Experience service since it's command line is "svchost.exe -k netsvcs."
The Application Experience service performs various operations. One of its operations records the executable's full path in the RecentFileCache.bcf file as shown below:
The structure of the RecentFileCache.bcf file is fairly basic. Each entry only has the file path length and file path itself.
Harlan Carvey wrote a Perl script to parse the RecentFileCache.bcf file; his script makes it easier to read its contents than using a hex editor. Below is the script's output parsing a file from a test system:
c:\windows\devcon\64bits\devconx64.exe
c:\tcopy.exe
c:\users\ anon\appdata\local\microsoft\windows\temporary internet files\content.ie5\hjhf58cm\tweb.exe
Application Experience and RecentFileCache.bcf
The Application Experience service records the file names of executables in the RecentFileCache.bcf file. These executables were newly created on the system whether if it was copied, downloaded, or extracted from another executable. The storage of this information in the RecentFileCache.bcf file is only temporary since it is cleared when the Application Experience ProgramDataUpdater task is ran.
side note: I have seen references to executables that executed from removable media (i.e. my collection tools) as well as executables that were present on the system but never executed referenced in this artifact. However, I was unable to reproduce either with my testing.
Continuing on with the demonstration, the system was monitored to see what else interacted with the RecentFileCache.bcf file. The services.exe started the Application Experience ProgramDataUpdater task as shown in the two screenshots below.
Shortly thereafter, the Application Experience service started interacting with the ProgramDataUpdater task as shown below.
Further monitoring of the ProgramDataUpdater task showed it accessed the RecentFileCache.bcf file and cleared its contents.
The ProgramDataUpdater task runs at least once a day and knowing it clears the cache means the programs listed in the RecentFileCache.bcf file executed fairly recently. The ProgramDataUpdater task also interacted with the xml file stored inside the AppCompat\Programs folder. This xml file appears to contain the program's publisher information.
RecentFileCache.bcf File's Relevance
The RecentFileCache.bcf file is another artifact that shows program execution. I have found this artifact helpful when investigating systems shortly after they became infected. The artifact is a quick way to locate malware - such as droppers and downloaders - on the system as I briefly mentioned in the post Triaging Malware Incidents. The relevance of the executables listed in this artifact mean the following:
1. The program is probably new to the system.
2. The program executed on the system.
3. The program executed on the system some time after the ProgramDataUpdater task was last ran.
*** Update ***
In the Windows 8 operating system the RecentFilceCache.bcf has been replaced by a registry hive named Amcache.hve. Yogesh Khatri digs into this new artifact and provided an excellent overview about what it contains in the post Amcache.hve in Windows 8 - Goldmine for malware hunters. The Amcache.hve stores a ton of information compared to the RcentFileCache.bcf; one of which are executables SHA-1 hashes.
Labels:
program execution
Posts
Corey,
It's clear that you have put a lot of work into this research; thanks for sharing it!
Very cool stuff - I echo Harlan's thanks for making your research available. This is going to be very useful for IR work.
Corey,
This is how research should be done. No "started expensive court approve forensics tool and processed . . ." You focused on what Windows was doing, the files created or modified, the processed and services involved. This research is usable regardless expensive tools, and is repeatable by skilled forensics practitioners. Simply excellent.
Troy
Very interesting and valuable post Corey.
Good corey!! Thank you for your effort,,
Thank you corey, very good article. If i may ask, Can i know the tool you used to view the Ps, PID , Path and result.
@anon,
I used procmon by Microsoft to capture that information