Compromised Root Cause Analysis Model Revisited

Wednesday, March 11, 2015 Posted by Corey Harrell
How? The one question that is easy to ask but can be very difficult to answer. It's the question I kept asking myself over and over. Reading article after article where publicized breaches and compromises were discussed. Each article alluded to the answer about how the breach or compromise occurred in the first place but each one left something out. Every single one left out the details that influenced their conclusions. As a result, I was left wondering how they figure out how the attack occurred in the first place. It was the question everyone alluded to and everyone said to perform root cause to determine the answer. They didn’t elaborate on how to actually do root cause analysis though. Most incident response literature echoes the same sentiment; do root cause analysis while omitting the most critical piece explaining how to do it.  I asked my question to a supposed "incident responder" and their response was along the lines "you will know it when you see it." Their answer along with every other answer on the topic was not good enough. What was needed was a repeatable methodical process one can use to perform root cause analysis. The type of methodical process found in the Compromise Root Cause Analysis Model.

I developed the Compromise Root Cause Analysis Model three years ago to fulfill the need for a repeatable investigative process for doing root cause analysis. In this post I'm revisiting the model and demonstrating its usefulness by outlining the following:

        - Exploring Locard’s Exchange Principle
        - Exploring Temporal Context
        - Exploring Attack Vectors
        - Exploring the Compromise Root Cause Analysis Model
        - The Model is Cyclical
        - Applying the Compromise Root Cause Analysis Model
                * Webserver Compromise

Exploring Locard’s Exchange Principle


The essential principle in the Compromise Root Cause Analysis Model is Locard’s Exchange Principle. This principle states “when two objects come into contact, something is exchanged from one to the other.” Locard’s Exchange Principle is typically explained using examples from the physical world. When one object – such as someone’s hand – comes in to contact with another object – such as a glass – something is exchanged from one to the other. In this example, on the glass are traces of oils from the person’s hand, skin flakes, and even fingerprints.

The principle is not only limited to the physical world; it applies to the digital world as well. Harlan Carvey’s example demonstrated the principle in the digital world as follows: “well, in essence, whenever two computers come "into contact" and interact, they exchange something from each other.” The principle is not only limited to computers; it applies to everything such as routers, switches, firewalls, or mobile devices. The essence of this principle for the Compromise Root Cause Analysis Model is:

When an attacker goes after another system; the exchange will leave remnants of the attack on the systems involved. There is a transfer between the attacker’s system(s), the targeted system(s), and the networking devices connecting them together.

The transfer between the systems and networks involved in the attack will indicate the actual attack used. By identifying and exploring the remnants left by the transfer is what enables the question of “how did the attack occur in the first place” to be answered.

Exploring Temporal Context


The second principle and one that supports the Compromise Root Cause Analysis Model is the psychology principle of proximity. The principle of proximity is one of the Gestalt laws of grouping and states that “when we see stimuli or objects that are in close proximity to each other, we tend to perceive them as being grouped together.” As it relates to the Compromise Root Cause Analysis Model, the grouping is based on the temporal relationship between each object. Temporal proximity impacts the model in two ways.

The first way temporal proximity impacts the Compromise Root Cause Analysis Model is by enabling the grouping of remnants related to an attack. When an attacker goes after another system, remnants are left is various places within system and the network the system is a part of. Networking devices logs showing the network activity, application logs showing what the intruder was doing, and remnants on the system showing what the intruder accomplished are only a few of the places where these artifacts could be located. The attacker’s actions are not the only remnants left within the network and system. The organization and its employees are creating remnants every day from their activity as well as remnants left by the normal operation of the information technology devices themselves. Temporal proximity enables the grouping of the remnants left by an attacker throughout a network and system by their temporal relationship to each other. Remnants that occur within a short timeframe of each other can be grouped together while remnants outside of this timeframe are excluded. Other factors are involved to identify the attacker’s remnants amongst normal activity but temporal proximity is one of the most significant factors.

The second way temporal proximity impacts the Compromise Root Cause Analysis Model is the time that lapses between when an attacker attacks the system and an investigation is conducted affects the ability to identify and group the remnants left by the attacker. The reason for this impact is that “time is what permits other forces to have an effect on the persistence of data.” The remnants left by the attacker is in the form of data on information technology devices. The more time that goes by after these remnants are left the more opportunity there is for them to be changed and/or removed. Logs can be overwritten, files modified, or files deleted through activities of the organization and its employees along with the normal operation of the information technology devices. The more time that lapses between when the attack occurred and when the investigation begins the greater the opportunity for remnants to disappear and the inability to group the remaining remnants together. The Compromise Root Cause Analysis Model can still be used to identify these remnants and group them but it is much more difficult as more time lapses between the initial attack and investigation.

Exploring Attack Vectors


Root cause analysis is trying to determine how an attacker went after another system or network by identifying and understanding the remnants they left on the systems involved during the attack. In essence, the analysis is identifying the attack vector used to compromise the system. It is crucial to explore what an attack vector is to see how it applies to the Compromise Root Cause Analysis Model.

SearchSecurity defines an attack vector as "a path or means by which a hacker (or cracker) can gain access to a computer or network server in order to deliver a payload or malicious outcome." Based on this definition, the attack vector can be broken down into three separate components. The path or means is the exploit used, the payload is the outcome of the exploit, and the delivery mechanism is what delivers the exploit and/or the payload to the target. The definition combines the delivery mechanism and exploit together but in reality these are separated. The exploit, payload, and delivery mechanism can all leave remnants (or artifacts) on the compromised system and network and these artifacts are used to identify the attack vector used.

Exploit


An exploit is defined as "a piece of software, a chunk of data, or a sequence of commands that takes advantage of a bug, glitch or vulnerability in order to cause unintended or unanticipated behavior to occur on computer software, hardware, or something electronic (usually computerized).." An exploit takes advantage of a weakness in a system to cause a desirable activity on that system for the attacker. Exploits can target vulnerabilities in either operating systems, applications, or the people using the system. In accordance with Locard’s Exchange Principle, when the exploit comes in contact with the system containing the weakness remnants are left by the attacker. Identifying these exploit artifacts left on a system are one piece of the puzzle for identifying the attack vector.

Payload


A payload is defined (in security) as “the cargo of a data transmission.” A payload is the desirable activity on a system for the attacker that was caused by an exploit taking advantage of a weakness. In accordance with Locard’s Exchange Principle, when the payload comes in contact with the system remnants are left by the attacker. Identifying these payload artifacts left on a system are another piece of the puzzle for identifying the attack vector.

Delivery Mechanism


A delivery mechanism is defined as “a method by which malicious software places a payload into a target computer or computer network.” The delivery mechanism is what delivers the exploit and/or payload to the system to enable the desirable activity to occur on the system for the attacker. Similar to the exploit and payload, when the delivery mechanisms come in contact with the system remnants are left by the attacker. Identifying these delivery mechanisms artifacts left on a system are the last piece of the puzzle for identifying the attack vector.

Exploring the Compromise Root Cause Analysis Model


When an attacker goes after another system; the exchange leaves artifacts of the attack on the systems involved. These artifacts are identified during an investigation and grouped together based on their temporal proximity to one another. Root cause analysis identifies the attack vector used by determining what of the identified artifacts are related to the exploit, payload, and delivery mechanism(s). The Compromise Root Cause Analysis Model is a methodical process for organizing information and identified artifacts during an investigation to make it easier to answer the question of how did a compromise occur. The model is a not a replacement for any existing models; rather it’s a complimentary model to help discover information related to a system compromise. The Compromise Root Cause Analysis Model organizes the artifacts left on a network and/or system after being attacked into the following categories: source, delivery mechanism, exploit, payload, and indicators. The relationship between the categories are shown below.



Source


At the core of the model is the source of the attack. The source is where the attack originated from. Attacks can originate from outside or within an organization’s network; it all depends on who the attacker is. An external source is anything residing outside the control of an organization or person. An example is attacks against a web application coming from the Internet. Attacks can also be internal, which is within the network and under the control of an organization or person. An example is an employee who is stealing data from a company file server.

The artifacts left behind by the attacker on the system is used to determine where the attack came from. For example, if the attack originated from the Internet then the data left on the systems indicate this. Firewall logs, web application logs, proxy server logs, authentication logs, and email logs all will point to the attacker’s location outside of the organization’s network.

Delivery Mechanism


Proceeding to the next layer is the first delivery mechanism. This is the mechanism used to send the exploit to the system. The mechanism used is dependent on the attacker’s location. Attackers external to the organization may use avenues such as email, network services (i.e. HTTP, SSH, FTP, etc..), or removable media. Attackers internal to the organization may use avenues such as physical access or file sharing protocols.

The artifacts left behind by the attacker on the system is used to determine how they sent the exploit to the system. Where and what the artifacts are is solely dependent on the method used. If the method was HTTP then either web proxy, web browser histories, or web application logs will contain the remnants from the attacker. If the method was email then the email gateway logs, client email storage file, or user activity involving email will contain the remnants from the attacker.

Exploit


Continuing outward to the next layer is the exploit. The exploit is what was sent to take advantage of a vulnerability. As mentioned previously, vulnerabilities can be present in a range of items: from operating systems to applications to databases to network services to the person using the computer.
When vulnerabilities are exploited it leaves specific artifacts on the system and these artifacts can identify the weakness targeted by the attacker. Where and what the artifacts are is solely dependent on what weakness is targeted. The Applying the Model section illustrates this artifact for one vulnerability.

Delivery Mechanism


The next layer is the second delivery mechanism. A successful exploit may result in a payload being sent to the system. This is what the outer delivery mechanism is for. If the payload has to be sent to then there may be artifacts showing this activity. This is the one layer that may not always be present. There are times when the payload is bundled with the exploit or the payload just provides access to the system. Similar to the exploit, where and what the artifacts are present solely dependent on what the exploit was.

Payload


The next layer outlines the desired end result in any attack; to deliver a payload or malicious outcome to the system. The payload can include a number of actions ranging from unauthorized access to denial of service to remote code execution to escalation of privileges. The payload artifacts left behind will be dependent on what action was taken.

Indicators


The last layer in the model is the indicators layer. The layer is not only where the information and artifacts about how the attack was detected would go but it also encompasses all of the artifacts showing the post compromise activity. The reason for organizing all the other remnants left by the attacker into this layer is to make it easier to identify the attack vector artifacts (exploit, payload, and delivery mechanisms.) This results in the layer being broad since it contains all of the post compromise artifacts such as downloading files, malware executing, network traversal, or data exfiltration.

The Model is Cyclical


The Compromise Root Cause Analysis Model is a way to organize information and artifacts to make it easier to answer questions about an attack. More specifically to answer: how and when did the compromise occur? Information or artifacts about the compromise are discovered by completing examination steps against any relevant systems involved with the attack. The model is cyclical; as each new system is discovered the model is used to determine how the system was compromised. This ongoing process continues until each system involved with an attack is examined to confirm if it truly was a part of the attack.

To illustrate, take the hypothetical scenario of an IDS alert indicating an organization’s employee laptop is infected with malware. The IDS signature that flagged the network traffic is shown below (signature was obtained from the Emerging Threats emerging-botcc.rules.) As can be seen in the rule, the laptop was flagged for visiting an IP address associated with the Zeus Trojan.

 
The network packet captured in the IDS alert indicates the employee is a remote user connected through the organization’s VPN. The network diagram below shows the organization’s network layout and where this employee’s laptop is located.


The investigation into the employee’s laptop - remotely over the network – located the Zeus Trojan on the laptop. The examination continued by doing root cause analysis to determine how the laptop became infected in the first place. The employee was surfing the Internet prior to connecting to the organization’s network through the VPN. A drive-by attack successfully compromised the laptop when the employee visited the organization’s website. The IDS alerted on the infection once the laptop connected through the VPN. The investigation now uncovered another system involved with the attack (organization’s web server) and its location is shown below.


The organization’s main website is compromised and serving malware to its visitors. The investigation continues by moving to the compromised web server. The Root Cause Analysis Model is applied to the server to determine how it became compromised in the first place. The answer was an attacker found the webserver was running an outdated Joomla plug-in and exploited it. The attacker eventually leveraged the compromised web server to deliver malware to its visitors.

In this hypothetical scenario, the Compromise Root Cause Analysis Model was initially applied to a compromised laptop. The source of the attack pointed to another system under the control of the organization. The investigation continued to the newly discovered system by applying the Compromise Root Cause Analysis Model against it. The attack vector pointed to an attacker from the Internet so at this point all of the systems involved in the attack have been investigated and the root cause identified. If there were more systems involved then the cyclical process continues until all systems are investigated. The Compromise Root Cause Analysis Model enabled the attack vector for each system to be determined and the incident information discovered can then be further organized using other models. For example, the overall attack can be described using the Lockheed Martin's Cyber Kill Chain model.

Applying the Compromise Root Cause Analysis Model


The Compromise Root Cause Analysis Model is a way to organize information and artifacts to make it easier to answer questions about a compromise. The model can be applied to systems to either confirm how they were compromised or to determine if they were compromised. The article Malware Root Cause Analysis goes in to detail about how to apply the model for a malicious code incident involving a single system. However, the model is not limited to only malicious code incidents. It can be applied to any type of security incident including: unauthorized access, denial of service, malicious network traffic, phishing, and compromised user accounts. To demonstrate the model’s versatility, it will be applied to a hypothetical security incident using data from a published article. The incident is for a compromised Coldfusion webserver as described in the An Eye on Forensics's article A Cold Day in E-Commerce - Guest Post. The data referenced below either was influenced/borrowed from either the previously mentioned article, the Coldfusion for Pentesters presentation, or made up to appear realistic.

Webserver Compromise


An IDS alert flags some suspicious network traffic for an external system trying to connect to an organization’s Coldfusion web server located in their DMZ. The organization monitors for access attempts to the Coldfusion administrator web panel including access to features such as scheduling tasks. The external system triggered the IDS signature shown below because it accessed the Coldfusion’s scheduleedit located at hxxp://www.fake_site.com/CFIDE/administrator/scheduler/scheduleedit.cfm on an established session.


The reason the IDS alert is concerning is because what accessing scheduleedit means. One method an attacker can use to upload code on to a compromised Coldfusion server is by leveraging the scheduled tasks. The attacker can schedule a task, point it to their program’s location on a different server, and then have the task save it locally to the Coldfusion server for them to use (see page 85 in this presentation.) Accessing the interface to edit scheduled tasks is reflected by “scheduleedit” appearing in the URL. The IDS alert is triaged to determine if the Coldfusion server was successfully compromised and if an attacker was uploading anything to the server using the scheduled tasks feature.

The Coldfusion instance is running on a Windows 2008 server with IIS and its IP address is 192.168.0.1. The IIS log was reviewed for the time in question to see the activity around the time the IDS alert triggered.

2015-03-10 22:09:00 192.168.0.1 GET /CFIDE/Administrator/scheduler/scheduletasks.cfm - 80 – X.X.X.X fake-useragent  200 0 0 5353

2015-03-10 22:09:10 192.168.0.1 GET /CFIDE/Administrator/scheduler/scheduleedit.cfm submit=Schedule+New+Task 80 - X.X.X.X fake-useragent 200 0 0 5432

2015-03-10 22:09:15 192.168.0.1 GET /CFIDE/Administrator/scheduler/scheduletasks.cfm runtask=z&timeout=0 80 – X.X.X.X fake-useragent 200 0 0 1000

2015-03-10 22:11:15 192.168.0.1 GET /CFIDE/shell.cfm - 80 – X.X.X.X fake-useragent 200 0 0 432

The IIS logs showed the activity that tripped the IDS sensor occurred at 2015-03-10 22:09:10 when the external system with IP address X.X.X.X scheduled a new task successfully. Notice the 200 HTTP status code indicating the request completed successfully. This single entry answers one of the questions. The attacker did compromise the Coldfusion server and has administrator rights to the Coldfusion instance because they were able to access the schedule tasks area within the administrator panel. The next log entry shows the scheduled task named “z” executed at 2015-03-10 22:09:15 and shortly thereafter the attacker accessed a file named “shell.cfm”. Applying the Root Cause Analysis Model to this incident results in this activity along with the IDS alert being organized into the indicators layer. The activity is post compromise activity and the model is being used to identify the attack vector. The investigation continues to see what remnants the attacker left in the logs just prior to tripping the sensor while trying to upload their web shell.

The IIS log was reviewed to see what occurred prior to 2015-03-10 22:09:10 for the attackers IP address X.X.X.X. The few records are listed below:

2015-03-10 22:08:30 192.168.0.1 GET /CFIDE/adminapi/administrator.cfc method=login&adminpassword=&rdsPasswordAllowed=true 80 – X.X.X.X fake-useragent 200 0 0 432

2015-03-10 22:08:40 192.168.0.1 GET /CFIDE/administrator/images/icon.jpg 80 – X.X.X.X fake-useragent 200 0 0 432

The prior activity shows the attacker requesting a strange URL followed by successfully accessing the icon.jpg image file. Searching on the strange URL reveals it’s an Adobe ColdFusion Administrative Login Bypass exploit and when successful it provides access to the admin panel. This remnant is organized into the exploit layer. The payload of this exploit is direct access to the admin panel. There is no delivery mechanism for the payload. When the admin panel is accessed certain files are loaded such as images. In this scenario one of the images loaded by default is the file icon.jpg. This remnant indicates the attacker successfully accessed the admin panel so it means the exploit worked and the payload was admin access. The access to the icon.jpg file is organized into the payload layer. At this point the following layers in the Root Cause Analysis have been completed: indicators, payload, deliver mechanism, and exploit. The remaining layers are the delivery mechanism for the exploit and source. The attacker used a tool or web browser to attack the server so the delivery mechanism for the exploit is HTTP and the source of the attack is somewhere from the Internet.

The Compromise Root Cause Analysis Model was applied to the hypothetical web compromise security incident and it made it easier to review the remnants left by the attacker to identify the attack vector they used.

Root Cause Analysis Is Easier with a Methodical Process


The Compromise Root Cause Analysis Model is a cyclical methodical process one can use to perform root cause analysis. The model is way to organize information and artifacts discovered during investigations for each system involved in the attack. The model is a repeatable investigation process enabling the questions of how and when did the compromise occur to be answered.



References

Carvey, H. (2005). Locard's Exchange Principle in the Digital World. Retrieved from http://windowsir.blogspot.com/2005/01/locards-exchange-principle-in-digital.html

Harrell, C. (2010). Attack Vector Artifacts. Retrieved from http://journeyintoir.blogspot.com/2010/11/attack-vector-artifacts.html

Harrell, C. (2012). Compromise Root Cause Analysis Model. Retrieved from http://journeyintoir.blogspot.com/2012/06/compromise-root-cause-analysis-model.html

Harrell, C. (2012). Malware Root Cause Analysis. Retrieved from http://journeyintoir.blogspot.com/2012/07/malware-root-cause-analysis.html

Harrell, C. (2014). Malware Root Cause Analysis Dont Be a Bone Head Slide Deck. Retrieved from http://journeyintoir.blogspot.com/2014/06/malware-root-cause-analysis-dont-be.html

Hogfly. (2008). Footprints in the snow. Retrieved from http://forensicir.blogspot.com/2008/12/footprints-in-snow.html
  1. Sunil Varkey

    Very good and informative work. Thank you

  2. Corey,

    Your post is yet another prime example of what's missed in a great many organizations. There is no investment into an RCA process, and as such, organizations that get compromised throw out a SWAG as to the IIV, and never actually validate the root cause. This leads to poor business decisions being made based on bad technical information, and the organization continues to be compromised.

    I presented my findings regarding a RAT infection at a conference for "APT researchers" a while back. I described the RAT, and most in the room were familiar with it. When I asked, "what is the infection vector?", everyone that responded said, "spear phish". That's the easy route...see the IDS/IPS alert for the RAT on the wire, assume it was the result of a phish. This one was installed by the user, from a USB drive...information which radically changes the approach and the solution.

    Everything that is new and unfamiliar is "hard" and "takes too much time". Invest in the capability to do this, either by developing staff or signing a retainer contract. The other option is to just turn off your IPS, disable logging, etc. After all, if you're not going to do the RCA, what's the point of all that instrumentation?

  3. @Harlan,

    Great points; thanks for commenting. I think one of the big advantages to doing root cause analysis (besides the knowing what an attacker did) is to help make better business decisions. Years ago when I just started doing RCA on a consistent basis I was helping an acquaintance’s organization with a malware infection. In this organization, an assumption was made about how the infection occurred. This resulted in advice being provided to the user and a general announcement to the whole organization with similar advice. After I did RCA I determined the infection vector was different than the assumption. In the end the wrong advice was provided and the time/resources spent doing it would have zero impact on reducing a similar infection from re-occurring.

  4. Ionut Ionescu

    Dear Corey,
    Good article, especially as you've added examples, which bring the power of the methodical approach of the CoRCAM model to the fore.
    I agree with you and Harlan, many organisations see RCA as "too hard", or "too techie and time consuming" and senior management expects quick solutions.
    Sometimes, after organisations invest in tools and instrumentation, they forget to invest (or to keep investing) in people and therefore, they miss the insights that good RCA would bring them. They rarely have a good threat taxonomy and so they cannot focus the security budget on the important areas. Sometimes, they even outsource operational security and incident response and say they will concentrate on risk management instead, without ensuring that the managed services provider has the skills and capabilities to do RCA and to really understand how to protect them on an on-going basis (i.e. not just to administer the protective controls). Security can be a lonely job sometimes.

Post a Comment