Journey Into Incident Response

Helping train a computer security incident response team (CSIRT) comes with the territory when building out an enterprise incident response process. As I was reading No Starch's recently released Penetration Testing A Hands-On Introduction to Hacking book by Georgia Weldman I saw an opportunity. The book target's audience are beginners. In the author's words "I thought an introduction to pentesting would make the biggest impact on the audience I most wanted to reach." The opportunity I saw was the book can also be leveraged by people new to incident response. Specifically, new to responding to systems hit with either client-side or web application attacks. To see how a beginner pen testing book can become a beginner incident response book read on.

Practical Approach

No Starch Press has a series of books with the word "practical" in their title. These books are the equivalent of an entire training for the cost of a book. The reason is because they take a practical approach. They provide supplemental material which allows the reader to do the same hands-on activities they are reading about throughout the book. Not only does the reader learn through reading but they learn by doing using the same tools and data described in the books. The Penetration Testing book may not have "practical" in its title but it follows the same approach, which is why it is a great book for beginners.

The book starts out by having the reader configure four different virtual machines (VMs). One VM is a Kali box, which is the platform the reader launches attacks from. The three other VMs (Windows XP, Windows 7, and Ubuntu) are configured very vulnerable to make it easier to attack them. As the reader progresses through the book they read about performing various attacks followed by actually doing the attacks against the vulnerable VMs. It's an excellent approach to expose beginners to pen testing. On the flip side, it's also an excellent approach to expose beginners to incident response. Not only does the reader get familiar with different attacks, vulnerabilities, and the payloads but they are creating compromised vulnerable VMs to examine after the fact. So the reader can compromise a system then examine the system to see what attack artifacts are present.

The book does cover numerous vulnerabilities and exploits. However, were two attack categories I thought excel at helping incident response beginners.  Over the years, attacks targeting client-side and web applications have been a significant issue for organizations. These attacks are so common that sooner or later they will be encountered by most incident responders. As a result, exploring these attacks is of utmost important for people new to the IR field and the book addresses these attacks thought-out different chapters. The reader may gain a thing or two from the other attacks (i.e. MS08-067 exploitation) but the exploits used in those attacks are something they may not see outside of training environments due to their age.

One Configuration Change for XP VM

One step to perform in addition to what is outlined in the Setting Up Your Virtual Lab chapter is to enable logging for Filezilla (FTP service) on the XP VM. To do so, access FileZill'a admin panel then select Edit -> Settings to get to the configuration window. In the logging section check the Enable Logging selection.

Where to Look

Before diving into the client-side or web application attacks I need to take a moment to outline where the reader needs to look for attack artifacts. I won't go into detail about all of the artifacts and will only cover the pertinent ones to explore the attacks for the Windows VMs.

Windows XP

One of the vulnerable applications installed on Windows XP is XAMPP. XAMPP is an "Apache distribution containing MySQL, PHP and Perl; this provides the VM with vulnerable web services. The following are the logs and artifacts of interest the reader would want to review after the attacks:

Apache logs location:  C:\xampp\apache\logs
FileZilla log location: C:\xampp\FileZillaFTP\Logs

The Windows XP operating system artifacts of interest are:

NTFS Master File Table ($MFT)
NTFS Logfile ($Logfile)
Internet explorer browser artifacts

Windows 7

One of the applications installed on the Windows 7 VM is the web service IIS. The book provides a vulnerable web application (called BookApp) that runs on top of IIS. The ISS logging settings can be left in their default configuration. The following are the logs and artifacts of interest the reader would want to review after the attacks:

IIS logs location: C:\inetpub\logs\LogFiles\W3SVC1
Windows HTTP.sys error logging location: C:\Windows\System32\LogFiles\HTTPERR

The Windows 7 operating system artifacts of interest are:

NTFS Master File Table ($MFT)
NTFS Logfile ($Logfile)
NTFS Change Journal ($USNJrnl)

Stay Organized

The last tidbit I'll cover before diving into the attacks is the need to stay organized. As I worked my way through the book I took notes to record when and what attack I performed against what vulnerable VM. This reference made things easier as I was examining the artifacts; it was easier to match up attacks against activity in the artifacts. The screenshot below shows a portion of the notes I kept


Keeping notes is not a necessity but something I highly recommend for beginners.

Client side attacks

Artifacts left by client-side attacks have been well documented on this blog. To name a few: Java was documented here and here, Silverlight discussed here and here while Adobe Reader was mentioned here. Even though some artifacts have already been documented replicating client-side attacks is more beneficial than only reading research done by others. Actually performing the attack against a vulnerable VM shows a completely different perspective and one that can't be attained by just reading about artifacts. The image below shows the Windows XP parsed MFT around the time the system was compromised with the Aurora Internet Explorer exploit (the exploit is the htm page.)

Web Application attacks

The book covers numerous attacks against web applications on the Windows XP, Windows 7, and Ubuntu VMs. The variety of the attacks I thought provided a good starting point for exploring the different ways web servers and applications are compromised. In this post I'm only discussing a few items to demonstrate the traces left in certain artifacts showing these attacks took place.

Vulnerability and Nmap Script Scan Against Windows 7

Chapter 6 walks through a process for finding vulnerabilities on a VM. The chapter has the reader use the following methods: Nessus (vulnerability scanner), Nmap scripting engine (port scanner), Metasploit (exploitation framework), Nikto (web application scanner), and one manual technique. Exploring this aspect for incident response is important since for a system to be compromised a vulnerability had to be exploited. Seeing the traces left by the different methods provides an idea about what one may come across during examinations.

The image below shows the Nessus vulnerability scan activity in the Windows 7 IIS logs. The image is only a small portion of the logs but it illustrates how noisy vulnerability scanners are. This is obvious activity that most monitoring tools will detect fairly quickly. It's also activity that will most likely be tied to people authorized to perform this work for an organization and not an attacker.


In addition to the IIS logs in the W3SVC1 folder, the HTTP.sys error logging provides additional clues about the vulnerability scanning activity as shown below.


The Nmap script scan activity leaves traces in the logs as well but nothing close to the amount left by a vulnerability scanner. The image below shows the Windows 7 IIS logs where the Nmap script scan activity was located (note: the activity will be dependent on the script ran and won't always appear as it does in the image.)

Command Execution Vulnerability Against Windows 7 IIS

Part III in the book has eight different chapters focused on different types of attacks. I won't rehash what all of the attacks are since the book's table of contents is adequate enough. One of the attacks was against a command execution vulnerability, which allows a person to execute commands against the system through the web application. In order to see the command's output - since the web application doesn't display - it requires the attacker (aka reader) to redirect it to a file on the server and then the file is accessed.

The image below shows the command being executed though the "subscribe to a newsletter" functionality (line in the middle.) It's not too apparent what is taking place since the HTTP method used is a POST to send data to the server. The data is not reflected in the URL. However, shortly after the POST there is a GET request for a file on the server named test.txt and the 200 status code means OK (file is there). The POST method may not be too revealing but the GET method reveals a file is on the server.


The server's MFT (parsed into a timeline) provides more clarity about the attack as shown below. The ipconfig.exe program executed at the same time as a new user (georgia) subscribed to a newsletter. Looking at the contents of test.txt shows it contains the ipconfig command output.

PHPAdmin SQL Command Against Windows XP

Another attack that was discussed is leveraging the PHPAdmin panel to execute SQL commands directly against a backend database. The attack involved executing a SQL command that creates a simple web shell on the system; shell only lets you execute commands in a supplied URL.

The Apache logs reveal the attack as seen in the image below. The line for the GET request at 21:17:01 (EDT) shows the URL containing the exact SQL command that creates the file called shell.php. The other highlighted lines show a request for the shell.php file followed by the various commands that were executed through the shell.


The server's MFT parsed into a timeline provides more clarity about the attack as shown below (time is in UTC).

Conclusion

As described in the book's introduction it is for beginners. Those who are new to pen testing field. I also see value in the book for incident response. For those who are either new to incident response without a background in offensive security or those who haven't been exposed to compromised web servers and applications. One shouldn't expect to know everything about investigating systems impacted by client-side and web attacks after working their way through the book. Heck, the reader may never see these exact attacks (outside of pen testers) against production systems. However, the reader will have a better understanding about attack vectors. The means by which an attacker can gain access to a system in order to deliver a payload or malicious outcome. The reader will have an even better understanding about the attack vector artifacts left on those systems. For anyone fitting what I just described should definitely pick up this book, build the VMs, attack the VMs, and then discover what traces they can find. It won't be the last step in their journey but it will be a good first one.

Security information and event management (SIEM) has been an area where I have spent considerable time researching. My research started out as curiosity to see if the technology could solve some problems then continued to get organization buy-in followed by going all in to architect, implement, and manage a SIEM for my organization. Needless to say, I did my homework to ensure our organization would not follow in the footsteps of others who either botched their SIEM deployments and/or ended up with a SIEM solution that doesn't meet expectations. In this linkz post I'm sharing my bookmarks for all things SIEM.

We Need a Response Team

In the movie Avengers after everything else failed Nick Fury was there saying "we need a response team." Regardless what the World Security Council said Nick kept saying "we need a response team." The Avengers is a great movie with many parallels to incident response (yes, I cut up the movie to use for in-house training and it's on the deck for my next presentation). Deploying a SIEM - as with any detection technology - will result in things being detected. After things are detected then someone will need to respond to it to investigate it. As a result, my initial focus for my SIEM research was on designing and implementing an enterprise-scale incident response (IR) process. For a bunch of IR linkz see my post Linkz for Incident Response. My initial focus on IR wasn't solely because things will be detected. I also see IR activities merging with security monitoring activities. To see more about this thought process refer to one of the links I shared in my post by Anton Chuvakin titled Fusion of Incident Response and Security Monitoring?

General SIEM Information

The first few linkz provide general information about SIEM technology. Securosis did a series about SIEMs and three of their posts provide an overview about what they are. The posts are: Understanding and Selecting SIEM/Log Management: Introduction and Understanding,  Selecting SIEM/LM: Data Collection, and Understanding and Selecting SIEM/LM: Aggregation, Normalization, and Enrichment.

Hands down the best content I found about SIEM was written by Anton Chuvakin who is a Gartner analyst. This post links to a lot of his material and the following posts are self explanatory: SIEM Analytics Histories and Lessons and SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?

Rounding out my general links is another one by Securosis. In their series this post actually came a lot later (after the articles I listed in the Planning the SIEM Project section) but I think the content is more important to have up front. To get anywhere with a SIEM in an organization someone has to agree to it. Someone who has the ability to purchase it. This is where Securosis's next article comes into play since it provides examples of the justifications one could use. For more see the post Understanding and Selecting SIEM/LM: Business Justification.

Planning the SIEM Project

The next round of links is what I found to be gold designing my organization's SIEM solution. One thing I didn't want to happen was to follow in the footsteps of so many companies before me. They buy and acquire a SIEM then end up with a solution that doesn't solve any of their problems. Bumbling a SIEM project is not something I wanted in my rear view mirror. To avoid this from happening I spent considerable time researching how to be successful in SIEM deployments so I could avoid the pitfalls that others have fallen in. Sitting where I am today and reflecting back I'm really glad I did my homework upfront as our SIEM project continues along addressing our use cases.

The best reference I found to help architect a SIEM solution is a slide deck by Anton Chuvakin. The presentation is Five Best and Five Worst Practices for SIEM and it outlines the major areas to include in your SIEM project (16 to be exact). It may not cover everything -such as building rules, alarms, and establishing triage processes - but it does an outstanding job outlining how to avoid the pitfalls others have fallen in. If anyone is considering a SIEM deployment or in the midst of a SIEM deployment then this is the one link they will want to read.

Continuing on with links from Anton that provide additional details are the following: On Broken SIEM Deployments, Detailed SIEM Use Case Example, On Large-scale SIEM Architecture, On SIEM Deployment Evolution, and Popular SIEM Starter Use Cases. All of these posts are worth taking the time to read.

Similar to the amount of information Anton makes public Securosis also has a wealth of great SIEM posts. The following posts are great since they discuss use cases: Understanding and Selecting SIEM/LM: Use Cases, Part 1 and Understanding and Selecting SIEM/LM: Use Cases, Part 2.

Selecting a SIEM

At some point a SIEM may be bought and it is helpful to know what should be taken into consideration. Again, Anton and Securosis have posts addressing this as well. Anton has the two posts Top 10 Criteria for a SIEM? and On Choosing SIEM while Securosis has their white paper Understanding and Selecting SIEM/Log Management.

The last reference to use for SIEM selection is the analysis done by Gartner. Regardless what people may think about how Gartner comes to their conclusions the company does publish quality research. The SIEM Magic Quadrant analyzes the various SIEM products, ranks them, and discusses their pros/cons. To get the quadrant you need to download it from a SIEM vendor and yes that vendor will start contacting you. To find where to download it just google "SIEM Magic Quadrant 2014" for this year's report.

Managing a SIEM

Up to this point there are a lot of references one can use to help in: exploring SIEM technology, selecting a SIEM, and designing a SIEM solution. However, I started to see a drop in information for things that occur after a SIEM is up and running. There appears to be very little related to SIEM operations and how others are leveraging SIEM to solve security issues. I only have two linkz both by Anton which are On SIEM Processes/Practices and On SIEM Tool and Operation Metrics.

Due to the lack of SIEM operational literature I branched out to look at resources related to security monitoring. To a certain extent this was helpful but again not exactly what I was looking for. What I've been looking to find is literature focused on detection, intelligence, and response. What I came across was more of the general information as opposed to operational information. One slide deck I found helpful for identifying operational areas to consider was Brandie Anderson's 2013 SANs DFIR Summit slide deck Building, Maturing & Rocking a Security Operations Center.

Keep Waiting for a Decent SIEM Book

The book Security Information and Event Management (SIEM) Implementation is the only SIEM book on the market. I saw the poor Amazon reviews but I opted to take a chance on the book. I was designing and implementing a SIEM so I wanted to read anything I could on the subject. I gave this book two chances but in the end it was a waste of my money. The book doesn't address how to design and implement a SIEM solution nor does it properly cover SIEM operational processes. For those looking for a SIEM book I'd keep waiting and in the meantime read the linkz in this post. I wanted to give others considering this book a heads up.