Background

The Windows Registry file format consists of a set of allocation units, known as HBIN blocks. Each block is subdivided into cells, which form the basic unit of content in the format. Although cells may contain pure binary data, many cells are used solely for metadata storage. These include: nk-records (nodes in the tree-like structure of the Windows Registry), lf-records (point to children of a node), and vk-records (describe a single Registry value). Python-registry exposes a set of classes in the RegistryParse module that parse and read these low-level structures. One RegistryParse class maps to each structure, and each exposes a set of convenient methods for accessing metadata and other referenced structures.

For example, once a Windows Registry file has been loaded by python-registry, you can select an HBIN block, and iterate through each cell. The following code listing shows how you can identify free cells, which may be extracted so you can search for deleted Registry keys.

f = open("NTUSER.DAT")
buf = f.read()
regf = RegistryParse.REGFBlock(buf, 0, false) 

for HBIN in regf.hbins():
    for cell in HBIN.cells():
        if cell.is_free():
            print "Unallocated cell at offset 0x%x" % (cell.offset())

There are few libraries in the wild that parse Windows Registry files and many fewer still that encourage interaction with the underlying structure. Enterprising forensicators should recognize the opportunity for using python-registry to enable future research in Registry key recovery.

In addition to low-level parsing classes, python-registry provides a high-level interface similar to those exposed by the Microsoft Windows API. The Registry module abstraction layer removes from the user details about allocation units and records and provides familiar sounding RegistryKey and RegistryValue classes. Once a Windows Registry file has been loaded, instances of the RegistryKey class are organized into a tree structure, and may be associated with a list of RegistryValues. This tree mirrors the structure you would see if you were to explore the corresponding Windows Registry with RegEdit.

Forensics with python-registry

Similar to Harlan Carvey’s RegRipper, python-registry is particularly suited for forensic analysis. Python-registry works directly with Registry hive files — not the live hives — so it can easily be used on a forensic workstation after an acquisition.

Recently, I used python-registry to help identify compromised servers during an incident response from Mandiant’s New York City office. I suspected a system was infected with malware based on remote login events found on another system and wanted to do a deeper dive. I used Mandiant Intelligent Response® (MIR) to acquire the Registry hive files, including C:\WINDOWS\system32\config\software, from the remote system and downloaded them to my Linux laptop.

Next, I wrote the following ten-line Python script using python-registry to print out each key and its last modified date, while filtering on a suspicious time window.

from Registry import Registry

def rec(key):
    if MIN_DATE < key.timestamp() < MAX_DATE:
        print "%s    %s" % (key.timestamp(), key.path())
    for subkey in key.subkeys():
        rec(subkey)

f = open("software", "rb")
r = Registry.Registry(f)
rec(r.root())

Immediately an entry with the path “$$$PROTO.HIV\Microsoft\Windows\CurrentVersion\Run” caught my eye. What was changed during the time window?

The python-registry package contains a set of tools that have been built using python-registry to demonstrate its usage and are contained in the samples directory. The regview.py GUI tool (similar to RegEdit.exe) can be used to explore the “Run” key as seen in the following figure.

A malicious entry showed that an attacker had gained persistence by adding his malware to execute on startup. The system was definitely compromised. I continued the Registry analysis with a few more experimental scripts I had been developing, including a shellbag parser, and the client was happy.

Python-registry is effective because it is flexible and intuitive. The package is a library that is easily integrated into both one-off scripts and larger projects. For example, I’ve found that using python-registry with an interactive Python console (like IPython) is an effective environment for rapid triage and analysis. Python-registry converts Registry values into native data types, which makes manipulation of data familiar to a Python programmer. This means it is easy to pass the output of a python-registry method call to a decoding function developed elsewhere.

Using the library
Let’s explore how you can easily make use of this library. You can recurse across all RegistryKeys and perform an action as seen in the following code snippet. This utility encapsulates this logic by applying a function to each RegistryKey.

def rec(key, f):
    """
    Recurses across all RegistryKeys and applies the function f.

    key : A Registry.RegistryKey
    f : A function taking one argument, a Registry.RegistryKey
    returns : None
    """
    f(key)
    for subkey in key.subkeys():
        rec(subkey, f)

Metadata about a particular RegistryKey can be accessed through the methods .name(), .path(), and .timestamp(). RegistryKey.subkeys() returns a list of child RegistryKey objects that are lazily parsed. By lazily parsing the file, python-registry reduces memory consumption and minimizes initial load time. The following code listing shows how to print out all Registry key paths.

from Registry import Registry

def print_key(key):
    """
    Print the path of a RegistryKey.

    key : A Registry.RegistryKey
    returns : None
    """
    print key.path()

f = open("NTUSER.DAT", "rb")
reg = Registry.Registry(f)
rec(reg.root(), print_key)

In python-registry, a Registry file is initially loaded by constructing a Registry object. The .root() method returns the root RegistryKey, and serves as the starting point for most functions. Alternatively, a script may use the .open() method to attempt to open a RegistryKey by path.

I could have used a modified version of the print_key() function in the earlier example to identify the malicious “Run” key; however, a slightly more interesting application might be to print out all Registry values of string type that contain given string. In the following code listing, we iterate over the RegistryValues associated with each RegistryKey while filtering out non-string values. We apply the find_microsoft() function to each RegistryKey using our recursive utility rec(), and it prints out the RegistryValue hits.

def find_microsoft(key):
    """
    Prints Registry keys whose values contain the string “microsoft”.

    key : A Registry.RegistryKey
    returns : None
    """
    for value in [v.value() for v in key.values()
                                  if v.value_type() == Registry.RegSZ
                                  or v.value_type() == Registry.RegExpandSZ]:
        if "microsoft" in value:
            print key.path()

rec(reg.root(), find_microsoft)

The .values() method of a RegistryKey returns a list of RegistryValues associated with the key. RegistryValues are a conceptually a tuple of (name, type, data), which map to the methods .name(), .value_type(), and .data(). The type of a RegistryValue is represented by an integer ranging from 0×0 to 0xB and may mean something like “string”, “binary”, or “dword”. Fortunately, python-registry provides a set of constants, like Registry.RegSZ, Registry.RegBin, and Registry.RegDWord, to improve readability. When the data is requested from the RegistryValue using the .data() method, it is first converted into the native Python datatype. For example, strings are converted from ASCII or Unicode into native strings, and numeric types into integers.

Documentation

Live documentation for python-registry can be found here. Most classes and methods have been documented; however, for the few that have not, the source code is written to be self-documenting. Python-registry is released under the Apache 2 license, so all users should feel encouraged to browse basic structure of the source. I hope that clear and accessible source code that implements the Windows Registry file format can serve to be a central repository for knowledge regarding the format.

Conclusion

Python-registry is a package written for those who enjoy Python and Windows Registry forensics, and potentially platforms other than Microsoft Windows. It implements each of the lower level structures, yet also presents a high-level interface that encourages both one-off scripts and substantial processing. Since python-registry is released under the Apache 2 license, all are encouraged to examine and patch the source code. The homepage for python-registry is found here, and the latest source can be downloaded from GitHub here.

Acknowledgements

• “WinReg.txt” by B.D. found at http://pogostick.net/~pnh/ntpasswd/WinReg.txt
• “The Windows NT Registry File Format” by Timothy Morgan found at http://www.sentinelchicken.com/research/registry_format
• “The Internal Structure of the Windows Registry” by Peter Morris found at http://amnesia.gtisc.gatech.edu/~moyix/suzibandit.ltd.uk/MSc/

I look forward to seeing everyone at the conference. Rob Lee has put together what I believe everyone will find is an informative show. Do not forget to catch Kris Harms’ talk and see if you can find evil or not.

Speakers: Jamie Butler and Peter Silberman
Date: Tuesday, July 7, 3:10pm – 3:50pm
Title: Memory Forensics and Analysis

The memory in today’s business desktops is now larger than the hard drives that were in systems just a few years ago. Traditionally, forensic analysis has meant taking an image of the hard drive and sifting through files. This is only half of the story and can no longer be considered sufficient. Attackers are writing less to disk and hiding more in the ample memory users now enjoy. Memory analysis – once a niche function performed by only the most advanced forensic investigators – is now mainstream and common in professional investigations. Tools have been written to make memory analysis as easy for the investigator if not easier than hard drive analysis and in a fraction of the time. In this talk, we will show you how to quickly identify suspicious things in memory without having to be a reverse engineer. This talk will feature research, use cases, and real world examples.

Speaker: Kris Harms
Date: Tuesday, July 7, 9:30am – 10:30am
Title: Evil or Not? Rapid Confirmation of Compromised Hosts Via Live Incident Response

During this presentation, attendees will learn practical, tried, and true methods to review live incident response information. You will obtain the skillful eye required to quickly confirm or dispel if a system is compromised. Recent case data from PCI credit card breaches as well as the Advanced Persistent Threat (APT) will be used as samples. Armed with this knowledge, you will excel as an initial responder to any incident.

I recently spoke at the DoD cybercrime conference on Advanced Persistent Threat (APT) forensics.  During the presentation I talked about several ways you can use forensics to answer difficult questions that arise once an APT incident is identified.  Some of these questions are:

• What was the initial vector?

• What did the attackers do exactly?

• Was any sensitive data exposed for exfiltrated?

• How do we successfully respond to the incident?

These questions can usually be answered easily if the response team has the right tools and methodology.  This is where the APT M-unition pack will help.  In this package are templates for forensic methodology, EnScripts to help with analysis, and the presentation given at DoD cybercrime. The forensic methodology template can be opened with NoteCase. NoteCase is available at the following link:

NoteCase

If anyone has questions on the use of the EnScripts or steps in the methodology feel free to contact me by email at kelcey.tietjen@mandiant.com. The APT M-unition pack can be acquired from below:

APT M-unition Pack

Kelcey

So I searched the Internet and asked colleagues in search of an application that would allow me to quickly remove lines from a text file. I wanted to be able to scroll through the file, and as I identified text that was irrelevant, remove lines from the display that contained that text. Sounds simple enough, right? But after searching for about a week, it seemed that no one knew of such a tool. Many suggested using a series of “grep -v” commands under Linux or with the Win32 Unix tools. Even though I am an avid command line user and a fan of using grep and Linux, that solution was a bit too clunky and not the sort of streamlined workflow I was looking for. A week more frustrated, I couldn’t find any app like the one I was searching for, so I decided I would have to make it myself.

Over two days I wrote a very basic C# application using Microsoft Visual Studio Express. The application had a single function – load a text file into a textbox, let me select text, and remove all lines with that text from being displayed. The original file was never modified, but they weren’t shown to me.

I used my new tool on a selection of the Windows event logs and immediately saw the benefit; with some files, this technique of removing lines quickly eliminated about 80-90% of the events. This let me focus closely on the remaining events, which allowed me to find evil and solve crime faster than ever!

After a little use I realized that thought it would be cool if I not only removed lines, but also found where certain strings occurred throughout a file. I started with the idea of statistical analysis on the file – generate information about each word that indicated frequency, distribution, etc. The problem with that is that I couldn’t come up with any good way to represent the results. After explaining the idea to my Mandiant colleague, Lindsey Lack, he simply said “I’m a graphical person. Why don’t you make a visual representation of the file and display information graphically?”. GENIUS!

Our idea was to depict the file as a graphic on which we could highlight areas on the graphic that corresponded to a key word or phrase. The depiction would immediately give you a sense for frequency and distribution. So with help from one of Mandiant’s Intelligent Response developers, Matt Frazier, we created a C# control that displays the file as a graphic. The graphic represents a sort of super zoomed-out version of the file. Lines from the original file are displayed as graphics lines (no text) on the screen. The lines displayed are proportional to the line lengths in the file. So you have a graphic on the screen next to the text box that proportionally represents the entire file. So, back to the Windows event logs.

See the line numbers jump in the text window. Hidden lines are indicated in the overview with grey lines.

I opened the log and selected a username in question identified through the previous analysis I did. I right-clicked and selected the new function – “Highlight”. The graphic lit up with small red lines (highlights), indicating each exact location that username appeared in the file. I immediately noticed something odd – the red highlights appeared in a fairly regular pattern, except around a certain spot, where there were a number of red highlights that just appeared out-of-place in comparison to the rest. We made the graphic clickable, so I clicked in that area and the textbox advanced to that portion of the file. The log entries that came up were very late at night – a time when this user should not have been accessing this system. Further investigation revealed the user’s account was compromised, malware was installed, and a number of other things happened that day.

The lines highlighted in the salmon color in the text box correspond with the yellow highlights in the overview window.

Evil found. Crimes being solved.

http://www.mandiant.com/software/highlighter.htm

http://mandiant.invisionzone.com/index.php?showforum=15