Memory Mapped I/O Trace

Mmiotrace is a toolbox for tracing memory mapped I/O access within the kernel. It can be used to record how a driver module uses MMIO reads and writes, for reverse engineering and debugging.

The supported architectures are x86 and x86_64, they both seem to be pretty stable. If you get a crash, see if there's any information about it in your log files and contact PekkaPaalanen by email or at #mmio-trace IRC channel on Freenode. Old debugging notes are in page MmioTraceDebugging.

Features:

timestamped events
a mechanism to inject markers into the log while tracing
no kernel patching
collect block-reads/writes in mmio-parse
use register names (Rules-ng)

The modern format of an mmiotrace log is described in MmioTraceLogFormat.

Instructions for specific kernel versions

The latest kernel (or at least 2.6.28) is the recommended one, if you can choose. The installation and usage instructions depend on your kernel version.

These kernel version notes are for people doing mmiotrace dumps. People analysing mmiotrace dumps likely want to get the out-of-tree mmiotrace, too, because it contains useful analysis tools.

2.6.27 and later

Mmiotrace is in the kernel, no need to fetch or patch anything extra. Please read the instructions in the file Documentation/tracers/mmiotrace.txt or Documentation/trace/mmiotrace.txt in your kernel source tree. No special programs are needed to record the output. For submission guidelines, see MmioTraceHowto section "Sending results".

2.6.29: Crash fixes. If you had trouble getting a trace due to the machine freezing, that is hopefully fixed now.
2.6.28: Added marker support, fixed the 8-bit write bug.
2.6.27: The first kernel carrying mmiotrace. No marker support. Has a bug related to 8-bit writes showing incorrect data.

2.6.25

Mmiotrace must be patched into the kernel, see MmioTrace2_6_25. Kernel 2.6.26 is not supported, but the same patch could probably be adapted.

2.6.24

The 2.6.24 kernel needs a patch, after which you can use the out-of-tree mmiotrace. 0001-x86-Add-a-list-for-custom-page-fault-handlers.patch.txt. Do enable CONFIG_PAGE_FAULT_HANDLERS. Then follow MmioTraceHowto.

2.6.19 to 2.6.23

Use the out-of-tree mmiotrace, see MmioTraceHowto.

The page fault notifier hooks required by the kernel-patchless mmiotrace were introduced in Linus's kernel tree in 2.6.19-rc1, commits 474c256841074b913e76e392082373e12103a75d and 273819a2d982faace30e587b86a0683882251fe7.

How mmiotrace works inside

Kernel functions ioremap, ioremap_nocache and iounmap are replaced (for the driver module only) with wrappers to record MMIO areas. In ioremap the pages for the MMIO area are marked as not present, causing access to those addresses generate a page fault. In the page fault handler the mmio-traced addresses are detected and the attempted action recorded. The page is marked present and the page-faulting code is single-stepped to execute the instruction doing MMIO. Then, the page is marked again as not present.

The recording works by calling pre and post functions in mmio.ko before and after the single-stepping. mmio.ko uses relayfs and debugfs to relay the data to user space.

Unfortunately the legacy ISA address range 0xa0000 - 0x100000 cannot be traced this way because marking those pages as not present crashes the kernel. There can also be machine instructions that are not decoded properly, but so far they have been rare enough.

An Alternative idea

While discussing about x86 instruction emulation, Avi Kivity proposed the following, quote:

However there is a simpler (for you) solution: run the driver-to-be-reverse-engineered in a kvm guest, and modify kvm userspace to log accesses to mmio regions. This requires the not-yet-merged pci passthrough support. You can reverse engineer Windows drivers with this as well.

Reference: http://lkml.org/lkml/2008/4/5/13

Who would like to take that project?

Out-of-tree mmiotrace and analysis tools

The main out-of-kernel-tree source of mmio-trace is pq's (PekkaPaalanen) git tree: git://people.freedesktop.org/~pq/mmio-trace (browse)

There are two branches: master and binformat. Use branch master if you can, and binformat only if you absolutely have to. binformat is mostly for people wanting to reparse old binary mmio logs. Further description of the branches follows later.

out-of-kernel-tree mmio-trace consists of:

kernel module to process MMIO accesses, mmio.ko (deprecated)
user space program to record the logs, mmio-trace (deprecated)
user space program to interpret the logs, mmio-parse
user space program to re-execute the logs, mmio-replay
user space program to convert between some log formats, mmio-convert
user space program to validate testmmiotrace.ko output, test-check
kernel module to test mmiotrace, testmmiotrace.ko

In addition the inspected driver module has to have some symbols it calls from the kernel core rerouted, there is a script to do this.

CONFIG_DEBUG_FS and CONFIG_RELAY kernel options are required to be enabled in your kernel.

Old discussion: RFC mmiotrace full patch, preview 1, RFC mmiotrace full patch, preview 2, RFC 1/3 mmiotrace full patch, preview 3 http://lkml.org/lkml/2008/3/22/70

pq's master branch

Compared to older versions, this branch has redesigned data formats for both kernel-userspace communication via relayfs and on-disk log format. This is completely incompatible with the old versions.

The log format is text based, so it is readable without any special tools, and also avoids incompatibilities between architectures. Benefits of the rewritten kernel data format are variable length messages from kernel to user space and logging correct physical addresses of memory operations.

Additional features:

records the correct physical address of an event
stores lspci output and /proc/bus/pci/devices information, the latter is used to convert physical addresses into offsets usable with Rules-ng
mmio-parse on any arch can parse logs from any other arch
mmio-replay can replay register writes from logs. Dangerous!

This branch contains a version of mmio-convert written by jwstolk. This tool can convert dump files from one format to another, but the results are not always quite accurate. If you have an old dump that you cannot redo, take a look at mmio-convert.

pq's binformat branch

Here lives the older mmio-trace version which uses a binary on-disk log format. No new development happens here, but it is kept in somewhat working order for people who for one reason or another cannot use the master branch.

jrmuizel's git tree

The old source at jrmuizel's git tree: git://people.freedesktop.org/~jrmuizel/mmio-trace (browse) is no longer actively developed. jrmuizel is the one who resurrected mmio-trace and brought it to Nouveau when the developers did not yet have a way to see what the nvidia kernel blob did. pq's git tree is a fork of jrmuizel's master branch as of Aug 24th, 2007.

Usage notes (out-of-kernel-tree)

You can inject markers (text lines) into the trace log by echo 'X is running' > /proc/mmio-marker
Only one active CPU is supported, do not use in multiprocessing system. You can disable extra processors/cores during boot time with a kernel argument or runtime via sysFS entries.
Do not restart ./mmio-trace because it will get old data from kernel buffers at first. You need to unload and reload mmio.ko before starting ./mmio-trace again to clear the kernel relay buffers.
After tracing, check your kernel log for cpu 0 buffer full!!! errors. If you have any, almost certainly some events were lost. You should redo the trace with bigger relay buffers. Relay buffer size is controlled by n_subbufs parameter to the mmio kernel module. The default value is 128, and means the number of 256kB pieces reserved for the buffer, so 128 means 32MB. Increase the number, until you no longer get the buffer full message.
Low ISA range tracing, experimental: apply 0001-ioremap-do-not-handle-the-low-ISA-range-specially.patch to your kernel, and insmod mmio.ko ISA_trace=1 and let PekkaPaalanen know what happened. If you do not patch your kernel, your machine will crash, if the blob maps the ISA range. This is an untested feature.
Old Nvidia driver with 2.6.25 - To build the nvidia driver with 2.6.25 then you need the patches from nvnews, http://www.nvnews.net/vbulletin/showthread.php?t=110088.
Another thing is that you will probably see messages like NVRM: bad caching on address XXXXXXX: actual 0x173 != expected 0x17b in dmesg. This didn't happen with 2.6.25-rc6 but it now exists in 2.6.25-rc7 because of commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=d546b67a940eb42a99f56b86c5cd8d47c8348c2a.

Rules-ng connection

RulesNG (CVS) is a database definition describing hardware registers and a set of tools for using the database. One of the tools is staticdb. mmio-parse supports staticdb plugins.

Staticdb is a Rules-ng database converted into C-code and compiled into a dynamically loaded library. It allows to convert raw addresses into symbolic register names, track indexed registers, and convert register values into a more human readable format. To use it, you need to get Rules-ng from CVS, and say make in staticdb/ directory to get the database library. The library is used with the command ./mmio-parse -m NN -s path/to/libnvidia-mmio.so where NN is your card type (in Rules-ng terms variant).

To do, suggestions and known issues

support large pages
complete instruction support: get rid of "unknown type" --- or refactor KVM instruction decoding and merge there (Argh!)
support tracing access from user space
event filtering based on device and BAR, maybe address ranges
Changes to the log format:
- backtraces
- cpu identifier?
PPC support?
think about how to trace ISA region; David suggests that commenting out the ISA region checks in arch/i386/mm/ioremap.c would be enough to enable ISA range tracing. Is it really so? Does it break something?
improve mmio-parse
- (output fifo traces to replace renouvau data collector (requires tracing user space access))
make the mapping from PCI resources to staticdb plugins configurable (currently hardcoded for Nvidia BAR 0)