Valgrind-mmt is a Valgrind modification which allows tracing application accesses to mmaped memory (which is how userspace parts of graphics drivers communicate with hardware). It was created by Dave Airlie and then extended/fixed by others.

Installation:

git clone https://github.com/envytools/valgrind.git
git clone https://github.com/envytools/VEX.git valgrind/VEX
cd valgrind/VEX
git checkout -f VEX_3_9_BRANCH
cd ..
git checkout -f mmt-3.9
./autogen.sh
./configure --prefix=...
make
make install

Update:

cd valgrind
git pull
cd VEX
git pull
cd ..

Use:

valgrind --tool=mmt --mmt-trace-file=/dev/nvidia0 --mmt-trace-file=/dev/nvidiactl --mmt-trace-nvidia-ioctls glxgears

or (preferred):

valgrind --tool=mmt --mmt-trace-file=/dev/nvidia0 --mmt-trace-file=/dev/nvidiactl --mmt-trace-nvidia-ioctls --mmt-output-binary --log-file=file.log glxgears

Options:

--mmt-trace-file=path      trace loads and stores to memory mapped for this file (e.g. /dev/nvidia0) (you can pass this option multiple times)
--mmt-trace-all-files      trace loads and stores to memory mapped for all files
--mmt-trace-nvidia-ioctls  trace nvidia ioctls on /dev/nvidiactl
--mmt-trace-nouveau-ioctls trace nouveau ioctls on /dev/dri/cardX
--mmt-trace-all-opens      trace all 'open' syscalls
--mmt-trace-marks          send mmiotrace marks before and after ioctls
--mmt-output-binary        output in binary format (readable by demmt and mmt_bin2dedma)

Some notes about tracing Xorg:

  • valgrind can't trace suid binaries, you have to copy Xorg to e.g. Xorg-nosuid
  • running plain Xorg won't load any windowmanager - before starting X, start on another console:
    export DISPLAY=:0; while [ true ]; do xterm; sleep 1; done
  • nvidia ioctl tracing used to be fragile, now it's pretty good
  • when valgrind crashes, Xorg leaves only black screen - you need to press Alt-SysRq-R, switch to some free console (ctrl-alt-fxx), blindly login and start normal Xorg (startx)

What to do with generated trace: All tools come from envytools repository. Text log can be parsed by "dedma":

./dedma -m nvXX -v $(mapid) file.log

where $(mapid) is a buffer number. Note that dedma won't properly decode buffers other than main one. You need binary mmt trace and demmt to see them.

Binary output can be converted back to text output with "mmt_bin2text" tool (if for some reason demmt does not work):

./mmt_bin2dedma < file-bin.log > file-txt.log

and fed back to dedma or directly parsed by demmt:

./demmt -m nvXX -f < file-bin.log

which will output "possible IB buffer: $id" or "possible USER buffer: $id" and then:

./demmt -m nvXX -n $id < file-bin.log

If -f can't find IB/USER buffer (it's a bug, please report it), you can skip -n, but demmt output will be less reliable (but still better than dedma).

All demmt options:

-m 'chipset'    set chipset version (required)
-f              find possible pushbuf pointers (IB / USER)
-n id           set pushbuf pointer to "id"
-g              print gpu addresses
-o              dump ioctl data
-q              (quiet) print only the most important data (pushbufs from IB / USER and disassembled code)
-a              disable shader disassembly
-c              enable colors
-l file         use "file" as input

-s              do not "compress" obvious buffer clears
-i              do not guess invalid pushbufs
-d              hide invalid pushbufs
-e              do not decode invalid pushbufs