From: eLinux.org
Kernel Trace Systems
Here are some links to information about different kernel tracing systems:
Contents
- 1 General Purpose tracing systems
- 2 Special Purpose tracing systems
- 3 Trace Infrastructure
- 4 Sampling Systems
- 5 Related facilities
- 6 Other Systems
- 7 Collaboration Efforts
General Purpose tracing systems
Some major Linux general-purpose tracing systems are:
- ptrace - ability to trace syscall entry and exit, and signal
delivery, to a process (also used for debugging a process)
- see "man ptrace" and "man strace"
- Ftrace
- Ftrace Function Graph
ARM -
presentations and patches by Tim Bird to add function graph and
duration tracing to ARM systems
- The presentation has good introductory material on ftrace, as well as links to additional resources
- tracer for kernel functions
- can also be used for debugging or analyzing latencies and performance issues
- in mainline since 2.6.27
- See Measuring Function Duration with
FTrace
- outline of presentation by Tim Bird for Linux Symposium 2009
- Ftrace Function Graph
ARM -
presentations and patches by Tim Bird to add function graph and
duration tracing to ARM systems
- System Tap - System Tap is a system for building and executing tracing and sampling systems that can be applied to a running Linux system
- LTTng - Linux Trace Toolkit, next generation
- LKST - Linux Kernel State Tracer
Special Purpose tracing systems
There are some other notable special-purpose kernel tracing systems:
- KFT - Kernel Function Trace - traces functions to show function durations and call graphs
- latency trace - RT-preempt tool for measuring interrupt and mutex
latency
- The latency tracer is embedded in the RT-preempt patch - see Realtime Preemption and RT-preempt Article
- block tracer (blktrace) - allows you to see exactly what is going on
in the block layer for a given queue
- Introduction by Jens Axboe: Introduction
- Execellent presentation: blktrace.pdf
- Guide to using is at: blktrace guide
- This appears to have been mainlined as of 2.6.17
- Timeline utility (blktrace post-processing tool): blktrace timeline utility
- delay accounting patches - collect statistics about the delays that are experienced by each task on the system
Trace Infrastructure
KProbes - grew out of dprobes, with information at: dprobes
- see an excellent tutorial at: kprobes
- The mainline version of the KProbes supports x86,Alpha and PPC64 architectures. A MIPS implementation has been completed on the 2.6.16 kernel and tested on the Toshiba TX49 platform. Patch is available in the Patch Archive.
[would be nice to have some djprobe stuff here]
Sampling Systems
Note that profile systems (or "sampling systems") are slightly different, in that they involve sampling instead of event tracing. Some major ones for Linux are:
- top - provides a dynamic real-time view of a running system,
including processes
- see "man top"
- see also ksysguard, Gnome system Monitor
- OProfile - system-wide profiler for Linux systems
- see oprofile
- also: oprofile at IBM
- BootChart - samples bootup and provides visualization of process
startup and system utilization
- see Bootchart
Related facilities
- in-kernel statistics infrastructure - proposal for a generic
implementation of statistics facilities inside the kernel
- see inkernel stats
- perfmon2 - interfaces to hardware performance monitoring features of
the CPU
- see perfmon
- inotify - inotify
Other Systems
Here are some systems I haven't classified yet:
- Datastreams - a system for creating and monitoring tracepoints - see datastreams
Collaboration Efforts
Some trace system project leaders are trying to collaborate: see Tracing Collaboration Project