43.5. Analyzing the Data

Periodically, the OProfile daemon, oprofiled collects the samples and writes them to the /var/lib/oprofile/samples/ directory. Before reading the data, make sure all data has been written to this directory by executing the following command as root:

opcontrol --dump

Each sample file name is based on the name of the executable, with a closing curly bracket (}) replacing each forward slash (/). The file name ends with a hash mark (#), followed by the counter number used for that sample file. For example, the following file includes the sample data for the /sbin/syslogd executable collected with counter 0:


The following tools are available to profile the sample data once it has been collected:

Use these tools, along with the binaries profiled, to generate reports that can be further analyzed.


The executable being profiled must be used with these tools to analyze the data. If it must change after the data is collected, backup the executable used to create the samples as well as the sample files.

Samples for each executable are written to a single sample file. Samples from each dynamically linked library are also written to a single sample file. While OProfile is running, if the executable being monitored changes and a sample file for the executable exists, the existing sample file is automatically deleted. Thus, if the existing sample file is needed, it must be backed up, along with the executable used to create it before replacing the executable with a new version. Refer to Section 43.4 Saving Data for details on how to backup the sample file.

43.5.1. Using op_time

The op_time tool provides an overview of all the executables being profiled.

The following is part of an example output:

581        0.2949  0.0000 /usr/bin/oprofiled
966        0.4904  0.0000 /usr/sbin/cupsd
1028       0.5218  0.0000 /usr/sbin/irqbalance
1187       0.6026  0.0000 /bin/bash
1480       0.7513  0.0000 /usr/bin/slocate
2039       1.0351  0.0000 /usr/lib/rpm/rpmq
6249       3.1722  0.0000 /usr/X11R6/bin/XFree86
8842       4.4885  0.0000 /bin/sed
31342     15.9103  0.0000 /usr/bin/gdmgreeter
58283     29.5865  0.0000 /no-vmlinux
82853     42.0591  0.0000 /usr/bin/perl

Each executable is listed on its own line. The first column is the number of samples recorded for the executable. The second column is the percentage of samples relative to the total number of samples. The third column is unused, and the fourth is the name of the executable.

Refer to the op_time man page for a list of available command line options such as the -r option used to sort the output from the executable with the largest number of samples to the one with the smallest number of samples. The -c option is also useful for specifying a counter number.

43.5.2. Using oprofpp

To retrieve more detailed information about a specific executable, use oprofpp:

oprofpp <mode> <executable>

<executable> must be the full path to the executable to be analyzed. <mode> must be one of the following:


List sample data by symbols. For example, the following is part of the output from running the command oprofpp -l /usr/X11R6/bin/XFree86:

vma      samples  %           symbol name
08195d10 4        3.0303      miComputeCompositeClip
080b9180 5        3.78788     Dispatch
080cdce0 5        3.78788     FreeResource
080ce4a0 5        3.78788     LegalNewID
080ce640 5        3.78788     SecurityLookupIDByClass
080dd470 9        6.81818     WaitForSomething
080e1360 12       9.09091     StandardReadRequestFromClient

The first column is the starting virtual memory address (vma). The second column is the number of samples for the symbol. The third column is the percentage of samples for this symbol relative to the overall samples for the executable, and the fourth column is the symbol name.

To sort the output from the largest number of samples to the smallest (reverse order), use -r in conjunction with the -l option.

-s <symbol-name>

List sample data specific to a symbol name. For example, the following output is from the command oprofpp -s StandardReadRequestFromClient /usr/X11R6/bin/XFree86:

vma      samples  %           symbol name
080e1360 12       100         StandardReadRequestFromClient
080e1360 1        8.33333
080e137f 1        8.33333
080e13bb 1        8.33333
080e13f4 1        8.33333
080e13fb 1        8.33333
080e144a 1        8.33333
080e15aa 1        8.33333
080e1668 1        8.33333
080e1803 1        8.33333
080e1873 1        8.33333
080e190a 2        16.6667

The first line is a summary for the symbol/executable combination.

The first column consists of the virtual memory addresses sampled. The second column is the number of samples for the memory address. The third column is the percentage of samples for the memory address relative to the total number of samples for the symbol.


List sample data by symbols with more details than -l. For example:

vma      samples  %           symbol name
08083630 2        1.51515     xf86Wakeup
 08083641 1        50
 080836a1 1        50
080b8150 1        0.757576    Ones
 080b8179 1        100
080b8fb0 2        1.51515     FlushClientCaches
 080b8fb9 1        50
 080b8fba 1        50

The data is the same as the -l option except that for each symbol, each virtual memory address used is shown. For each virtual memory address, the number of samples and percentage of samples relative to the number of samples for the symbol is displayed.

-g <file-name>

Generate output to a file in gprof format. If the generated file is named gmon.out, gprof can be used to further analyze the data. Refer to the gprof man page for details.

Other options to further restrict the data are as follows:

-f <file-name>

Use the specified sample file <file-name>. By default, the sample file in /var/lib/oprofile/samples/ is used. Use this option to specify a sample file from a previous session.

-i <file-name>

Use <file-name> as the name of the executable for which to retrieve data.


Demangle C++ symbol names.


Demangle C++ symbol names, and simplify STL library demangled names.

--counter <number>

Gather information from a specific counter. The default counter is 0 if not specified.


Display the line number in the source code for each sample. When the executable was compiled, GCC's -g option should have been used. Otherwise, this option can not display the line numbers. None of the Red Hat Enterprise Linux executables are compiled with this option by default.

vma      samples  %      symbol name        linear info
0806cbb0 0        0      _start             ../sysdeps/i386/elf/start.S:47
-e <symbol-name>

Exclude the comma-separated list of symbols from the output.


Display an additional column containing the shared library. This option only produces results if the --separate=library option to opcontrol is specified when configuring OProfile and if the --dump-gprof-file option is not used in conjunction with this option.

-t <format>

Display the output in a specific column order. This option can not be used with -g.

Use the following letters to represent the columns:

vVirtual memory address
sNumber of samples
SCumulative number of samples
pPercentage of samples relative to total number of samples for the executable
PCumulative percentage of samples relative to total number of samples for the executable
qPercentage of samples relative to all executables sampled
QCumulative percentage of samples relative to all executables sampled
nSymbol name
lFile name of source file and line number, including full path
LBase name of the source code file name and line number
iName of the executable, including full path
IBase name of the executable
dDetails of the sample
hDisplay column headers

Table 43-4. Letters for Column Order

--session <name>

Specify the full path to the session or a directory relative to the /var/lib/oprofile/samples/ directory.

-p <path-list>

Specify a comma-separated list of paths in which the executables to be analyzed are located.

43.5.3. Using op_to_source

The op_to_source tool tries to match the samples for particular instructions to the corresponding lines in the source code. The resulting files generated should have the samples for the lines at the left. It also puts in a comment at the beginning of each function listing the total samples for the function.

For this utility to work, the executable must be compiled with GCC's -g option. By default, Red Hat Enterprise Linux packages are not compiled with this option.

The general syntax for op_to_source is as follows:

op_to_source --source-dir <src-dir> <executable>

The directory containing the source code and the executable to be analyzed must be specified. Refer to the op_to_source man page for a list of additional command line options.

43.5.4. Using op_merge

If multiple sample files exist for the exact same executable or library, the sample files can be merged for easier analysis.

For example, to merge files for the library /usr/lib/library-1.2.3.so, execute the following command as root:

op_merge /usr/lib/library-1.2.3.so

The resulting file is /var/lib/oprofile/samples/}usr}lib}library-1.2.3.so.

To limit the samples merged to a specific counter, use the -c option followed by the counter number.