Enlightenment Debugging

This tutorial aims at providing a comprehensive and reproducible documentation to debug Enlightenment. It is divided in two sections:

Before debugging make sure your debug symbols are enabled, if not, go to the Enable debug symbols section.

Debugging Enlightenment using GDB

In order to use GDB, we will simulate a crash in Enlightenment.

Start by running Enlightenment on a machine. Then switch to another tty (CTRL+ALT+F1 for tty1).

At that point, 2 processes are interesting, enlightenment and enlightenment_start. Indeed, enlightenment is traced by enlightenment_start.

Then setup SEGV to segfault enlightenment.

kill -SIGSEGV $(pidof enlightenment)

enlightenment_start will open a pop-up named Enlightenment Error. This pop-up indicates that Enlightenment segfaulted and allows to recover or log out but also detaches from the child process (Enlightenment) and let advanced users use GDB to debug it.

#gdb on the running enlightenment process
gdb enlightenment $(pidof enlightenment)

After a bit a prompt is available, if you want, you can save the traces in a log:

#save the traces in log.txt
set logging file log.txt
set logging on
Backtrace

Use the backtrace command to get information about frames to know where the segfault is coming from.

(gdb) bt
#0  0xb7d539f8 in select () from /lib/tls/libc.so.6
#1  0xb7dff66a in _XEnq () from /usr/X11R6/lib/libX11.so.6
#2  0xb7dffa7e in _XRead () from /usr/X11R6/lib/libX11.so.6
#3  0xb7e01795 in _XReadEvents () from /usr/X11R6/lib/libX11.so.6
#4  0xb7defa88 in XNextEvent () from /usr/X11R6/lib/libX11.so.6
#5  0x0809b698 in e_alert_show (
    text=0x80a34f0 "This is very bad. Enlightenment has segfaulted.\nThis
    is not meant to happen and is likely a
    sign of a\nbug in Enlightenment
    or the libraries it relies on.\n\nYou can gdb attach to this process
    now to try"...)
    at e_alert.c:136
#6  0x0808f706 in e_sigseg_act (x=11, info=0x80a9fb0, data=0x80aa030)
    at e_signals.c:54
#7  <signal handler called>
#8  0xb7d539f8 in select () from /lib/tls/libc.so.6
#9  0xb7f814ee in _ecore_main_select (timeout=0)
    at ecore_main.c:338
#10 0xb7f819ba in _ecore_main_loop_iterate_internal (once_only=0)
    at ecore_main.c:575
#11 xb7f81a2b in ecore_main_loop_begin () at ecore_main.c:79
#12 0x08059bb3 in main (argc=1, argv=0xbffff144) at e_main.c:551

As you can see in the stack trace, GDB finds the segfault in libc and pops to the main function in e_main. But it doesn't seem credible to have a bug in libc or x, the important thing is Enlightenment has its own segfault handler which is very explicit on frame 5.

The e_sigseg_act() function at frame 6 is called directly from the kernel when the program segfaults, meaning enlightenment causes the segfault. So, the segfault comes from the select function (a libc function) at frame 8 called in _e_core_main_select_function at frame 9.

Go in a frame

So, the reason of segfault seems to be at frame 9, when select function is called. Let's go to frame 9:

fr 9
#9  0xb7f814ee in _ecore_main_select (timeout=0) at ecore_main.c:338
338        ret = select(max_fd + 1, &rfds, &wfds, &exfds, t);
(gdb) l
333               }
334          }
335     #ifndef WIN32
336        if (_ecore_signal_count_get()) return -1;
337     #endif
338        ret = select(max_fd + 1, &rfds, &wfds, &exfds, t);
339        if (ret < 0)
340          {
341             if (errno == EINTR) return -1;
342          }

The first command fr 9 gives useful information: name of the file, number of the line, the function called, … The second command l lists the code around the called function. Another useful command allows to print the variables, the parameters, helping you to find out the problem, a wrong parameter, a null pointer…

(gdb) p ret
$1 = -4
(gdb) p rfds
$2 = {__fds_bits = {1280, 0 <repeats 31 times>}}
(gdb) p wfds
$3 = {__fds_bits = {0 <repeats 32 times>}}
(gdb) p exfds
$4 = {__fds_bits = {0 <repeats 32 times>}}

GDB is important to start debugging, it will help you to resize the problem even if sometimes is not enough.


Valgrind

Valgrind aims at finding memory problems but for that Enlightenment needs to be run through Valgrind.

Prerequisites

This tutorial will present 3 different ways to run enlightenment through Valgrind:

  • Remote debugging
  • Xephyr invocation
  • Xinit invocation

The easiest way is certainly Xephyr because it allows to target a window on a X server host, sadly Xephyr doesn't yet support OpenGL and any issue that may be related to it will need to use the Xinit version.

Remote Debugging

Enlightenment_start launcher will handle setting up environment variables, paths, and launching any other required services before Enlightenment starts. Fortunately, there are some options in enlightenment_start that allow to run Enlightenment through Valgrind:

$enlightenment_start --help
Options:
    -valgrind[=MODE]
        Run enlightenment from inside valgrind, mode is OR of:
           1 = plain valgrind to catch crashes (default)
           2 = trace children (thumbnailer, efm slaves, ...)
           4 = check leak
           8 = show reachable after processes finish.
         all = all of above
    -massif
        Run enlightenment from inside massif valgrind tool.
    -callgrind
        Run enlightenment from inside callgrind valgrind tool.
    -valgrind-log-file=<FILENAME>
        Save valgrind log to file, see valgrind's --log-f

First of all, get the IP address of your host machine and connect to it, then on the distant machine, launch X:

#launch X on tty1
sudo X -ac :1 &

For example, if you want to check leak and save traces in a log file:

export DISPLAY=:1
enlightenment_start -valgrind=4 -valgrind-log-file=log.txt

At that point, Enlightenment should have started on your host machine. If the session is closed then the summary of Valgrind should look like:

==1488==
==1488== HEAP SUMMARY:
==1488==     in use at exit: 4,479,487 bytes in 24,302 blocks
==1488==   total heap usage: 336,239 allocs, 311,937 frees, 88,068,674 bytes allocated
==1488==
==1488== LEAK SUMMARY:
==1488==    definitely lost: 825 bytes in 15 blocks
==1488==    indirectly lost: 39 bytes in 2 blocks
==1488==      possibly lost: 384 bytes in 1 blocks
==1488==    still reachable: 4,478,239 bytes in 24,284 blocks
==1488==         suppressed: 0 bytes in 0 blocks
==1488== Rerun with --leak-check=full to see details of leaked memory
==1488==
==1488== For counts of detected and suppressed errors, rerun with: -v
==1488== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Valgrind Invocation - Xinit

Create a file called .xinitrc-debug in your home with the following content:

#!/bin/sh
 
ulimit -c unlimited
 
log_file="$HOME/enlightenment-xinit-debug-valgrind.log"
#path of you installation
enlightenment_install_path=/usr/local
 
#set vars
LD_LIBRARY_PATH="$enlightenment_install_path/lib"
PATH="$enlightenment_install_path/bin:$PATH"
 
#log to file
echo "using installation at $enlightenment_install_path"
echo "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" >  $log_file
echo "PATH=$enlightenment_install_path/bin: ... " >> $log_file
 
#start e with valgrind
$enlightenment_install_path/bin/enlightenment_start -valgrind 2>&1 | tee -a "$log_file"

You can now start a debugging session, after X was closed if it is not the case log out:

/usr/bin/xinit ~/.xinitrc-debug -- :1 -nolisten tcp
Valgrind Invocation - Xephyr

Xephyr is a display server implementing the X11 display server protocol which targets a window on a X Server host. So, for this it is important that X is launched, you can even do it in your usual window manager under X.

Of course, you need to install Xephyr.

Xephyr -ac -br -noreset -screen 800x600 :1
  • ac: disable access control restrictions
  • br: create root window with black background
  • noreset: don't reset after last client exists
  • screen 800×600: Specify screen characteristics

A black screen should now be displayed, and the interesting thing for us is that you can launch a window manager on it with Valgrind in addition:

DISPLAY=:1 enlightenment_start -valgrind