ubuntu+systemtap进行Linux内核和用户空间开发测试
Sailor_forever sailing_9806#163.com
(本原创文章发表于Sailor_forever 的个人blog，未经本人许可，不得用于商业用途。任何个人、媒体、其他网站不得私自抄袭；网络媒体转载请注明出处，增加原文链接，否则属于侵权行为。如有任何问题，请留言或者发邮件给sailing_9806＃163.com)
【摘要】本文主要介绍在ubuntu平台 + 自定义内核上如何安装systemtap工具包及解决编译加载运行过程中的一些问题，如何利用systemtap工具监测分析内核函数，内核模块及用户态程序。
【关键字】ubuntu, systemtap, utrace, backtrace, dtrace,callstack
1.2.1 卸载通过apt-get自动安装的systemtap. 2
6 Systemtap加载KO异常，Unknown symbol in module. 9
7 error:implicit declaration of function ‘unregister_uprobe’13
Systemtap 是学习内核一个必不可少的工具，它不仅可以侦测内核空间的相关信息，还可以侦测用户空间的信息，是研究内核源代码、优化系统性能及调试诊断系统问题的一个必备工具。相关原理信息可参见附录https://wiki.ubuntu.com/Kernel/Systemtap及“Linux 下的一个全新的性能测量和调式诊断工具 Systemtap, 第 3 部分: Systemtap”
http://www.ibm.com/developerworks/cn/linux/l-cn-systemtap3/#main
用系统工具自动安装systemtap时，用户不用考虑各个软件包的依赖关系，系统会自动根据当前内核版本下载相应的软件包。
dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudoapt-get install systemtap
。。
The following extra packages will beinstalled:
libdw1 systemtap-commonsystemtap-runtime
The following NEW packages will beinstalled:
libdw1 systemtap systemtap-common systemtap-runtime
。
Get:1 http://us.archive.ubuntu.com/ubuntu/quantal-updates/main libdw1 i386 0.153-1ubuntu1.1 [216 kB]
。。
如上可知，系统自动安装了相关的4个软件包libdw1 systemtap-common systemtap-runtime
有时候系统自动安装的不一定能用，这个时候需要升级，此时只能通过下载源代码自己编译
首先要卸载系统之前安装的相关package，避免后续带来其他副作用。
dd@ubuntu:/mnt/hgfs/systemtap$ sudo apt-getremove systemtap systemtap-common systemtap-runtime
。。
Removing systemtap ...
Removing systemtap-common ...
Removing systemtap-runtime ...
Processing triggers for man-db ...
下载相应版本的源码包。源码包并不是越新越好，因为他们之间有一定的依赖关系，最简单的是按照内核源码包发布的时间来选择systemtap。
https://fedorahosted.org/releases/e/l/elfutils/
ftp://sources.redhat.com/pub/systemtap/releases/
dd@ubuntu:/mnt/hgfs/systemtap/systemtap-2.0$./configure --with-elfutils=../elfutils-0.156 --prefix=/usr/local
。。。。。
configure: ./configure ‘--with-elfutils=../elfutils-0.156‘‘--prefix=/usr/local‘ --prefix=/home/dd/systemtap-2.0-4155
configure: Running systemtap uninstalled, entirely out of the buildtree,
configure: is not supported.
重启系统之后再执行，仍然如此，不知是否有其他副作用
dd@ubuntu:/mnt/hgfs/systemtap/systemtap-2.0$sudo make
。。
ln -fs libdw.so libdw.so.1
ln: failed to create symbolic link `libdw.so.1‘: Operation notsupported
make[4]: *** [libdw.so] Error 1
windows不支持符号链接，因此system包不能放在windows目录编译
dd@ubuntu:~/systemtap/systemtap-2.0$ sudomake
。。。
m4 -Di386 -DDISASSEMBLER../../../elfutils-0.156/libcpu/defs/i386 > i386_defsT
/bin/bash: m4: command not found
System调试内核或者application，必须有debuginfo，因此必须安装内核debug symbol。具体参见附录。
Systemtap安装完毕后，进行简单的测试。如果能打印出helloworld则认为基本功能OK。
dd@ubuntu:/usr/src$ sudo stap -ve ‘probebegin { log("hello world") exit() }‘
Pass 1: parsed user script and 81 libraryscript(s) using 22640virt/13560res/2192shr kb, in 80usr/10sys/93real ms.
Pass 2: analyzed script: 1 probe(s), 2function(s), 0 embed(s), 0 global(s) using 22904virt/14088res/2264shr kb, in0usr/0sys/3real ms.
Pass 3: translated to C into"/tmp/stapNgfERv/stap_202537bef6262d2233e5759ff826c9d3_744_src.c"using 22904virt/14364res/2508shr kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into"stap_202537bef6262d2233e5759ff826c9d3_744.ko" in4760usr/900sys/6360real ms.
Pass 5: starting run.
hello world
Pass 5: run completed in 0usr/20sys/359realms.
dd@ubuntu:/usr/src$ sudo stap -e ‘probekernel.function("sys_open") {log("hello world") exit()}‘
hello world
dd@ubuntu:/mnt/hgfs/systemtap/example$ cat./timer.stp
#!/usr/bin/stap
probekernel.function("do_timer").return {
print_backtrace();
printf("\n");
exit();
}
dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./timer.stp
Returning from: 0xc1091520 : do_timer+0x0/0x40 [kernel]
Returning to : 0xc1097e4a : tick_do_update_jiffies64+0xca/0x130 [kernel]
0xc1098074 : tick_sched_timer+0xc4/0xd0[kernel]
0xc1068f70 : __run_hrtimer+0x70/0x190 [kernel]
0xc1069be7 : hrtimer_interrupt+0xe7/0x2a0 [kernel]
0xc15d0769 :smp_apic_timer_interrupt+0x59/0x8d [kernel]
0xc15c9815 : apic_timer_interrupt+0x31/0x38[kernel]
0xc1036f35 : native_safe_halt+0x5/0x10[kernel]
0xc1017b66 : default_idle+0x46/0x190 [kernel]
0xc1018716 : cpu_idle+0xb6/0xe0 [kernel]
0xc159f715 : rest_init+0x5d/0x68 [kernel]
0xc18ae9be : start_kernel+0x35d/0x363 [kernel]
0xc18ae303 : i386_start_kernel+0xa6/0xad[kernel]
dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap ./vmx_inject_irq.stp
[sudo] password for dd:
semantic error: while resolving probepoint: identifier ‘kernel‘ at ./vmx_inject_irq.stp:1:7
source: probe kernel.function("vmx_inject_irq") {
^
semantic error: no match
Pass 2: analysis failed. Try again with another ‘--vp 01‘ option.
dd@ubuntu:/mnt/hgfs/systemtap/kvm$ cat/proc/kallsyms | grep vmx_inject_irq
00000000 t vmx_inject_irq [kvm_intel]
kvm_intel以模块形式加载，其symbol不能以kernel.function形式访问。
Ubuntu平台上需要修正动态module的debug info信息
dd@ubuntu:/mnt/hgfs/systemtap/kvm$ catfix-module.sh
#!/bin/sh
for file in `find /usr/lib/debug -name‘*.ko‘ -print`
do
buildid=`eu-readelf-n $file| grep Build.ID: | awk ‘{print $3}‘`
dir=`echo$buildid | cut -c1-2`
fn=`echo$buildid | cut -c3-`
mkdir-p /usr/lib/debug/.build-id/$dir
ln-s $file /usr/lib/debug/.build-id/$dir/$fn
ln-s $file /usr/lib/debug/.build-id/$dir/${fn}.debug
done
dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap -l ‘module("*kvm*").function("*")‘ >kvm-module-sysm.log
dd@ubuntu:/mnt/hgfs/systemtap/kvm$ catkvm-module-sysm.log | grep vmx_inject_irq
module("kvm_intel").function("vmx_inject_irq@/build/buildd/linux-3.5.0/arch/x86/kvm/vmx.c:4036")
dd@ubuntu:/mnt/hgfs/systemtap/kvm$ catvmx_inject_irq.stp
probemodule("kvm_intel").function("vmx_inject_irq") {
printf("-------------------------------------\n")
print_backtrace()
printf("-------------------------------------\n")
}
很多符号都无法解析啊？？？？？？？ 根本原因待分析。
-------------------------------------
0xf86e4230 : vmx_inject_irq+0x0/0xe0[kvm_intel]
0xf998e694 [kvm]
0xc10654cd (inexact)
0xf997b25b [kvm] (inexact)
0xf998f11f [kvm] (inexact)
0xf997922b [kvm] (inexact)
0xc1075dee (inexact)
0xc1075ebf (inexact)
0xc1099a12 (inexact)
0xc109be08 (inexact)
0xf9978dd0 [kvm] (inexact)
0xc115e8fa (inexact)
0xc1069c65 (inexact)
0xc109c120 (inexact)
0xc115ee88 (inexact)
0xc15cff5f (inexact)
0xc15c0000 (inexact)
-------------------------------------
用户态监控需要utrace支持，但utrace默认只有redhat系列的kernel支持，如fedora等。其他发行版Linux需要patch
Systemtap的README中有关于utrace支持的描述：
- Consider applying the utrace kernelpatches, if you wish to probe
user-space applications. http://sourceware.org/systemtap/wiki/utrace
Orif your kernel is near 3.5, apply the uprobes and related patches
(see NEWS). Or if your kernel is>= 3.5, enjoy the built-in uprobes.
- Build the kernel using your normalprocedures. Enable
CONFIG_DEBUG_INFO, CONFIG_KPROBES, CONFIG_RELAY, CONFIG_DEBUG_FS,
CONFIG_MODULES, CONFIG_MODULE_UNLOAD, CONFIG_UTRACE if able
- % make modules_install install headers_install
- Boot into the kernel.
Ubuntu 要支持utrace有两种方式：
1. 自己给 Ubuntu 自带的老 kernel 应用 utrace 补丁，并重新编译。但好像redhat提供的patch不能访问了
http://people.redhat.com/roland/utrace/
Ubuntu 11.10 (Oneiric)上编译带utrace补丁的内核
http://chaoslawful.iteye.com/blog/1463564
2. 将 kernel 升级到官方最新的 3.5 或以上的版本。最新的kernel 默认包含了 uprobes 机制，不再需要 utrace 补丁了。
Kernel hacking --->
[*] Tracers --->
[*] Enable uprobes-based dynamic events
新的内核里已经没有选项CONFIG_UTRACE，选择utrace相关的选项即可，包括CONFIG_ARCH_SUPPORTS_UPROBESCONFIG_UPROBES CONFIG_UPROBE_EVENT CONFIG_KPROBES CONFIG_DEBUG_FS 等
dd@ubuntu:/mnt/hgfs/systemtap/example$ cat/boot/config-3.5.0-17-generic | grep UPROBE
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_UPROBES=y
CONFIG_UPROBE_EVENT=y
dd@ubuntu:/mnt/hgfs/systemtap/example$ cat/boot/config-3.5.0-17-generic | grep KPROBE
CONFIG_KPROBES=y
CONFIG_HAVE_KPROBES=y
# CONFIG_KPROBES_SANITY_TEST is not set
CONFIG_KPROBE_EVENT=y
如果不是必须用某一个版本，建议直接升级到3.5以后的内核版本，如Ubuntu 12.10
首先写了一个用户态的测试函数，可参考如下
http://blog.csdn.net/astrotycoon/article/details/8142588
Utrace调试的程序需要在编译时加 -g 选项
dd@ubuntu:/mnt/hgfs/systemtap/example$./utrace-test
Obtained 8 stack frames.nm
./utrace-test() [0x80485c1]
[0xb77a3400]
./utrace-test() [0x804869f]
./utrace-test() [0x80486ac]
./utrace-test() [0x80486b6]
./utrace-test() [0x80486eb]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75ff4d3]
./utrace-test() [0x80484b1]
有函数调用栈，但是没有函数名称，莫非是编译优化导致无符号表？经查需要传递(-rdynamic)，-rdynamic可用来通知链接器将所有符号添加到动态符号表中
dd@ubuntu:/mnt/hgfs/systemtap/example$ gcc-g -rdynamic utrace-test.c -o utrace-test
dd@ubuntu:/mnt/hgfs/systemtap/example$./utrace-test
Obtained 7 stack frames.
./utrace-test(print_trace+0x1f) [0x80487ab]
./utrace-test(dummy_function_3+0xb)[0x8048840]
./utrace-test(test2+0xb) [0x804884d]
./utrace-test(test1+0xb) [0x804885a]
./utrace-test(main+0xb) [0x8048867]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75f44d3]
./utrace-test() [0x80486c1]
OK，能够正确打印出函数调用顺序，参考基准已经建立起来了。
如下测试显示stap可以监测到待调试程序utrace-test的函数列表。
dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap -l ‘process("./utrace-test").function("*")‘
process("/mnt/hgfs/systemtap/example/utrace-test").function("dummy_function_3@/mnt/hgfs/systemtap/example/utrace-test.c:26")
process("/mnt/hgfs/systemtap/example/utrace-test").function("main@/mnt/hgfs/systemtap/example/utrace-test.c:39")
process("/mnt/hgfs/systemtap/example/utrace-test").function("print_trace@/mnt/hgfs/systemtap/example/utrace-test.c:7")
process("/mnt/hgfs/systemtap/example/utrace-test").function("test1@/mnt/hgfs/systemtap/example/utrace-test.c:34")
process("/mnt/hgfs/systemtap/example/utrace-test").function("test2@/mnt/hgfs/systemtap/example/utrace-test.c:30")
当任意一个函数被调用时，打印其函数名及参数
dd@ubuntu:/mnt/hgfs/systemtap/example$ catutrace.stp
#!/usr/bin/stap
probeprocess("./utrace-test").function("*") {
printf("%s(%s)\n", probefunc(), $$parms);
}
实际检测到的函数调用顺序与实际匹配
dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./utrace.stp
_start()
__libc_csu_init()
__i686.get_pc_thunk.bx()
_init()
frame_dummy()
register_tm_clones()
main()
test1()
test2()
dummy_function_3()
print_trace()
__do_global_dtors_aux()
deregister_tm_clones()
_fini()
利用print_backtrace打印监测点的函数调用栈。
dd@ubuntu:/mnt/hgfs/systemtap/example$ catutrace-stack.stp
#!/usr/bin/stap
probeprocess("./utrace-test").function("dummy_function_3"){
printf("%s(%s)\n", probefunc(),$$parms);
print_backtrace();
printf("\n");
}
dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./utrace-stack.stp
semantic error: while resolving probepoint: identifier ‘process‘ at ./utrace.stp:3:7
source: probeprocess("./utrace-test").function("dummy_function_3").return{
semantic error: process return probes not available withinode-based uprobes
Pass 2: analysis failed. Try again with another ‘--vp 01‘ option.
去掉return，再执行
dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./utrace-stack.stp
WARNING: no or bad debug frame hdr
WARNING: No binary search table for eh frame, doing slow linear searchfor stap_3fb2bcd721ab82a6de577773eff766b_10596
dummy_function_3()
0xf8887e10 [stap_3fb2bcd721ab82a6de577773eff766b_10596+0xbe10/0x0]
0xf8888209[stap_3fb2bcd721ab82a6de577773eff766b_10596+0xc209/0x0]
0xf88868b0[stap_3fb2bcd721ab82a6de577773eff766b_10596+0xa8b0/0x0]
0xc113eb3f
0xc104e51b (inexact)
0xc161ac9d (inexact)
没有打印出用户态的调用堆栈，具体原因待分析。
加载动态编译出来的module时失败Unknown symbol inmodule
dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap -ve ‘probe begin { log("hello world") exit() }‘
。。
Pass 4: compiled C into"stap_aeecf973d1b00387ff715fbb9a761ae9_688.ko" in6010usr/1080sys/10345real ms.
Pass 5: starting run.
Error inserting module‘/tmp/stapoo63sX/stap_aeecf973d1b00387ff715fbb9a761ae9_688.ko‘: Unknown symbolin module
WARNING: /usr/bin/staprun exited withstatus: 1
Pass 5: run completed in 0usr/0sys/11realms.
Pass 5: run failed. Try again with another ‘--vp 00001‘ option.
dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap -e ‘probe kernel.function("sys_open") {log("helloworld") exit()}‘
Error inserting module‘/tmp/stapPq4Mh5/stap_b3efd67e86a8df14579c7f419ddcd07f_826.ko‘: Unknown symbolin module
WARNING: /usr/bin/staprun exited withstatus: 1
Pass 5: run failed. Try again with another ‘--vp 00001‘ option
查看内核log，分析Unknown symbol inmodule的意义，这种问题通常都是某个symbol无法找到。
dd@ubuntu:~/systemtap/systemtap-2.5$ dmesg| tail
…
[ 76.855961] stap_880c6f2fc0076129ea6feb7db23da703_899: Unknown parameter `_stp_bufsize‘
“-k”参数保留运行过程中生产的文件，查找线索
dd@ubuntu:~$ sudo stap -k -ve ‘probe begin{ log("hello world") exit() }‘
。。
Pass 5: starting run.
Error inserting module‘/tmp/staplCMn6Z/stap_4270.ko‘: Unknown symbol in module
WARNING: /usr/bin/staprun exited withstatus: 1
Pass 5: run completed in 0usr/0sys/3realms.
Pass 5: run failed. Try again with another ‘--vp 00001‘ option.
Keeping temporary directory"/tmp/staplCMn6Z"
dd@ubuntu:~$ sudo nm/tmp/staplCMn6Z/stap_4270.ko | grep _stp_bufsize
00000032 r __mod__stp_bufsize76
00000050 r __mod__stp_bufsizetype75
00000000 r __param__stp_bufsize
00000515 r __param_str__stp_bufsize
00000130 b _stp_bufsize
“--vp00005”启动systemtap的调试选项获得更多的log。 具体用法可参考http://blog.163.com/digoal@126/blog/static/163877040201391434530674/
dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap --vp 00005 -ve‘probe begin{printf("hello world\n") exit()}‘
。。。
Pass 5: starting run.
Running /usr/local/bin/staprun -v -v -v -v -v-R /tmp/stapwlfJOn/stap_333448f0b355beeb59e41a501a5421fd_817.ko
staprun:parse_modpath:386
。。。
staprun:insert_module:71 inserting module/tmp/stapwlfJOn/stap_333448f0b355beeb59e41a501a5421fd_817.ko
staprun:insert_module:97module options: _stp_bufsize=0
staprun:insert_module:105 module pathcanonicalized to ‘/tmp/stapwlfJOn/stap_333448f0b355beeb59e41a501a5421fd_817.ko‘
。。
staprun:insert_module:183 Modulestap_333448f0b355beeb59e41a501a5421f_6340 inserted from file/tmp/stapwlfJOn/stap_333448f0b355beeb59e41a501a5421fd_817.ko
ERROR:Couldn‘t insert module ‘/tmp/stapwlfJOn/stap_333448f0b355beeb59e41a501a5421fd_817.ko‘:Unknown symbol in module
Spawn waitpid result (0x100): 1
WARNING: /usr/local/bin/staprun exited withstatus: 1
Pass 5: run completed in 0usr/0sys/2real ms.
Pass 5: run failed. [man error::pass5]
Running rm -rf /tmp/stapwlfJOn
Spawn waitpid result (0x0): 0
Removed temporary directory"/tmp/stapwlfJOn"
dd@ubuntu:~/systemtap/systemtap-2.5$ grep -nr"_stp_bufsize" ./
Binary file ./staprun/staprun matches
Binary file ./staprun/staprun-staprun.omatches
./staprun/staprun.c:190: /* Add the _stp_bufsize option. */
./staprun/staprun.c:192: "_stp_bufsize=%d",buffer_size))
./runtime/transport/transport.c:75:static int_stp_bufsize;
./runtime/transport/transport.c:76:module_param(_stp_bufsize,int, 0);
./runtime/transport/transport.c:77:MODULE_PARM_DESC(_stp_bufsize,"buffer size");
./runtime/transport/transport.c:408: if (_stp_bufsize) {
./runtime/transport/transport.c:409: unsigned size = _stp_bufsize * 1024 *1024;
./runtime/transport/ring_buffer.c:83: unsigned long buffer_size = _stp_bufsize* 1024 * 1024;
./runtime/transport/transport.h:48:static int_stp_bufsize;
dd@ubuntu:/mnt/hgfs/systemtap/example$ sudomodinfo stapoMK4aC/stap_12809.ko [sudo] password for dd:
filename: stapoMK4aC/stap_12809.ko
。
parm: _stp_bufsize:buffer size (int)
从上面的代码看，模块参数_stp_bufsize确实存在，且加载的过程没有问题啊，很奇怪为什么会fail。难道是版本不匹配？尝试卸载2.0版本，安装2.5版本，依然有问题
于是尝试分析其内在原理，看是否能真正fix或者workaround这个问题。
./runtime/transport/transport.c接收模块参数_stp_bufsize，但是从代码看并没有特别的用处；另外通过“staprun:insert_module:97module options: _stp_bufsize=0”知道./staprun/staprun.c:192实际传入的参数值是0，因此可去掉special_options参数试试
static int insert_stap_module(privilege_t*user_credentials)
{
charspecial_options[128] = {0};
#if 0
/* Add the _stp_bufsize option. */
if (snprintf_chk(special_options,sizeof (special_options),
"_stp_bufsize=%d", buffer_size))
return -1;
#endif
stap_module_inserted= insert_module(modpath, special_options,
modoptions,
assert_stap_module_permissions,
user_credentials);
if (stap_module_inserted != 0)
err("Error insertingmodule ‘%s‘: %s\n", modpath, moderror(errno));
returnstap_module_inserted;
}
重新编译加载后，一切正常。
dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap -ve ‘probe process("./utrace-test").function("*"){printf("-------------------------------------\n") print_backtrace()printf("-------------------------------------\n")}‘
。。。
In file included from/tmp/stapRI0noH/stap_db3a2cd3f06ff0a78a55d936a32da8b0_1325_src.c:184:0:
/usr/share/systemtap/runtime/uprobes-inode.c:In function ‘stp_inode_uprobes_unreg’:
/usr/share/systemtap/runtime/uprobes-inode.c:94:3:error: implicit declaration of function ‘unregister_uprobe’[-Werror=implicit-function-declaration]
/usr/share/systemtap/runtime/uprobes-inode.c:In function ‘stp_inode_uprobes_reg’:
/usr/share/systemtap/runtime/uprobes-inode.c:107:3:error: implicit declaration of function ‘register_uprobe’ [-Werror=implicit-function-declaration]
cc1: all warnings being treated as errors
make[1]: ***[/tmp/stapRI0noH/stap_db3a2cd3f06ff0a78a55d936a32da8b0_1325_src.o] Error 1
make: *** [_module_/tmp/stapRI0noH] Error 2
WARNING: make exited with status: 2
Pass 4: compiled C into"stap_db3a2cd3f06ff0a78a55d936a32da8b0_1325.ko" in280usr/170sys/681real ms.
Pass 4: compilation failed. Try again with another ‘--vp 0001‘ option.
https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1075772
stap fails with: "error: implicitdeclaration of function ‘unregister_uprobe’"
dd@ubuntu:/mnt/hgfs/systemtap$ stap -V
Systemtap translator/driver (version 1.7/0.153 Debian version1.7-1ubuntu1 (quantal))
Copyright (C) 2005-2012 Red Hat, Inc. andothers
This is free software; see the source forcopying conditions.
enabled features: AVAHI LIBSQLITE3 NSSBOOST_SHARED_PTR TR1_UNORDERED_MAP NLS
内核中没有systemtap/runtime引用的API register_uprobe。在新版本的systemtap中已经fix了，因此需要下载源代码手动升级
dd@ubuntu:/mnt/hgfs/systemtap$ sudo grep"\suprobe.*register" /proc/kallsyms
c10fe1d0 T uprobe_register
c10fe430 T uprobe_unregister
升级到2.0版本之后，正常
dd@ubuntu:~/systemtap/systemtap-2.0$ stap-V
Systemtap translator/driver (version2.0/0.156, non-git sources)
Copyright (C) 2005-2012 Red Hat, Inc. andothers
This is free software; see the source forcopying conditions.
enabled features: TR1_UNORDERED_MAP NLS
Systemtap WiKi
https://wiki.ubuntu.com/Kernel/Systemtap
A guide on how to install Systemtap on anUbuntu system
https://sourceware.org/systemtap/wiki/SystemtapOnUbuntu
Linux 下的一个全新的性能测量和调式诊断工具Systemtap, 第 3 部分: Systemtap
http://www.ibm.com/developerworks/cn/linux/l-cn-systemtap3/#main
使用systemtap调试Linux内核示例
http://lenky.info/archives/2013/02/2209
User-Space Probing
https://sourceware.org/systemtap/SystemTap_Beginners_Guide/userspace-probing.html
Dynamic Tracing with DTrace and SystemTap
https://gist.github.com/dhedlund/9635035
http://guiquanz.me/2013/04/21/systemtap-intro/
Systemtap : stap PROCESSING 5 stepsintroduce
http://blog.163.com/digoal@126/blog/static/163877040201391434530674/
ubuntu+systemtap进行Linux内核和用户空间开发测试,布布扣,bubuko.com
ubuntu+systemtap进行Linux内核和用户空间开发测试
原文：http://blog.csdn.net/sailor_8318/article/details/25076745