问题描述
机房临时硬件问题, 机器掉电. 恢复后, 重启mysql进程, 结果出现core dump. 而且一直循环, 无法终止, 只能kill掉. backtrace如下.
# service mysqld restart Stopping mysqld: [ OK ] *** glibc detected *** /usr/libexec/mysqld: malloc(): memory corruption: 0x0000000002f3d370 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3fea675e66] /lib64/libc.so.6[0x3fea679904] /lib64/libc.so.6(__libc_malloc+0x71)[0x3fea67a6b1] /usr/libexec/mysqld(my_malloc+0x32)[0x83e372] /usr/libexec/mysqld(alloc_root+0xd5)[0x83eba5] /usr/libexec/mysqld(strmake_root+0x22)[0x83ec92] /usr/libexec/mysqld(_Z14open_table_defP3THDP14st_table_sharej+0xb86)[0x610836] /usr/libexec/mysqld(_Z19mysql_rm_tmp_tablesv+0x267)[0x601247] /usr/libexec/mysqld(main+0x550)[0x5c1e90] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3fea61ed5d] /usr/libexec/mysqld[0x5098f9] ======= Memory map: ======== 00400000-00a63000 r-xp 00000000 ca:00 3682 /usr/libexec/mysqld 00c62000-00d3d000 rw-p 00662000 ca:00 3682 /usr/libexec/mysqld 00d3d000-00d52000 rw-p 00000000 00:00 0 00f3c000-00fb7000 rw-p 0073c000 ca:00 3682 /usr/libexec/mysqld 02643000-0425f000 rw-p 00000000 00:00 0 [heap] 3fe9e00000-3fe9e20000 r-xp 00000000 ca:00 29584 /lib64/ld-2.12.so 3fea01f000-3fea020000 r--p 0001f000 ca:00 29584 /lib64/ld-2.12.so 3fea020000-3fea021000 rw-p 00020000 ca:00 29584 /lib64/ld-2.12.so 3fea021000-3fea022000 rw-p 00000000 00:00 0 3fea200000-3fea202000 r-xp 00000000 ca:00 30363 /lib64/libdl-2.12.so 3fea202000-3fea402000 ---p 00002000 ca:00 30363 /lib64/libdl-2.12.so 3fea402000-3fea403000 r--p 00002000 ca:00 30363 /lib64/libdl-2.12.so 3fea403000-3fea404000 rw-p 00003000 ca:00 30363 /lib64/libdl-2.12.so 3fea600000-3fea78a000 r-xp 00000000 ca:00 30356 /lib64/libc-2.12.so 3fea78a000-3fea98a000 ---p 0018a000 ca:00 30356 /lib64/libc-2.12.so 3fea98a000-3fea98e000 r--p 0018a000 ca:00 30356 /lib64/libc-2.12.so 3fea98e000-3fea98f000 rw-p 0018e000 ca:00 30356 /lib64/libc-2.12.so 3fea98f000-3fea994000 rw-p 00000000 00:00 0 3feaa00000-3feaa17000 r-xp 00000000 ca:00 30371 /lib64/libpthread-2.12.so 3feaa17000-3feac17000 ---p 00017000 ca:00 30371 /lib64/libpthread-2.12.so 3feac17000-3feac18000 r--p 00017000 ca:00 30371 /lib64/libpthread-2.12.so 3feac18000-3feac19000 rw-p 00018000 ca:00 30371 /lib64/libpthread-2.12.so 3feac19000-3feac1d000 rw-p 00000000 00:00 0 3feae00000-3feae15000 r-xp 00000000 ca:00 30368 /lib64/libz.so.1.2.3 3feae15000-3feb014000 ---p 00015000 ca:00 30368 /lib64/libz.so.1.2.3 3feb014000-3feb015000 r--p 00014000 ca:00 30368 /lib64/libz.so.1.2.3 3feb015000-3feb016000 rw-p 00015000 ca:00 30368 /lib64/libz.so.1.2.3 3feb200000-3feb283000 r-xp 00000000 ca:00 30369 /lib64/libm-2.12.so 3feb283000-3feb482000 ---p 00083000 ca:00 30369 /lib64/libm-2.12.so 3feb482000-3feb483000 r--p 00082000 ca:00 30369 /lib64/libm-2.12.so 3feb483000-3feb484000 rw-p 00083000 ca:00 30369 /lib64/libm-2.12.so 3feb600000-3feb616000 r-xp 00000000 ca:00 29152 /lib64/libnsl-2.12.so 3feb616000-3feb815000 ---p 00016000 ca:00 29152 /lib64/libnsl-2.12.so 3feb815000-3feb816000 r--p 00015000 ca:00 29152 /lib64/libnsl-2.12.so 3feb816000-3feb817000 rw-p 00016000 ca:00 29152 /lib64/libnsl-2.12.so 3feb817000-3feb819000 rw-p 00000000 00:00 0 3feba00000-3feba1d000 r-xp 00000000 ca:00 30370 /lib64/libselinux.so.1 3feba1d000-3febc1c000 ---p 0001d000 ca:00 30370 /lib64/libselinux.so.1 3febc1c000-3febc1d000 r--p 0001c000 ca:00 30370 /lib64/libselinux.so.1 3febc1d000-3febc1e000 rw-p 0001d000 ca:00 30370 /lib64/libselinux.so.1 3febc1e000-3febc1f000 rw-p 00000000 00:00 0 3febe00000-3febe16000 r-xp 00000000 ca:00 29132 /lib64/libgcc_s-4.4.7-20120601.so.1 3febe16000-3fec015000 ---p 00016000 ca:00 29132 /lib64/libgcc_s-4.4.7-20120601.so.1 3fec015000-3fec016000 rw-p 00015000 ca:00 29132 /lib64/libgcc_s-4.4.7-20120601.so.1 3fec200000-3fec207000 r-xp 00000000 ca:00 30365 /lib64/libcrypt-2.12.so 3fec207000-3fec407000 ---p 00007000 ca:00 30365 /lib64/libcrypt-2.12.so 3fec407000-3fec408000 r--p 00007000 ca:00 30365 /lib64/libcrypt-2.12.so 3fec408000-3fec409000 rw-p 00008000 ca:00 30365 /lib64/libcrypt-2.12.so 3fec409000-3fec437000 rw-p 00000000 00:00 0 3fec600000-3fec602000 r-xp 00000000 ca:00 30364 /lib64/libfreebl3.so 3fec602000-3fec801000 ---p 00002000 ca:00 30364 /lib64/libfreebl3.so 3fec801000-3fec802000 r--p 00001000 ca:00 30364 /lib64/libfreebl3.so 3fec802000-3fec803000 rw-p 00002000 ca:00 30364 /lib64/libfreebl3.so 3feca00000-3feca16000 r-xp 00000000 ca:00 30384 /lib64/libresolv-2.12.so 3feca16000-3fecc16000 ---p 00016000 ca:00 30384 /lib64/libresolv-2.12.so 3fecc16000-3fecc17000 r--p 00016000 ca:00 30384 /lib64/libresolv-2.12.so 3fecc17000-3fecc18000 rw-p 00017000 ca:00 30384 /lib64/libresolv-2.12.so 3fecc18000-3fecc1a000 rw-p 00000000 00:00 0 3fece00000-3fecee8000 r-xp 00000000 ca:00 195 /usr/lib64/libstdc++.so.6.0.13 3fecee8000-3fed0e8000 ---p 000e8000 ca:00 195 /usr/lib64/libstdc++.so.6.0.13 3fed0e8000-3fed0ef000 r--p 000e8000 ca:00 195 /usr/lib64/libstdc++.so.6.0.13 3fed0ef000-3fed0f1000 rw-p 000ef000 ca:00 195 /usr/lib64/libstdc++.so.6.0.13 3fed0f1000-3fed106000 rw-p 00000000 00:00 0 3fed200000-3fed203000 r-xp 00000000 ca:00 30388 /lib64/libcom_err.so.2.1 3fed203000-3fed402000 ---p 00003000 ca:00 30388 /lib64/libcom_err.so.2.1 3fed402000-3fed403000 r--p 00002000 ca:00 30388 /lib64/libcom_err.so.2.1 3fed403000-3fed404000 rw-p 00003000 ca:00 30388 /lib64/libcom_err.so.2.1 3feea00000-3feebb9000 r-xp 00000000 ca:00 1235 /usr/lib64/libcrypto.so.1.0.1e 3feebb9000-3feedb8000 ---p 001b9000 ca:00 1235 /usr/lib64/libcrypto.so.1.0.1e 3feedb8000-3feedd3000 r--p 001b8000 ca:00 1235 /usr/lib64/libcrypto.so.1.0.1e 3feedd3000-3feeddf000 rw-p 001d3000 ca:00 1235 /usr/lib64/libcrypto.so.1.0.1e 3feeddf000-3feede3000 rw-p 00000000 00:00 0 3fef200000-3fef20a000 r-xp 00000000 ca:00 30386 /lib64/libkrb5support.so.0.1 3fef20a000-3fef409000 ---p 0000a000 ca:00 30386 /lib64/libkrb5support.so.0.1 3fef409000-3fef40a000 r--p 00009000 ca:00 30386 /lib64/libkrb5support.so.0.1 3fef40a000-3fef40b000 rw-p 0000a000 ca:00 30386 /lib64/libkrb5support.so.0.1 3fef600000-3fef629000 r-xp 00000000 ca:00 30387 /lib64/libk5crypto.so.3.1 3fef629000-3fef829000 ---p 00029000 ca:00 30387 /lib64/libk5crypto.so.3.1 3fef829000-3fef82a000 r--p 00029000 ca:00 30387 /lib64/libk5crypto.so.3.1 3fef82a000-3fef82b000 rw-p 0002a000 ca:00 30387 /lib64/libk5crypto.so.3.1 3fef82b000-3fef82c000 rw-p 00000000 00:00 0 3fefa00000-3fefa41000 r-xp 00000000 ca:00 30390 /lib64/libgssapi_krb5.so.2.2 3fefa41000-3fefc41000 ---p 00041000 ca:00 30390 /lib64/libgssapi_krb5.so.2.2 3fefc41000-3fefc42000 r--p 00041000 ca:00 30390 /lib64/libgssapi_krb5.so.2.2 3fefc42000-3fefc44000 rw-p 00042000 ca:00 30390 /lib64/libgssapi_krb5.so.2.2 3fefe00000-3fefedb000 r-xp 00000000 ca:00 30389 /lib64/libkrb5.so.3.3 3fefedb000-3ff00da000 ---p 000db000 ca:00 30389 /lib64/libkrb5.so.3.3 3ff00da000-3ff00e4000 r--p 000da000 ca:00 30389 /lib64/libkrb5.so.3.3 3ff00e4000-3ff00e6000 rw-p 000e4000 ca:00 30389 /lib64/libkrb5.so.3.3 3ff0200000-3ff0202000 r-xp 00000000 ca:00 30274 /lib64/libkeyutils.so.1.3 3ff0202000-3ff0401000 ---p 00002000 ca:00 30274 /lib64/libkeyutils.so.1.3 3ff0401000-3ff0402000 r--p 00001000 ca:00 30274 /lib64/libkeyutils.so.1.3 3ff0402000-3ff0403000 rw-p 00002000 ca:00 30274 /lib64/libkeyutils.so.1.3 3ff0600000-3ff0662000 r-xp 00000000 ca:00 1236 /usr/lib64/libssl.so.1.0.1e 3ff0662000-3ff0861000 ---p 00062000 ca:00 1236 /usr/lib64/libssl.so.1.0.1e 3ff0861000-3ff0865000 r--p 00061000 ca:00 1236 /usr/lib64/libssl.so.1.0.1e 3ff0865000-3ff086c000 rw-p 00065000 ca:00 1236 /usr/lib64/libssl.so.1.0.1e 7f8c8fdee000-7f8c8fdef000 ---p 00000000 00:00 0 7f8c8fdef000-7f8c905ef000 rw-p 00000000 00:00 0 [stack:1338] 7f8c905ef000-7f8c905f0000 ---p 00000000 00:00 0 7f8c905f0000-7f8c90df0000 rw-p 00000000 00:00 0 [stack:1337] 7f8c90df0000-7f8c90df1000 ---p 00000000 00:00 0 7f8c90df1000-7f8c915f1000 rw-p 00000000 00:00 0 [stack:1336] 7f8c915f1000-7f8c915f2000 ---p 00000000 00:00 0 7f8c915f2000-7f8c91df2000 rw-p 00000000 00:00 0 [stack:1335] 7f8c91df2000-7f8c91df3000 ---p 00000000 00:00 0 7f8c91df3000-7f8c927f8000 rw-p 00000000 00:00 0 7f8c927f8000-7f8c927f9000 ---p 00000000 00:00 0 7f8c927f9000-7f8c92ff9000 rw-p 00000000 00:00 0 [stack:1328] 7f8c92ff9000-7f8c92ffa000 ---p 00000000 00:00 0 7f8c92ffa000-7f8c937fa000 rw-p 00000000 00:00 0 [stack:1327] 7f8c937fa000-7f8c937fb000 ---p 00000000 00:00 0 7f8c937fb000-7f8c9c000000 rw-p 00000000 00:00 0 [stack:1326] 7f8c9c000000-7f8c9c021000 rw-p 00000000 00:00 0 7f8c9c021000-7f8ca0000000 ---p 00000000 00:00 0 7f8ca0155000-7f8ca01dd000 rw-p 00000000 00:00 0 7f8ca05ba000-7f8ca05bb000 ---p 00000000 00:00 0 7f8ca05bb000-7f8ca3293000 rw-p 00000000 00:00 0 [stack:1339] 7f8ca3293000-7f8ca329f000 r-xp 00000000 ca:00 29672 /lib64/libnss_files-2.12.so 7f8ca329f000-7f8ca349f000 ---p 0000c000 ca:00 29672 /lib64/libnss_files-2.12.so 7f8ca349f000-7f8ca34a0000 r--p 0000c000 ca:00 29672 /lib64/libnss_files-2.12.so 7f8ca34a0000-7f8ca34a1000 rw-p 0000d000 ca:00 29672 /lib64/libnss_files-2.12.so 7f8ca34a1000-7f8ca34a2000 ---p 00000000 00:00 0 7f8ca34a2000-7f8ca3cad000 rw-p 00000000 00:00 0 [stack:1325] 7f8ca3cb1000-7f8ca3cb9000 rw-p 00000000 00:00 0 7fffa76b9000-7fffa76da000 rw-p 00000000 00:00 0 [stack] 7fffa7770000-7fffa7772000 r--p 00000000 00:00 0 [vvar] 7fffa7772000-7fffa7774000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] *** glibc detected *** /usr/libexec/mysqld: malloc(): memory corruption: 0x0000000002f3d370 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3fea675e66] /lib64/libc.so.6[0x3fea679904] /lib64/libc.so.6(__libc_malloc+0x71)[0x3fea67a6b1] /lib64/libc.so.6(__backtrace_symbols+0x119)[0x3fea6feb19] /usr/libexec/mysqld(my_print_stacktrace+0x4b)[0x8509ab] /usr/libexec/mysqld(handle_fatal_signal+0x483)[0x6a3ef3] /lib64/libpthread.so.0[0x3feaa0f710] /lib64/libc.so.6(gsignal+0x35)[0x3fea632625] /lib64/libc.so.6(abort+0x175)[0x3fea633e05] /lib64/libc.so.6[0x3fea670537] /lib64/libc.so.6[0x3fea675e66] /lib64/libc.so.6[0x3fea679904] /lib64/libc.so.6(__libc_malloc+0x71)[0x3fea67a6b1] /usr/libexec/mysqld(my_malloc+0x32)[0x83e372] /usr/libexec/mysqld(alloc_root+0xd5)[0x83eba5] /usr/libexec/mysqld(strmake_root+0x22)[0x83ec92] /usr/libexec/mysqld(_Z14open_table_defP3THDP14st_table_sharej+0xb86)[0x610836] /usr/libexec/mysqld(_Z19mysql_rm_tmp_tablesv+0x267)[0x601247] /usr/libexec/mysqld(main+0x550)[0x5c1e90] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3fea61ed5d] /usr/libexec/mysqld[0x5098f9] ======= Memory map: ======== 00400000-00a63000 r-xp 00000000 ca:00 3682 /usr/libexec/mysqld 00c62000-00d3d000 rw-p 00662000 ca:00 3682 /usr/libexec/mysqld 00d3d000-00d52000 rw-p 00000000 00:00 0 00f3c000-00fb7000 rw-p 0073c000 ca:00 3682 /usr/libexec/mysqld 02643000-0425f000 rw-p 00000000 00:00 0 [heap] 3fe9e00000-3fe9e20000 r-xp 00000000 ca:00 29584 /lib64/ld-2.12.so 3fea01f000-3fea020000 r--p 0001f000 ca:00 29584 /lib64/ld-2.12.so 3fea020000-3fea021000 rw-p 00020000 ca:00 29584 /lib64/ld-2.12.so 3fea021000-3fea022000 rw-p 00000000 00:00 0 3fea200000-3fea202000 r-xp 00000000 ca:00 30363 /lib64/libdl-2.12.so 3fea202000-3fea402000 ---p 00002000 ca:00 30363 /lib64/libdl-2.12.so 3fea402000-3fea403000 r--p 00002000 ca:00 30363 /lib64/libdl-2.12.so 3fea403000-3fea404000 rw-p 00003000 ca:00 30363 /lib64/libdl-2.12.so 3fea600000-3fea78a000 r-xp 00000000 ca:00 30356 /lib64/libc-2.12.so 3fea78a000-3fea98a000 ---p 0018a000 ca:00 30356 /lib64/libc-2.12.so 3fea98a000-3fea98e000 r--p 0018a000 ca:00 30356 /lib64/libc-2.12.so 3fea98e000-3fea98f000 rw-p 0018e000 ca:00 30356 /lib64/libc-2.12.so 3fea98f000-3fea994000 rw-p 00000000 00:00 0 3feaa00000-3feaa17000 r-xp 00000000 ca:00 30371 /lib64/libpthread-2.12.so 3feaa17000-3feac17000 ---p 00017000 ca:00 30371 /lib64/libpthread-2.12.so 3feac17000-3feac18000 r--p 00017000 ca:00 30371 /lib64/libpthread-2.12.so 3feac18000-3feac19000 rw-p 00018000 ca:00 30371 /lib64/libpthread-2.12.so 3feac19000-3feac1d000 rw-p 00000000 00:00 0 3feae00000-3feae15000 r-xp 00000000 ca:00 30368 /lib64/libz.so.1.2.3 3feae15000-3feb014000 ---p 00015000 ca:00 30368 /lib64/libz.so.1.2.3 3feb014000-3feb015000 r--p 00014000 ca:00 30368 /lib64/libz.so.1.2.3 3feb015000-3feb016000 rw-p 00015000 ca:00 30368 /lib64/libz.so.1.2.3 3feb200000-3feb283000 r-xp 00000000 ca:00 30369 /lib64/libm-2.12.so 3feb283000-3feb482000 ---p 00083000 ca:00 30369 /lib64/libm-2.12.so 3feb482000-3feb483000 r--p 00082000 ca:00 30369 /lib64/libm-2.12.so 3feb483000-3feb484000 rw-p 00083000 ca:00 30369 /lib64/libm-2.12.so 3feb600000-3feb616000 r-xp 00000000 ca:00 29152 /lib64/libnsl-2.12.so 3feb616000-3feb815000 ---p 00016000 ca:00 29152 /lib64/libnsl-2.12.so 3feb815000-3feb816000 r--p 00015000 ca:00 29152 /lib64/libnsl-2.12.so 3feb816000-3feb817000 rw-p 00016000 ca:00 29152 /lib64/libnsl-2.12.so 3feb817000-3feb819000 rw-p 00000000 00:00 0 3feba00000-3feba1d000 r-xp 00000000 ca:00 30370 /lib64/libselinux.so.1 3feba1d000-3febc1c000 ---p 0001d000 ca:00 30370 /lib64/libselinux.so.1 3febc1c000-3febc1d000 r--p 0001c000 ca:00 30370 /lib64/libselinux.so.1 3febc1d000-3febc1e000 rw-p 0001d000 ca:00 30370 /lib64/libselinux.so.1 3febc1e000-3febc1f000 rw-p 00000000 00:00 0 3febe00000-3febe16000 r-xp 00000000 ca:00 29132 /lib64/libgcc_s-4.4.7-20120601.so.1 3febe16000-3fec015000 ---p 00016000 ca:00 29132 /lib64/libgcc_s-4.4.7-20120601.so.1 3fec015000-3fec016000 rw-p 00015000 ca:00 29132 /lib64/libgcc_s-4.4.7-20120601.so.1 3fec200000-3fec207000 r-xp 00000000 ca:00 30365 /lib64/libcrypt-2.12.so 3fec207000-3fec407000 ---p 00007000 ca:00 30365 /lib64/libcrypt-2.12.so 3fec407000-3fec408000 r--p 00007000 ca:00 30365 /lib64/libcrypt-2.12.so 3fec408000-3fec409000 rw-p 00008000 ca:00 30365 /lib64/libcrypt-2.12.so 3fec409000-3fec437000 rw-p 00000000 00:00 0 3fec600000-3fec602000 r-xp 00000000 ca:00 30364 /lib64/libfreebl3.so 3fec602000-3fec801000 ---p 00002000 ca:00 30364 /lib64/libfreebl3.so 3fec801000-3fec802000 r--p 00001000 ca:00 30364 /lib64/libfreebl3.so 3fec802000-3fec803000 rw-p 00002000 ca:00 30364 /lib64/libfreebl3.so 3feca00000-3feca16000 r-xp 00000000 ca:00 30384 /lib64/libresolv-2.12.so 3feca16000-3fecc16000 ---p 00016000 ca:00 30384 /lib64/libresolv-2.12.so 3fecc16000-3fecc17000 r--p 00016000 ca:00 30384 /lib64/libresolv-2.12.so 3fecc17000-3fecc18000 rw-p 00017000 ca:00 30384 /lib64/libresolv-2.12.so 3fecc18000-3fecc1a000 rw-p 00000000 00:00 0 3fece00000-3fecee8000 r-xp 00000000 ca:00 195 /usr/lib64/libstdc++.so.6.0.13 3fecee8000-3fed0e8000 ---p 000e8000 ca:00 195 /usr/lib64/libstdc++.so.6.0.13 3fed0e8000-3fed0ef000 r--p 000e8000 ca:00 195 /usr/lib64/libstdc++.so.6.0.13 3fed0ef000-3fed0f1000 rw-p 000ef000 ca:00 195 /usr/lib64/libstdc++.so.6.0.13 3fed0f1000-3fed106000 rw-p 00000000 00:00 0 3fed200000-3fed203000 r-xp 00000000 ca:00 30388 /lib64/libcom_err.so.2.1 3fed203000-3fed402000 ---p 00003000 ca:00 30388 /lib64/libcom_err.so.2.1 3fed402000-3fed403000 r--p 00002000 ca:00 30388 /lib64/libcom_err.so.2.1 3fed403000-3fed404000 rw-p 00003000 ca:00 30388 /lib64/libcom_err.so.2.1 3feea00000-3feebb9000 r-xp 00000000 ca:00 1235 /usr/lib64/libcrypto.so.1.0.1e 3feebb9000-3feedb8000 ---p 001b9000 ca:00 1235 /usr/lib64/libcrypto.so.1.0.1e 3feedb8000-3feedd3000 r--p 001b8000 ca:00 1235 /usr/lib64/libcrypto.so.1.0.1e 3feedd3000-3feeddf000 rw-p 001d3000 ca:00 1235 /usr/lib64/libcrypto.so.1.0.1e 3feeddf000-3feede3000 rw-p 00000000 00:00 0 3fef200000-3fef20a000 r-xp 00000000 ca:00 30386 /lib64/libkrb5support.so.0.1 3fef20a000-3fef409000 ---p 0000a000 ca:00 30386 /lib64/libkrb5support.so.0.1 3fef409000-3fef40a000 r--p 00009000 ca:00 30386 /lib64/libkrb5support.so.0.1 3fef40a000-3fef40b000 rw-p 0000a000 ca:00 30386 /lib64/libkrb5support.so.0.1 3fef600000-3fef629000 r-xp 00000000 ca:00 30387 /lib64/libk5crypto.so.3.1 3fef629000-3fef829000 ---p 00029000 ca:00 30387 /lib64/libk5crypto.so.3.1 3fef829000-3fef82a000 r--p 00029000 ca:00 30387 /lib64/libk5crypto.so.3.1 3fef82a000-3fef82b000 rw-p 0002a000 ca:00 30387 /lib64/libk5crypto.so.3.1 3fef82b000-3fef82c000 rw-p 00000000 00:00 0 3fefa00000-3fefa41000 r-xp 00000000 ca:00 30390 /lib64/libgssapi_krb5.so.2.2 3fefa41000-3fefc41000 ---p 00041000 ca:00 30390 /lib64/libgssapi_krb5.so.2.2 3fefc41000-3fefc42000 r--p 00041000 ca:00 30390 /lib64/libgssapi_krb5.so.2.2 3fefc42000-3fefc44000 rw-p 00042000 ca:00 30390 /lib64/libgssapi_krb5.so.2.2 3fefe00000-3fefedb000 r-xp 00000000 ca:00 30389 /lib64/libkrb5.so.3.3 3fefedb000-3ff00da000 ---p 000db000 ca:00 30389 /lib64/libkrb5.so.3.3 3ff00da000-3ff00e4000 r--p 000da000 ca:00 30389 /lib64/libkrb5.so.3.3 3ff00e4000-3ff00e6000 rw-p 000e4000 ca:00 30389 /lib64/libkrb5.so.3.3 3ff0200000-3ff0202000 r-xp 00000000 ca:00 30274 /lib64/libkeyutils.so.1.3 3ff0202000-3ff0401000 ---p 00002000 ca:00 30274 /lib64/libkeyutils.so.1.3 3ff0401000-3ff0402000 r--p 00001000 ca:00 30274 /lib64/libkeyutils.so.1.3 3ff0402000-3ff0403000 rw-p 00002000 ca:00 30274 /lib64/libkeyutils.so.1.3 3ff0600000-3ff0662000 r-xp 00000000 ca:00 1236 /usr/lib64/libssl.so.1.0.1e 3ff0662000-3ff0861000 ---p 00062000 ca:00 1236 /usr/lib64/libssl.so.1.0.1e 3ff0861000-3ff0865000 r--p 00061000 ca:00 1236 /usr/lib64/libssl.so.1.0.1e 3ff0865000-3ff086c000 rw-p 00065000 ca:00 1236 /usr/lib64/libssl.so.1.0.1e 7f8c8fdee000-7f8c8fdef000 ---p 00000000 00:00 0 7f8c8fdef000-7f8c905ef000 rw-p 00000000 00:00 0 [stack:1338] 7f8c905ef000-7f8c905f0000 ---p 00000000 00:00 0 7f8c905f0000-7f8c90df0000 rw-p 00000000 00:00 0 [stack:1337] 7f8c90df0000-7f8c90df1000 ---p 00000000 00:00 0 7f8c90df1000-7f8c915f1000 rw-p 00000000 00:00 0 [stack:1336] 7f8c915f1000-7f8c915f2000 ---p 00000000 00:00 0 7f8c915f2000-7f8c91df2000 rw-p 00000000 00:00 0 [stack:1335] 7f8c91df2000-7f8c91df3000 ---p 00000000 00:00 0 7f8c91df3000-7f8c927f8000 rw-p 00000000 00:00 0 7f8c927f8000-7f8c927f9000 ---p 00000000 00:00 0 7f8c927f9000-7f8c92ff9000 rw-p 00000000 00:00 0 [stack:1328] 7f8c92ff9000-7f8c92ffa000 ---p 00000000 00:00 0 7f8c92ffa000-7f8c937fa000 rw-p 00000000 00:00 0 [stack:1327] 7f8c937fa000-7f8c937fb000 ---p 00000000 00:00 0 7f8c937fb000-7f8c9c000000 rw-p 00000000 00:00 0 [stack:1326] 7f8c9c000000-7f8c9c021000 rw-p 00000000 00:00 0 7f8c9c021000-7f8ca0000000 ---p 00000000 00:00 0 7f8ca0155000-7f8ca01dd000 rw-p 00000000 00:00 0 7f8ca05ba000-7f8ca05bb000 ---p 00000000 00:00 0 7f8ca05bb000-7f8ca3293000 rw-p 00000000 00:00 0 [stack:1339] 7f8ca3293000-7f8ca329f000 r-xp 00000000 ca:00 29672 /lib64/libnss_files-2.12.so 7f8ca329f000-7f8ca349f000 ---p 0000c000 ca:00 29672 /lib64/libnss_files-2.12.so 7f8ca349f000-7f8ca34a0000 r--p 0000c000 ca:00 29672 /lib64/libnss_files-2.12.so 7f8ca34a0000-7f8ca34a1000 rw-p 0000d000 ca:00 29672 /lib64/libnss_files-2.12.so 7f8ca34a1000-7f8ca34a2000 ---p 00000000 00:00 0 7f8ca34a2000-7f8ca3cad000 rw-p 00000000 00:00 0 [stack:1325] 7f8ca3cb0000-7f8ca3cb1000 rw-p 00000000 00:00 0 7f8ca3cb2000-7f8ca3cb9000 rw-p 00000000 00:00 0 7fffa76b9000-7fffa76da000 rw-p 00000000 00:00 0 [stack] 7fffa7770000-7fffa7772000 r--p 00000000 00:00 0 [vvar] 7fffa7772000-7fffa7774000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] *** glibc detected *** /usr/libexec/mysqld: malloc(): memory corruption: 0x0000000002bc6370 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3fea675e66] /lib64/libc.so.6[0x3fea679904] /lib64/libc.so.6(__libc_malloc+0x71)[0x3fea67a6b1] /usr/libexec/mysqld(my_malloc+0x32)[0x83e372] /usr/libexec/mysqld(alloc_root+0xd5)[0x83eba5] /usr/libexec/mysqld(strmake_root+0x22)[0x83ec92] /usr/libexec/mysqld(_Z14open_table_defP3THDP14st_table_sharej+0xb86)[0x610836] /usr/libexec/mysqld(_Z19mysql_rm_tmp_tablesv+0x267)[0x601247] /usr/libexec/mysqld(main+0x550)[0x5c1e90] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3fea61ed5d] /usr/libexec/mysqld[0x5098f9] ======= Memory map: ======== 00400000-00a63000 r-xp 00000000 ca:00 3682 /usr/libexec/mysqld 00c62000-00d3d000 rw-p 00662000 ca:00 3682 /usr/libexec/mysqld 00d3d000-00d52000 rw-p 00000000 00:00 0 00f3c000-00fb7000 rw-p 0073c000 ca:00 3682 /usr/libexec/mysqld 022cc000-03ee8000 rw-p 00000000 00:00 0 [heap] 3fe9e00000-3fe9e20000 r-xp 00000000 ca:00 29584 /lib64/ld-2.12.so 3fea01f000-3fea020000 r--p 0001f000 ca:00 29584 /lib64/ld-2.12.so 3fea020000-3fea021000 rw-p 00020000 ca:00 29584 /lib64/ld-2.12.so 3fea021000-3fea022000 rw-p 00000000 00:00 0 3fea200000-3fea202000 r-xp 00000000 ca:00 30363 /lib64/libdl-2.12.so 3fea202000-3fea402000 ---p 00002000 ca:00 30363 /lib64/libdl-2.12.so 3fea402000-3fea403000 r--p 00002000 ca:00 30363 /lib64/libdl-2.12.so 3fea403000-3fea404000 rw-p 00003000 ca:00 30363 /lib64/libdl-2.12.so 3fea600000-3fea78a000 r-xp 00000000 ca:00 30356 /lib64/libc-2.12.so 3fea78a000-3fea98a000 ---p 0018a000 ca:00 30356 /lib64/libc-2.12.so 3fea98a000-3fea98e000 r--p 0018a000 ca:00 30356 /lib64/libc-2.12.so
根据backtrace中的_Z19mysql_rm_tmp_tablesv 可以确定的是mysql重启时, 清除临时表的时候, 崩溃. 猜测是机器掉电的时候, 把临时表的数据或者结构搞乱了.
此后, 处理此问题时候, 由于数据有备份, 所以处理此问题时, 暴力处理, 重装mysql, 删掉corrupt的数据(/var/lib/mysql, 我们是有数据备份的, 所以可以删, 如果没有备份万不可删). 问题依然存在, 猜测其他地方依然有mysql的数据. 查看/tmp文件夹, 发现了$sql*的数据, 清除. 重启mysql问题解决.
此问题可以深入研究, TBD.
原文:http://www.cnblogs.com/randywithj/p/4356842.html