MySQL Bug导致异常宕机的分析流程

本文主要通过一个bug来记录一下如何分析一个MySQL bug的崩溃信息。
版本:Percona 5.7.17-11

一、数据库重启日志分析

terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check
04:10:09 UTC - mysqld got signal 6 ;
mysqld got signal 6 ;
......
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/dbdata/mysql3306/bin/mysqld(my_print_stacktrace+0x35)[0xf3e175]
/dbdata/mysql3306/bin/mysqld(handle_fatal_signal+0x4b4)[0x7c3b94]
/lib64/libpthread.so.0(+0xf7e0)[0x7f79f28e87e0]
/lib64/libc.so.6(gsignal+0x35)[0x7f79f137d495]
/lib64/libc.so.6(abort+0x175)[0x7f79f137ec75]
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x7f79f1c37a8d]
/usr/lib64/libstdc++.so.6(+0xbcbe6)[0x7f79f1c35be6]
/usr/lib64/libstdc++.so.6(+0xbcc13)[0x7f79f1c35c13]
/usr/lib64/libstdc++.so.6(+0xbcd32)[0x7f79f1c35d32]
/usr/lib64/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x67)[0x7f79f1bdadb7]
/dbdata/mysql3306/bin/mysqld[0x11d8f15]
/dbdata/mysql3306/bin/mysqld[0x11d99d5]
/dbdata/mysql3306/bin/mysqld(_Z17dict_stats_updateP12dict_table_t23dict_stats_upd_option_t+0x9dc)[0x11de0cc]
/dbdata/mysql3306/bin/mysqld(dict_stats_thread+0x4f2)[0x11e0512]
/lib64/libpthread.so.0(+0x7aa1)[0x7f79f28e0aa1]
/lib64/libc.so.6(clone+0x6d)[0x7f79f1433bcd]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

这部分是数据库崩溃的时候的栈帧，因为收到的是信号6 SIGABRT，只要捕获信号后改变其行为即可。这部分在MySQL官方文档中叫做Stack Trace，参考：

28.5.1.5 Using a Stack Trace

实际上在这里我们已经可以看到大约是统计数据收集的问题，因为我们看到了dict_stats_thread，这是统计收集线程的回调函数。

二、生成更加可视化的Stack Trace

1、通过Stack Trace获得内存地址

获取如下：

[0xf3e175]
[0x7c3b94]
[0x7f79f28e87e0]
[0x7f79f137d495]
[0x7f79f137ec75]
[0x7f79f1c37a8d]
[0x7f79f1c35be6]
[0x7f79f1c35c13]
[0x7f79f1c35d32]
[0x7f79f1bdadb7]
[0x11d8f15]
[0x11d99d5]
[0x11de0cc]
[0x11e0512]
[0x7f79f28e0aa1]
[0x7f79f1433bcd]

2、将这些地址放入一个文件

如：vi /tmp/err0222.log放入即可

3、通nm命令获取库文件链接文件

如：nm -D -n ./mysqld > /tmp/mysqld.sym

4、使用mysql工具resolve_stack_dump得到输出

如下：

[root@dyzsdb2 bin]# ./resolve_stack_dump -s /tmp/mysqld.sym -n /tmp/err0222.log | c++filt
0xf3e175 my_print_stacktrace + 53
0x7c3b94 handle_fatal_signal + 1204
0x7f79f28e87e0 _end + -258115144
0x7f79f137d495 _end + -280574355
0x7f79f137ec75 _end + -280568243
0x7f79f1c37a8d _end + -271422363
0x7f79f1c35be6 _end + -271430210
0x7f79f1c35c13 _end + -271430165
0x7f79f1c35d32 _end + -271429878
0x7f79f1bdadb7 _end + -271802481
0x11d8f15 dict_stats_analyze_index_for_n_prefix(dict_index_t*, unsigned long, std::vector<unsigned long, ut_allocator<unsigned long> > const*, n_diff_data_t*, mtr_t*) + 4949
0x11d99d5 dict_stats_analyze_index(dict_index_t*) + 2693
0x11de0cc dict_stats_update(dict_table_t*, dict_stats_upd_option_t) + 2524
0x11e0512 dict_stats_thread + 1266
0x7f79f28e0aa1 _end + -258147207
0x7f79f1433bcd _end + -279827035

实际上到这里通过函数的调用我们可以发现是统计数据收集出现了问题。

三、通过官方网站查询Bug

在报错信息中提起比较代表性的信息在官方网站中进行搜索通过在percona中查看发现本bug由上游MySQL代码造成BUG号：Bug #84940
并且发现在5.7.18中得到修复同时给出了内部BUG号如下：

[10 Feb 2017 8:12] Shane Bester
Oli, Umesh, this would be same as internal:
Bug 24585978 - INNODB: ASSERTION TOTAL_RECS > 0 FAILURE IN FILE DICT0STATS.CC

四、查询Bug到底修改了什么地方

这里请教了阿里的印风兄才知道怎么查看，首先拿到了内部bug号：24585978
然后在git的commit log中搜索得到
git --no-pager log >/root/commitlog
vi /root/commitlog 找到commit号为：

git show 29acdaaaeef9afe32b42785f1da3d79d56ed7e59
如下是这个bug的修复地方：

commit 29acdaaaeef9afe32b42785f1da3d79d56ed7e59
Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com>
Date:   Wed Feb 8 12:00:52 2017 +0530

    Bug #24585978       INNODB: ASSERTION TOTAL_RECS > 0 FAILURE
                        IN FILE DICT0STATS.CC
    
    Analysis:
    ========
    There was missing bracket for IF conditon in dict_stats_analyze_index_level()
    and it leads to wrong result.
    
    Fix:
    ====
    Fix the IF condition in dict_stats_analyze_index_level() so that it satisfied
    the if condtion only if level is zero.
    
    Reviewed-by : Jimmy Yang <jimmy.yang@oracle.com>

diff --git a/storage/innobase/dict/dict0stats.cc b/storage/innobase/dict/dict0stats.cc
index 3494070..55a2626 100644
--- a/storage/innobase/dict/dict0stats.cc
+++ b/storage/innobase/dict/dict0stats.cc
@@ -1099,10 +1099,10 @@ dict_stats_analyze_index_level(
                leaf-level delete marks because delete marks on
                non-leaf level do not make sense. */
 
-               if (level == 0 && srv_stats_include_delete_marked? 0:
+               if (level == 0 && (srv_stats_include_delete_marked ? 0:
                    rec_get_deleted_flag(
                            rec,
-                           page_is_comp(btr_pcur_get_page(&pcur)))) {
+                           page_is_comp(btr_pcur_get_page(&pcur))))) {
 
                        if (rec_is_last_on_page
                            && !prev_rec_is_copied

五、为什么这么修改

这里是我的浅显的分析，不对的地方的还请见谅。
我们发现这里实际上修改就是多了一个括号而已，但是意义是相当重要的。

if (level == 0 && srv_stats_include_delete_marked ? 0:
            rec_get_deleted_flag(
                rec,
                page_is_comp(btr_pcur_get_page(&pcur)))) 
修改为了
if (level == 0 && (srv_stats_include_delete_marked ? 0:
            rec_get_deleted_flag(
                rec,
                page_is_comp(btr_pcur_get_page(&pcur)))))

修改前：
如果level != 0 不管innodb_stats_include_delete_marked参数如何设置必然触发判断是否存在del_flag,然后通过设置偏移量的方式跳过这行，但是随后的(*total_recs)++; 将不会触发，极端情况下可能为0。
而在上层代码dict_stats_analyze_index中的found_level:地方实际上是需要非叶子结点行数不为0的如下：

        /* if any of these is 0 then there is exactly one page in the
        B-tree and it is empty and we should have done full scan and
        should not be here */
        ut_ad(total_recs > 0);
        ut_ad(n_diff_on_level[n_prefix - 1] > 0);

六、如何规避

在官网查看的时候有如下方式可以规避这个Bug

升级到5.7.18
设置参数

innodb-stats-persistent             = 0
innodb-stats-transient-sample-pages = 20
innodb-stats-auto-recalc            = 0

设置这些参数后实际上是使用的老的非固化的统计数据收集的方式，而不会通过线程dict_stats_thread收集下面是逻辑片段位于row_update_statistics_if_needed中如下：

    if (dict_stats_is_persistent_enabled(table)) { //参数innodb-stats-persistent 作用
        if (counter > n_rows / 10 /* 10% */
            && dict_stats_auto_recalc_is_enabled(table)) {//参数innodb-stats-auto-recalc 作用

            dict_stats_recalc_pool_add(table);
            table->stat_modified_counter = 0;
        }
        return;
    }

    /* Calculate new statistics if 1 / 16 of table has been modified
    since the last time a statistics batch was run.
    We calculate statistics at most every 16th round, since we may have
    a counter table which is very small and updated very often. */

    if (counter > 16 + n_rows / 16 /* 6.25% */) {

        ut_ad(!mutex_own(&dict_sys->mutex));
        /* this will reset table->stat_modified_counter to 0 */
        dict_stats_update(table, DICT_STATS_RECALC_TRANSIENT);
    }

这样做的话肯定不会调用到触发bug的函数，有兴趣的可以看看dict_stats_update(table, DICT_STATS_RECALC_TRANSIENT);的逻辑。实际上使用的是老的方式断点可以打在btr_estimate_number_of_different_key_vals函数上。