背景：

    开发找到说某张表只有到早上的数据

排查过程：

    登录数据库发现复制出现错误，index  表 faild，无法update，desc  表显示没那张表，查看相关data目录显示存在ibd文件

    登录线上数据库查看哪张表没有问题，此时线下数据突然crash

    数据库无法启动，查看错误日志显示：

      23. InnoDB: Reading tablespace information from the .ibd files...

  24. InnoDB: Restoring possible half-written data pages from the doublewrite

  25. InnoDB: buffer...

  26. InnoDB: Doing recovery: scanned up to log sequence number 9120034833

  27. 150125 16:12:51 InnoDB: Starting an apply batch of log records to the database...

  28. InnoDB: Progress in percents: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 150125 16:12:51 [ERROR] mysqld got signal 11 ;

  29. This could be because you hit a bug. It is also possible that this binary

  30. or one of the libraries it was linked against is corrupt, improperly built,

  31. or misconfigured. This error can also be caused by malfunctioning hardware.

  32. To report this bug, see http://kb.askmonty.org/en/reporting-bugs

  33. We will try our best to scrape up some info that will hopefully help

  34. diagnose the problem, but since we have already crashed,

  35. something is definitely wrong and this may fail.

  36. Server version: 5.5.37-MariaDB-log

  37. key_buffer_size=268435456

  38. read_buffer_size=1048576

  39. max_used_connections=0

  40. max_threads=1002

  41. thread_count=0

  42. It is possible that mysqld could use up to

  43. key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2332093 K bytes of memory

  44. 41 Hope that.

  无头绪....

  如果使用备份恢复，备份时间是早上8点的，说明备份数据表也有问题了，再次恢复意义不大，放弃...

  采取恢复救援模式启动mysql：修改参数文件的 innodb_force_recovery =1，会自动跳过检查corrupt页，数据库起来

  drop  那张表

  修改参数文件innodb_force_recovery =0

  正常启动mysql

  查看binlog日志找到具体的复制断点，到其它机器dump出这个表的数据导入进去

  start  slave；

  开始同步直至追上线上数据

  ##上述步骤仅限于有其它机器备份的时候

  下面适用于无其它机器可以拷贝损坏表数据的方法

  采用非常规手段,首先修改innodb_force_recovery参数，使mysqld跳过恢复步骤，将mysqld 启动,将数据导出来然后重建数据库

  <参考文档> http://blog.itpub.net/22664653/viewspace-1441389/