有次突然观察到线上很多获取图片出现响应码500的错误。由于历史遗留原因，以前的图片文件保存在MongoDB的GridFS里面。这次突然出现获取图片500的错误，很有可能MongoDB出问题了。

用mongo命令连接到PRIMARY节点上去查看副本集的状态，如下：

ptcupload:PRIMARY> db.version()
2.4.8
ptcupload:PRIMARY> rs.status()
{
    "set" : "ptcupload",
    "date" : ISODate("2016-04-11T15:56:08Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "ptcupload-mongodb-001:27018",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 6355878,
            "optime" : Timestamp(1460390155, 1),
            "optimeDate" : ISODate("2016-04-11T15:55:55Z"),
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "ptcupload-mongodb-002:27018",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 45351,
            "optime" : Timestamp(1459863715, 2),
            "optimeDate" : ISODate("2016-04-05T13:41:55Z"),
            "lastHeartbeat" : ISODate("2016-04-11T15:56:08Z"),
            "lastHeartbeatRecv" : ISODate("2016-04-11T15:56:08Z"),
            "pingMs" : 0,
            "lastHeartbeatMessage" : "syncThread: 10334 BSONObj size: 1853163520 (0x0008756E) is invalid. Size must be between 0 and 16793600(16MB) First element: que: ?type=105",
            "syncingTo" : "ptcupload-mongodb-001:27018"
        },
        {
            "_id" : 2,
            "name" : "ptcupload-mongodb-003:37018",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 5992973,
            "lastHeartbeat" : ISODate("2016-04-11T15:56:07Z"),
            "lastHeartbeatRecv" : ISODate("2016-04-11T15:56:07Z"),
            "pingMs" : 0
        }
    ],
    "ok" : 1
}

整个副本集由一个PRIMARY、一个SECONDARY和一个ARBITER节点组成。从上面输出的副本集状态可以看到SECONDARY节点同步出现了问题：

"lastHeartbeatMessage" : "syncThread: 10334 BSONObj size: 1853163520 (0x0008756E) is invalid. Size must be between 0 and 16793600(16MB) First element: que: ?type=105"

还观察到SECONDARY节点的uptime为45351秒（12.59小时），远小于PRIMARY节点的uptime. 由此，怀疑12.59小时前SECONDARY节点宕机自动重启过。

我司MongoDB数据库有专门的数据库管理员管理，我们业务使用方没有查看服务器相关日志的权限，因此不方便调查故障原因。不过，对于这种情况，有一种修复办法，即将有问题的SECONDARY节点撤掉，重新加一个新的SECONDARY节点。