ceph backfill

第一次backfill 循环

step1: 
     new_backfill: 1
     get_backfill_targets(): 2
     last_backfill_started: MIN
     local pg_info_t: 1.0( v 20"20000 (20"9900,20"20000"] local-lis/les=32/33 n=20000 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
     earliest_backfill(): MIN

step2: before update_range 
     backfill_info: BackfillInfo(MIN-MIN 0 version: 0"0 objects)"
     pending_backfill_updates: {}
     backfills_in_flight: 
     peer: 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MIN-MIN 0 version: 0"0 objects)"
         peer pg_info_t: 1.0( v 20"20000 (20"10000,20"20000"] lb MIN local-lis/les=32/33 n=0 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
step3: after update_range 
     local backfill_info: BackfillInfo(MIN-1:067f54e0:::test19063.txt:head 511 version: 20"20000 objects 
     {1:00001e5d:::test4054.txt:head=20"4054,
     1:0005c6e0:::test18392.txt:head=20"18392,
     1:000a2176:::test11801.txt:head=20"11801,
     1:000d9c27:::test1033.txt:head=20"1033,
     ,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})              
step4: befor while (ops < max)
     local backfill_info: BackfillInfo(1:00001e5d:::test4054.txt:head-1:067f54e0:::test19063.txt:head 511 version: 20"20000 objects {1:00001e5d:::test4054.txt:head=20"4054,1:0005c6e0:::test18392.txt:head=20"18392,1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     earliest_peer_backfill(): MIN     
     peer 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MIN-MIN 0 version: 0"0 objects)"

step8 in while send to peer scan
     peer: 2
     pbi: BackfillInfo(MIN-MIN 0 version: 0"0 objects)"

step 10:  out while:  before update peer last_backfill:
     backfills_in_flight: 
     backfill_info: BackfillInfo(1:00001e5d:::test4054.txt:head-1:067f54e0:::test19063.txt:head 511 version: 20"20000 objects {1:00001e5d:::test4054.txt:head=20"4054,1:0005c6e0:::test18392.txt:head=20"18392,1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     pending_backfill_updates: {}
     last_backfill_started: MIN
     add_to_stat: 
     next_backfill_to_complete: MIN
     new_last_backfill: MIN
     backfill_pos MIN
     peer: 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MIN-MIN 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 20"20000 (20"10000,20"20000] lb MIN local-lis/les=32/33 n=0 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
step : the end compute new_last_backfill
     new_last_backfill MIN         
     peer: 2
         peer last_backfill: MIN
         new_last_backfill > pinfo.last_backfill : 0
         pbi: BackfillInfo(MIN-MIN 0 version: 0"0 objects)"

do_scan 返回：
return in OP_SCAN_DIGEST BackfillInfo(MIN-MAX 0 version: 0"0 objects) from 2

第二次循环：

(PG::finish_recovery_op(hobject_t const&, bool)+0x13c)
void PrimaryLogPG::do_scan(OpRequestRef op,ThreadPool::TPHandle &handle)

step1: 
     new_backfill: 0
     get_backfill_targets(): 2
     last_backfill_started: MIN
     local pg_info_t: 1.0( v 20"20000 (20"9900,20"20000] local-lis/les=32/33 n=20000 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
     earliest_backfill(): MIN         
step2: before update_range 
     backfill_info: BackfillInfo(MIN-1:067f54e0:::test19063.txt:head 511 version: 20"20000 objects {1:00001e5d:::test4054.txt:head=20"4054,1:0005c6e0:::test18392.txt:head=20"18392,1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     pending_backfill_updates: {}
     backfills_in_flight: 
    peer: 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MIN-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 20"20000 (20"10000,20"20000] lb MIN local-lis/les=32/33 n=0 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)          
step3: after update_range 
     local backfill_info: BackfillInfo(MIN-1:067f54e0:::test19063.txt:head 511 version: 20"20000 objects {1:00001e5d:::test4054.txt:head=20"4054,1:0005c6e0:::test18392.txt:head=20"18392,1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
step4: befor while (ops < max)
     local backfill_info: BackfillInfo(1:00001e5d:::test4054.txt:head-1:067f54e0:::test19063.txt:head 511 version: 20"20000 objects {1:00001e5d:::test4054.txt:head=20"4054,1:0005c6e0:::test18392.txt:head=20"18392,1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     earliest_peer_backfill(): MAX         
     peer 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)     
step inwhile:
     process file backfill_info.begin 1:00001e5d:::test4054.txt:head
     check MAX
     need_ver_targs= []
     keep_ver_targs= []
     missing_targs= [2]
     skip_targs= []         
step 9: in while after process one file
     lcoal backfill_info: BackfillInfo(1:0005c6e0:::test18392.txt:head-1:067f54e0:::test19063.txt:head 510 version: 20"20000 objects {1:0005c6e0:::test18392.txt:head=20"18392,1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     last_backfill_started 1:00001e5d:::test4054.txt:head     
     peer 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
step 10:  out while:  before update peer last_backfill:
     backfills_in_flight: 1:00001e5d:::test4054.txt:head
     backfill_info: BackfillInfo(1:0005c6e0:::test18392.txt:head-1:067f54e0:::test19063.txt:head 510 version: 20"20000 objects {1:0005c6e0:::test18392.txt:head=20"18392,1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     pending_backfill_updates: {1:00001e5d:::test4054.txt:head=0"0}
     last_backfill_started: 1:00001e5d:::test4054.txt:head
     add_to_stat: 1:00001e5d:::test4054.txt:head
     next_backfill_to_complete: 1:00001e5d:::test4054.txt:head
     new_last_backfill: MIN
     backfill_pos 1:0005c6e0:::test18392.txt:head
     peer: 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 20"20000 (20"10000,20"20000] lb MIN local-lis/les=32/33 n=0 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)         

    pending_backfill_updates < next_backfill_to_complete 
    才会更新 new_last_backfill 为pending_backfill_updates的元素

step : the end compute new_last_backfill
     new_last_backfill MIN
     peer: 2
         peer last_backfill: MIN
         new_last_backfill > pinfo.last_backfill : 0
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)

处理了一个文件之后，开始第三次循环：

5: (PG::queue_recovery()+0x304) 
6: (PG::finish_recovery_op(hobject_t const&, bool)+0x13c) 
7: (PrimaryLogPG::on_global_recover(hobject_t const&, object_stat_sum_t const&, bool)+0x264)
8: (ReplicatedBackend::handle_push_reply(pg_shard_t, PushReplyOp const&, PushOp*)+0x660)
9: (ReplicatedBackend::do_push_reply(boost::intrusive_ptr<OpRequest>)+0xe8) 
10: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x1a4)

如果文件比较大，数据和omap的kv实际大小，如果过大会分多批次进行PushOP,如果最终处理完成，就触发下一个文件
osd_recovery_max_chunk 8M
osd_recovery_max_omap_entries_per_chunk 8096

step1: 
     new_backfill: 0
     get_backfill_targets(): 2
     last_backfill_started: 1:00001e5d:::test4054.txt:head
     local pg_info_t: 1.0( v 20"20000 (20"9900,20"20000] local-lis/les=32/33 n=20000 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
     earliest_backfill(): MIN
step2: before update_range 
     backfill_info: BackfillInfo(1:00001e5d:::test4054.txt:head-1:067f54e0:::test19063.txt:head 510 version: 20"20000 objects {1:0005c6e0:::test18392.txt:head=20"18392,1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     pending_backfill_updates: {1:00001e5d:::test4054.txt:head=0"0}
     backfills_in_flight: 
     peer: 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 20"20000 (20"10000,20"20000] lb MIN local-lis/les=32/33 n=0 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)     
step3: after update_range 
     local backfill_info: BackfillInfo(1:00001e5d:::test4054.txt:head-1:067f54e0:::test19063.txt:head 510 version: 20"20000 objects {1:0005c6e0:::test18392.txt:head=20"18392,1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
step4: befor while (ops < max)
     local backfill_info: BackfillInfo(1:0005c6e0:::test18392.txt:head-1:067f54e0:::test19063.txt:head 510 version: 20"20000 objects {1:0005c6e0:::test18392.txt:head=20"18392,1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     earliest_peer_backfill(): MAX         
     peer 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
step inwhile:
     process file backfill_info.begin 1:0005c6e0:::test18392.txt:head
     check MAX
     need_ver_targs= []
     keep_ver_targs= []
     missing_targs= [2]
     skip_targs= []
step 9: in while after process one file
     lcoal backfill_info: BackfillInfo(1:000a2176:::test11801.txt:head-1:067f54e0:::test19063.txt:head 509 version: 20"20000 objects {1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     last_backfill_started 1:0005c6e0:::test18392.txt:head
     peer 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)         
step 10:  out while:  before update peer last_backfill:
     backfills_in_flight: 1:0005c6e0:::test18392.txt:head
     backfill_info: BackfillInfo(1:000a2176:::test11801.txt:head-1:067f54e0:::test19063.txt:head 509 version: 20"20000 objects {1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     pending_backfill_updates: {1:00001e5d:::test4054.txt:head=0"0,1:0005c6e0:::test18392.txt:head=0"0}
     last_backfill_started: 1:0005c6e0:::test18392.txt:head
     add_to_stat: 1:0005c6e0:::test18392.txt:head
     next_backfill_to_complete: 1:0005c6e0:::test18392.txt:head
     new_last_backfill: MIN
     backfill_pos 1:000a2176:::test11801.txt:head
     peer: 2
         peer last_backfill: MIN
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 20"20000 (20"10000,20"20000] lb MIN local-lis/les=32/33 n=0 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
step : the end compute new_last_backfill
     new_last_backfill 1:00001e5d:::test4054.txt:head
     peer: 2
         peer last_backfill: MIN
         new_last_backfill > pinfo.last_backfill : 1
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)

第四次循环：关注 peer last_backfill

step1: 
     new_backfill: 0
     get_backfill_targets(): 2
     last_backfill_started: 1:0005c6e0:::test18392.txt:head
     local pg_info_t: 1.0( v 20"20000 (20"9900,20"20000] local-lis/les=32/33 n=20000 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
     earliest_backfill(): 1:00001e5d:::test4054.txt:head

step2: before update_range 
     backfill_info: BackfillInfo(1:0005c6e0:::test18392.txt:head-1:067f54e0:::test19063.txt:head 509 version: 20"20000 objects {1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     pending_backfill_updates: {1:0005c6e0:::test18392.txt:head=0"0}
     backfills_in_flight: 
     peer: 2
         peer last_backfill: 1:00001e5d:::test4054.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 20"20000 (20"10000,20"20000] lb 1:00001e5d:::test4054.txt:head local-lis/les=32/33 n=1 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
step3: after update_range 
     local backfill_info: BackfillInfo(1:0005c6e0:::test18392.txt:head-1:067f54e0:::test19063.txt:head 509 version: 20"20000 objects {1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})

step4: befor while (ops < max)
     local backfill_info: BackfillInfo(1:000a2176:::test11801.txt:head-1:067f54e0:::test19063.txt:head 509 version: 20"20000 objects {1:000a2176:::test11801.txt:head=20"11801,1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     earliest_peer_backfill(): MAX         
     peer 2
         peer last_backfill: 1:00001e5d:::test4054.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
step inwhile:
     process file backfill_info.begin 1:000a2176:::test11801.txt:head
     check MAX
     need_ver_targs= []
     keep_ver_targs= []
     missing_targs= [2]
     skip_targs= []

step 9: in while after process one file
     lcoal backfill_info: BackfillInfo(1:000d9c27:::test1033.txt:head-1:067f54e0:::test19063.txt:head 508 version: 20"20000 objects {1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     last_backfill_started 1:000a2176:::test11801.txt:head
     peer 2
         peer last_backfill: 1:00001e5d:::test4054.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)         
step 10:  out while:  before update peer last_backfill:
     backfills_in_flight: 1:000a2176:::test11801.txt:head
     backfill_info: BackfillInfo(1:000d9c27:::test1033.txt:head-1:067f54e0:::test19063.txt:head 508 version: 20"20000 objects {1:000d9c27:::test1033.txt:head=20"1033,,,,,,,,,1:06738052:::test4708.txt:head=20"4708,1:067e20fb:::test10941.txt:head=20"10941})
     pending_backfill_updates: {1:0005c6e0:::test18392.txt:head=0"0,1:000a2176:::test11801.txt:head=0"0}
     last_backfill_started: 1:000a2176:::test11801.txt:head
     add_to_stat: 1:000a2176:::test11801.txt:head
     next_backfill_to_complete: 1:000a2176:::test11801.txt:head
     new_last_backfill: 1:00001e5d:::test4054.txt:head
     backfill_pos 1:000d9c27:::test1033.txt:head
     peer: 2
         peer last_backfill: 1:00001e5d:::test4054.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 20"20000 (20"10000,20"20000] lb 1:00001e5d:::test4054.txt:head local-lis/les=32/33 n=1 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
step : the end compute new_last_backfill
     new_last_backfill 1:0005c6e0:::test18392.txt:head
     peer: 2
         peer last_backfill: 1:00001e5d:::test4054.txt:head
         new_last_backfill > pinfo.last_backfill : 1
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)

511 个数据，已经遍历空了 : bi 为空之后的循环:

step1: 
     new_backfill: 0
     get_backfill_targets(): 2
     last_backfill_started: 1:067e20fb:::test10941.txt:head
     local pg_info_t: 1.0( v 20"20000 (20"9900,20"20000] local-lis/les=32/33 n=20000 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
     earliest_backfill(): 1:06738052:::test4708.txt:head

step2: before update_range 
     backfill_info: BackfillInfo(1:067e20fb:::test10941.txt:head-1:067f54e0:::test19063.txt:head 0 version: 20"20000 objects)
     pending_backfill_updates: {1:067e20fb:::test10941.txt:head=0"0}
     backfills_in_flight: 
     peer: 2
         peer last_backfill: 1:06738052:::test4708.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 20"20000 (20"10000,20"20000] lb 1:06738052:::test4708.txt:head local-lis/les=32/33 n=510 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
step3: after update_range 
     local backfill_info: BackfillInfo(1:067e20fb:::test10941.txt:head-1:067f54e0:::test19063.txt:head 0 version: 20"20000 objects)

step4: befor while (ops < max)
     local backfill_info: BackfillInfo(1:067f54e0:::test19063.txt:head-1:067f54e0:::test19063.txt:head 0 version: 20"20000 objects)
     earliest_peer_backfill(): MAX
     peer 2
         peer last_backfill: 1:06738052:::test4708.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)

step5: in while befor update_range 1
    local backfill_info: BackfillInfo(1:067f54e0:::test19063.txt:head-1:067f54e0:::test19063.txt:head 0 version: 20"20000 objects)

step5: in while befor update_range 2
     local backfill_info: BackfillInfo(1:067f54e0:::test19063.txt:head-MAX 0 version: 0"0 objects)
step7: in while after update_range 3
     local backfill_info: BackfillInfo(1:067f54e0:::test19063.txt:head-1:0c924fff:::test14189.txt:head 512 version: 20"20000 objects {1:067f54e0:::test19063.txt:head=20"19063,1:0680e042:::test10674.txt:head=20"10674,,,,,,,,,,1:0c8ef72a:::test19518.txt:head=20"19518})

step inwhile:
     process file backfill_info.begin 1:067f54e0:::test19063.txt:head
     check MAX
     need_ver_targs= []
     keep_ver_targs= []
     missing_targs= [2]
     skip_targs= []

step 9: in while after process one file
     lcoal backfill_info: BackfillInfo(1:0680e042:::test10674.txt:head-1:0c924fff:::test14189.txt:head 511 version: 20"20000 objects {1:0680e042:::test10674.txt:head=20"10674,,,,,,,,,,1:0c8ef72a:::test19518.txt:head=20"19518})
     last_backfill_started 1:067f54e0:::test19063.txt:head
     peer 2
         peer last_backfill: 1:06738052:::test4708.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)

step 10:  out while:  before update peer last_backfill:
     backfills_in_flight: 1:067f54e0:::test19063.txt:head
     backfill_info: BackfillInfo(1:0680e042:::test10674.txt:head-1:0c924fff:::test14189.txt:head 511 version: 20"20000 objects {1:0680e042:::test10674.txt:head=20"10674,,,,,,,,,,1:0c8ef72a:::test19518.txt:head=20"19518})
     pending_backfill_updates: {1:067e20fb:::test10941.txt:head=0"0,1:067f54e0:::test19063.txt:head=0"0}
     last_backfill_started: 1:067f54e0:::test19063.txt:head
     add_to_stat: 1:067f54e0:::test19063.txt:head
     next_backfill_to_complete: 1:067f54e0:::test19063.txt:head
     new_last_backfill: 1:06738052:::test4708.txt:head
     backfill_pos 1:0680e042:::test10674.txt:head
     peer: 2
         peer last_backfill: 1:06738052:::test4708.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 20"20000 (20"10000,20"20000] lb 1:06738052:::test4708.txt:head local-lis/les=32/33 n=510 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
step : the end compute new_last_backfill
     new_last_backfill 1:067e20fb:::test10941.txt:head
     peer: 2
         peer last_backfill: 1:06738052:::test4708.txt:head
         new_last_backfill > pinfo.last_backfill : 1
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)

在backfill的过程中,写入新的数据,查看双边osd的行为,如何保障backfill的一致性

每次写入一个文件 projected_last_update 版本就更新为 epoch + version_t, 39"20001,39"20002; update_range 需要和最新更新的文件ptlog 进行状态merge
实际写入aaa.txt.1-aaa.txt.7 7个文件, 只有aaa.txt.3 在osd2中落盘了, 因为osd 2 last_backfill 是： 1:18b5b223:::test10094.txt:head
按照hash计算, 只有aaa.txt.3 的hash reverse小于 osd2的last_backfill,只有它落盘了。

aaa.txt.1 0x6a06655f  0xfaa66056
aaa.txt.2 0xeea16c69  0x96368577   
aaa.txt.3 0xbc5572b0  0x0d4eaa3d   
aaa.txt.4 0x361a48ac  0x3512586c   
aaa.txt.5 0xc1a4b22a  0x544d2583   
aaa.txt.6 0xbdc7c15e  0x7a83e3bd   
aaa.txt.7 0x6e517bcb  0xd3de8a76   

test10094.txt 0xc44dad18  0x18b5b223

./bin/ceph-kvstore-tool bluestore-kv ./dev/osd2 list O
./bin/ceph-kvstore-tool bluestore-kv ./dev/osd2 dump O
使用工具查看osd2的数据过滤aaa.txt

O       %7f%80%00%00%00%00%00%00%01%0dN%aa%3d%21aaa.txt.3%21%3d%ff%ff%ff%ff%ff%ff%ff%fe%ff%ff%ff%ff%ff%ff%ff%ffo

当恢复的数据量比较多时,停止backfill,然后再次写入7个bbb.txt.1-7 查看osd2的落盘情况:
new_last_backfill 1:f85b51eb:::test6800.txt:head

bbb.txt.1 0x49065466  0x662a6092   
bbb.txt.2 0x39ae3912  0x489c759c   
bbb.txt.3 0xba7464b2  0x4d262e5d   
bbb.txt.4 0xc99ba421  0x8425d993   
bbb.txt.5 0x5fbf94b7  0xed29fdfa   
bbb.txt.6 0x55dd7d8b  0xd1bebbaa   
bbb.txt.7 0xd828a671  0x8e65141b

osd.2 再次过滤aaa.txt 除了aaa.txt.1 不存在,其他都已经存在
osd.2 过滤bbb.txt 7个文件都存在, 说明了last_backfill 在backfill中的作用, 有点过于严谨了

step1: 
     new_backfill: 0
     get_backfill_targets(): 2
     last_backfill_started: 1:18b725a9:::test10814.txt:head
     local pg_info_t: 1.0( v 39"20001 (20"10000,39"20001] local-lis/les=32/33 n=20001 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)
     earliest_backfill(): 1:18b5b223:::test10094.txt:head

step2: before update_range 
     backfill_info: BackfillInfo(1:18b725a9:::test10814.txt:head-1:19fba18e:::test12743.txt:head 88 version: 20"20000 objects {1:18b7eb0c:::test15864.txt:head=20"15864,,,,,,,,,,1:19f875c7:::test17364.txt:head=20"17364})
     pending_backfill_updates: {1:18b725a9:::test10814.txt:head=0"0}
     backfills_in_flight: 
     peer: 2
         peer last_backfill: 1:18b5b223:::test10094.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 39"20001 (20"10000,39"20001] lb 1:18b5b223:::test10094.txt:head local-lis/les=32/33 n=1958 ec=3/3 lis/c=32/19 les/c/f=33/20/0 sis=32)

2023-11-23T18:30:37.812+0800 ffff73bde9c0 10 update_range: bi is old, (20"20000) can be updated with log to projected_last_update 39"20001
2023-11-23T18:30:37.812+0800 ffff73bde9c0 10 scanning pg log first
2023-11-23T18:30:37.812+0800 ffff73bde9c0 10 operator(): updating from version 39"20001
2023-11-23T18:30:37.812+0800 ffff73bde9c0 10 scanning projected log
2023-11-23T18:30:37.812+0800 ffff73bde9c0 10 
每次有更新操作, bi的 version 会比 log_tail 大,但是比projected_last_update 要小,需要进行更新
这里的更新只是从pg log中,从后往前查找到 上次bi version之后的操作,然后再把所有的操作应用到 bi的object中

step3: after update_range 
     local backfill_info: BackfillInfo(1:18b725a9:::test10814.txt:head-1:19fba18e:::test12743.txt:head 88 version: 39"20001 objects {1:18b7eb0c:::test15864.txt:head=20"15864,,,,,,,,,,1:19f875c7:::test17364.txt:head=20"17364})

step4: befor while (ops < max)
     local backfill_info: BackfillInfo(1:18b7eb0c:::test15864.txt:head-1:19fba18e:::test12743.txt:head 88 version: 39"20001 objects {1:18b7eb0c:::test15864.txt:head=20"15864,,,,,,,,,,1:19f875c7:::test17364.txt:head=20"17364})
     earliest_peer_backfill(): MAX
     peer 2
         peer last_backfill: 1:18b5b223:::test10094.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)

倒数第二次

step1: 
     new_backfill: 0
     get_backfill_targets(): 2
     last_backfill_started: 1:fffc981f:::test18600.txt:head
     local pg_info_t: 1.0( v 64"20014 (20"10000,64"20014] local-lis/les=69/70 n=20014 ec=3/3 lis/c=69/19 les/c/f=70/20/0 sis=69)
     earliest_backfill(): 1:fffbd9ba:::test8865.txt:head
step2: before update_range 
     backfill_info: BackfillInfo(1:fffc981f:::test18600.txt:head-MAX 0 version: 64"20014 objects)
     pending_backfill_updates: {1:fffc981f:::test18600.txt:head=0"0}
     backfills_in_flight: 
     peer: 2
         peer last_backfill: 1:fffbd9ba:::test8865.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 64"20014 (20"10000,64"20014] lb 1:fffbd9ba:::test8865.txt:head local-lis/les=69/70 n=20013 ec=3/3 lis/c=69/19 les/c/f=70/20/0 sis=69)
2023-11-24T10:34:05.935+0800 ffff7c2e99c0 10 update_range: bi is current 64"20014 project_last_update 64"20014
step3: after update_range 
     local backfill_info: BackfillInfo(1:fffc981f:::test18600.txt:head-MAX 0 version: 64"20014 objects)

step4: befor while (ops < max)
     local backfill_info: BackfillInfo(MAX-MAX 0 version: 64"20014 objects)
     earliest_peer_backfill(): MAX
     peer 2
         peer last_backfill: 1:fffbd9ba:::test8865.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
2023-11-24T10:34:05.935+0800 ffff7c2e99c0 10  reached end for both local and all peers

step 10:  out while:  before update peer last_backfill:
     backfills_in_flight: 
     backfill_info: BackfillInfo(MAX-MAX 0 version: 64"20014 objects)
     pending_backfill_updates: {1:fffc981f:::test18600.txt:head=0"0}
     last_backfill_started: 1:fffc981f:::test18600.txt:head
     add_to_stat: 
     next_backfill_to_complete: MAX
     new_last_backfill: 1:fffbd9ba:::test8865.txt:head
     backfill_pos MAX
     peer: 2
         peer last_backfill: 1:fffbd9ba:::test8865.txt:head
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 64"20014 (20"10000,64"20014] lb 1:fffbd9ba:::test8865.txt:head local-lis/les=69/70 n=20013 ec=3/3 lis/c=69/19 les/c/f=70/20/0 sis=69)

step : the end compute new_last_backfill
     new_last_backfill MAX
     peer: 2
         peer last_backfill: 1:fffbd9ba:::test8865.txt:head
         new_last_backfill > pinfo.last_backfill : 1
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)

2023-11-24T10:34:05.935+0800 ffff7c2e99c0 10 update peer lastbackfill MAX
发送pg_backfill(finish)

最终backfill结束：

5: (PG::queue_recovery()+0x304) [0xaaaaec7934d4]
6: (PG::finish_recovery_op(hobject_t const&, bool)+0x13c) [0xaaaaec794170]
7: (PrimaryLogPG::do_backfill(boost::intrusive_ptr<OpRequest>)+0x3e4) [0xaaaaec842de8]

step1: 
     new_backfill: 0
     get_backfill_targets(): 2
     last_backfill_started: MAX
     local pg_info_t: 1.0( v 64"20014 (20"10000,64"20014] local-lis/les=69/70 n=20014 ec=3/3 lis/c=69/19 les/c/f=70/20/0 sis=69)
     earliest_backfill(): MAX

step2: before update_range 
     backfill_info: BackfillInfo(MAX-MAX 0 version: 64"20014 objects)
     pending_backfill_updates: {}
     backfills_in_flight: 
     peer: 2
         peer last_backfill: MAX
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 64"20014 (20"10000,64"20014] local-lis/les=69/70 n=20014 ec=3/3 lis/c=69/19 les/c/f=70/20/0 sis=69)

step3: after update_range 
     local backfill_info: BackfillInfo(MAX-MAX 0 version: 64"20014 objects)

step4: befor while (ops < max)
     local backfill_info: BackfillInfo(MAX-MAX 0 version: 64"20014 objects)
     earliest_peer_backfill(): MAX
     peer 2
         peer last_backfill: MAX
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
2023-11-24T10:34:06.025+0800 ffff782e19c0 10  reached end for both local and all peers

step 10:  out while:  before update peer last_backfill:
     backfills_in_flight: 
     backfill_info: BackfillInfo(MAX-MAX 0 version: 64"20014 objects)
     pending_backfill_updates: {}
     last_backfill_started: MAX
     add_to_stat: 
     next_backfill_to_complete: MAX
     new_last_backfill: MAX
     backfill_pos MAX
     peer: 2
         peer last_backfill: MAX
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)
         peer pg_info_t: 1.0( v 64"20014 (20"10000,64"20014] local-lis/les=69/70 n=20014 ec=3/3 lis/c=69/19 les/c/f=70/20/0 sis=69)

step : the end compute new_last_backfill
     new_last_backfill MAX
     peer: 2
         peer last_backfill: MAX
         new_last_backfill > pinfo.last_backfill : 0
         pbi: BackfillInfo(MAX-MAX 0 version: 0"0 objects)

人面猴
序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
沈念sama阅读 206,968评论 6赞 482
死咒
序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
沈念sama阅读 88,601评论 2赞 382
救了他两次的神仙让他今天三更去死
文/潘晓璐我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
开封第一讲书人阅读 153,220评论 0赞 344
道士缉凶录：失踪的卖姜人
文/不坏的土叔我叫张陵，是天一观的道长。经常有香客问我，道长，这世上最难降的妖魔是什么？我笑而不...
开封第一讲书人阅读 55,416评论 1赞 279
港岛之恋（遗憾婚礼）
正文为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
茶点故事阅读 64,425评论 5赞 374
恶毒庶女顶嫁案：这布局不是一般人想出来的
文/花漫我一把揭开白布。她就那样静静地躺着，像睡着了一般。火红的嫁衣衬着肌肤如雪。梳的纹丝不乱的头发上，一...
开封第一讲书人阅读 49,144评论 1赞 285
城市分裂传说
那天，我揣着相机与录音，去河边找鬼。笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。我是一名探鬼主播，决...
沈念sama阅读 38,432评论 3赞 401
双鸳鸯连环套：你想象不到人心有多黑
文/苍兰香墨我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
开封第一讲书人阅读 37,088评论 0赞 261
万荣杀人案实录
序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
沈念sama阅读 43,586评论 1赞 300
护林员之死
正文独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋以下内容为张勋视角年9月15日...
茶点故事阅读 36,028评论 2赞 325
白月光启示录
正文我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
茶点故事阅读 38,137评论 1赞 334
活死人
序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
沈念sama阅读 33,783评论 4赞 324
日本核电站爆炸内幕
正文年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
茶点故事阅读 39,343评论 3赞 307
男人毒药：我在死后第九天来索命
文/蒙蒙一、第九天我趴在偏房一处隐蔽的房顶上张望。院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
开封第一讲书人阅读 30,333评论 0赞 19
一桩弑父案，背后竟有这般阴谋
文/苍兰香墨我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。一阵脚步声响...
开封第一讲书人阅读 31,559评论 1赞 262
情欲美人皮
我被黑心中介骗来泰国打工，没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。一个月前我还...
沈念sama阅读 45,595评论 2赞 355
代替公主和亲
正文我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
茶点故事阅读 42,901评论 2赞 345