From 4c45115bdf2c5145ca3779c38a06e90024f013f0 Mon Sep 17 00:00:00 2001 From: Fan Yong Date: Sat, 17 Oct 2015 02:55:33 +0800 Subject: [PATCH] LU-6895 lfsck: drop bad OI files after MDT file-level restore For a new formatted Lustre system, it does not guarantee that all the on-disk inodes/blocks are initialized. Instead, they are marked as free in related inode/block bitmaps. When the inode is allocated to some object, it will be initialized at that time. Such process accelerates the format. But it may cause trouble for MDT file-level backup/restore. For example: the sysadmin backup the MDT via server side file level tar, then reformat the MDT device, and then restore the MDT from the backup. Assume some object_A, its FID is the FID_A, and it was mapped to the inode_B before the backup, such OI mapping was recorded in the OI file. After the restore, another inode_C is assigned to the object_A, but before OI scrub rebuilding related OI mapping, the stale mapping "FID_A => inode_B" is still in the OI file, and can be found by OI lookup. Generally, even if someone found the stale OI mapping, it is not trouble, because the OSD will verify whether FID-in-LMA for the indoe_B matches the FID_A or not. But if the inode_B is NOT allocated after the restore, and because we did not initialize inode_B during reformat, then the FID-in-LMA for the indoe_B is still kept as the same before the backup, means it matches the FID_A, then the OSD will think that the inode_B is still assigned to the object_A after the restore. That is wrong. In futher, although the inode_B is not allocated, but some of the blocks that were assigned to it may have been allocated to others. Then accessing the blocks via the inode_B may access some invalid data, and may trigger some assertion, such as this issue. So the key issues are two: 1) Some FID based operation may access stale OI mapping after MDT file-level backup/restore. 2) The OSD-ldiskfs may get some non-allocated inode with the give ino#/generation. So as long as we can resovle one of them, then the trouble in the ticket can be resolved. The solutions for 1): 1.1) Avoid FID based operation before OI scrub done. That is not easy, because fid2path cannot be covered by some name based operation. 1.2) Remove the OI files after MDT file-level backup/restore. It is more direct solution. Another benefit is that even if OI scrub rebuilt the OI files, it only guarantees that all the FIDs' OI mappings have been refrshed. But it does not clean the stale FIDs' OI mappings. Because the OI scrub only does inode-table based scanning, not OI files scanning. Removing the OI files can resolve related trouble completely. The solutions for 2): 2.1) New ldiskfs patch to make ldiskfs_iget() to return "-ENOENT" for the case of loading non-allocated inode by checking the inode bitmap. 2.2) Check the inode's valid inside OSD via related inode bitmap. Generally, less ldiskfs patches is better. It will safe a lot of effort when kernel upgrade. So 2.1) is not the best solution. As for 2.2), it is not good to access the inode bitmap directly in OSD unless we have to. Relatively, the solution 1.2) is more efficient and benefit. That is the current patch does. On the other hand, this patch also makes some effort for the 1.1): When start LFSCK, to hold the potential orphans in subsequent system check, the start process will verify the .lustre/lost+found/ and its sub-directories firstly. For each MDT, there is one sub-dir named as .lustre/lost+found/MDTxxxx, which FID is recorded in the LFSCK trace bookmark file. Originally, the start process uses such FID to locate the MDTxxxx object. It is not problem for most of cases, but if the MDT is just restored from file-level backup, and the low layer (OSD) OI mapping may be invalid, then locating the MDTxxxx object will hit -EREMCHG failure, and then may cause more troubles: 1) It will try to start OI scrub because of bad OI mapping detected, but because we are already in starting the LFSCK, then the logic of triggering OI scrub because of LFSCK is confused and difficult to be handled. 2) It will misguide the up layer LFSCK to think that the MDTxxxx obj does not exist or crashed, then will take some unexpected actions to repair such "fake" inconsistency. The patch make some adjustment of the LFSCK start: a) Trigger low layer OI scrub before verifying .lustre/lost+found. b) To verify the .lustre/lost+found/ and its sub-dirs, scanning the .lustre/lost+found/ directory via namespace dt_lookup, and check the returned FID with the FID in the LFSCK trace bookmark file. With such way, even though the MDT is restored from file-level backup, the low layer still can locate the right object (inode). Test-Parameters: alwaysuploadlogs envdefinitions=SLOW=yes,ENABLE_QUOTA=yes mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs clientdistro=el7 ossdistro=el7 mdsdistro=el7 mdtcount=1 testlist=sanity-lfsck,sanity-lfsck,sanity-lfsck Signed-off-by: Fan Yong Change-Id: Icfdab19e69b75400bb39279924c0c3e0b400b4e0 Reviewed-on: http://review.whamcloud.com/17403 Tested-by: Jenkins Tested-by: Maloo Reviewed-by: Alex Zhuravlev Reviewed-by: Andreas Dilger --- lustre/lfsck/lfsck_engine.c | 21 +++--- lustre/lfsck/lfsck_lib.c | 146 ++++++++++++++++++++------------------- lustre/osd-ldiskfs/osd_handler.c | 2 + lustre/osd-ldiskfs/osd_oi.c | 55 ++++++++++++++- lustre/osd-ldiskfs/osd_oi.h | 3 +- lustre/osd-ldiskfs/osd_scrub.c | 14 ++-- lustre/tests/sanity-scrub.sh | 33 +++++---- 7 files changed, 169 insertions(+), 105 deletions(-) diff --git a/lustre/lfsck/lfsck_engine.c b/lustre/lfsck/lfsck_engine.c index 6daaf80..d1053c1 100644 --- a/lustre/lfsck/lfsck_engine.c +++ b/lustre/lfsck/lfsck_engine.c @@ -1006,6 +1006,18 @@ int lfsck_master_engine(void *args) int rc; ENTRY; + /* There will be some objects verification during the LFSCK start, + * such as the subsequent lfsck_verify_lpf(). Trigger low layer OI + * OI scrub before that to handle the potential inconsistence. */ + oit_di = oit_iops->init(env, oit_obj, lfsck->li_args_oit); + if (IS_ERR(oit_di)) { + rc = PTR_ERR(oit_di); + CDEBUG(D_LFSCK, "%s: master engine fail to init iteration: " + "rc = %d\n", lfsck_lfsck2name(lfsck), rc); + + GOTO(fini_args, rc); + } + if (lfsck->li_master && (!list_empty(&lfsck->li_list_scan) || !list_empty(&lfsck->li_list_double_scan))) { @@ -1021,15 +1033,6 @@ int lfsck_master_engine(void *args) lfsck_lfsck2name(lfsck), rc); } - oit_di = oit_iops->init(env, oit_obj, lfsck->li_args_oit); - if (IS_ERR(oit_di)) { - rc = PTR_ERR(oit_di); - CDEBUG(D_LFSCK, "%s: master engine fail to init iteration: " - "rc = %d\n", lfsck_lfsck2name(lfsck), rc); - - GOTO(fini_args, rc); - } - spin_lock(&lfsck->li_lock); lfsck->li_di_oit = oit_di; spin_unlock(&lfsck->li_lock); diff --git a/lustre/lfsck/lfsck_lib.c b/lustre/lfsck/lfsck_lib.c index 9f340e8..7158337 100644 --- a/lustre/lfsck/lfsck_lib.c +++ b/lustre/lfsck/lfsck_lib.c @@ -1435,61 +1435,13 @@ int lfsck_verify_lpf(const struct lu_env *env, struct lfsck_instance *lfsck) lfsck_lfsck2name(lfsck), rc); } - if (!fid_is_zero(&bk->lb_lpf_fid)) { - if (unlikely(!fid_is_norm(&bk->lb_lpf_fid))) { - struct lu_fid tfid = bk->lb_lpf_fid; - - /* Invalid FID record in the bookmark file, reset it. */ - fid_zero(&bk->lb_lpf_fid); - rc = lfsck_bookmark_store(env, lfsck); - - CDEBUG(D_LFSCK, "%s: reset invalid LPF fid "DFID - " in the bookmark file: rc = %d\n", - lfsck_lfsck2name(lfsck), PFID(&tfid), rc); - - if (rc != 0) - GOTO(put, rc); - } else { - child1 = lfsck_object_find_bottom(env, lfsck, - &bk->lb_lpf_fid); - if (IS_ERR(child1)) { - child1 = NULL; - goto find_child2; - } - - if (unlikely(!dt_object_exists(child1) || - dt_object_remote(child1)) || - !S_ISDIR(lfsck_object_type(child1))) { - /* Invalid FID record in the bookmark file, - * reset it. */ - fid_zero(&bk->lb_lpf_fid); - rc = lfsck_bookmark_store(env, lfsck); - - CDEBUG(D_LFSCK, "%s: reset invalid LPF fid "DFID - " in the bookmark file: rc = %d\n", - lfsck_lfsck2name(lfsck), - PFID(lfsck_dto2fid(child1)), rc); - - if (rc != 0) - GOTO(put, rc); - - lfsck_object_put(env, child1); - child1 = NULL; - } else if (unlikely(!dt_try_as_dir(env, child1))) { - GOTO(put, rc = -ENOTDIR); - } - } - } - -find_child2: + /* child2 */ snprintf(name, 8, "MDT%04x", node); rc = dt_lookup(env, parent, (struct dt_rec *)cfid, (const struct dt_key *)name); if (rc == -ENOENT) { - if (!fid_is_zero(&bk->lb_lpf_fid)) - goto check_child1; - - GOTO(put, rc = 0); + rc = 0; + goto find_child1; } if (rc != 0) @@ -1501,7 +1453,7 @@ find_child2: if (rc != 0) GOTO(put, rc); - goto check_child1; + goto find_child1; } child2 = lfsck_object_find_bottom(env, lfsck, cfid); @@ -1515,23 +1467,20 @@ find_child2: if (rc != 0) GOTO(put, rc); - goto check_child1; + goto find_child1; } - if (unlikely(!dt_try_as_dir(env, child2))) - GOTO(put, rc = -ENOTDIR); + if (unlikely(!dt_try_as_dir(env, child2))) { + lfsck_object_put(env, child2); + child2 = NULL; + rc = -ENOTDIR; + } - if (child1 == NULL) { - rc = lfsck_verify_lpf_pairs(env, lfsck, child2, name, - pfid, LVLT_BY_NAMEENTRY); - } else if (!lu_fid_eq(cfid, &bk->lb_lpf_fid)) { - rc = lfsck_verify_lpf_pairs(env, lfsck, child1, name, - pfid, LVLT_BY_BOOKMARK); - if (!lu_fid_eq(pfid, &LU_LPF_FID)) - rc = lfsck_verify_lpf_pairs(env, lfsck, child2, - name, pfid, - LVLT_BY_NAMEENTRY); - } else { +find_child1: + if (fid_is_zero(&bk->lb_lpf_fid)) + goto check_child2; + + if (likely(lu_fid_eq(cfid, &bk->lb_lpf_fid))) { if (lfsck->li_lpf_obj == NULL) { lu_object_get(&child2->do_lu); lfsck->li_lpf_obj = child2; @@ -1539,14 +1488,69 @@ find_child2: cname = lfsck_name_get_const(env, name, strlen(name)); rc = lfsck_verify_linkea(env, child2, cname, &LU_LPF_FID); + + GOTO(put, rc); } - GOTO(put, rc); + if (unlikely(!fid_is_norm(&bk->lb_lpf_fid))) { + struct lu_fid tfid = bk->lb_lpf_fid; -check_child1: - if (child1 != NULL) - rc = lfsck_verify_lpf_pairs(env, lfsck, child1, name, - pfid, LVLT_BY_BOOKMARK); + /* Invalid FID record in the bookmark file, reset it. */ + fid_zero(&bk->lb_lpf_fid); + rc = lfsck_bookmark_store(env, lfsck); + + CDEBUG(D_LFSCK, "%s: reset invalid LPF fid "DFID + " in the bookmark file: rc = %d\n", + lfsck_lfsck2name(lfsck), PFID(&tfid), rc); + + if (rc != 0) + GOTO(put, rc); + + goto check_child2; + } + + child1 = lfsck_object_find_bottom(env, lfsck, &bk->lb_lpf_fid); + if (IS_ERR(child1)) { + child1 = NULL; + goto check_child2; + } + + if (unlikely(!dt_object_exists(child1) || + dt_object_remote(child1)) || + !S_ISDIR(lfsck_object_type(child1))) { + /* Invalid FID record in the bookmark file, reset it. */ + fid_zero(&bk->lb_lpf_fid); + rc = lfsck_bookmark_store(env, lfsck); + + CDEBUG(D_LFSCK, "%s: reset invalid LPF fid "DFID + " in the bookmark file: rc = %d\n", + lfsck_lfsck2name(lfsck), + PFID(lfsck_dto2fid(child1)), rc); + + if (rc != 0) + GOTO(put, rc); + + lfsck_object_put(env, child1); + child1 = NULL; + goto check_child2; + } + + if (unlikely(!dt_try_as_dir(env, child1))) { + lfsck_object_put(env, child1); + child1 = NULL; + rc = -ENOTDIR; + goto check_child2; + } + + rc = lfsck_verify_lpf_pairs(env, lfsck, child1, name, pfid, + LVLT_BY_BOOKMARK); + if (lu_fid_eq(pfid, &LU_LPF_FID)) + GOTO(put, rc); + +check_child2: + if (child2 != NULL) + rc = lfsck_verify_lpf_pairs(env, lfsck, child2, name, + pfid, LVLT_BY_NAMEENTRY); GOTO(put, rc); diff --git a/lustre/osd-ldiskfs/osd_handler.c b/lustre/osd-ldiskfs/osd_handler.c index bf46774..619662a 100644 --- a/lustre/osd-ldiskfs/osd_handler.c +++ b/lustre/osd-ldiskfs/osd_handler.c @@ -815,6 +815,8 @@ trigger: result = -EINPROGRESS; else result = -EREMCHG; + } else { + result = -EREMCHG; } if (fid_is_on_ost(info, dev, fid, OI_CHECK_FLD)) diff --git a/lustre/osd-ldiskfs/osd_oi.c b/lustre/osd-ldiskfs/osd_oi.c index 6f15760..6b6942f 100644 --- a/lustre/osd-ldiskfs/osd_oi.c +++ b/lustre/osd-ldiskfs/osd_oi.c @@ -338,7 +338,54 @@ osd_oi_table_open(struct osd_thread_info *info, struct osd_device *osd, RETURN(count); } -int osd_oi_init(struct osd_thread_info *info, struct osd_device *osd) +static int osd_remove_oi_one(struct dentry *parent, const char *name, + int namelen) +{ + struct dentry *child; + int rc; + + child = ll_lookup_one_len(name, parent, namelen); + if (IS_ERR(child)) { + rc = PTR_ERR(child); + } else { + rc = ll_vfs_unlink(parent->d_inode, child); + dput(child); + } + + return rc == -ENOENT ? 0 : rc; +} + +static int osd_remove_ois(struct osd_thread_info *info, struct osd_device *osd) +{ + char name[16]; + int namelen; + int rc; + int i; + + for (i = 0; i < osd->od_scrub.os_file.sf_oi_count; i++) { + namelen = snprintf(name, sizeof(name), "%s.%d", + OSD_OI_NAME_BASE, i); + rc = osd_remove_oi_one(osd_sb(osd)->s_root, name, namelen); + if (rc != 0) { + CERROR("%.16s: fail to remove the stale OI file %s: " + "rc = %d\n", + LDISKFS_SB(osd_sb(osd))->s_es->s_volume_name, + name, rc); + return rc; + } + } + + namelen = snprintf(name, sizeof(name), "%s", OSD_OI_NAME_BASE); + rc = osd_remove_oi_one(osd_sb(osd)->s_root, name, namelen); + if (rc != 0) + CERROR("%.16s: fail to remove the stale OI file %s: rc = %d\n", + LDISKFS_SB(osd_sb(osd))->s_es->s_volume_name, name, rc); + + return rc; +} + +int osd_oi_init(struct osd_thread_info *info, struct osd_device *osd, + bool restored) { struct osd_scrub *scrub = &osd->od_scrub; struct scrub_file *sf = &scrub->os_file; @@ -346,6 +393,12 @@ int osd_oi_init(struct osd_thread_info *info, struct osd_device *osd) int rc; ENTRY; + if (restored) { + rc = osd_remove_ois(info, osd); + if (rc != 0) + return rc; + } + OBD_ALLOC(oi, sizeof(*oi) * OSD_OI_FID_NR_MAX); if (oi == NULL) RETURN(-ENOMEM); diff --git a/lustre/osd-ldiskfs/osd_oi.h b/lustre/osd-ldiskfs/osd_oi.h index fbdcffa..6d5fae3 100644 --- a/lustre/osd-ldiskfs/osd_oi.h +++ b/lustre/osd-ldiskfs/osd_oi.h @@ -139,7 +139,8 @@ enum oi_check_flags { }; int osd_oi_mod_init(void); -int osd_oi_init(struct osd_thread_info *info, struct osd_device *osd); +int osd_oi_init(struct osd_thread_info *info, struct osd_device *osd, + bool restored); void osd_oi_fini(struct osd_thread_info *info, struct osd_device *osd); int osd_oi_lookup(struct osd_thread_info *info, struct osd_device *osd, const struct lu_fid *fid, struct osd_inode_id *id, diff --git a/lustre/osd-ldiskfs/osd_scrub.c b/lustre/osd-ldiskfs/osd_scrub.c index ff639e2..80f2794 100644 --- a/lustre/osd-ldiskfs/osd_scrub.c +++ b/lustre/osd-ldiskfs/osd_scrub.c @@ -2453,7 +2453,8 @@ int osd_scrub_setup(const struct lu_env *env, struct osd_device *dev) struct file *filp; struct inode *inode; struct lu_fid *fid = &info->oti_fid; - int dirty = 0; + bool dirty = false; + bool restored = false; int rc = 0; ENTRY; @@ -2507,7 +2508,7 @@ int osd_scrub_setup(const struct lu_env *env, struct osd_device *dev) * have their FID mapping in OI files already. */ if (dev->od_maybe_new) sf->sf_internal_flags = SIF_NO_HANDLE_OLD_FID; - dirty = 1; + dirty = true; } else if (rc != 0) { GOTO(cleanup_inode, rc); } else { @@ -2530,14 +2531,15 @@ int osd_scrub_setup(const struct lu_env *env, struct osd_device *dev) old_uuid->uuid, new_uuid->uuid); } osd_scrub_file_reset(scrub, es->s_uuid,SF_INCONSISTENT); - dirty = 1; + dirty = true; + restored = true; if (old_uuid != NULL) OBD_FREE_PTR(old_uuid); if (new_uuid != NULL) OBD_FREE_PTR(new_uuid); } else if (sf->sf_status == SS_SCANNING) { sf->sf_status = SS_CRASHED; - dirty = 1; + dirty = true; } } @@ -2546,14 +2548,14 @@ int osd_scrub_setup(const struct lu_env *env, struct osd_device *dev) else scrub->os_pos_current = LDISKFS_FIRST_INO(sb) + 1; - if (dirty != 0) { + if (dirty) { rc = osd_scrub_file_store(scrub); if (rc != 0) GOTO(cleanup_inode, rc); } /* Initialize OI files. */ - rc = osd_oi_init(info, dev); + rc = osd_oi_init(info, dev, restored); if (rc < 0) GOTO(cleanup_inode, rc); diff --git a/lustre/tests/sanity-scrub.sh b/lustre/tests/sanity-scrub.sh index 8e91ad4..5b3e2a7 100644 --- a/lustre/tests/sanity-scrub.sh +++ b/lustre/tests/sanity-scrub.sh @@ -7,8 +7,7 @@ set -e ONLY=${ONLY:-"$*"} -#Bug number for excepting test 6705 -ALWAYS_EXCEPT="$SANITY_SCRUB_EXCEPT 1c 5 10" +ALWAYS_EXCEPT="$SANITY_SCRUB_EXCEPT" [ "$SLOW" = "no" ] && EXCEPT_SLOW="" # UPDATE THE COMMENT ABOVE WITH BUG NUMBERS WHEN CHANGING ALWAYS_EXCEPT! @@ -409,7 +408,7 @@ test_3() { echo "starting MDTs with OI scrub disabled" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" scrub_check_status 3 init - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent } #run_test 3 "Do not trigger OI scrub when MDT mounts if 'noscrub' specified" @@ -418,7 +417,7 @@ test_4a() { scrub_backup_restore 1 echo "starting MDTs with OI scrub disabled" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent mount_client $MOUNT || error "(5) Fail to start client!" scrub_enable_auto full_scrub_ratio 0 @@ -452,7 +451,7 @@ test_4b() { scrub_backup_restore 1 echo "starting MDTs with OI scrub disabled" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent mount_client $MOUNT || error "(5) Fail to start client!" scrub_enable_auto full_scrub_ratio 10 @@ -518,7 +517,7 @@ test_4c() { scrub_backup_restore 1 echo "starting MDTs with OI scrub disabled" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent mount_client $MOUNT || error "(5) Fail to start client!" scrub_enable_auto full_scrub_ratio 2 @@ -588,7 +587,7 @@ test_5() { echo "starting MDTs with OI scrub disabled (1)" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" scrub_check_status 3 init - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent mount_client $MOUNT || error "(5) Fail to start client!" scrub_enable_auto @@ -655,7 +654,7 @@ test_6() { scrub_backup_restore 1 echo "starting MDTs with OI scrub disabled" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent mount_client $MOUNT || error "(5) Fail to start client!" scrub_enable_auto @@ -733,7 +732,7 @@ test_7() { scrub_backup_restore 1 echo "starting MDTs with OI scrub disabled" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent mount_client $MOUNT || error "(5) Fail to start client!" scrub_enable_auto @@ -751,7 +750,7 @@ test_7() { done scrub_check_status 8 scanning - scrub_check_flags 9 inconsistent,auto + scrub_check_flags 9 recreated,inconsistent,auto do_nodes $(comma_list $(mdts_nodes)) \ $LCTL set_param fail_loc=0 fail_val=0 @@ -766,7 +765,7 @@ test_8() { scrub_backup_restore 1 echo "starting MDTs with OI scrub disabled" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent #define OBD_FAIL_OSD_SCRUB_DELAY 0x190 do_nodes $(comma_list $(mdts_nodes)) \ @@ -798,7 +797,7 @@ test_9() { echo "starting MDTs with OI scrub disabled" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent local BASE_SPEED1=100 local RUN_TIME1=10 @@ -872,7 +871,7 @@ test_10a() { scrub_backup_restore 1 echo "starting mds$n with OI scrub disabled (1)" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent mount_client $MOUNT || error "(5) Fail to start client!" scrub_enable_auto @@ -907,7 +906,7 @@ test_10b() { scrub_backup_restore 1 echo "starting MDTs with OI scrub disabled" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent #define OBD_FAIL_OSD_SCRUB_DELAY 0x190 do_nodes $(comma_list $(mdts_nodes)) \ @@ -1096,7 +1095,7 @@ test_15() { echo "starting MDTs with OI scrub disabled" scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB" scrub_check_status 3 init - scrub_check_flags 4 inconsistent + scrub_check_flags 4 recreated,inconsistent # run under dryrun mode if [ $server_version -lt $(version_code 2.5.58) ]; then @@ -1105,7 +1104,7 @@ test_15() { scrub_start 5 --dryrun fi scrub_check_status 6 completed - scrub_check_flags 7 inconsistent + scrub_check_flags 7 recreated,inconsistent scrub_check_params 8 dryrun scrub_check_repaired 9 20 @@ -1116,7 +1115,7 @@ test_15() { scrub_start 10 --dryrun fi scrub_check_status 11 completed - scrub_check_flags 12 inconsistent + scrub_check_flags 12 recreated,inconsistent scrub_check_params 13 dryrun scrub_check_repaired 14 20 -- 1.8.3.1