LU-6895 lfsck: drop bad OI files after MDT file-level restore

For a new formatted Lustre system, it does not guarantee that all
the on-disk inodes/blocks are initialized. Instead, they are marked
as free in related inode/block bitmaps. When the inode is allocated
to some object, it will be initialized at that time. Such process
accelerates the format. But it may cause trouble for MDT file-level
backup/restore. For example: the sysadmin backup the MDT via server
side file level tar, then reformat the MDT device, and then restore
the MDT from the backup. Assume some object_A, its FID is the FID_A,
and it was mapped to the inode_B before the backup, such OI mapping
was recorded in the OI file. After the restore, another inode_C is
assigned to the object_A, but before OI scrub rebuilding related
OI mapping, the stale mapping "FID_A => inode_B" is still in the
OI file, and can be found by OI lookup. Generally, even if someone
found the stale OI mapping, it is not trouble, because the OSD will
verify whether FID-in-LMA for the indoe_B matches the FID_A or not.
But if the inode_B is NOT allocated after the restore, and because
we did not initialize inode_B during reformat, then the FID-in-LMA
for the indoe_B is still kept as the same before the backup, means
it matches the FID_A, then the OSD will think that the inode_B is
still assigned to the object_A after the restore. That is wrong.
In futher, although the inode_B is not allocated, but some of the
blocks that were assigned to it may have been allocated to others.
Then accessing the blocks via the inode_B may access some invalid
data, and may trigger some assertion, such as this issue.

So the key issues are two:

1) Some FID based operation may access stale OI mapping after MDT
   file-level backup/restore.

2) The OSD-ldiskfs may get some non-allocated inode with the give
   ino#/generation.

So as long as we can resovle one of them, then the trouble in the
ticket can be resolved.

The solutions for 1):

1.1) Avoid FID based operation before OI scrub done. That is not
     easy, because fid2path cannot be covered by some name based
     operation.

1.2) Remove the OI files after MDT file-level backup/restore. It
     is more direct solution. Another benefit is that even if OI
     scrub rebuilt the OI files, it only guarantees that all the
     FIDs' OI mappings have been refrshed. But it does not clean
     the stale FIDs' OI mappings. Because the OI scrub only does
     inode-table based scanning, not OI files scanning. Removing
     the OI files can resolve related trouble completely.

The solutions for 2):

2.1) New ldiskfs patch to make ldiskfs_iget() to return "-ENOENT"
     for the case of loading non-allocated inode by checking the
     inode bitmap.

2.2) Check the inode's valid inside OSD via related inode bitmap.

Generally, less ldiskfs patches is better. It will safe a lot of
effort when kernel upgrade. So 2.1) is not the best solution. As
for 2.2), it is not good to access the inode bitmap directly in
OSD unless we have to.

Relatively, the solution 1.2) is more efficient and benefit. That
is the current patch does.

On the other hand, this patch also makes some effort for the 1.1):

When start LFSCK, to hold the potential orphans in subsequent system
check, the start process will verify the .lustre/lost+found/ and its
sub-directories firstly. For each MDT, there is one sub-dir named as
.lustre/lost+found/MDTxxxx, which FID is recorded in the LFSCK trace
bookmark file. Originally, the start process uses such FID to locate
the MDTxxxx object. It is not problem for most of cases, but if the
MDT is just restored from file-level backup, and the low layer (OSD)
OI mapping may be invalid, then locating the MDTxxxx object will hit
-EREMCHG failure, and then may cause more troubles:

1) It will try to start OI scrub because of bad OI mapping detected,
   but because we are already in starting the LFSCK, then the logic
   of triggering OI scrub because of LFSCK is confused and difficult
   to be handled.

2) It will misguide the up layer LFSCK to think that the MDTxxxx obj
   does not exist or crashed, then will take some unexpected actions
   to repair such "fake" inconsistency.

The patch make some adjustment of the LFSCK start:

a) Trigger low layer OI scrub before verifying .lustre/lost+found.

b) To verify the .lustre/lost+found/ and its sub-dirs, scanning the
   .lustre/lost+found/ directory via namespace dt_lookup, and check
   the returned FID with the FID in the LFSCK trace bookmark file.
   With such way, even though the MDT is restored from file-level
   backup, the low layer still can locate the right object (inode).

Test-Parameters: alwaysuploadlogs envdefinitions=SLOW=yes,ENABLE_QUOTA=yes mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs clientdistro=el7 ossdistro=el7 mdsdistro=el7 mdtcount=1 testlist=sanity-lfsck,sanity-lfsck,sanity-lfsck
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Icfdab19e69b75400bb39279924c0c3e0b400b4e0
Reviewed-on: http://review.whamcloud.com/17403
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

diff --git a/lustre/lfsck/lfsck_engine.c b/lustre/lfsck/lfsck_engine.c
index 6daaf80..d1053c1 100644 (file)
--- a/lustre/lfsck/lfsck_engine.c
+++ b/lustre/lfsck/lfsck_engine.c
@@ -1006,6 +1006,18 @@ int lfsck_master_engine(void *args)
        int                       rc;
        ENTRY;
 
+       /* There will be some objects verification during the LFSCK start,
+        * such as the subsequent lfsck_verify_lpf(). Trigger low layer OI
+        * OI scrub before that to handle the potential inconsistence. */
+       oit_di = oit_iops->init(env, oit_obj, lfsck->li_args_oit);
+       if (IS_ERR(oit_di)) {
+               rc = PTR_ERR(oit_di);
+               CDEBUG(D_LFSCK, "%s: master engine fail to init iteration: "
+                      "rc = %d\n", lfsck_lfsck2name(lfsck), rc);
+
+               GOTO(fini_args, rc);
+       }
+
        if (lfsck->li_master &&
            (!list_empty(&lfsck->li_list_scan) ||
             !list_empty(&lfsck->li_list_double_scan))) {
@@ -1021,15 +1033,6 @@ int lfsck_master_engine(void *args)
                               lfsck_lfsck2name(lfsck), rc);
        }
 
-       oit_di = oit_iops->init(env, oit_obj, lfsck->li_args_oit);
-       if (IS_ERR(oit_di)) {
-               rc = PTR_ERR(oit_di);
-               CDEBUG(D_LFSCK, "%s: master engine fail to init iteration: "
-                      "rc = %d\n", lfsck_lfsck2name(lfsck), rc);
-
-               GOTO(fini_args, rc);
-       }
-
        spin_lock(&lfsck->li_lock);
        lfsck->li_di_oit = oit_di;
        spin_unlock(&lfsck->li_lock);

diff --git a/lustre/lfsck/lfsck_lib.c b/lustre/lfsck/lfsck_lib.c
index 9f340e8..7158337 100644 (file)
--- a/lustre/lfsck/lfsck_lib.c
+++ b/lustre/lfsck/lfsck_lib.c
@@ -1435,61 +1435,13 @@ int lfsck_verify_lpf(const struct lu_env *env, struct lfsck_instance *lfsck)
                               lfsck_lfsck2name(lfsck), rc);
        }
 
-       if (!fid_is_zero(&bk->lb_lpf_fid)) {
-               if (unlikely(!fid_is_norm(&bk->lb_lpf_fid))) {
-                       struct lu_fid tfid = bk->lb_lpf_fid;
-
-                       /* Invalid FID record in the bookmark file, reset it. */
-                       fid_zero(&bk->lb_lpf_fid);
-                       rc = lfsck_bookmark_store(env, lfsck);
-
-                       CDEBUG(D_LFSCK, "%s: reset invalid LPF fid "DFID
-                              " in the bookmark file: rc = %d\n",
-                              lfsck_lfsck2name(lfsck), PFID(&tfid), rc);
-
-                       if (rc != 0)
-                               GOTO(put, rc);
-               } else {
-                       child1 = lfsck_object_find_bottom(env, lfsck,
-                                                         &bk->lb_lpf_fid);
-                       if (IS_ERR(child1)) {
-                               child1 = NULL;
-                               goto find_child2;
-                       }
-
-                       if (unlikely(!dt_object_exists(child1) ||
-                                    dt_object_remote(child1)) ||
-                                    !S_ISDIR(lfsck_object_type(child1))) {
-                               /* Invalid FID record in the bookmark file,
-                                * reset it. */
-                               fid_zero(&bk->lb_lpf_fid);
-                               rc = lfsck_bookmark_store(env, lfsck);
-
-                               CDEBUG(D_LFSCK, "%s: reset invalid LPF fid "DFID
-                                      " in the bookmark file: rc = %d\n",
-                                      lfsck_lfsck2name(lfsck),
-                                      PFID(lfsck_dto2fid(child1)), rc);
-
-                               if (rc != 0)
-                                       GOTO(put, rc);
-
-                               lfsck_object_put(env, child1);
-                               child1 = NULL;
-                       } else if (unlikely(!dt_try_as_dir(env, child1))) {
-                               GOTO(put, rc = -ENOTDIR);
-                       }
-               }
-       }
-
-find_child2:
+       /* child2 */
        snprintf(name, 8, "MDT%04x", node);
        rc = dt_lookup(env, parent, (struct dt_rec *)cfid,
                       (const struct dt_key *)name);
        if (rc == -ENOENT) {
-               if (!fid_is_zero(&bk->lb_lpf_fid))
-                       goto check_child1;
-
-               GOTO(put, rc = 0);
+               rc = 0;
+               goto find_child1;
        }
 
        if (rc != 0)
@@ -1501,7 +1453,7 @@ find_child2:
                if (rc != 0)
                        GOTO(put, rc);
 
-               goto check_child1;
+               goto find_child1;
        }
 
        child2 = lfsck_object_find_bottom(env, lfsck, cfid);
@@ -1515,23 +1467,20 @@ find_child2:
                if (rc != 0)
                        GOTO(put, rc);
 
-               goto check_child1;
+               goto find_child1;
        }
 
-       if (unlikely(!dt_try_as_dir(env, child2)))
-               GOTO(put, rc = -ENOTDIR);
+       if (unlikely(!dt_try_as_dir(env, child2))) {
+               lfsck_object_put(env, child2);
+               child2 = NULL;
+               rc = -ENOTDIR;
+       }
 
-       if (child1 == NULL) {
-               rc = lfsck_verify_lpf_pairs(env, lfsck, child2, name,
-                                           pfid, LVLT_BY_NAMEENTRY);
-       } else if (!lu_fid_eq(cfid, &bk->lb_lpf_fid)) {
-               rc = lfsck_verify_lpf_pairs(env, lfsck, child1, name,
-                                           pfid, LVLT_BY_BOOKMARK);
-               if (!lu_fid_eq(pfid, &LU_LPF_FID))
-                       rc = lfsck_verify_lpf_pairs(env, lfsck, child2,
-                                                   name, pfid,
-                                                   LVLT_BY_NAMEENTRY);
-       } else {
+find_child1:
+       if (fid_is_zero(&bk->lb_lpf_fid))
+               goto check_child2;
+
+       if (likely(lu_fid_eq(cfid, &bk->lb_lpf_fid))) {
                if (lfsck->li_lpf_obj == NULL) {
                        lu_object_get(&child2->do_lu);
                        lfsck->li_lpf_obj = child2;
@@ -1539,14 +1488,69 @@ find_child2:
 
                cname = lfsck_name_get_const(env, name, strlen(name));
                rc = lfsck_verify_linkea(env, child2, cname, &LU_LPF_FID);
+
+               GOTO(put, rc);
        }
 
-       GOTO(put, rc);
+       if (unlikely(!fid_is_norm(&bk->lb_lpf_fid))) {
+               struct lu_fid tfid = bk->lb_lpf_fid;
 
-check_child1:
-       if (child1 != NULL)
-               rc = lfsck_verify_lpf_pairs(env, lfsck, child1, name,
-                                           pfid, LVLT_BY_BOOKMARK);
+               /* Invalid FID record in the bookmark file, reset it. */
+               fid_zero(&bk->lb_lpf_fid);
+               rc = lfsck_bookmark_store(env, lfsck);
+
+               CDEBUG(D_LFSCK, "%s: reset invalid LPF fid "DFID
+                      " in the bookmark file: rc = %d\n",
+                      lfsck_lfsck2name(lfsck), PFID(&tfid), rc);
+
+               if (rc != 0)
+                       GOTO(put, rc);
+
+               goto check_child2;
+       }
+
+       child1 = lfsck_object_find_bottom(env, lfsck, &bk->lb_lpf_fid);
+       if (IS_ERR(child1)) {
+               child1 = NULL;
+               goto check_child2;
+       }
+
+       if (unlikely(!dt_object_exists(child1) ||
+                    dt_object_remote(child1)) ||
+                    !S_ISDIR(lfsck_object_type(child1))) {
+               /* Invalid FID record in the bookmark file, reset it. */
+               fid_zero(&bk->lb_lpf_fid);
+               rc = lfsck_bookmark_store(env, lfsck);
+
+               CDEBUG(D_LFSCK, "%s: reset invalid LPF fid "DFID
+                      " in the bookmark file: rc = %d\n",
+                      lfsck_lfsck2name(lfsck),
+                      PFID(lfsck_dto2fid(child1)), rc);
+
+               if (rc != 0)
+                       GOTO(put, rc);
+
+               lfsck_object_put(env, child1);
+               child1 = NULL;
+               goto check_child2;
+       }
+
+       if (unlikely(!dt_try_as_dir(env, child1))) {
+               lfsck_object_put(env, child1);
+               child1 = NULL;
+               rc = -ENOTDIR;
+               goto check_child2;
+       }
+
+       rc = lfsck_verify_lpf_pairs(env, lfsck, child1, name, pfid,
+                                   LVLT_BY_BOOKMARK);
+       if (lu_fid_eq(pfid, &LU_LPF_FID))
+               GOTO(put, rc);
+
+check_child2:
+       if (child2 != NULL)
+               rc = lfsck_verify_lpf_pairs(env, lfsck, child2, name,
+                                           pfid, LVLT_BY_NAMEENTRY);
 
        GOTO(put, rc);
 

diff --git a/lustre/osd-ldiskfs/osd_handler.c b/lustre/osd-ldiskfs/osd_handler.c
index bf46774..619662a 100644 (file)
--- a/lustre/osd-ldiskfs/osd_handler.c
+++ b/lustre/osd-ldiskfs/osd_handler.c
@@ -815,6 +815,8 @@ trigger:
                                        result = -EINPROGRESS;
                                else
                                        result = -EREMCHG;
+                       } else {
+                               result = -EREMCHG;
                        }
 
                        if (fid_is_on_ost(info, dev, fid, OI_CHECK_FLD))

diff --git a/lustre/osd-ldiskfs/osd_oi.c b/lustre/osd-ldiskfs/osd_oi.c
index 6f15760..6b6942f 100644 (file)
--- a/lustre/osd-ldiskfs/osd_oi.c
+++ b/lustre/osd-ldiskfs/osd_oi.c
@@ -338,7 +338,54 @@ osd_oi_table_open(struct osd_thread_info *info, struct osd_device *osd,
        RETURN(count);
 }
 
-int osd_oi_init(struct osd_thread_info *info, struct osd_device *osd)
+static int osd_remove_oi_one(struct dentry *parent, const char *name,
+                            int namelen)
+{
+       struct dentry *child;
+       int rc;
+
+       child = ll_lookup_one_len(name, parent, namelen);
+       if (IS_ERR(child)) {
+               rc = PTR_ERR(child);
+       } else {
+               rc = ll_vfs_unlink(parent->d_inode, child);
+               dput(child);
+       }
+
+       return rc == -ENOENT ? 0 : rc;
+}
+
+static int osd_remove_ois(struct osd_thread_info *info, struct osd_device *osd)
+{
+       char name[16];
+       int namelen;
+       int rc;
+       int i;
+
+       for (i = 0; i < osd->od_scrub.os_file.sf_oi_count; i++) {
+               namelen = snprintf(name, sizeof(name), "%s.%d",
+                                  OSD_OI_NAME_BASE, i);
+               rc = osd_remove_oi_one(osd_sb(osd)->s_root, name, namelen);
+               if (rc != 0) {
+                       CERROR("%.16s: fail to remove the stale OI file %s: "
+                              "rc = %d\n",
+                              LDISKFS_SB(osd_sb(osd))->s_es->s_volume_name,
+                              name, rc);
+                       return rc;
+               }
+       }
+
+       namelen = snprintf(name, sizeof(name), "%s", OSD_OI_NAME_BASE);
+       rc = osd_remove_oi_one(osd_sb(osd)->s_root, name, namelen);
+       if (rc != 0)
+               CERROR("%.16s: fail to remove the stale OI file %s: rc = %d\n",
+                      LDISKFS_SB(osd_sb(osd))->s_es->s_volume_name, name, rc);
+
+       return rc;
+}
+
+int osd_oi_init(struct osd_thread_info *info, struct osd_device *osd,
+               bool restored)
 {
        struct osd_scrub  *scrub = &osd->od_scrub;
        struct scrub_file *sf = &scrub->os_file;
@@ -346,6 +393,12 @@ int osd_oi_init(struct osd_thread_info *info, struct osd_device *osd)
        int                rc;
        ENTRY;
 
+       if (restored) {
+               rc = osd_remove_ois(info, osd);
+               if (rc != 0)
+                       return rc;
+       }
+
        OBD_ALLOC(oi, sizeof(*oi) * OSD_OI_FID_NR_MAX);
        if (oi == NULL)
                RETURN(-ENOMEM);

diff --git a/lustre/osd-ldiskfs/osd_oi.h b/lustre/osd-ldiskfs/osd_oi.h
index fbdcffa..6d5fae3 100644 (file)
--- a/lustre/osd-ldiskfs/osd_oi.h
+++ b/lustre/osd-ldiskfs/osd_oi.h
@@ -139,7 +139,8 @@ enum oi_check_flags {
 };
 
 int osd_oi_mod_init(void);
-int osd_oi_init(struct osd_thread_info *info, struct osd_device *osd);
+int osd_oi_init(struct osd_thread_info *info, struct osd_device *osd,
+               bool restored);
 void osd_oi_fini(struct osd_thread_info *info, struct osd_device *osd);
 int  osd_oi_lookup(struct osd_thread_info *info, struct osd_device *osd,
                   const struct lu_fid *fid, struct osd_inode_id *id,

diff --git a/lustre/osd-ldiskfs/osd_scrub.c b/lustre/osd-ldiskfs/osd_scrub.c
index ff639e2..80f2794 100644 (file)
--- a/lustre/osd-ldiskfs/osd_scrub.c
+++ b/lustre/osd-ldiskfs/osd_scrub.c
@@ -2453,7 +2453,8 @@ int osd_scrub_setup(const struct lu_env *env, struct osd_device *dev)
        struct file                *filp;
        struct inode               *inode;
        struct lu_fid              *fid    = &info->oti_fid;
-       int                         dirty  = 0;
+       bool                        dirty  = false;
+       bool                        restored = false;
        int                         rc     = 0;
        ENTRY;
 
@@ -2507,7 +2508,7 @@ int osd_scrub_setup(const struct lu_env *env, struct osd_device *dev)
                 * have their FID mapping in OI files already. */
                if (dev->od_maybe_new)
                        sf->sf_internal_flags = SIF_NO_HANDLE_OLD_FID;
-               dirty = 1;
+               dirty = true;
        } else if (rc != 0) {
                GOTO(cleanup_inode, rc);
        } else {
@@ -2530,14 +2531,15 @@ int osd_scrub_setup(const struct lu_env *env, struct osd_device *dev)
                                       old_uuid->uuid, new_uuid->uuid);
                        }
                        osd_scrub_file_reset(scrub, es->s_uuid,SF_INCONSISTENT);
-                       dirty = 1;
+                       dirty = true;
+                       restored = true;
                        if (old_uuid != NULL)
                                OBD_FREE_PTR(old_uuid);
                        if (new_uuid != NULL)
                                OBD_FREE_PTR(new_uuid);
                } else if (sf->sf_status == SS_SCANNING) {
                        sf->sf_status = SS_CRASHED;
-                       dirty = 1;
+                       dirty = true;
                }
        }
 
@@ -2546,14 +2548,14 @@ int osd_scrub_setup(const struct lu_env *env, struct osd_device *dev)
        else
                scrub->os_pos_current = LDISKFS_FIRST_INO(sb) + 1;
 
-       if (dirty != 0) {
+       if (dirty) {
                rc = osd_scrub_file_store(scrub);
                if (rc != 0)
                        GOTO(cleanup_inode, rc);
        }
 
        /* Initialize OI files. */
-       rc = osd_oi_init(info, dev);
+       rc = osd_oi_init(info, dev, restored);
        if (rc < 0)
                GOTO(cleanup_inode, rc);
 

diff --git a/lustre/tests/sanity-scrub.sh b/lustre/tests/sanity-scrub.sh
index 8e91ad4..5b3e2a7 100644 (file)
--- a/lustre/tests/sanity-scrub.sh
+++ b/lustre/tests/sanity-scrub.sh
@@ -7,8 +7,7 @@
 set -e
 
 ONLY=${ONLY:-"$*"}
-#Bug number for excepting test         6705
-ALWAYS_EXCEPT="$SANITY_SCRUB_EXCEPT    1c 5 10"
+ALWAYS_EXCEPT="$SANITY_SCRUB_EXCEPT"
 
 [ "$SLOW" = "no" ] && EXCEPT_SLOW=""
 # UPDATE THE COMMENT ABOVE WITH BUG NUMBERS WHEN CHANGING ALWAYS_EXCEPT!
@@ -409,7 +408,7 @@ test_3() {
        echo "starting MDTs with OI scrub disabled"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
        scrub_check_status 3 init
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
 }
 #run_test 3 "Do not trigger OI scrub when MDT mounts if 'noscrub' specified"
 
@@ -418,7 +417,7 @@ test_4a() {
        scrub_backup_restore 1
        echo "starting MDTs with OI scrub disabled"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
        mount_client $MOUNT || error "(5) Fail to start client!"
        scrub_enable_auto
        full_scrub_ratio 0
@@ -452,7 +451,7 @@ test_4b() {
        scrub_backup_restore 1
        echo "starting MDTs with OI scrub disabled"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
        mount_client $MOUNT || error "(5) Fail to start client!"
        scrub_enable_auto
        full_scrub_ratio 10
@@ -518,7 +517,7 @@ test_4c() {
        scrub_backup_restore 1
        echo "starting MDTs with OI scrub disabled"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
        mount_client $MOUNT || error "(5) Fail to start client!"
        scrub_enable_auto
        full_scrub_ratio 2
@@ -588,7 +587,7 @@ test_5() {
        echo "starting MDTs with OI scrub disabled (1)"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
        scrub_check_status 3 init
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
        mount_client $MOUNT || error "(5) Fail to start client!"
        scrub_enable_auto
 
@@ -655,7 +654,7 @@ test_6() {
        scrub_backup_restore 1
        echo "starting MDTs with OI scrub disabled"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
        mount_client $MOUNT || error "(5) Fail to start client!"
        scrub_enable_auto
 
@@ -733,7 +732,7 @@ test_7() {
        scrub_backup_restore 1
        echo "starting MDTs with OI scrub disabled"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
        mount_client $MOUNT || error "(5) Fail to start client!"
        scrub_enable_auto
 
@@ -751,7 +750,7 @@ test_7() {
        done
 
        scrub_check_status 8 scanning
-       scrub_check_flags 9 inconsistent,auto
+       scrub_check_flags 9 recreated,inconsistent,auto
 
        do_nodes $(comma_list $(mdts_nodes)) \
                $LCTL set_param fail_loc=0 fail_val=0
@@ -766,7 +765,7 @@ test_8() {
        scrub_backup_restore 1
        echo "starting MDTs with OI scrub disabled"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
 
        #define OBD_FAIL_OSD_SCRUB_DELAY         0x190
        do_nodes $(comma_list $(mdts_nodes)) \
@@ -798,7 +797,7 @@ test_9() {
 
        echo "starting MDTs with OI scrub disabled"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
 
        local BASE_SPEED1=100
        local RUN_TIME1=10
@@ -872,7 +871,7 @@ test_10a() {
        scrub_backup_restore 1
        echo "starting mds$n with OI scrub disabled (1)"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
        mount_client $MOUNT || error "(5) Fail to start client!"
        scrub_enable_auto
 
@@ -907,7 +906,7 @@ test_10b() {
        scrub_backup_restore 1
        echo "starting MDTs with OI scrub disabled"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
 
        #define OBD_FAIL_OSD_SCRUB_DELAY         0x190
        do_nodes $(comma_list $(mdts_nodes)) \
@@ -1096,7 +1095,7 @@ test_15() {
        echo "starting MDTs with OI scrub disabled"
        scrub_start_mds 2 "$MOUNT_OPTS_NOSCRUB"
        scrub_check_status 3 init
-       scrub_check_flags 4 inconsistent
+       scrub_check_flags 4 recreated,inconsistent
 
        # run under dryrun mode
        if [ $server_version -lt $(version_code 2.5.58) ]; then
@@ -1105,7 +1104,7 @@ test_15() {
                scrub_start 5 --dryrun
        fi
        scrub_check_status 6 completed
-       scrub_check_flags 7 inconsistent
+       scrub_check_flags 7 recreated,inconsistent
        scrub_check_params 8 dryrun
        scrub_check_repaired 9 20
 
@@ -1116,7 +1115,7 @@ test_15() {
                scrub_start 10 --dryrun
        fi
        scrub_check_status 11 completed
-       scrub_check_flags 12 inconsistent
+       scrub_check_flags 12 recreated,inconsistent
        scrub_check_params 13 dryrun
        scrub_check_repaired 14 20