LU-6895 lfsck: drop bad OI files after MDT file-level restore

LU-6895 lfsck: drop bad OI files after MDT file-level restore

For a new formatted Lustre system, it does not guarantee that all
the on-disk inodes/blocks are initialized. Instead, they are marked
as free in related inode/block bitmaps. When the inode is allocated
to some object, it will be initialized at that time. Such process
accelerates the format. But it may cause trouble for MDT file-level
backup/restore. For example: the sysadmin backup the MDT via server
side file level tar, then reformat the MDT device, and then restore
the MDT from the backup. Assume some object_A, its FID is the FID_A,
and it was mapped to the inode_B before the backup, such OI mapping
was recorded in the OI file. After the restore, another inode_C is
assigned to the object_A, but before OI scrub rebuilding related
OI mapping, the stale mapping "FID_A => inode_B" is still in the
OI file, and can be found by OI lookup. Generally, even if someone
found the stale OI mapping, it is not trouble, because the OSD will
verify whether FID-in-LMA for the indoe_B matches the FID_A or not.
But if the inode_B is NOT allocated after the restore, and because
we did not initialize inode_B during reformat, then the FID-in-LMA
for the indoe_B is still kept as the same before the backup, means
it matches the FID_A, then the OSD will think that the inode_B is
still assigned to the object_A after the restore. That is wrong.
In futher, although the inode_B is not allocated, but some of the
blocks that were assigned to it may have been allocated to others.
Then accessing the blocks via the inode_B may access some invalid
data, and may trigger some assertion, such as this issue.

So the key issues are two:

1) Some FID based operation may access stale OI mapping after MDT
   file-level backup/restore.

2) The OSD-ldiskfs may get some non-allocated inode with the give
   ino#/generation.

So as long as we can resovle one of them, then the trouble in the
ticket can be resolved.

The solutions for 1):

1.1) Avoid FID based operation before OI scrub done. That is not
     easy, because fid2path cannot be covered by some name based
     operation.

1.2) Remove the OI files after MDT file-level backup/restore. It
     is more direct solution. Another benefit is that even if OI
     scrub rebuilt the OI files, it only guarantees that all the
     FIDs' OI mappings have been refrshed. But it does not clean
     the stale FIDs' OI mappings. Because the OI scrub only does
     inode-table based scanning, not OI files scanning. Removing
     the OI files can resolve related trouble completely.

The solutions for 2):

2.1) New ldiskfs patch to make ldiskfs_iget() to return "-ENOENT"
     for the case of loading non-allocated inode by checking the
     inode bitmap.

2.2) Check the inode's valid inside OSD via related inode bitmap.

Generally, less ldiskfs patches is better. It will safe a lot of
effort when kernel upgrade. So 2.1) is not the best solution. As
for 2.2), it is not good to access the inode bitmap directly in
OSD unless we have to.

Relatively, the solution 1.2) is more efficient and benefit. That
is the current patch does.

On the other hand, this patch also makes some effort for the 1.1):

When start LFSCK, to hold the potential orphans in subsequent system
check, the start process will verify the .lustre/lost+found/ and its
sub-directories firstly. For each MDT, there is one sub-dir named as
.lustre/lost+found/MDTxxxx, which FID is recorded in the LFSCK trace
bookmark file. Originally, the start process uses such FID to locate
the MDTxxxx object. It is not problem for most of cases, but if the
MDT is just restored from file-level backup, and the low layer (OSD)
OI mapping may be invalid, then locating the MDTxxxx object will hit
-EREMCHG failure, and then may cause more troubles:

1) It will try to start OI scrub because of bad OI mapping detected,
   but because we are already in starting the LFSCK, then the logic
   of triggering OI scrub because of LFSCK is confused and difficult
   to be handled.

2) It will misguide the up layer LFSCK to think that the MDTxxxx obj
   does not exist or crashed, then will take some unexpected actions
   to repair such "fake" inconsistency.

The patch make some adjustment of the LFSCK start:

a) Trigger low layer OI scrub before verifying .lustre/lost+found.

b) To verify the .lustre/lost+found/ and its sub-dirs, scanning the
   .lustre/lost+found/ directory via namespace dt_lookup, and check
   the returned FID with the FID in the LFSCK trace bookmark file.
   With such way, even though the MDT is restored from file-level
   backup, the low layer still can locate the right object (inode).

Test-Parameters: alwaysuploadlogs envdefinitions=SLOW=yes,ENABLE_QUOTA=yes mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs clientdistro=el7 ossdistro=el7 mdsdistro=el7 mdtcount=1 testlist=sanity-lfsck,sanity-lfsck,sanity-lfsck
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Icfdab19e69b75400bb39279924c0c3e0b400b4e0
Reviewed-on: http://review.whamcloud.com/17403
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>