LU-16623 lod: handle object allocation consistently

LU-16623 lod: handle object allocation consistently

Consistently handle the various OS_STATFS_* flags that indicate
an OST or MDT is full or otherwise marked ineligible for use.

Fix lod_statfs_check() so it skips MDTs with OS_STATFS_ENOINO
for allocating dir stripes instead of only checking OST targets.

In the LOD code, ltd_active=0 indicates that the device is not
usable for new object allocations for a variety of reasons. That
includes out of space or inodes, read-only, max_create_count=0,
or disconnected export, not *only* that the OSP is disconnected
from the OST as with imp_deactive.  Targets marked ltd_active=0
will not be counted in ld_active_tgt_count, so these OSTs will
not count toward stripe_count for stripe_count=-1 files.

Set flags = LOD_USES_DEFAULT_STRIPE in lod_qos_prep_create() for
stripe_count = -1 layouts and pass it to lod_stripe_count_min()
to avoid use of *all* OSTs when free space is imbalanced or OSTs
are not available, and be happy with allocations on 3/4 of OSTs.
It looks like this functionality was missed when object allocations
transitioned from the LOV to LOD module.  Put the LOV_USES_* into
an enum and rename to LOD_USES_* for consistency with current code.

Apply the lod.*.max_stripe_count limits to PFL components as well
as plain file layouts in lod_comp_entry_stripe_count().

Rename ltd_connecting to ltd_discon, since there is no guarantee
that this target is actually *connecting*, only that it is currently
disconnected.  Use ltd_discon in places that checked ltd_active to
decide if the OSP was disconnected from the OST, which shouldn't be
skipped just because the OST is full or has creates disabled.

Fixes: 7b124fef76 ("LU-4277 lod: handle os_state as a flag, check READONLY")
Fixes: 5b147e47de ("LU-11115 lod: skip max_create_count=0 OST in QoS and RR algorithms")
Fixes: c7f2e70a27 ("LU-1303 lod: QoS allocation policy")
Fixes: c1d0a355a6 ("LU-12624 lod: alloc dir stripes by QoS")
Fixes: 3c9580931d ("LU-9162 lod: option to set max stripe count per filesystem")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: Ifb9443fe6c80b4d7f82b442060db7ac8423ebbe5
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50250
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>