LU-5946 lprocfs: free expired jobstats after /proc read

LU-5946 lprocfs: free expired jobstats after /proc read

Currently, jobstats for expired jobids are only freed when a new
jobid arrives.  If no new jobids have arrived in a long time, the
expired jobstats won't be freed even though the /proc/.../job_stats
file is read repeatedly.  This shouldn't be a problem for steady job
submission patterns, but if there is a big gap in new jobid arrival
then expired jobids shouldn't be left around such a long time.

Free expired stats in lprocfs_jobstats_seq_release() after job_stats
is read, and move normal expiry to the arrival of new jobids instead
of on every stat to reduce overhead.  If jobs arrive frequently then
expired jobs will also be freed frequently, and if jobs arrive
slowly then there wan't be many expired jobs accumulating.

Rename job_iter_callback() to job_cleanup_iter_callback() for name
consistency with its caller and to make it more clear what it is for.

Instead of just passing a "bool force" param to lprocfs_job_cleanup(),
pass the number of seconds before which stats should be expired.
That allows lprocfs_job_cleanup() to be used at all callers instead
of having to have custom iterators in several places.

Avoid having multiple threads do cleanup of the jobstats concurrently.
This is not itself dangerous, since hash locking avoids races, but it
is inefficient to have multiple threads contending on the hash locks.

Add comment blocks for modified functions.

Remove some LASSERT() calls and replace with checks and error returns,
preferably at initialization time rather than during later usage.

Fix miscellaneous coding style issues.

Add a test for jobstats expiry after /proc read, and add a DNE test.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Icfd65ed7cd7beb26a9b566667e2686afe23ebbe5
Reviewed-on: http://review.whamcloud.com/12821
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>