LU-5297 osp: process unsuccessful osp sync records properly

LU-5297 osp: process unsuccessful osp sync records properly

Unsuccessful records can be classifed into two types: failed and
invalid. And they should be handled differently.

This patch improves the handling process by the following fixes.

First, correct the return values of osp_sync_new_xxx_job() to separate
the record types:
- 0 on success
- negative on error
- 1 on invalid record

Second, process these two types of records differently:
- When a failed record is processed, opd_syn_{changes,rpc_in_flight,
  rpc_in_progess} should be decreased, and opd_syn_last_processed_id
  should be bumped.
- When an invalid record is processed, besides above process should
  be taken, this record should be deleted at the end. ("unknown record
  type is treated as invalid record".)

Third, simplify the sending rec process in osp_sync_process_queues(),
remove the unnecessary loop waiting and continue processing other
records directly.

Also, OBD_FAIL_OSP_CHECK_INVALID_REC and OBD_FAIL_OSP_CHECK_ENOMEM are
defined and used in sanity.sh test_239a/b respectively to verify the
fix.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I9c55f43f160a3d9e51892a2dc2f45a52f9b6f2c8
Reviewed-on: http://review.whamcloud.com/14925
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>