LU-11610 target: Fix correct return value in out_handler.c

LU-11610 target: Fix correct return value in out_handler.c

out_handler.c has a pattern:

    if (IS_ERR(x) || expression) {
        ...
        RETURN(PTR_ERR(x))
    }

which if x is a valid pointer but the expression that represents an
error would cause the function to return PTR_ERR(x) for a valid pointer.
Fix this by either returning PTR_ERR(x) or -EPROTO depending on what the error
is.

Signed-off-by: Ronnie Sahlberg <rsahlberg@whamcloud.com>
Change-Id: Ic9f3427be31a9ba84b46f349ba90cc6aea379845
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/56260
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>