LU-10391 lnet: remove 'fatal' arg from lnet_{sock_}connect

LU-10391 lnet: remove 'fatal' arg from lnet_{sock_}connect

The 'fatal' arg for indicating if an error is fatal doesn't
help make the code easier to read. In each case where it is
tested, it would be more clear to directly tested for the
particular errors that are not fatal.

So remove the arg and change the tests.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I905e0c2652b3148d93219a32a6b93515af323159
Reviewed-on: https://review.whamcloud.com/37700
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>