LU-11643 libext2fs: don't use O_DIRECT for files on tmpfs

LU-11643 libext2fs: don't use O_DIRECT for files on tmpfs

If a filesystem image is on tmpfs, opening it with O_DIRECT for
reading the MMP will fail.  This is unnecessary, since the image
file can't really be open on another node at this point.  If the
open with O_DIRECT fails, retry without it when plausible.

Remove the special-casing of tmpfs from the mmp test cases.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I41f4b31657b06f62f10be8d6e524d303dd3ebbe5
Reviewed-on: https://review.whamcloud.com/37680
Tested-by: jenkins <devops@whamcloud.com>