Whamcloud - gitweb
LU-1669 llite: Replace write mutex with range lock
Testing has shown the ll_inode_inode's lli_write_mutex to be a
limiting factor with single shared file write performance, when using
many writing threads on a single machine. Even if each thread is
writing to a unique portion of the file, the lli_write_mutex will
prevent no more than a single thread to ever write to the file
simultaneously.
This change attempts to remove this bottle neck, by replacing this
mutex with a range lock. This should allow multiple threads to write
to a single file simultaneously iff the threads are writing to unique
regions of the file.
Performance testing shows this change to garner a significant
performance boost to write bandwidth. Using FIO on a single machine
with 32 GB of RAM, write performance tests were run with and without
this change applied; the results are below:
+---------+-----------+---------+--------+--------+--------+
| fio v2.0.13 | Write Bandwidth (KB/s) |
+---------+-----------+---------+--------+--------+--------+
| # Tasks | GB / Task | Test 1 | Test 2 | Test 3 | Test 4 |
+---------+-----------+---------+--------+--------+--------+
| 1 | 64 | 452446 | 454623 | 457653 | 463737 |
| 2 | 32 | 850318 | 565373 | 602498 | 733027 |
| 4 | 16 | 1058900 | 463546 | 529107 | 976284 |
| 8 | 8 | 1026300 | 468190 | 576451 | 963404 |
| 16 | 4 | 1065500 | 503160 | 462902 | 830065 |
| 32 | 2 | 1068600 | 462228 | 466963 | 749733 |
| 64 | 1 | 991830 | 556618 | 557863 | 710912 |
+---------+-----------+---------+--------+--------+--------+
* Test 1: Lustre client running 04ec54f. File per process write
workload. This test was used as a baseline for what we
_could_ achieve in the single shared file tests if the
bottle necks were removed.
* Test 2: Lustre client running 04ec54f. Single shared file
workload, each task writing to a unique region.
* Test 3: Lustre client running 04ec54f + I0023132b. Single shared
file workload, each task writing to a unique region.
* Test 4: Lustre client running 04ec54f + this patch.
Single shared file workload, each task writing to a unique
region.
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I71e060c190065d87a20dc8df3104f898883d0583
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-on: http://review.whamcloud.com/6320
Tested-by: Jenkins
Reviewed-by: Hiroya Nozaki <nozaki.hiroya@jp.fujitsu.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>