Whamcloud - gitweb
LU-13134 obdclass: use slab allocation for cl_page 25/37225/17
authorWang Shilong <wshilong@ddn.com>
Thu, 16 Jan 2020 07:05:08 +0000 (15:05 +0800)
committerOleg Drokin <green@whamcloud.com>
Tue, 7 Apr 2020 17:18:58 +0000 (17:18 +0000)
commit03364148c01ffe7bb2f2abfde84bbc36dc4d8237
treef9a2e7998089ed5dda91b58474255b2444965125
parent21ccfd99de46e1ca1572b16db0d3cef9cd805f4f
LU-13134 obdclass: use slab allocation for cl_page

Currently we use kmalloc() for cl_page allocation, this is because
cl_page size could be different with different objects.

For most cases (except obdecho object) we have 408 bytes for cl_page.
kmalloc() uses 512 bytes slab for it. So for 4KB PAGE_SIZE, we could
allocate 8 cl_page in a single page.

If we use private slab cache, it can fit 10 cl_page for 4KB PAGE_SIZE,
and even more struct cl_page can fit if we reduce the struct size.
For 4GB of pages the cl_page memory usage will shrink from 512M to 408M.

With clients already having 100GB of memory, it will save us 2.5GB
memory usage for cl_page, and some systems have as much as 24TB today!

This patch uses a static array with N (currently 16) entries storing
the { size, slab pointer }, with only 2-3 entries in current usage.
If the static array is not large enough, then fall back to kmalloc().

Benchmark numbers:

    4KB Random Read
    fio -iodepth=128 -direct=1 -size=2g -runtime=60 -numjobs=256
        -group_reporting  -directory=/ai400/out -create_serialize=0
        -filename_format='f.$jobnum.$filenum'

                        master+     master+
                        LU-4198     LU-4198+
                                    LU-13134   delta
      QD=1, numjobs=1     4518       4538      +0.44%
    QD=128, numjobs=256   2177K      2207k     +1.38%

Change-Id: I565b09616b22706f93d4c0fdc0df396d06cd51cc
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37225
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
lustre/include/cl_object.h
lustre/obdclass/cl_internal.h
lustre/obdclass/cl_object.c
lustre/obdclass/cl_page.c