4 This survey script does sequential I/O with varying numbers of threads and
5 objects (files) by using lctl to drive the echo_client connected
6 to local or remote obdfilter instances, or remote obdecho instances.
8 It can be used to characterise the performance of the following lustre
11 1. The Object Storage Targets.
13 Here the script directly exercises one or more instances of obdfilter.
14 They may be running on 1 or more nodes, e.g. when they are all attached
15 to the same multi-ported disk subsystem.
17 You need to tell the script all the names of the obdfilter instances.
18 These should be up and running already . If some are on different
19 nodes, you need to specify their hostnames too (e.g. node1:ost1).
21 You just need to pass parameter case=disk to the script. The script will
22 automatically detect the local obdfilter instances.
24 All the obdfilter instances are driven directly. The script
25 automatically loads the obdecho module if required and creates one
26 instance of echo_client for each obdfilter instance.
30 Here the script drives one or more instances of obdecho server via instances
31 of echo_client running on 1 or more nodes.
33 You just need to pass parameters case=network and
34 targets="<hostname/ip_of_server>" to the script. The script will do the
35 required setup for network case.
37 3. The Stripe F/S over the Network.
39 Here the script drives one or more instances of obdfilter via instances
40 of echo_client running on 1 or more nodes.
42 You need to tell the script all the names of the OSC's, which should be
45 You just need to pass parameter case=netdisk to the script. The script will
46 use all of the local OSCs.
48 Note that the script is _NOT_ scalable to 100s of nodes since it is only
49 intended to measure individual servers, not the scalability of the system
55 The script must be customised according to the components under test and
56 where it should keep its working files. Customization variables are
57 described clearly at Customization variables Section in the script.
58 Please see maximum suported value ranges for customization variables
61 To run against a local disk:
62 ---------------------------
63 - Create a Lustre configuraton using your normal methods
66 Setup the Lustre filesystem with required OST's. Make sure that obdecho.ko
67 module is present. Then invoke the obdfilter-survey script with parameter
69 e.g. : $ nobjhi=2 thrhi=2 size=1024 case=disk sh obdfilter-survey
74 - You do not need to specify and MDS or LOV
75 - List all OSTs that you wish to test
76 - On all OSS machines:
77 Remember, write tests are destructive! This test should be run prior to
78 startup of your actual Lustre filesystem. If that is the case, you will not
79 need to reformat to restart Lustre - however, if the test is terminated before
80 completion, you may have to remove objects from the disk.
82 - Determine the obdfilter instance names on all the clients, column 4
83 of 'lctl dl'. For example:
85 # pdsh -w oss[01-02] lctl dl |grep obdfilter |sort
86 oss01: 0 UP obdfilter oss01-sdb oss01-sdb_UUID 3
87 oss01: 2 UP obdfilter oss01-sdd oss01-sdd_UUID 3
88 oss02: 0 UP obdfilter oss02-sdi oss02-sdi_UUID 3
91 Here the obdfilter instance names are oss01-sdb, oss01-sdd, oss02-sdi.
93 Since you are driving obdfilter instances directly, set the shell array
94 variable 'targets' to the names of the obdfilter instances.
98 targets='oss01:oss01-sdb oss01:oss01-sdd oss02:oss02-sdi' \
101 To run against a network:
102 ------------------------
103 For the second case i.e. obdfilter-survey over network, following setup
105 - Install all lustre modules including obdecho.
106 - Start lctl and check for the device list. The device list must be empty.
107 - It is suggested that there should be passwordless enrty between client
108 and server machine to avoid typing password.
110 To run obdfilter-surevy against network you just need to pass parameter
111 case=netdisk and targets="<hostname/ip_of_server>" to the script.
113 e.g. $ nobjhi=2 thrhi=2 size=1024 targets="<hostname/ip_of_server>" \
114 case=network sh obdfilter-survey
116 On server side you can see the stats at :
117 /proc/fs/lustre/obdecho/<echo_srv>/stats
118 where, 'echo_srv' is the obdecho server created through script.
120 NOTE: In network test only automated run is supported.
122 To run against network-disk:
123 ----------------------------
124 - Create a Lustre configuraton using your normal methods
127 Setup the lustre with required OST's. Make sure that obdecho.ko module is
128 present. Then invoke the obdfilter-survey script with parameter case=netdisk.
129 e.g. : $ nobjhi=2 thrhi=2 size=1024 case=netdisk sh obdfilter-survey
132 While running manually you need to tell the script all the names of the
133 echo_client instances, which should already be up and running.
134 e.g. $ nobjhi=2 thrhi=2 size=1024 targets="<osc_name> ..." \
141 When the script runs, it creates a number of working files and a pair of
142 result files. All files start with the prefix given by ${rslt}.
144 ${rslt}.summary same as stdout
145 ${rslt}.script_* per-host test script files
146 ${rslt}.detail_tmp* per-ost result files
147 ${rslt}.detail collected result files for post-mortem
149 The script iterates over the given numbers of threads and objects
150 performing all the specified tests and checking that all test processes
151 completed successfully.
153 Note that the script may not clean up properly if it is aborted or if it
154 encounters an unrecoverable error. In this case, manual cleanup may be
155 required, possibly including killing any running instances of 'lctl' (local
156 or remote), removing echo_client instances created by the script and
163 The summary file and stdout contain lines like...
165 ost 8 sz 67108864K rsz 1024 obj 8 thr 8 write 613.54 [ 64.00, 82.00]
167 ost 8 is the total number of OSTs under test.
168 sz 67108864K is the total amount of data read or written (in KB).
169 rsz 1024 is the record size (size of each echo_client I/O, in KB).
170 obj 8 is the total number of objects over all OSTs
171 thr 8 is the total number of threads over all OSTs and objects
172 write is the test name. If more tests have been specified they
173 all appear on the same line.
174 613.54 is the aggregate bandwidth over all OSTs measured by
175 dividing the total number of MB by the elapsed time.
176 [64.00, 82.00] are the minimum and maximum instantaneous bandwidths seen on
179 Note that although the numbers of threads and objects are specifed per-OST
180 in the customization section of the script, results are reported aggregated
187 I've found it most useful to import the summary data (it's fixed width)
188 into gnuplot, Excel (or any graphing package) and graph bandwidth v.
189 # threads for varying numbers of concurrent regions. This shows how
190 the OSS performs for a given number of concurrently accessed objects
191 (i.e. files) with varying numbers of I/Os in flight.
193 It is also extremely useful to record average disk I/O sizes during each
194 test. These numbers help find pathologies in file the file system block
195 allocator and the block device elevator.
197 The included plot-obdfilter script is an example of processing the output
198 files to a .csv format and plotting graph using gnuplot.