From: Emoly Liu Date: Wed, 10 Jan 2018 05:58:15 +0000 (+0800) Subject: LUDOC-328 tbf: update TBF section to reflect the new changes X-Git-Tag: 2.11.0~15 X-Git-Url: https://git.whamcloud.com/?a=commitdiff_plain;h=fec5d9ee1eb79df935878f04f059ab21d90124e2;p=doc%2Fmanual.git LUDOC-328 tbf: update TBF section to reflect the new changes Update NRS-TBF policy section to reflect the following changes: - change TBF command format, e.g. rate=, nid={nid},… - Lustre-commit: d09237044382361b289c01a9f7412818b3599e79 - Lustre-change: http://review.whamcloud.com/19499 - specify ordering of TBF policy rules, e.g. rank= - Lustre-commit: aa14b0b9a1521b3deb26a07eb8f7bb28ce793001 - Lustre-change: http://review.whamcloud.com/19476 - add opcode based TBF policy, e.g. opcode={opcode} - Lustre-commit: d2c403363f65337166dc745b58d0a4529a534b84 - Lustre-change: http://review.whamcloud.com/11918 - support complex TBF rules, e.g. opcode={}&jobid={},nid={} - Lustre-commit: 949bc00ad1ec0786940bd00ba53479d88c23ea14 - Lustre-change: http://review.whamcloud.com/17345 - support wildcard in JobID TBF rule, e.g. jobid={prog*.10*} - Lustre-commit: d5b5a07a8b07ab51a2215de593cb8bfce7e589d0 - Lustre-change: http://review.whamcloud.com/24523 Change-Id: Iac8f735374cedf7065fdd445571fedcc01554d4c Signed-off-by: Emoly Liu Reviewed-on: https://review.whamcloud.com/30705 Tested-by: Jenkins Reviewed-by: Yingjin Qian Reviewed-by: Joseph Gmitter --- diff --git a/LustreTuning.xml b/LustreTuning.xml index d7f2e04..055fe93 100644 --- a/LustreTuning.xml +++ b/LustreTuning.xml @@ -1628,119 +1628,246 @@ ost.OSS.ost_io.nrs_orr_supported=reg_supported:reads_and_writes knows its RPC token rate. A rule can be added to or removed from the list at run time. Whenever the list of rules is changed, the queues will update their matched rules. - - - - ost.OSS.ost_io.nrs_tbf_rule - - The format of the rule start command of TBF policy is as - follows: - -$ lctl set_param x.x.x.nrs_tbf_rule= - "[reg|hp] start rule_name arguments..." - - The ' - rule_name' argument is a string which - identifies a rule. The format of the ' - arguments' is changing according to the - type of the TBF policy. For the NID based TBF policy, its format is - as follows: - -$ lctl set_param x.x.x.nrs_tbf_rule= - "[reg|hp] start rule_name {nidlist} rate" - - The format of ' - nidlist' argument is the same as the - format when configuring LNet route. The ' - rate' argument is the RPC rate of the - rule, means the upper limit number of requests per second. - Following commands are valid. Please note that a newly started - rule is prior to old rules, so the order of starting rules is - critical too. - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule= - "start other_clients {192.168.*.*@tcp} 50" - - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule= - "start loginnode {192.168.1.1@tcp} 100" - - General rule can be replaced by two rules (reg and hp) as - follows: - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule= - "reg start loginnode {192.168.1.1@tcp} 100" - - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule= - "hp start loginnode {192.168.1.1@tcp} 100" - - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule= - "start computes {192.168.1.[2-128]@tcp} 500" - - The above rules will put an upper limit for servers to process - at most 5x as many RPCs from compute nodes as login nodes. - For the JobID (please see - for more details) based TBF - policy, its format is as follows: - -$ lctl set_param x.x.x.nrs_tbf_rule= - "[reg|hp] start name {jobid_list} rate" - - Following commands are valid: - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule= - "start user1 {iozone.500 dd.500} 100" - - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule= - "start iozone_user1 {iozone.500} 100" - - Same as nid, could use reg and hp rules separately: - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule= - "hp start iozone_user1 {iozone.500} 100" - - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule= - "reg start iozone_user1 {iozone.500} 100" - - The format of the rule change command of TBF policy is as - follows: - -$ lctl set_param x.x.x.nrs_tbf_rule= - "[reg|hp] change rule_name rate" - - Following commands are valid: - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="change loginnode 200" - - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="reg change loginnode 200" - - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="hp change loginnode 200" - - The format of the rule stop command of TBF policy is as - follows: - -$ lctl set_param x.x.x.nrs_tbf_rule="[reg|hp] stop -rule_name" - - Following commands are valid: - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="stop loginnode" +
+ Enable TBF policy + Command: + lctl set_param ost.OSS.ost_io.nrs_policies="tbf <policy>" + + For now, the RPCs can be classified into the different types + according to their NID, JOBID and OPCode. (UID/GID will be supported + soon.) When enabling TBF policy, you can specify one of the types, or + just use "tbf" to enable all of them to do a fine-grained RPC requests + classification. + Example: + $ lctl set_param ost.OSS.ost_io.nrs_policies="tbf" +$ lctl set_param ost.OSS.ost_io.nrs_policies="tbf nid" +$ lctl set_param ost.OSS.ost_io.nrs_policies="tbf jobid" +$ lctl set_param ost.OSS.ost_io.nrs_policies="tbf opcode" +
+
+ Start a TBF rule + The TBF rule is defined in the parameter + ost.OSS.ost_io.nrs_tbf_rule. + Command: + lctl set_param x.x.x.nrs_tbf_rule= +"[reg|hp] start rule_name arguments..." + + 'rule_name' is a string of the TBF + policy rule's name and 'arguments' is a + string to specify the detailed rule according to the different types. + + + Next, the different types of TBF policies will be described. + + NID based TBF policy + Command: + lctl set_param x.x.x.nrs_tbf_rule= +"[reg|hp] start rule_name nid={nidlist} rate=rate" + + 'nidlist' uses the same format + as configuring LNET route. 'rate' is + the (upper limit) RPC rate of the rule. + Example: + $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start other_clients nid={192.168.*.*@tcp} rate=50" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start computes nid={192.168.1.[2-128]@tcp} rate=500" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start loginnode nid={192.168.1.1@tcp} rate=100" + In this example, the rate of processing RPC requests from + compute nodes is at most 5x as fast as those from login nodes. + The output of ost.OSS.ost_io.nrs_tbf_rule is + like: + lctl get_param ost.OSS.ost_io.nrs_tbf_rule +ost.OSS.ost_io.nrs_tbf_rule= +regular_requests: +CPT 0: +loginnode {192.168.1.1@tcp} 100, ref 0 +computes {192.168.1.[2-128]@tcp} 500, ref 0 +other_clients {192.168.*.*@tcp} 50, ref 0 +default {*} 10000, ref 0 +high_priority_requests: +CPT 0: +loginnode {192.168.1.1@tcp} 100, ref 0 +computes {192.168.1.[2-128]@tcp} 500, ref 0 +other_clients {192.168.*.*@tcp} 50, ref 0 +default {*} 10000, ref 0 + Also, the rule can be written in reg and + hp formats: + $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"reg start loginnode nid={192.168.1.1@tcp} rate=100" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"hp start loginnode nid={192.168.1.1@tcp} rate=100" + + + JobID based TBF policy + For the JobID, please see + for more details. + Command: + lctl set_param x.x.x.nrs_tbf_rule= +"[reg|hp] start name jobid={jobid_list} rate=rate" + + Wildcard is supported in + {jobid_list}. + Example: + $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start iozone_user jobid={iozone.500} rate=100" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start dd_user jobid={dd.*} rate=50" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start user1 jobid={*.600} rate=10" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start user2 jobid={io*.10* *.500} rate=200" + Also, the rule can be written in reg and + hp formats: + $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"hp start iozone_user1 jobid={iozone.500} rate=100" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"reg start iozone_user1 jobid={iozone.500} rate=100" + + + Opcode based TBF policy + Command: + $ lctl set_param x.x.x.nrs_tbf_rule= +"[reg|hp] start name opcode={opcode_list} rate=rate" + + Example: + $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start user1 opcode={ost_read} rate=100" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start iozone_user1 opcode={ost_read ost_write} rate=200" + Also, the rule can be written in reg and + hp formats: + $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"hp start iozone_user1 opcode={ost_read} rate=100" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"reg start iozone_user1 opcode={ost_read} rate=100" + + + Policy combination + To support rules with complex expressions of NID/JOBID/OPCode + conditions, TBF classifier is extented to classify RPC in a more + fine-grained way. This feature supports logical conditional + conjunction and disjunction operations among different types. + In the rule: + "&" represents the conditional conjunction and + "," represents the conditional disjunction. + Example: + $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start comp_rule opcode={ost_write}&jobid={dd.0},\ +nid={192.168.1.[1-128]@tcp 0@lo} rate=100" + In this example, those RPCs whose opcode is + ost_write and jobid is dd.0, or + nid satisfies the condition of + {192.168.1.[1-128]@tcp 0@lo} will be processed at the rate of 100 + req/sec. + The output of ost.OSS.ost_io.nrs_tbf_ruleis like: + + $ lctl get_param ost.OSS.ost_io.nrs_tbf_rule +ost.OSS.ost_io.nrs_tbf_rule= +regular_requests: +CPT 0: +comp_rule opcode={ost_write}&jobid={dd.0},nid={192.168.1.[1-128]@tcp 0@lo} 100, ref 0 +default * 10000, ref 0 +CPT 1: +comp_rule opcode={ost_write}&jobid={dd.0},nid={192.168.1.[1-128]@tcp 0@lo} 100, ref 0 +default * 10000, ref 0 +high_priority_requests: +CPT 0: +comp_rule opcode={ost_write}&jobid={dd.0},nid={192.168.1.[1-128]@tcp 0@lo} 100, ref 0 +default * 10000, ref 0 +CPT 1: +comp_rule opcode={ost_write}&jobid={dd.0},nid={192.168.1.[1-128]@tcp 0@lo} 100, ref 0 +default * 10000, ref 0 + + +
+
+ Change a TBF rule + Command: + lctl set_param x.x.x.nrs_tbf_rule= +"[reg|hp] change rule_name rate=rate" + + Example: + $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"change loginnode rate=200" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"reg change loginnode rate=200" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"hp change loginnode rate=200" - +
+
+ Stop a TBF rule + Command: + lctl set_param x.x.x.nrs_tbf_rule="[reg|hp] stop +rule_name" + Example: + $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="stop loginnode" $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="reg stop loginnode" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="hp stop loginnode" +
+
+ Rule options + To support more flexible rule conditions, the following options + are added. + + + Reordering of TBF rules + By default, a newly started rule is prior to the old ones, + but by specifying the argument 'rank=' when + inserting a new rule with "start" command, + the rank of the rule can be changed. Also, it can be changed by + "change" command. + + Command: + lctl set_param ost.OSS.ost_io.nrs_tbf_rule= +"start rule_name arguments... rank=obj_rule_name" +lctl set_param ost.OSS.ost_io.nrs_tbf_rule= +"change rule_name rate=rate rank=obj_rule_name" - -$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="hp stop loginnode" - - - + By specifying the existing rule + 'obj_rule_name', the new rule + 'rule_name' will be moved to the front of + 'obj_rule_name'. + Example: + $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start computes nid={192.168.1.[2-128]@tcp} rate=500" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start user1 jobid={iozone.500 dd.500} rate=100" +$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\ +"start iozone_user1 opcode={ost_read ost_write} rate=200 rank=computes" + In this example, rule "iozone_user1" is added to the front of + rule "computes". We can see the order by the following command: + + $ lctl get_param ost.OSS.ost_io.nrs_tbf_rule +ost.OSS.ost_io.nrs_tbf_rule= +regular_requests: +CPT 0: +user1 jobid={iozone.500 dd.500} 100, ref 0 +iozone_user1 opcode={ost_read ost_write} 200, ref 0 +computes nid={192.168.1.[2-128]@tcp} 500, ref 0 +default * 10000, ref 0 +CPT 1: +user1 jobid={iozone.500 dd.500} 100, ref 0 +iozone_user1 opcode={ost_read ost_write} 200, ref 0 +computes nid={192.168.1.[2-128]@tcp} 500, ref 0 +default * 10000, ref 0 +high_priority_requests: +CPT 0: +user1 jobid={iozone.500 dd.500} 100, ref 0 +iozone_user1 opcode={ost_read ost_write} 200, ref 0 +computes nid={192.168.1.[2-128]@tcp} 500, ref 0 +default * 10000, ref 0 +CPT 1: +user1 jobid={iozone.500 dd.500} 100, ref 0 +iozone_user1 opcode={ost_read ost_write} 200, ref 0 +computes nid={192.168.1.[2-128]@tcp} 500, ref 0 +default * 10000, ref 0 + + +