3 # vim:expandtab:shiftwidth=4:softtabstop=4:tabstop=4:
6 # lustre_config.sh - format and set up multiple lustre servers from a csv file
8 # This script is used to parse each line of a spreadsheet (csv file) and
9 # execute remote commands to format (mkfs.lustre) every Lustre target
10 # that will be part of the Lustre cluster.
12 # In addition, it can also verify the network connectivity and hostnames in
13 # the cluster, configure Linux MD/LVM devices and produce High-Availability
14 # software configurations for Heartbeat or CluManager.
16 ################################################################################
22 Usage: `basename $0` [options] <csv file>
24 This script is used to format and set up multiple lustre servers from a
29 -a select all the nodes from the csv file to operate on
30 -w hostname,hostname,...
31 select the specified list of nodes (separated by commas) to
32 operate on rather than all the nodes in the csv file
33 -x hostname,hostname,...
34 exclude the specified list of nodes (separated by commas)
35 -t HAtype produce High-Availability software configurations
36 The argument following -t is used to indicate the High-
37 Availability software type. The HA software types which
38 are currently supported are: hbv1 (Heartbeat version 1)
39 and hbv2 (Heartbeat version 2).
40 -n no net - don't verify network connectivity and hostnames
42 -d configure Linux MD/LVM devices before formatting the
44 -f force-format the Lustre targets using --reformat option
45 -m no fstab change - don't modify /etc/fstab to add the new
47 If using this option, then the value of "mount options"
48 item in the csv file will be passed to mkfs.lustre, else
49 the value will be added into the /etc/fstab.
51 csv file a spreadsheet that contains configuration parameters
52 (separated by commas) for each target in a Lustre cluster
62 This script is used to parse each line of a spreadsheet (csv file) and
63 execute remote commands to format (mkfs.lustre) every Lustre target
64 that will be part of the Lustre cluster.
66 It can also optionally:
67 * verify the network connectivity and hostnames in the cluster
68 * configure Linux MD/LVM devices
69 * modify /etc/modprobe.conf to add Lustre networking info
70 * add the Lustre server info to /etc/fstab
71 * produce configurations for Heartbeat or CluManager.
73 There are 5 kinds of line formats in the csv file. They represent the following
77 hostname,MD,md name,operation mode,options,raid level,component devices
79 hostname hostname of the node in the cluster
80 MD marker of MD device line
81 md name MD device name, e.g. /dev/md0
82 operation mode create or remove, default is create
83 options a "catchall" for other mdadm options, e.g. "-c 128"
84 raid level raid level: 0,1,4,5,6,10,linear and multipath
85 component devices block devices to be combined into the MD device
86 Multiple devices are separated by space or by using
87 shell expansions, e.g. "/dev/sd{a,b,c}"
89 2) Linux LVM PV (Physical Volume)
91 hostname,PV,pv names,operation mode,options
93 hostname hostname of the node in the cluster
95 pv names devices or loopback files to be initialized for later
96 use by LVM or to be wiped the label, e.g. /dev/sda
97 Multiple devices or files are separated by space or by
98 using shell expansions, e.g. "/dev/sd{a,b,c}"
99 operation mode create or remove, default is create
100 options a "catchall" for other pvcreate/pvremove options
103 3) Linux LVM VG (Volume Group)
105 hostname,VG,vg name,operation mode,options,pv paths
107 hostname hostname of the node in the cluster
109 vg name name of the volume group, e.g. ost_vg
110 operation mode create or remove, default is create
111 options a "catchall" for other vgcreate/vgremove options
113 pv paths physical volumes to construct this VG, required by
115 Multiple PVs are separated by space or by using
116 shell expansions, e.g. "/dev/sd[k-m]1"
118 4) Linux LVM LV (Logical Volume)
120 hostname,LV,lv name,operation mode,options,lv size,vg name
122 hostname hostname of the node in the cluster
124 lv name name of the logical volume to be created (optional)
125 or path of the logical volume to be removed (required
127 operation mode create or remove, default is create
128 options a "catchall" for other lvcreate/lvremove options
130 lv size size [kKmMgGtT] to be allocated for the new LV
131 Default unit is megabytes.
132 vg name name of the VG in which the new LV will be created
136 hostname,module_opts,device name,mount point,device type,fsname,mgs nids,index,
137 format options,mkfs options,mount options,failover nids
139 hostname hostname of the node in the cluster, must match "uname -n"
140 module_opts Lustre networking module options
141 device name Lustre target (block device or loopback file)
142 mount point Lustre target mount point
143 device type Lustre target type (mgs, mdt, ost, mgs|mdt, mdt|mgs)
144 fsname Lustre filesystem name, should be limited to 8 characters
146 mgs nids NID(s) of remote mgs node, required for mdt and ost targets
147 If this item is not given for an mdt, it is assumed that
148 the mdt will also be an mgs, according to mkfs.lustre.
149 index Lustre target index
150 format options a "catchall" contains options to be passed to mkfs.lustre
151 "--device-size", "--param", etc. all goes into this item.
152 mkfs options format options to be wrapped with --mkfsoptions="" and
153 passed to mkfs.lustre
154 mount options If this script is invoked with "-m" option, then the value of
155 this item will be wrapped with --mountfsoptions="" and passed
156 to mkfs.lustre, else the value will be added into /etc/fstab.
157 failover nids NID(s) of failover partner node
159 All the NIDs in one node are delimited by commas (','). When multiple nodes are
160 specified, they are delimited by a colon (':').
162 Items left blank will be set to defaults.
164 Example 1 - Simple, with combo MGS/MDT:
165 -------------------------------------------------------------------------------
167 lustre-mgs,options lnet networks=tcp,/tmp/mgs,/mnt/mgs,mgs|mdt,,,,--device-size=10240
170 lustre-ost,options lnet networks=tcp,/tmp/ost0,/mnt/ost0,ost,,lustre-mgs@tcp0,,--device-size=10240
173 lustre-ost,options lnet networks=tcp,/tmp/ost1,/mnt/ost1,ost,,lustre-mgs@tcp0,,--device-size=10240
174 -------------------------------------------------------------------------------
176 Example 2 - Separate MGS/MDT, two networks interfaces:
177 -------------------------------------------------------------------------------
179 lustre-mgs1,options lnet 'networks="tcp,elan"',/dev/sda,/mnt/mgs,mgs,,,,--quiet --param="sys.timeout=50",,"defaults,noauto","lustre-mgs2,2@elan"
182 lustre-mdt1,options lnet 'networks="tcp,elan"',/dev/sdb,/mnt/mdt,mdt,lustre2,"lustre-mgs1,1@elan:lustre-mgs2,2@elan",,--quiet --param="lov.stripesize=4194304",-J size=16,"defaults,noauto",lustre-mdt2
185 lustre-ost1,options lnet 'networks="tcp,elan"',/dev/sdc,/mnt/ost,ost,lustre2,"lustre-mgs1,1@elan:lustre-mgs2,2@elan",,--quiet,-I 512,"defaults,noauto",lustre-ost2
186 -------------------------------------------------------------------------------
188 Example 3 - with combo MGS/MDT failover pair and OST failover pair:
189 -------------------------------------------------------------------------------
191 lustre-mgs1,options lnet networks=tcp,/tmp/mgs,/mnt/mgs,mgs|mdt,,,,--quiet --device-size=10240,,,lustre-mgs2@tcp0
193 # combo mgs/mdt backup (--noformat)
194 lustre-mgs2,options lnet networks=tcp,/tmp/mgs,/mnt/mgs,mgs|mdt,,,,--quiet --device-size=10240 --noformat,,,lustre-mgs1@tcp0
197 lustre-ost1,options lnet networks=tcp,/tmp/ost1,/mnt/ost1,ost,,"lustre-mgs1@tcp0:lustre-mgs2@tcp0",,--quiet --device-size=10240,,,lustre-ost2@tcp0
199 # ost backup (--noformat) (note different device name)
200 lustre-ost2,options lnet networks=tcp,/tmp/ost2,/mnt/ost2,ost,,"lustre-mgs1@tcp0:lustre-mgs2@tcp0",,--quiet --device-size=10240 --noformat,,,lustre-ost1@tcp0
201 -------------------------------------------------------------------------------
203 Example 4 - Configure Linux MD/LVM devices before formatting Lustre targets:
204 -------------------------------------------------------------------------------
205 # MD device on mgsnode
206 mgsnode,MD,/dev/md0,,-q,1,/dev/sda1 /dev/sdb1
208 # MD/LVM devices on ostnode
209 ostnode,MD,/dev/md0,,-q -c 128,5,"/dev/sd{a,b,c}"
210 ostnode,MD,/dev/md1,,-q -c 128,5,"/dev/sd{d,e,f}"
211 ostnode,PV,/dev/md0 /dev/md1
212 ostnode,VG,ost_vg,,-s 32M,/dev/md0 /dev/md1
213 ostnode,LV,ost0,,-i 2 -I 128,300G,ost_vg
214 ostnode,LV,ost1,,-i 2 -I 128,300G,ost_vg
217 mgsnode,options lnet networks=tcp,/dev/md0,/mnt/mgs,mgs|mdt,,,,--quiet
220 ostnode,options lnet networks=tcp,/dev/ost_vg/ost0,/mnt/ost0,ost,,mgsnode,,--quiet
223 ostnode,options lnet networks=tcp,/dev/ost_vg/ost1,/mnt/ost1,ost,,mgsnode,,--quiet
224 -------------------------------------------------------------------------------
230 # Get the library of functions
231 . @scriptlibdir@/lc_common.sh
233 #***************************** Global variables *****************************#
234 declare -a MGS_NODENAME # node names of the MGS servers
235 declare -a MGS_IDX # indexes of MGSs in the global arrays
236 declare -i MGS_NUM # number of MGS servers in the cluster
239 declare -a NODE_NAMES # node names in the failover group
240 declare -a TARGET_OPTS # target services in one failover group
242 # All the items in the csv file
243 declare -a HOST_NAME MODULE_OPTS DEVICE_NAME MOUNT_POINT DEVICE_TYPE FS_NAME
244 declare -a MGS_NIDS INDEX FORMAT_OPTIONS MKFS_OPTIONS MOUNT_OPTIONS FAILOVERS
246 # Corresponding to MGS_NIDS and FAILOVERS arrays,
247 # IP addresses in which were converted to hostnames
248 declare -a MGS_NIDS_NAMES FAILOVERS_NAMES
254 # Get and check the positional parameters
255 while getopts "aw:x:t:ndfmhv" OPTION; do
258 [ -z "${SPECIFIED_NODELIST}" ] && [ -z "${EXCLUDED_NODELIST}" ] \
260 NODELIST_OPT="${NODELIST_OPT} -a"
264 SPECIFIED_NODELIST=$OPTARG
265 NODELIST_OPT="${NODELIST_OPT} -w ${SPECIFIED_NODELIST}"
269 EXCLUDED_NODELIST=$OPTARG
270 NODELIST_OPT="${NODELIST_OPT} -x ${EXCLUDED_NODELIST}"
274 if [ "${HATYPE_OPT}" != "${HBVER_HBV1}" ] \
275 && [ "${HATYPE_OPT}" != "${HBVER_HBV2}" ] \
276 && [ "${HATYPE_OPT}" != "${HATYPE_CLUMGR}" ]; then
277 echo >&2 $"`basename $0`: Invalid HA software type" \
289 REFORMAT_OPTION=$"--reformat "
306 # Toss out the parameters we've already processed
307 shift `expr $OPTIND - 1`
309 # Here we expect the csv file
310 if [ $# -eq 0 ]; then
311 echo >&2 $"`basename $0`: Missing csv file!"
315 # Check the items required for OSTs, MDTs and MGS
317 # When formatting an OST, the following items: hostname, module_opts,
318 # device name, device type and mgs nids, cannot have null value.
320 # When formatting an MDT or MGS, the following items: hostname,
321 # module_opts, device name and device type, cannot have null value.
324 if [ $# -eq 0 ]; then
325 echo >&2 $"`basename $0`: check_item() error: Missing argument"\
326 "for function check_item()!"
332 # Check hostname, module_opts, device name and device type
333 if [ -z "${HOST_NAME[i]}" ]||[ -z "${MODULE_OPTS[i]}" ]\
334 ||[ -z "${DEVICE_NAME[i]}" ]||[ -z "${DEVICE_TYPE[i]}" ]; then
335 echo >&2 $"`basename $0`: check_item() error: Some required"\
336 "item has null value! Check hostname, module_opts,"\
337 "device name and device type!"
342 if [ "${DEVICE_TYPE[i]}" = "ost" ]&&[ -z "${MGS_NIDS[i]}" ]; then
343 echo >&2 $"`basename $0`: check_item() error: OST's mgs nids"\
344 "item has null value!"
349 if [ -z "${MOUNT_POINT[i]}" ]; then
350 echo >&2 $"`basename $0`: check_item() error: mount"\
351 "point item of target ${DEVICE_NAME[i]} has null value!"
358 # Get the number of MGS nodes in the cluster
361 MGS_NUM=${#MGS_NODENAME[@]}
362 [ -z "${MGS_NODENAME[0]}" ] && let "INIT_IDX += 1" \
363 && let "MGS_NUM += 1"
366 # is_mgs_node hostname
367 # Verify whether @hostname is a MGS node
373 for ((i = ${INIT_IDX}; i < ${MGS_NUM}; i++)); do
374 [ "${MGS_NODENAME[i]}" = "${host_name}" ] && return 0
380 # Check whether the MGS nodes are in the same failover group
388 for ((i = ${INIT_IDX}; i < ${MGS_NUM}; i++)); do
389 mgs_node=${MGS_NODENAME[i]}
390 for ((j = ${INIT_IDX}; j < ${MGS_NUM}; j++)); do
391 [ "${MGS_NODENAME[j]}" = "${mgs_node}" ] && continue 1
394 if [ "${FAILOVERS_NAMES[idx]#*$mgs_node*}" = "${FAILOVERS_NAMES[idx]}" ]
396 echo >&2 $"`basename $0`: check_mgs_group() error:"\
397 "MGS node ${mgs_node} is not in the ${HOST_NAME[idx]}"\
407 # Get and check MGS servers.
408 # There should be no more than one MGS specified in the entire csv file.
412 declare -i exp_idx # Index of explicit MGS servers
413 declare -i imp_idx # Index of implicit MGS servers
414 local is_exp_mgs is_imp_mgs
417 # Initialize the MGS_NODENAME and MGS_IDX arrays
423 for ((i = 0; i < ${#HOST_NAME[@]}; i++)); do
427 # Check whether this node is an explicit MGS node
429 if [ "${DEVICE_TYPE[i]#*mgs*}" != "${DEVICE_TYPE[i]}" ]; then
430 verbose_output "Explicit MGS target" \
431 "${DEVICE_NAME[i]} in host ${HOST_NAME[i]}."
435 if [ "${DEVICE_TYPE[i]}" = "mdt" -a -z "${MGS_NIDS[i]}" ]; then
436 verbose_output "Implicit MGS target" \
437 "${DEVICE_NAME[i]} in host ${HOST_NAME[i]}."
441 # Get and check MGS servers
442 if ${is_exp_mgs} || ${is_imp_mgs}; then
443 # Check whether more than one MGS target in one MGS node
444 if is_mgs_node ${HOST_NAME[i]}; then
445 echo >&2 $"`basename $0`: check_mgs() error:"\
446 "More than one MGS target in the same node -"\
447 "\"${HOST_NAME[i]}\"!"
451 # Get and check primary MGS server and backup MGS server
452 if [ "${FORMAT_OPTIONS[i]}" = "${FORMAT_OPTIONS[i]#*noformat*}" ]
455 if [ -z "${MGS_NODENAME[0]}" ]; then
456 if [ "${is_exp_mgs}" = "true" -a ${imp_idx} -gt 1 ] \
457 || [ "${is_imp_mgs}" = "true" -a ${exp_idx} -gt 1 ]; then
458 echo >&2 $"`basename $0`: check_mgs() error:"\
459 "There exist both explicit and implicit MGS"\
460 "targets in the csv file!"
463 MGS_NODENAME[0]=${HOST_NAME[i]}
466 mgs_node=${MGS_NODENAME[0]}
467 if [ "${FAILOVERS_NAMES[i]#*$mgs_node*}" = "${FAILOVERS_NAMES[i]}" ]
469 echo >&2 $"`basename $0`: check_mgs() error:"\
470 "More than one primary MGS nodes in the csv" \
471 "file - ${MGS_NODENAME[0]} and ${HOST_NAME[i]}!"
473 echo >&2 $"`basename $0`: check_mgs() error:"\
474 "MGS nodes ${MGS_NODENAME[0]} and ${HOST_NAME[i]}"\
475 "are failover pair, one of them should use"\
476 "\"--noformat\" in the format options item!"
480 else # Backup MGS server
481 if [ "${is_exp_mgs}" = "true" -a ${imp_idx} -gt 1 ] \
482 || [ "${is_imp_mgs}" = "true" -a ${exp_idx} -gt 1 ]; then
483 echo >&2 $"`basename $0`: check_mgs() error:"\
484 "There exist both explicit and implicit MGS"\
485 "targets in the csv file!"
489 if ${is_exp_mgs}; then # Explicit MGS
490 MGS_NODENAME[exp_idx]=${HOST_NAME[i]}
492 exp_idx=$(( exp_idx + 1 ))
494 MGS_NODENAME[imp_idx]=${HOST_NAME[i]}
496 imp_idx=$(( imp_idx + 1 ))
499 fi #End of "if ${is_exp_mgs} || ${is_imp_mgs}"
502 # Check whether the MGS nodes are in the same failover group
503 if ! check_mgs_group; then
510 # Construct the command line of mkfs.lustre
511 construct_mkfs_cmdline() {
513 if [ $# -eq 0 ]; then
514 echo >&2 $"`basename $0`: construct_mkfs_cmdline() error:"\
515 "Missing argument for function construct_mkfs_cmdline()!"
520 local mgsnids mgsnids_str
521 local failnids failnids_str
524 MKFS_CMD=${MKFS_CMD}${REFORMAT_OPTION}
526 case "${DEVICE_TYPE[i]}" in
528 MKFS_CMD=${MKFS_CMD}$"--ost "
531 MKFS_CMD=${MKFS_CMD}$"--mdt "
534 MKFS_CMD=${MKFS_CMD}$"--mgs "
536 "mdt|mgs" | "mgs|mdt")
537 MKFS_CMD=${MKFS_CMD}$"--mdt --mgs "
540 echo >&2 $"`basename $0`: construct_mkfs_cmdline() error:"\
541 "Invalid device type - \"${DEVICE_TYPE[i]}\"!"
546 if [ -n "${FS_NAME[i]}" ]; then
547 MKFS_CMD=${MKFS_CMD}$"--fsname="${FS_NAME[i]}$" "
550 if [ -n "${MGS_NIDS[i]}" ]; then
551 mgsnids_str=${MGS_NIDS[i]}
552 for mgsnids in ${mgsnids_str//:/ }; do
553 MKFS_CMD=${MKFS_CMD}$"--mgsnode="${mgsnids}$" "
557 if [ -n "${INDEX[i]}" ]; then
558 MKFS_CMD=${MKFS_CMD}$"--index="${INDEX[i]}$" "
561 if [ -n "${FORMAT_OPTIONS[i]}" ]; then
562 MKFS_CMD=${MKFS_CMD}${FORMAT_OPTIONS[i]}$" "
565 if [ -n "${MKFS_OPTIONS[i]}" ]; then
566 MKFS_CMD=${MKFS_CMD}$"--mkfsoptions="$"\""${MKFS_OPTIONS[i]}$"\""$" "
569 if [ -n "${MOUNT_OPTIONS[i]}" ]; then
570 if ! ${MODIFY_FSTAB}; then
571 MKFS_CMD=${MKFS_CMD}$"--mountfsoptions="$"\""${MOUNT_OPTIONS[i]}$"\""$" "
575 if [ -n "${FAILOVERS[i]}" ]; then
576 failnids_str=${FAILOVERS[i]}
577 for failnids in ${failnids_str//:/ }; do
578 MKFS_CMD=${MKFS_CMD}$"--failnode="${failnids}$" "
582 MKFS_CMD=${MKFS_CMD}${DEVICE_NAME[i]}
586 # Get all the node names in this failover group
589 if [ $# -eq 0 ]; then
590 echo >&2 $"`basename $0`: get_nodenames() error: Missing"\
591 "argument for function get_nodenames()!"
599 # Initialize the NODE_NAMES array
602 NODE_NAMES[0]=${HOST_NAME[i]}
605 for nids in ${FAILOVERS_NAMES[i]//:/ }
607 NODE_NAMES[idx]=$(nids2hostname ${nids})
608 if [ ${PIPESTATUS[0]} -ne 0 ]; then
609 echo >&2 "${NODE_NAMES[idx]}"
619 # Verify whether the format line has HA items
623 [ -n "${FAILOVERS[i]}" ] && return 0
628 # Produce HA software's configuration files
636 HOSTNAME_OPT=${HOST_NAME[i]}
638 if ! get_nodenames $i; then
639 echo >&2 $"`basename $0`: gen_ha_config() error: Can not get the"\
640 "failover nodenames from failover nids - \"${FAILOVERS[i]}\" in"\
641 "the \"${HOST_NAME[i]}\" failover group!"
645 for ((idx = 1; idx < ${#NODE_NAMES[@]}; idx++)); do
646 HOSTNAME_OPT=${HOSTNAME_OPT}$":"${NODE_NAMES[idx]}
649 # Target devices option
650 DEVICE_OPT=" -d "${TARGET_OPTS[0]}
651 for ((idx = 1; idx < ${#TARGET_OPTS[@]}; idx++)); do
652 DEVICE_OPT=${DEVICE_OPT}" -d "${TARGET_OPTS[idx]}
655 # Construct the generation script command line
656 case "${HATYPE_OPT}" in
657 "${HBVER_HBV1}"|"${HBVER_HBV2}") # Heartbeat
658 cmd_line=${GEN_HB_CONFIG}$" -r ${HATYPE_OPT} -n ${HOSTNAME_OPT}"
659 cmd_line=${cmd_line}${DEVICE_OPT}${VERBOSE_OPT}
661 "${HATYPE_CLUMGR}") # CluManager
662 cmd_line=${GEN_CLUMGR_CONFIG}$" -n ${HOSTNAME_OPT}"
663 cmd_line=${cmd_line}${DEVICE_OPT}${VERBOSE_OPT}
667 # Execute script to generate HA software's configuration files
668 verbose_output "Generating HA software's configurations in"\
669 "${HOST_NAME[i]} failover group..."
670 verbose_output "${cmd_line}"
671 eval $(echo "${cmd_line}")
672 if [ ${PIPESTATUS[0]} -ne 0 ]; then
675 verbose_output "Generate HA software's configurations in"\
676 "${HOST_NAME[i]} failover group OK"
681 # Configure HA software
683 if [ -z "${HATYPE_OPT}" ]; then
688 declare -i prim_idx # Index for PRIM_HOSTNAMES array
689 declare -i target_idx # Index for TARGET_OPTS and HOST_INDEX arrays
691 declare -a PRIM_HOSTNAMES # Primary hostnames in all the failover
692 # groups in the lustre cluster
693 declare -a HOST_INDEX # Indices for the same node in all the
694 # format lines in the csv file
697 # Initialize the PRIM_HOSTNAMES array
701 # Get failover groups and generate HA configuration files
702 for ((i = 0; i < ${#HOST_NAME[@]}; i++)); do
703 prim_host=${HOST_NAME[i]}
705 for ((j = 0; j < ${#PRIM_HOSTNAMES[@]}; j++)); do
706 [ "${prim_host}" = "${PRIM_HOSTNAMES[j]}" ] && continue 2
712 for ((k = 0; k < ${#HOST_NAME[@]}; k++)); do
713 if [ "${prim_host}" = "${HOST_NAME[k]}" ] && is_ha_line "${k}"
715 HOST_INDEX[target_idx]=$k
716 TARGET_OPTS[target_idx]=${DEVICE_NAME[k]}:${MOUNT_POINT[k]}
717 target_idx=$(( target_idx + 1 ))
721 if [ ${#TARGET_OPTS[@]} -ne 0 ]; then
722 PRIM_HOSTNAMES[prim_idx]=${prim_host}
723 prim_idx=$(( prim_idx + 1 ))
725 if ! gen_ha_config ${HOST_INDEX[0]}; then
731 if [ ${#PRIM_HOSTNAMES[@]} -eq 0 ]; then
732 verbose_output "There are no \"failover nids\" items in the"\
733 "csv file. No HA configuration files are generated!"
740 # Get all the items in the csv file and do some checks.
743 if [ $# -eq 0 ]; then
744 echo >&2 $"`basename $0`: get_items() error: Missing argument"\
745 "for function get_items()!"
753 declare -i line_num=0
756 while read -r LINE; do
757 line_num=${line_num}+1
758 # verbose_output "Parsing line ${line_num}: $LINE"
760 # Get rid of the empty line
761 if [ -z "`echo ${LINE}|awk '/[[:alnum:]]/ {print $0}'`" ]; then
765 # Get rid of the comment line
766 if [ -z "`echo \"${LINE}\" | egrep -v \"([[:space:]]|^)#\"`" ]
771 # Skip the Linux MD/LVM line
772 marker=$(echo ${LINE} | cut -d, -f 2)
773 if [ "${marker}" = "${MD_MARKER}" -o "${marker}" = "${PV_MARKER}" ] \
774 || [ "${marker}" = "${VG_MARKER}" -o "${marker}" = "${LV_MARKER}" ]; then
778 # Skip the host which is not specified in the host list
779 if ! ${USE_ALLNODES}; then
780 hostname=$(echo ${LINE} | cut -d, -f 1)
781 ! host_in_hostlist ${hostname} ${NODES_TO_USE} && continue
784 # Parse the config line into CONFIG_ITEM
785 if ! parse_line "$LINE"; then
786 echo >&2 $"`basename $0`: parse_line() error: Occurred"\
787 "on line ${line_num} in ${CSV_FILE}: $LINE"
791 HOST_NAME[idx]=${CONFIG_ITEM[0]}
792 MODULE_OPTS[idx]=${CONFIG_ITEM[1]}
793 DEVICE_NAME[idx]=${CONFIG_ITEM[2]}
794 MOUNT_POINT[idx]=${CONFIG_ITEM[3]}
795 DEVICE_TYPE[idx]=${CONFIG_ITEM[4]}
796 FS_NAME[idx]=${CONFIG_ITEM[5]}
797 MGS_NIDS[idx]=${CONFIG_ITEM[6]}
798 INDEX[idx]=${CONFIG_ITEM[7]}
799 FORMAT_OPTIONS[idx]=${CONFIG_ITEM[8]}
800 MKFS_OPTIONS[idx]=${CONFIG_ITEM[9]}
801 MOUNT_OPTIONS[idx]=${CONFIG_ITEM[10]}
802 FAILOVERS[idx]=${CONFIG_ITEM[11]}
804 MODULE_OPTS[idx]=`echo "${MODULE_OPTS[idx]}" | sed 's/"/\\\"/g'`
806 # Convert IP addresses in NIDs to hostnames
807 MGS_NIDS_NAMES[idx]=$(ip2hostname_multi_node ${MGS_NIDS[idx]})
808 if [ ${PIPESTATUS[0]} -ne 0 ]; then
809 echo >&2 "${MGS_NIDS_NAMES[idx]}"
813 FAILOVERS_NAMES[idx]=$(ip2hostname_multi_node ${FAILOVERS[idx]})
814 if [ ${PIPESTATUS[0]} -ne 0 ]; then
815 echo >&2 "${FAILOVERS_NAMES[idx]}"
819 # Check some required items for formatting target
820 if ! check_item $idx; then
821 echo >&2 $"`basename $0`: check_item() error:"\
822 "Occurred on line ${line_num} in ${CSV_FILE}."
832 # check_lnet_connect hostname_index mgs_hostname
833 # Check whether the target node can contact the MGS node @mgs_hostname
834 # If @mgs_hostname is null, then it means the primary MGS node
835 check_lnet_connect() {
839 local COMMAND RET_STR
841 local nids nids_names
846 # Execute remote command to check that
847 # this node can contact the MGS node
848 verbose_output "Checking lnet connectivity between" \
849 "${HOST_NAME[i]} and the MGS node ${mgs_node}"
850 mgs_prim_nids=`echo ${MGS_NIDS[i]} | awk -F: '{print $1}'`
852 if [ -z "${mgs_node}" ]; then
853 nids_str=${mgs_prim_nids} # nids of primary MGS node
854 if [ -z "${nids_str}" ]; then
855 echo >&2 $"`basename $0`: check_lnet_connect() error:"\
856 "Check the mgs nids item of host ${HOST_NAME[i]}!"\
857 "Missing nids of the primary MGS node!"
861 for nids in ${MGS_NIDS[i]//:/ }; do
862 nids_names=$(ip2hostname_single_node ${nids})
863 if [ ${PIPESTATUS[0]} -ne 0 ]; then
864 echo >&2 "${nids_names}"
868 [ "${nids_names}" != "${nids_names#*$mgs_node*}" ]\
869 && nids_str=${nids} # nids of backup MGS node
871 if [ -z "${nids_str}" ]; then
872 echo >&2 $"`basename $0`: check_lnet_connect() error:"\
873 "Check the mgs nids item of host ${HOST_NAME[i]}!"\
874 "Can not figure out which nids corresponding to the MGS"\
875 "node ${mgs_node} from \"${MGS_NIDS[i]}\"!"
881 for mgs_nid in ${nids_str//,/ }
883 COMMAND=$"${LCTL} ping ${mgs_nid} 5 || echo failed 2>&1"
884 RET_STR=$(${REMOTE} ${HOST_NAME[i]} "${COMMAND}" 2>&1)
885 if [ ${PIPESTATUS[0]} -eq 0 -a "${RET_STR}" = "${RET_STR#*failed*}" ]
887 # This node can contact the MGS node
888 verbose_output "${HOST_NAME[i]} can contact the MGS" \
889 "node ${mgs_node} by using nid \"${mgs_nid}\"!"
895 if ! ${ping_mgs}; then
896 echo >&2 "`basename $0`: check_lnet_connect() error:" \
897 "${HOST_NAME[i]} cannot contact the MGS node ${mgs_node}"\
898 "with nids - \"${nids_str}\"! Check ${LCTL} command!"
905 # Start lnet network in the cluster node and check that
906 # this node can contact the MGS node
908 if ! ${VERIFY_CONNECT}; then
913 if [ $# -eq 0 ]; then
914 echo >&2 $"`basename $0`: check_lnet() error: Missing"\
915 "argument for function check_lnet()!"
921 local COMMAND RET_STR
923 # Execute remote command to start lnet network
924 verbose_output "Starting lnet network in ${HOST_NAME[i]}"
925 COMMAND="PATH=\$PATH:/sbin:/usr/sbin modprobe lnet; ${LCTL} network up 2>&1"
926 RET_STR=$(${REMOTE} ${HOST_NAME[i]} "${COMMAND}" 2>&1)
927 if [ ${PIPESTATUS[0]} -ne 0 -o "${RET_STR}" = "${RET_STR#*LNET configured*}" ]
929 echo >&2 "`basename $0`: check_lnet() error: remote" \
930 "${HOST_NAME[i]} error: ${RET_STR}"
934 if is_mgs_node ${HOST_NAME[i]}; then
938 # Execute remote command to check that
939 # this node can contact the MGS node
940 for ((j = 0; j < ${MGS_NUM}; j++)); do
941 if ! check_lnet_connect $i ${MGS_NODENAME[j]}; then
949 # Start lnet network in the MGS node
955 if [ -z "${MGS_NODENAME[0]}" -a -z "${MGS_NODENAME[1]}" ]; then
956 if ${USE_ALLNODES}; then
957 verbose_output "There is no MGS target in the ${CSV_FILE} file."
959 verbose_output "There is no MGS target in the node list \"${NODES_TO_USE}\"."
964 for ((i = ${INIT_IDX}; i < ${MGS_NUM}; i++)); do
965 # Execute remote command to add lnet options lines to
966 # the MGS node's modprobe.conf/modules.conf
968 COMMAND=$"echo \"${MODULE_OPTS[${idx}]}\"|${MODULE_CONFIG}"
969 verbose_output "Adding lnet module options to ${MGS_NODENAME[i]}"
970 ${REMOTE} ${MGS_NODENAME[i]} "${COMMAND}" >&2
971 if [ ${PIPESTATUS[0]} -ne 0 ]; then
972 echo >&2 "`basename $0`: start_mgs_lnet() error:"\
973 "Failed to execute remote command to" \
974 "add module options to ${MGS_NODENAME[i]}!"\
975 "Check ${MODULE_CONFIG}!"
979 # Start lnet network in the MGS node
980 if ! check_lnet ${idx}; then
988 # Execute remote command to add lnet options lines to remote nodes'
989 # modprobe.conf/modules.conf and format(mkfs.lustre) Lustre targets
992 declare -a REMOTE_PID
993 declare -a REMOTE_CMD
997 if [ ${#HOST_NAME[@]} -eq 0 ]; then
998 verbose_output "There are no Lustre targets to be formatted."
1002 # Start lnet network in the MGS node
1003 if ! start_mgs_lnet; then
1007 for ((i = 0; i < ${#HOST_NAME[@]}; i++)); do
1008 # Construct the command line of mkfs.lustre
1009 if ! construct_mkfs_cmdline $i; then
1013 # create the mount point on the node
1014 COMMAND="mkdir -p ${MOUNT_POINT[i]}"
1015 verbose_output "Creating the mount point ${MOUNT_POINT[i]} on" \
1017 ${REMOTE} ${HOST_NAME[i]} "${COMMAND}" >&2
1018 if [ ${PIPESTATUS[0]} -ne 0 ]; then
1019 echo >&2 "`basename $0`: mass_config() error:"\
1020 "Failed to execute remote command to"\
1021 "create the mountpoint on ${HOST_NAME[i]}!"
1025 if ! is_mgs_node ${HOST_NAME[i]}; then
1026 # Execute remote command to add lnet options lines to
1027 # modprobe.conf/modules.conf
1028 COMMAND=$"echo \"${MODULE_OPTS[i]}\"|${MODULE_CONFIG}"
1029 verbose_output "Adding lnet module options to" \
1031 ${REMOTE} ${HOST_NAME[i]} "${COMMAND}" >&2
1032 if [ ${PIPESTATUS[0]} -ne 0 ]; then
1033 echo >&2 "`basename $0`: mass_config() error:"\
1034 "Failed to execute remote command to"\
1035 "add module options to ${HOST_NAME[i]}!"
1039 # Check lnet networks
1040 if ! check_lnet $i; then
1045 # Execute remote command to format Lustre target
1046 verbose_output "Formatting Lustre target ${DEVICE_NAME[i]} on ${HOST_NAME[i]}..."
1047 REMOTE_CMD[${pid_num}]="${REMOTE} ${HOST_NAME[i]} \"(${EXPORT_PATH} ${MKFS_CMD})\""
1048 verbose_output "Format command line is: ${REMOTE_CMD[${pid_num}]}"
1049 ${REMOTE} ${HOST_NAME[i]} "(${EXPORT_PATH} ${MKFS_CMD})" >&2 &
1050 REMOTE_PID[${pid_num}]=$!
1051 pid_num=${pid_num}+1
1055 # Wait for the exit status of the background remote command
1056 verbose_output "Waiting for the return of the remote command..."
1057 fail_exit_status=false
1058 for ((pid_num = 0; pid_num < ${#REMOTE_PID[@]}; pid_num++)); do
1059 wait ${REMOTE_PID[${pid_num}]}
1060 if [ ${PIPESTATUS[0]} -ne 0 ]; then
1061 echo >&2 "`basename $0`: mass_config() error: Failed"\
1062 "to execute \"${REMOTE_CMD[${pid_num}]}\"!"
1063 fail_exit_status=true
1067 if ${fail_exit_status}; then
1071 verbose_output "All the Lustre targets are formatted successfully!"
1075 # get_mntopts hostname device_name failovers
1076 # Construct the mount options of Lustre target @device_name in host @hostname
1079 local device_name=$2
1084 [ -n "${failovers}" ] && mnt_opts=defaults,noauto || mnt_opts=defaults
1086 # Execute remote command to check whether the device
1087 # is a block device or not
1088 ret_str=$(${REMOTE} ${host_name} \
1089 "[ -b ${device_name} ] && echo block || echo loop" 2>&1)
1090 if [ ${PIPESTATUS[0]} -ne 0 -a -n "${ret_str}" ]; then
1091 echo "`basename $0`: get_mntopts() error:" \
1092 "remote command to ${host_name} error: ${ret_str}"
1096 if [ -z "${ret_str}" ]; then
1097 echo "`basename $0`: get_mntopts() error: remote error:" \
1098 "No results from remote!" \
1099 "Check network connectivity between the local host and ${host_name}!"
1103 [ "${ret_str}" != "${ret_str#*loop}" ] && mnt_opts=${mnt_opts},loop
1109 # Execute remote command to modify /etc/fstab to add the new Lustre targets
1112 local mntent mntopts device_name
1115 if ! ${MODIFY_FSTAB}; then
1119 for ((i = 0; i < ${#HOST_NAME[@]}; i++)); do
1120 verbose_output "Modify /etc/fstab of host ${HOST_NAME[i]}"\
1121 "to add Lustre target ${DEVICE_NAME[i]}"
1122 mntent=${DEVICE_NAME[i]}"\t\t"${MOUNT_POINT[i]}"\t\t"${FS_TYPE}
1125 if [ -n "${MOUNT_OPTIONS[i]}" ]; then
1126 # The mount options already specified in the csv file.
1127 mntopts=${MOUNT_OPTIONS[i]}
1129 mntopts=$(get_mntopts ${HOST_NAME[i]} ${DEVICE_NAME[i]}\
1131 if [ ${PIPESTATUS[0]} -ne 0 ]; then
1132 echo >&2 "${mntopts}"
1137 mntent=${mntent}"\t"${mntopts}"\t"0" "0
1138 verbose_output "`echo -e ${mntent}`"
1140 # Execute remote command to modify /etc/fstab
1141 device_name=${DEVICE_NAME[i]//\//\\/}
1142 COMMAND=". @scriptlibdir@/lc_common.sh; \
1143 sed -i \"/^${device_name}\t/d\" \$(fcanon /etc/fstab); \
1144 echo -e \"${mntent}\" >> \$(fcanon /etc/fstab)"
1145 ${REMOTE} ${HOST_NAME[i]} "${COMMAND}" >&2
1146 if [ ${PIPESTATUS[0]} -ne 0 ]; then
1147 echo >&2 "`basename $0`: modify_fstab() error:"\
1148 "Failed to modify /etc/fstab of host ${HOST_NAME[i]}"\
1149 "to add Lustre target ${DEVICE_NAME[i]}!"
1158 # Check the csv file
1159 if ! check_file $1; then
1163 # Get the list of nodes to be operated on
1164 NODES_TO_USE=$(get_nodelist)
1165 [ ${PIPESTATUS[0]} -ne 0 ] && echo >&2 "${NODES_TO_USE}" && exit 1
1167 # Check the node list
1168 check_nodelist ${NODES_TO_USE} || exit 1
1170 if ${VERIFY_CONNECT}; then
1171 # Check the network connectivity and hostnames
1172 echo "`basename $0`: Checking the cluster network connectivity"\
1174 if ! ${VERIFY_CLUSTER_NET} ${NODELIST_OPT} ${VERBOSE_OPT} ${CSV_FILE}; then
1177 echo "`basename $0`: Check the cluster network connectivity"\
1182 if ${CONFIG_MD_LVM}; then
1183 # Configure Linux MD/LVM devices
1184 echo "`basename $0`: Configuring Linux MD/LVM devices..."
1185 if ! ${SCRIPT_CONFIG_MD} ${NODELIST_OPT} ${VERBOSE_OPT} ${CSV_FILE}; then
1189 if ! ${SCRIPT_CONFIG_LVM} ${NODELIST_OPT} ${VERBOSE_OPT} ${CSV_FILE}; then
1192 echo "`basename $0`: Configure Linux MD/LVM devices OK!"
1196 # Configure the Lustre cluster
1197 echo "`basename $0`: ******** Lustre cluster configuration START ********"
1198 if ! get_items ${CSV_FILE}; then
1202 if ! check_mgs; then
1206 if ! mass_config; then
1210 if ! modify_fstab; then
1214 # Produce HA software's configuration files
1215 if ! config_ha; then
1220 echo "`basename $0`: ******** Lustre cluster configuration END **********"