1 *************************************************************************
3 * Myrinet Express Lustre Networking Driver (MXLND) documentation *
5 *************************************************************************
9 MXLND provides support for Myricom's Myrinet Express (MX) communication
12 MXLND may be used with either MX-10G or MX-2G. See MX's README for
17 1. Configuring and compiling
21 1. Systems with different page sizes
23 3. MX endpoint collision
31 MXLND is supported on Linux 2.6. It may be possible to run it on 2.4,
32 but it has not been tested. MXLND requires Myricom's MX version 1.2.8
33 or higher. See MX's README for the supported list of processors.
35 MXLND requires the optional MX kernel library interface. MX must be compiled
36 with --enable-kernel-lib.
38 1. Configuring and compiling
40 MXLND should be already integrated into the Lustre build process. To
41 build MXLND, you will need to set the path to your MX installation
42 in Lustre's ./configure:
46 replacing /opt with the actual path. Configure will check to ensure that
47 the MX version has the required functions. If not, it will fail to build.
48 To check if MXLND built, look for:
50 checking whether to enable Myrinet MX support... yes
52 in configure's output or the presence of Makefile in
53 $LUSTRE/lnet/klnds/mxlnd.
57 MXLND supports a number of load-time parameters using Linux's module
58 parameter system. On our test systems, we created the following file:
60 /etc/modprobe.d/kmxlnd
62 On some (older?) systems, you may need to modify /etc/modprobe.conf.
64 The available options are:
66 n_waitd # of completion daemons
67 max_peers maximum number of peers that may connect
68 cksum set non-zero to enable small message (< 4KB) checksums
69 ntx # of total tx message descriptors
70 credits # concurrent sends to a single peer
71 board index value of the Myrinet board (NIC)
73 polling Use 0 to block (wait). A value > 0 will poll that many times before blocking
75 You may want to vary the options to obtain the optimal performance for your
78 n_waitd sets the number of threads that process completed MX requests
79 (sends and receives). In our testing, the default of 1 performed best.
81 max_peers tells MXLND the upper limit of machines that it will need to
82 communicate with. This affects how many receives it will pre-post and each
83 receive will use one page of memory. Ideally, on clients, this value will
84 be equal to the total number of Lustre servers (MDS and OSS). On servers,
85 it needs to equal the total number of machines in the storage system.
87 cksum turns on small message checksums. It can be used to aid in trouble-
88 shooting. MX also provides an optional checksumming feature which can check
89 all messages (large and small). See the MX README for details.
91 ntx is the number of total sends in flight from this machine. In actuality,
92 MXLND reserves half of them for connect messages so make this value twice as large
93 as you want for the total number of sends in flight.
95 credits is the number of in-flight messages for a specific peer. This is part
96 of the flow-control system in Lustre. Increasing this value may improve performance
97 but it requires more memory since each message requires at least one page.
99 board is the index of the Myricom NIC. Hosts can have multiple Myricom NICs
100 and this identifies which one MXLND should use.
102 ep_id is the MX endpoint ID. Each process that uses MX is required to have at
103 least one MX endpoint to access the MX library and NIC. The ID is a simple index
104 starting at 0. When used on a server, the server will attempt to use this end-
105 point. When used on a client, it specifies the endpoint to connect to on the
108 polling determines whether this host will poll or block for MX request com-
109 pletions. A value of 0 blocks and any positive value will poll that many times
110 before blocking. Since polling increases CPU usage, we suggest you set this to
111 0 on the client and experiment with different values for servers.
113 =====================
114 II. MXLND Performance
115 =====================
117 On MX-2G systems, MXLND should easily saturate the link and use minimal CPU
118 (5-10% for read and write operations). On MX-10G systems, MXLND can saturate
119 the link and use moderate CPU resources (20-30% for read and write operations).
120 MX-10G relies on PCI-Express which is relatively new and performance varies
121 considerably by processor, motherboard and PCI-E chipset. Refer to Myricom's
122 website for the latest DMA read/write performance results by motherboard. The
123 DMA results will place an upper-bound on MXLND performance.
129 1. Systems with different page sizes
131 MXLND will set the maximum small message size equal to the kernel's page size.
132 This means that machines running MXLND that have different page sizes are not
133 able to communicate with each other. If you wish to run MXLND in this case,
134 send email to help@myri.com.
138 At this time, the MXLND does not support more than one interface at a time.
139 Thus, a single Lustre router cannot route between two MX-10G, between two
140 MX-2G, or between MX-10G and MX-2G fabrics.
142 3. MX endpoint collision
144 Each process that uses MX is required to have at least one MX endpoint to
145 access the MX library and NIC. Other processes may need to use MX and no two
146 processes can use the same endpoint ID. MPICH-MX dynamically chooses one at
147 MPI startup and should not interfere with MXLND. Sockets-MX, on the other hand,
148 is hard coded to use 0 for its ID. If it is possible that anyone will want to
149 run Sockets-MX on this system, use a non-0 value for MXLND's endpoint ID.
156 MXLND is copyright (C) 2006 of Myricom, Inc.
158 MXLND is part of Lustre, http://www.lustre.org.
160 MXLND is free software; you can redistribute it and/or modify it under the
161 terms of version 2 of the GNU General Public License as published by the Free
164 MXLND is distributed in the hope that it will be useful, but WITHOUT ANY
165 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
166 PARTICULAR PURPOSE. See the GNU General Public License for more details.
168 You should have received a copy of the GNU General Public License along with
169 Lustre; if not, write to the Free Software Foundation, Inc., 675 Mass Ave,
170 Cambridge, MA 02139, USA.
176 If you have questions about MXLND, please contact help@myri.com.