4 Before any other interaction can take place between a client and a
5 Lustre file system the client must 'mount' the file system, and Lustre
6 services must already be started on the servers. A file system
7 mount may be initiated via the 'mount()' system call, passing the
8 name of the MGS and Lustre filesystem to the kernel. The Lustre
9 client exchanges a series of messages with the servers, beginning with
10 messages that retrieve the file system configuration <<llog>> from the
11 management server (MGS). This provides the client with the identities
12 of all the metadata servers (MDSs) and targets (MDTs) as well as all
13 the object storage servers (OSSs) and targets (OSTs). The client then
14 sequences through each of the targets exchanging additional messages
15 to initiate the connections with them. The following sections present
16 the details of the Lustre operations that accomplish the file system
19 Messages Between the Client and the MGS
20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
22 In order to be able to mount the Lustre file system the client needs
23 to know the identities of the various servers and targets so that it
24 can initiate connections to them. The following sequence of operations
27 .Connect RPCs Between the Client and the MGS
28 [[client-mgs-connect-rpcs]
29 image::client_mgs_connect_rpcs.png["Connect RPCs Between the Client and the MGS", height=600]
31 //////////////////////////////////////////////////////////////////////
32 The client_mgs_connect_rpcs.png diagram resembles this text art:
35 Step Client MGS MDT OST
36 ------- ------- --- ---
37 1 MGS_CONNECT-------------------->
38 2 <--------------------MGS_CONNECT
39 3 LDLM_ENQUEUE------------------->
40 4 <-------------------LDLM_ENQUEUE
41 5 LLOG_ORIGIN_HANDLE_CREATE------>
42 6 <------LLOG_ORIGIN_HANDLE_CREATE
43 7 LDLM_ENQUEUE------------------->
44 8 <-------------------LDLM_ENQUEUE
45 9 LLOG_ORIGIN_HANDLE_CREATE------>
46 10 <------LLOG_ORIGIN_HANDLE_CREATE
47 11 LLOG_ORIGIN_HANDLE_READ_HEADER->
48 12 <-LLOG_ORIGIN_HANDLE_READ_HEADER
49 13 LLOG_ORIGIN_HANDLE_NEXT_BLOCK->
50 14 <-LLOG_ORIGIN_HANDLE_NEXT_BLOCK
51 15 LDLM_ENQUEUE------------------->
52 16 <-------------------LDLM_ENQUEUE
53 17 MGS_CONFIG_READ---------------->
54 18 <----------------MGS_CONFIG_READ
55 19 LDLM_ENQUEUE------------------->
56 20 <-------------------LDLM_ENQUEUE
57 21 LLOG_ORIGIN_HANDLE_CREATE------>
58 22 <------LLOG_ORIGIN_HANDLE_CREATE
59 23 LLOG_ORIGIN_HANDLE_READ_HEADER->
60 24 <-LLOG_ORIGIN_HANDLE_READ_HEADER
61 //////////////////////////////////////////////////////////////////////
63 Prior to any other interaction between a client and a Lustre server
64 (or between two servers) the client must establish a 'connection'. The
65 connection establishes shared state between the two hosts. On the
66 client this connection state information is called an 'import', and
67 there is an import on the client for each target it connects to. On
68 the server this connection state is referred to as an 'export', and
69 again the server has an export for each client that has connected to
72 *1 - Client issues an MGS_CONNECT to the MGS.*
74 .MGS_CONNECT Request Packet Structure
75 image::mgs-connect-request.png["MGS_CONNECT Request Packet Structure",height=50]
77 //////////////////////////////////////////////////////////////////////
78 The mgs-connect-request.png diagram resembles this text art:
81 --request--------------------------------------------------
82 | ptlrpc_body | target_uuid | client_uuid | lustre_handle |
83 -----------------------------------------------------------
86 //////////////////////////////////////////////////////////////////////
88 See <<mgs-connect-rpc>>.
90 The client begins by carrying out the <<mgs-connect-rpc,MGS_CONNECT>>
91 Lustre RPC, which establishes the connection (creates the import and
92 the export) between the client and the MGS. The connect message from
93 the client includes a 'lustre_handle' to uniquely identify itself
94 (note that this is distinct from the 'pb_handle' field of the
95 structure described in <<struct-ptlrpc-body>>, which will be 0 in a
96 connection request). The target notes the 'lustre_handle' value in its
97 'export'. The connection data from the client also proposes the set of
98 <<connect-flags,connection flags>> appropriate to connecting to an
101 .Flags for the client connection to an MGS
104 | obd_connect_data->ocd_connect_flags
105 | OBD_CONNECT_VERSION
108 | OBD_CONNECT_IMP_RECOV
109 | OBD_CONNECT_MNE_SWAB
110 | OBD_CONNECT_PINGLESS
113 *2 - The MGS sends an MGS_CONNECT reply to the client.*
115 .MGS_CONNECT Reply Packet Structure
116 image::mgs-connect-reply.png["MGS_CONNECT Reply Packet Structure",height=50]
118 //////////////////////////////////////////////////////////////////////
119 The mgs-connect-reply.png diagram resembles this text art:
122 --reply---------------------------
123 | ptlrpc_body | obd_connect_data |
124 ----------------------------------
125 //////////////////////////////////////////////////////////////////////
127 The MGS's reply to the connection request will include the
128 'lustre_handle' (in the 'ptlrpc_body' structure's 'pb_handle' field)
129 that the client will use (in its 'pb_handle') to identify this target
130 in subsequent messages. The MGS also replies with the same set of
133 Once the connection is established the client gets configuration
134 information for the file system from the MGS in four stages. First,
135 the two exchange messages establishing the file system wide security
136 policy that will be followed in all subsequent communications.
138 Each time the client is going to read information from server storage
139 it needs to first acquire the appropriate lock. Since the client is
140 only reading data, the locks will be 'concurrent read' locks. The
141 LDLM_ENQUEUE command communicates this lock request to the MGS
142 target. The request identifies the target via the connection-handle
143 from the connection reply, and identifies the client (itself) with the
144 client-handle from its original connection request. The MGS's reply
145 grants that lock, if appropriate. If other clients were making some
146 sort of modification to the MGS data then the lock exchange might
147 result in a delay while the client waits. More details about the
148 behavior of the <<ldlm,Distributed Lock Manager>> are in that
151 *3 - The Client sends an LDLM_ENQUEUE to the MGS.*
153 .LDLM_ENQUEUE Request Packet Structure
154 image::ldlm-enqueue-request.png["LDLM_ENQUEUE Request Packet Structure",height=50]
156 //////////////////////////////////////////////////////////////////////
157 The ldlm-enqueue-request.png diagram resembles this text art:
160 --request---------------------
161 | ptlrpc_body | ldlm_request |
162 ------------------------------
163 //////////////////////////////////////////////////////////////////////
165 See <<ldlm-enqueue-rpc>>.
167 The client seeks a 'concurrent read' lock on the MGS.
169 *4 - The MGS sends an LDLM_ENQUEUE reply to the client.*
171 .LDLM_ENQUEUE Reply Packet Structure
172 image::ldlm-enqueue-reply.png["LDLM_ENQUEUE Reply Packet Structure",height=50]
174 //////////////////////////////////////////////////////////////////////
175 The ldlm-enqueue-reply.png diagram resembles this text
179 --reply---------------------
180 | ptlrpc_body | ldlm_reply |
181 ----------------------------
182 //////////////////////////////////////////////////////////////////////
184 The MGS grants that lock.
186 *5 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*
188 .LLOG_ORIGIN_HANDLE_CREATE Request Packet Structure
189 image::llog-origin-handle-create-request.png["LLOG_ORIGIN_HANDLE_CREATE Request Packet Structure",height=50]
191 //////////////////////////////////////////////////////////////////////
192 The llog-origin-handle-create-request.png diagram resembles this text
195 LLOG_ORIGIN_HANDLE_CREATE:
196 --request----------------------------
197 | ptlrpc_body | llogd_body | string |
198 -------------------------------------
199 //////////////////////////////////////////////////////////////////////
201 See <<llog-origin-handle-create>>.
203 The first LLOG_ORIGIN_HANDLE_CREATE operation asks
204 for the security configuration file ("lfs-sptlrpc"). <<security>>
207 *6 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*
210 .LLOG_ORIGIN_HANDLE_CREATE Reply Packet Structure
211 image::llog-origin-handle-create-reply.png["LLOG_ORIGIN_HANDLE_CREATE Reply Packet Structure",height=50]
213 //////////////////////////////////////////////////////////////////////
214 The llog-origin-handle-create-reply.png diagram resembles this text
217 LLOG_ORIGIN_HANDLE_CREATE:
218 --reply---------------------
219 | ptlrpc_body | llogd_body |
220 ----------------------------
221 //////////////////////////////////////////////////////////////////////
223 In this case there is nothing to be done for security. That is,
224 subsequent messages will all use an "empty security flavor" and no
225 encryption will take place. In this case the MGS's reply ('pb_status'
226 == -2, ENOENT) indicated that there was no such file, so nothing
229 Next, the client gets a configuration log (See <<llog>>) starting with
230 a bitmap instructing it as to which among the configuration records on
231 the MGS it needs. Another LDLM_ENQUEUE and LLOG_ORIGIN_HANDLE_CREATE
232 pair of operations identifies the configuration client data
235 *7 - The Client sends an LDLM_ENQUEUE to the MGS.*
237 See <<ldlm-enqueue-rpc>>.
239 The client again seeks a 'concurrent read' lock on the MGS.
241 *8 - The MGS sends an LDLM_ENQUEUE reply to the client.*
243 The MGS grants that lock.
245 *9 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*
247 See <<llog-origin-handle-create>>.
249 The client asks for the 'lfs-client' log file, which holds a bitmap
250 indicating the available configuration records.
252 *10 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*
254 In this case there is data to read. The LLOG_ORIGIN_HANDLE_CREATE
255 reply identifies the actual object of interest on the MGS via the
256 'llog_logid' field in the 'struct llogd_body'.
258 *11 - The Client sends an LLOG_ORIGIN_HANDLE_READ_HEADER to the MGS.*
260 .LLOG_ORIGIN_HANDLE_READ_HEADER Request Packet Structure
261 image::llog-origin-handle-read-header-request.png["LLOG_ORIGIN_HANDLE_READ_HEADER Request Packet Structure",height=50]
263 //////////////////////////////////////////////////////////////////////
264 The llog-origin-handle-read-header-request.png diagram resembles this
267 LLOG_ORIGIN_HANDLE_READ_HEADER:
268 --request-------------------
269 | ptlrpc_body | llogd_body |
270 ----------------------------
271 //////////////////////////////////////////////////////////////////////
273 See <<llog-origin-handle-read-header>>.
275 The client asks for that log file's header. The MGS stores
276 configuration data in log records. A header at the beginning of
277 "lfs-client" uses a bitmap to identify the log records that are
278 actually needed. The header includes both which records to retrieve
279 and how large those records are. The LLOG_ORIGIN_HANDLE_READ_HEADER
280 request uses the 'llog_logid' to identify desired log file, and the
281 reply provides the bitmap and size information identifying the
282 records that are actually needed.
284 *12 - The MGS sends an LLOG_ORIGIN_HANDLE_READ_HEADER reply to the client.*
286 .LLOG_ORIGIN_HANDLE_READ_HEADER Reply Packet Structure
287 image::llog-origin-handle-read-header-reply.png["LLOG_ORIGIN_HANDLE_READ_HEADER Reply Packet Structure",height=50]
289 //////////////////////////////////////////////////////////////////////
290 The llog-origin-handle-read-header-reply.png diagram resembles this
293 LLOG_ORIGIN_HANDLE_READ_HEADER:
294 --reply--------------------------
295 | ptlrpc_body | llog_log_header |
296 ---------------------------------
297 //////////////////////////////////////////////////////////////////////
299 The MGs responds with the header, which holds the actual bitmap.
301 *13 - The Client sends an LLOG_ORIGIN_HANDLE_NEXT_BLOCK to the MGS.*
303 .LLOG_ORIGIN_HANDLE_NEXT_BLOCK Request Packet Structure
304 image::llog-origin-handle-next-block-request.png["LLOG_ORIGIN_HANDLE_NEXT_BLOCK Request Packet Structure",height=50]
306 //////////////////////////////////////////////////////////////////////
307 The llog-origin-handle-next-block-request.png diagram resembles this
310 LLOG_ORIGIN_HANDLE_NEXT_BLOCK:
311 --request-------------------
312 | ptlrpc_body | llogd_body |
313 ----------------------------
314 //////////////////////////////////////////////////////////////////////
316 See <<llog-origin-handle-next-block>>.
318 The client asks for the configuration record.
320 *14 - The MGS sends an LLOG_ORIGIN_HANDLE_NEXT_BLOCK reply to the client.*
322 .LLOG_ORIGIN_HANDLE_NEXT_BLOCK Reply Packet Structure
323 image::llog-origin-handle-next-block-reply.png["LLOG_ORIGIN_HANDLE_NEXT_BLOCK Reply Packet Structure",height=50]
325 //////////////////////////////////////////////////////////////////////
326 The llog-origin-handle-next-block-reply.png diagram resembles this
329 LLOG_ORIGIN_HANDLE_NEXT_BLOCK:
330 --reply------------------------------
331 | ptlrpc_body | llogd_body | eadata |
332 -------------------------------------
333 //////////////////////////////////////////////////////////////////////
335 The MGS replies with the details of where in the configuration data to
336 find the desired details.
338 Knowing the specific configuration records it wants, the client then
339 proceeds to retrieve them. This requires another LDLM_ENQUEUE
340 operation, followed this time by the MGS_CONFIG_READ operation, which
341 get the UUIDs for the servers and targets from the configuration log
344 *15 - The Client sends an LDLM_ENQUEUE to the MGS.*
346 See <<ldlm-enqueue-rpc>>.
348 The client again seeks a 'concurrent read' lock on the MGS.
350 *16 - The MGS sends an LDLM_ENQUEUE reply to the client.*
352 The MGS grants that lock.
354 *17 - The Client sends an MGS_CONFIG_READ to the MGS.*
356 .MGS_CONFIG_READ Request Packet Structure
357 image::mgs-config-read-request.png["MGS_CONFIG_READ Request Packet Structure",height=50]
359 //////////////////////////////////////////////////////////////////////
360 The mgs-config-read-request.png diagram resembles this text art:
363 --request------------------------
364 | ptlrpc_body | mgs_config_body |
365 ---------------------------------
366 //////////////////////////////////////////////////////////////////////
368 See <<mgs-config-read-rpc>>.
370 The client identifies the desired record in the 'lfs-cliir' file,
371 which contains the current details of the configuration for this file
374 *18 - The MGS sends an MGS_CONFIG_READ reply to the client.*
376 .MGS_CONFIG_READ Reply Packet Structure
377 image::mgs-config-read-reply.png["MGS_CONFIG_READ Reply Packet Structure",height=50]
379 //////////////////////////////////////////////////////////////////////
380 The mgs-config-read-reply.png diagram resembles this text art:
383 --reply-------------------------
384 | ptlrpc_body | mgs_config_res |
385 --------------------------------
386 //////////////////////////////////////////////////////////////////////
388 The MGS responds with the actual configuration data. This gives the
389 client the list of all the servers and targets it will need to
392 Finally, the client reads the cluster wide configuration data (the
393 sort that might be set at the client command line with a 'lctl
394 conf_param' command). The following paragraphs go into these four
395 stages in more detail.
397 *19 - The Client sends an LDLM_ENQUEUE to the MGS.*
399 See <<ldlm-enqueue-rpc>>.
401 The client again seeks a 'concurrent read' lock on the MGS.
403 *20 - The MGS sends an LDLM_ENQUEUE reply to the client.*
405 The MGS grants that lock.
407 *21 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*
409 See <<llog-origin-handle-create>>.
411 The client asks for the 'params' log file.
413 *22 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*
415 The MGS responds that the log file is available.
417 *23 - The Client sends an LLOG_ORIGIN_HANDLE_READ_HEADER to the MGS.*
419 See <<llog-origin-handle-read-header>>.
421 The client asks for that log file's header.
423 *24 - The MGS sends an LLOG_ORIGIN_HANDLE_READ_HEADER reply to the client.*
425 The MGs responds with the header, which holds the actual bitmap. In
426 this case there are no 'params' to report.
428 Messages Between the Client and the MDSs
429 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
431 After the foregoing interaction with the MGS the client has a list of
432 the MDSs and MDTs in the file system. Next, the client invokes four
433 Lustre operations with each MDT on the list.
435 .Connect RPCs Between the Client and each MDT
436 [[client-mdt-connect-rpcs]
437 image::client_mdt_connect_rpcs.png["Connect RPCs Between the Client and each MDT", height=200]
439 //////////////////////////////////////////////////////////////////////
440 The client_mdt_connect_rpcs.png diagram resembles this text art:
443 Step Client MGS MDT OST
444 ------- ------- -------- ------
445 1 MDS_CONNECT-------------------->
446 2 <--------------------MDS_CONNECT
447 3 MDS_STATFS--------------------->
448 4 <---------------------MDS_STATFS
449 5 MDS_GETSTATUS------------------>
450 6 <------------------MDS_GETSTATUS
451 7 MDS_GETATTR-------------------->
452 8 <--------------------MDS_GETATTR
453 //////////////////////////////////////////////////////////////////////
455 The MDS_CONNECT operation establishes a connection between the client
456 and a specific target (MDT) on an MDS. Thus, if an MDS has multiple
457 targets, there is a separate MDS_CONNECT operation for each. This
458 creates an import for the target on the client and an export for the
459 client and target on the MDS.
461 *1 - Client issues an MDS_CONNECT to each MDT.*
463 .MDS_CONNECT Request Packet Structure
464 image::mds-connect-request.png["MDS_CONNECT Request Packet Structure",height=50]
466 //////////////////////////////////////////////////////////////////////
467 The mds-connect-request.png diagram resembles this text art:
470 --request--------------------------------------------------
471 | ptlrpc_body | target_uuid | client_uuid | lustre_handle |
472 -----------------------------------------------------------
474 ---------------------
475 //////////////////////////////////////////////////////////////////////
477 See <<mds-connect-rpc>>.
479 As with the connect operation for the MGS, the connect message from
480 the client includes a 'lustre_handle' to uniquely identify this
481 connection. The connection data from the client also proposes the set
482 of <<connect-flags,connection flags>> appropriate to connecting to an
483 MDS. The following are the flags always included.
485 .Always included flags for the client connection to an MDS
488 | obd_connect_data->ocd_connect_flags
490 | OBD_CONNECT_VERSION
494 | OBD_CONNECT_NODEVOH
495 | OBD_CONNECT_ATTRFID
496 | OBD_CONNECT_CANCELSET
498 | OBD_CONNECT_RMT_CLIENT
499 | OBD_CONNECT_RMT_CLIENT_FORCE
500 | OBD_CONNECT_BRW_SIZE
501 | OBD_CONNECT_MDS_CAPA
502 | OBD_CONNECT_OSS_CAPA
503 | OBD_CONNECT_MDS_MDS
505 | LRU_RESIZE_CONNECT_FLAG
510 | OBD_CONNECT_64BITHASH
511 | OBD_CONNECT_JOBSTATS
512 | OBD_CONNECT_EINPROGRESS
513 | OBD_CONNECT_LIGHTWEIGHT
515 | OBD_CONNECT_LVB_TYPE
516 | OBD_CONNECT_LAYOUTLOCK
517 | OBD_CONNECT_PINGLESS
518 | OBD_CONNECT_MAX_EASIZE
519 | OBD_CONNECT_FLOCK_DEAD
520 | OBD_CONNECT_DISP_STRIPE
522 | OBD_CONNECT_OPEN_BY_FID
523 | OBD_CONNECT_DIR_STRIPE
526 .Optional flags for the client connection to an MDS
529 | obd_connect_data->ocd_connect_flags
531 | OBD_CONNECT_LRU_RESIZE
537 | OBD_CONNECT_RMT_CLIENT_FORCE
540 *2 - The MDS sends an MDS_CONNECT reply to the client.*
542 .MDS_CONNECT Reply Packet Structure
543 image::mds-connect-reply.png["MDS_CONNECT Reply Packet Structure",height=50]
545 //////////////////////////////////////////////////////////////////////
546 The mds-connect-reply.png diagram resembles this text art:
549 --reply---------------------------
550 | ptlrpc_body | obd_connect_data |
551 ----------------------------------
552 //////////////////////////////////////////////////////////////////////
554 The MDS replies to the connect message with a subset of the flags
555 proposed by the client, and the client notes those values in its
556 import. As with the MGS, the MDS's reply to the connection request
557 will include a 'lustre_handle' in the 'pb_handle' field that the
558 client will use to identify this target in subsequent messages.
560 *3 - The Client sends an MDS_STATFS to the MDS.*
562 .MDS_STATFS Request Packet Structure
563 image::mds-statfs-request.png["MDS_STATFS Request Packet Structure",height=50]
565 //////////////////////////////////////////////////////////////////////
566 The mds-statfs-request.png diagram resembles this text art:
572 //////////////////////////////////////////////////////////////////////
574 See <<mds-statfs-rpc>>.
576 The client next uses an MDS_STATFS operation to request 'statfs'
577 information from the target, and that data is returned in the reply
578 message. The actual fields closely resemble the results of a 'statfs'
579 system call. See the 'obd_statfs' structure in the <<data-structs,Data
580 Structures and Defines Section>>.
582 *4 - The MDS sends an MDS_STATFS reply to the client.*
584 .MDS_STATFS Reply Packet Structure
585 image::mds-statfs-reply.png["MDS_STATFS Reply Packet Structure",height=50]
587 //////////////////////////////////////////////////////////////////////
588 The mds-statfs-reply.png diagram resembles this text art:
591 --reply---------------------
592 | ptlrpc_body | obd_statfs |
593 ----------------------------
594 //////////////////////////////////////////////////////////////////////
596 The MDS replies with the 'statfs' information.
598 *5 - The Client sends an MDS_GETSTATUS to the MDS.*
600 .MDS_GETSTATUS Request Packet Structure
601 image::mds-getstatus-request.png["MDS_GETSTATUS Request Packet Structure",height=50]
603 //////////////////////////////////////////////////////////////////////
604 The mds-getstatus-request.png diagram resembles this text art:
607 --request-----------------
608 | ptlrpc_body | mdt_body |
609 --------------------------
610 //////////////////////////////////////////////////////////////////////
612 See <<mds-getstatus-rpc>>.
614 The client uses the MDS_GETSTATUS operation to request information
615 about the mount point of the file system.
617 *6 - The MDS sends an MDS_GETSTATUS reply to the client.*
619 .MDS_GETSTATUS Reply Packet Structure
620 image::mds-getstatus-reply.png["MDS_GETSTATUS Reply Packet Structure",height=50]
622 //////////////////////////////////////////////////////////////////////
623 The mds-getstatus-reply.png diagram resembles this text art:
626 --reply-------------------
627 | ptlrpc_body | mdt_body |
628 --------------------------
629 //////////////////////////////////////////////////////////////////////
631 The server reply contains the 'fid' of the root directory of the file
632 system being mounted. If there is a security policy the capabilities
633 of that security policy are included in the reply.
635 *7 - The Client sends an MDS_GETATTR to the MDS.*
637 .MDS_GETATTR Request Packet Structure
638 image::mds-getattr-request.png["MDS_GETATTR Request Packet Structure",height=50]
640 //////////////////////////////////////////////////////////////////////
641 The mds-getattr-request.png diagram resembles this text art:
644 --request-------------------------------
645 | ptlrpc_body | mdt_body | lustre_capa |
646 ----------------------------------------
647 //////////////////////////////////////////////////////////////////////
649 See <<mds-getattr-rpc>>.
651 The client then uses the MDS_GETATTR operation to get get further
652 information about the root directory of the file system. The request
653 message includes the above fid.
655 *8 - The MDS sends an MDS_GETATTR reply to the client.*
657 .MDS_GETATTR Reply Packet Structure
658 image::mds-getattr-reply.png["MDS_GETATTR Reply Packet Structure",height=50]
660 //////////////////////////////////////////////////////////////////////
661 The mds-getattr-reply.png diagram resembles this text art:
664 --reply------------------------------------------------
665 | ptlrpc_body | mdt_body | MDS_MD | ACL | lustre_capa |
666 -------------------------------------------------------
669 //////////////////////////////////////////////////////////////////////
671 It will also include the security capability (if appropriate). The
672 reply also holds the same fid, and in this case the 'mdt_body' has
673 several additional fields filled in. These include the mtime, atime,
674 ctime, mode, uid, and gid. It also includes the size of the extended
675 attributes and the size of the ACL information. The reply message also
676 includes the extended attributes and the ACL. From the extended
677 attributes the client can find out about striping information for the
680 Messages Between the Client and the OSSs
681 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
683 Additional CONNECT messages flow between the client and each OST
684 enumerated by the MGS.
686 .Connect RPCs Between the Client and each OST
687 [[client-ost-connect-rpcs]
688 image::client_ost_connect_rpcs.png["Connect RPCs Between the Client and each OST", height=75]
690 //////////////////////////////////////////////////////////////////////
691 The client_ost_connect_rpcs.png diagram resembles this text art:
694 Step Client MGS MDT OST
695 ------- ------- -------- ------
696 1 OST_CONNECT----------------------------------->
697 2 <-----------------------------------OST_CONNECT
698 //////////////////////////////////////////////////////////////////////
700 The OST_CONNECT operation establishes a connection between the client
701 and a specific target (OST) on an OST. Thus, if an OST has multiple
702 targets, there is a separate OST_CONNECT operation for each. This
703 creates an import for the target on the client and an export for the
704 client and target on the OST.
706 *1 - Client issues an OST_CONNECT to each OST.*
708 .OST_CONNECT Request Packet Structure
709 image::ost-connect-request.png["OST_CONNECT Request Packet Structure",height=50]
711 //////////////////////////////////////////////////////////////////////
712 The ost-connect-request.png diagram resembles this text art:
715 --request--------------------------------------------------
716 | ptlrpc_body | target_uuid | client_uuid | lustre_handle |
717 -----------------------------------------------------------
719 ---------------------
720 //////////////////////////////////////////////////////////////////////
722 See <<ost-connect-rpc>>.
724 As with the connect operations for the MGS and MDTs, the connect
725 message from the client includes a 'lustre_handle' to uniquely
726 identify this connection. The connection data from the client also
727 proposes the set of <<connect-flags,connection flags>> appropriate to
728 connecting to an OST. The following are the flags always included.
730 .Flags for the client connection to an OST
733 | obd_connect_data->ocd_connect_flags
735 | OBD_CONNECT_SRVLOCK
736 | OBD_CONNECT_VERSION
737 | OBD_CONNECT_REQPORTAL
738 | OBD_CONNECT_TRUNCLOCK
739 | OBD_CONNECT_RMT_CLIENT
740 | OBD_CONNECT_BRW_SIZE
741 | OBD_CONNECT_OSS_CAPA
742 | OBD_CONNECT_CANCELSET
744 | OBD_CONNECT_LRU_RESIZE
749 | OBD_CONNECT_LAYOUTLOCK
750 | OBD_CONNECT_64BITHASH
751 | OBD_CONNECT_MAXBYTES
752 | OBD_CONNECT_JOBSTATS
753 | OBD_CONNECT_EINPROGRESS
754 | OBD_CONNECT_LVB_TYPE
755 | OBD_CONNECT_PINGLESS
758 *2 - The OST sends an OST_CONNECT reply to the client.*
760 .OST_CONNECT Reply Packet Structure
761 image::ost-connect-reply.png["OST_CONNECT Reply Packet Structure",height=50]
763 //////////////////////////////////////////////////////////////////////
764 The ost-connect-reply.png diagram resembles this text art:
767 --reply---------------------------
768 | ptlrpc_body | obd_connect_data |
769 ----------------------------------
770 //////////////////////////////////////////////////////////////////////
772 The OST replies to the connect message with a subset of the flags
773 proposed by the client, and the client notes those values in its
774 import. The OST's reply to the connection request will include a UUID
775 that the server and client will both use to identify this connection
776 in subsequent messages. The flags that the server doesn't include are:
777 OBD_CONNECT_RMT_CLIENT, OBD_CONNECT_OSS_CAPA, and
778 OBD_CONNECT_PINGLESS.