4 Before any other interaction can take place between a client and a
5 Lustre file system the client must 'mount' the file system, and Lustre
6 services must already be started on the servers. A file system
7 mount may be initiated via the 'mount()' system call, passing the
8 name of the MGS and Lustre filesystem to the kernel. The Lustre
9 client exchanges a series of messages with the servers, beginning with
10 messages that retrieve the file system configuration <<llog>> from the
11 management server (MGS). This provides the client with the identities
12 of all the metadata servers (MDSs) and targets (MDTs) as well as all
13 the object storage servers (OSSs) and targets (OSTs). The client then
14 sequences through each of the targets exchanging additional messages
15 to initiate the connections with them. The following sections present
16 the details of the Lustre operations that accomplish the file system
19 Messages Between the Client and the MGS
20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
22 In order to be able to mount the Lustre file system the client needs
23 to know the identities of the various servers and targets so that it
24 can initiate connections to them. The following sequence of operations
27 .Connect RPCs Between the Client and the MGS
28 [[client-mgs-connect-rpcs]
29 image::client_mgs_connect_rpcs.png["Connect RPCs Between the Client and the MGS", height=600]
31 //////////////////////////////////////////////////////////////////////
32 The client_mgs_connect_rpcs.png diagram resembles this text art:
35 Step Client MGS MDT OST
36 ------- ------- --- ---
37 1 MGS_CONNECT-------------------->
38 2 <--------------------MGS_CONNECT
39 3 LDLM_ENQUEUE------------------->
40 4 <-------------------LDLM_ENQUEUE
41 5 LLOG_ORIGIN_HANDLE_CREATE------>
42 6 <------LLOG_ORIGIN_HANDLE_CREATE
43 7 LDLM_ENQUEUE------------------->
44 8 <-------------------LDLM_ENQUEUE
45 9 LLOG_ORIGIN_HANDLE_CREATE------>
46 10 <------LLOG_ORIGIN_HANDLE_CREATE
47 11 LLOG_ORIGIN_HANDLE_READ_HEADER->
48 12 <-LLOG_ORIGIN_HANDLE_READ_HEADER
49 13 LLOG_ORIGIN_HANDLE_NEXT_BLOCK->
50 14 <-LLOG_ORIGIN_HANDLE_NEXT_BLOCK
51 15 LDLM_ENQUEUE------------------->
52 16 <-------------------LDLM_ENQUEUE
53 17 MGS_CONFIG_READ---------------->
54 18 <----------------MGS_CONFIG_READ
55 19 LDLM_ENQUEUE------------------->
56 20 <-------------------LDLM_ENQUEUE
57 21 LLOG_ORIGIN_HANDLE_CREATE------>
58 22 <------LLOG_ORIGIN_HANDLE_CREATE
59 23 LLOG_ORIGIN_HANDLE_READ_HEADER->
60 24 <-LLOG_ORIGIN_HANDLE_READ_HEADER
61 //////////////////////////////////////////////////////////////////////
63 Prior to any other interaction between a client and a Lustre server
64 (or between two servers) the client must establish a 'connection'. The
65 connection establishes shared state between the two hosts. On the
66 client this connection state information is called an 'import', and
67 there is an import on the client for each target it connects to. On
68 the server this connection state is referred to as an 'export', and
69 again the server has an export for each client that has connected to
72 *1 - Client issues an MGS_CONNECT to the MGS.*
74 .MGS_CONNECT Request Packet Structure
75 image::mgs-connect-request.png["MGS_CONNECT Request Packet Structure",height=50]
77 //////////////////////////////////////////////////////////////////////
78 The mgs-connect-request.png diagram resembles this text art:
81 --request--------------------------------------------
82 | ptlrpc_body | obd_uuid | obd_uuid | lustre_handle |
83 -----------------------------------------------------
86 //////////////////////////////////////////////////////////////////////
88 The client begins by carrying out the <<mgs-connect-rpc,MGS_CONNECT>>
89 Lustre RPC, which establishes the connection (creates the
90 import and the export) between the client and the MGS. The connect
91 message from the client includes a 'lustre_handle' to uniquely
92 identify itself. Subsequent request messages to the MGS will refer
93 to that client-handle to verify that the client has already connected.
94 The connection data from the client also proposes the set of
95 <<connect-flags,connection flags>> appropriate to connecting to an MGS.
97 .Flags for the client connection to an MGS
100 | obd_connect_data->ocd_connect_flags
101 | OBD_CONNECT_VERSION
104 | OBD_CONNECT_IMP_RECOV
105 | OBD_CONNECT_MNE_SWAB
106 | OBD_CONNECT_PINGLESS
109 *2 - The MGS sends an MGS_CONNECT reply to the client.*
111 .MGS_CONNECT Reply Packet Structure
112 image::mgs-connect-reply.png["MGS_CONNECT Reply Packet Structure",height=50]
114 //////////////////////////////////////////////////////////////////////
115 The mgs-connect-reply.png diagram resembles this text art:
118 --reply---------------------------
119 | ptlrpc_body | obd_connect_data |
120 ----------------------------------
121 //////////////////////////////////////////////////////////////////////
123 The MGS's reply to the connection request will include the handle that
124 the server and client will both use to identify this connection in
125 subsequent messages. This is the 'connection-handle' (as opposed to
126 the client-handle mentioned a moment ago). The MGS also replies with
127 the same set of connection flags.
129 Once the connection is established the client gets configuration
130 information for the file system from the MGS in four stages. First,
131 the two exchange messages establishing the file system wide security
132 policy that will be followed in all subsequent communications.
134 Each time the client is going to read information from server storage
135 it needs to first acquire the appropriate lock. Since the client is
136 only reading data, the locks will be 'concurrent read' locks. The
137 LDLM_ENQUEUE command communicates this lock request to the MGS
138 target. The request identifies the target via the connection-handle
139 from the connection reply, and identifies the client (itself) with the
140 client-handle from its original connection request. The MGS's reply
141 grants that lock, if appropriate. If other clients were making some
142 sort of modification to the MGS data then the lock exchange might
143 result in a delay while the client waits. More details about the
144 behavior of the <<ldlm,Distributed Lock Manager>> are in that
147 *3 - The Client sends an LDLM_ENQUEUE to the MGS.*
149 .LDLM_ENQUEUE Request Packet Structure
150 image::ldlm-enqueue-request.png["LDLM_ENQUEUE Request Packet Structure",height=50]
152 //////////////////////////////////////////////////////////////////////
153 The ldlm-enqueue-request.png diagram resembles this text art:
156 --request---------------------
157 | ptlrpc_body | ldlm_request |
158 ------------------------------
159 //////////////////////////////////////////////////////////////////////
161 The client seeks a 'concurrent read' lock on the MGS.
163 *4 - The MGS sends an LDLM_ENQUEUE reply to the client.*
165 .LDLM_ENQUEUE Reply Packet Structure
166 image::ldlm-enqueue-reply.png["LDLM_ENQUEUE Reply Packet Structure",height=50]
168 //////////////////////////////////////////////////////////////////////
169 The ldlm-enqueue-reply.png diagram resembles this text
173 --reply---------------------
174 | ptlrpc_body | ldlm_reply |
175 ----------------------------
176 //////////////////////////////////////////////////////////////////////
178 The MGS grants that lock.
180 *5 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*
183 .LLOG_ORIGIN_HANDLE_CREATE Request Packet Structure
184 image::llog-origin-handle-create-request.png["LLOG_ORIGIN_HANDLE_CREATE Request Packet Structure",height=50]
186 //////////////////////////////////////////////////////////////////////
187 The llog-origin-handle-create-request.png diagram resembles this text
190 LLOG_ORIGIN_HANDLE_CREATE:
191 --request----------------------------
192 | ptlrpc_body | llogd_body | string |
193 -------------------------------------
194 //////////////////////////////////////////////////////////////////////
196 The first LLOG_ORIGIN_HANDLE_CREATE operation asks
197 for the security configuration file ("lfs-sptlrpc"). <<security>>
200 *6 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*
203 .LLOG_ORIGIN_HANDLE_CREATE Reply Packet Structure
204 image::llog-origin-handle-create-reply.png["LLOG_ORIGIN_HANDLE_CREATE Reply Packet Structure",height=50]
206 //////////////////////////////////////////////////////////////////////
207 The llog-origin-handle-create-reply.png diagram resembles this text
210 LLOG_ORIGIN_HANDLE_CREATE:
211 --reply---------------------
212 | ptlrpc_body | llogd_body |
213 ----------------------------
214 //////////////////////////////////////////////////////////////////////
216 In this case there is nothing to be done for security. That is,
217 subsequent messages will all use an "empty security flavor" and no
218 encryption will take place. In this case the MGS's reply ('pb_status'
219 == -2, ENOENT) indicated that there was no such file, so nothing
222 Next, the client gets a configuration <<llog>> starting with a bitmap
223 instructing it as to which among the configuration records on the MGS
224 it needs. Another LDLM_ENQUEUE and LLOG_ORIGIN_HANDLE_CREATE pair of
225 operations identifies the configuration client data ("lfs-client")
228 *7 - The Client sends an LDLM_ENQUEUE to the MGS.*
230 The client again seeks a 'concurrent read' lock on the MGS.
232 *8 - The MGS sends an LDLM_ENQUEUE reply to the client.*
234 The MGS grants that lock.
236 *9 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*
238 The client asks for the 'lfs-client' log file, which holds a bitmap
239 indicating the available configuration records.
241 *10 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*
243 In this case there is data to read. The LLOG_ORIGIN_HANDLE_CREATE
244 reply identifies the actual object of interest on the MGS via the
245 'llog_logid' field in the 'struct llogd_body'.
247 *11 - The Client sends an LLOG_ORIGIN_HANDLE_READ_HEADER to the MGS.*
249 .LLOG_ORIGIN_HANDLE_READ_HEADER Request Packet Structure
250 image::llog-origin-handle-read-header-request.png["LLOG_ORIGIN_HANDLE_READ_HEADER Request Packet Structure",height=50]
252 //////////////////////////////////////////////////////////////////////
253 The llog-origin-handle-read-header-request.png diagram resembles this
256 LLOG_ORIGIN_HANDLE_READ_HEADER:
257 --request-------------------
258 | ptlrpc_body | llogd_body |
259 ----------------------------
260 //////////////////////////////////////////////////////////////////////
262 The client asks for that log file's header. The MGS stores
263 configuration data in log records. A header at the beginning of
264 "lfs-client" uses a bitmap to identify the log records that are
265 actually needed. The header includes both which records to retrieve
266 and how large those records are. The LLOG_ORIGIN_HANDLE_READ_HEADER
267 request uses the 'llog_logid' to identify desired log file, and the
268 reply provides the bitmap and size information identifying the
269 records that are actually needed.
271 *12 - The MGS sends an LLOG_ORIGIN_HANDLE_READ_HEADER reply to the client.*
273 .LLOG_ORIGIN_HANDLE_READ_HEADER Reply Packet Structure
274 image::llog-origin-handle-read-header-reply.png["LLOG_ORIGIN_HANDLE_READ_HEADER Reply Packet Structure",height=50]
276 //////////////////////////////////////////////////////////////////////
277 The llog-origin-handle-read-header-reply.png diagram resembles this
280 LLOG_ORIGIN_HANDLE_READ_HEADER:
281 --reply--------------------------
282 | ptlrpc_body | llog_log_header |
283 ---------------------------------
284 //////////////////////////////////////////////////////////////////////
286 The MGs responds with the header, which holds the actual bitmap.
288 *13 - The Client sends an LLOG_ORIGIN_HANDLE_NEXT_BLOCK to the MGS.*
290 .LLOG_ORIGIN_HANDLE_NEXT_BLOCK Request Packet Structure
291 image::llog-origin-handle-next-block-request.png["LLOG_ORIGIN_HANDLE_NEXT_BLOCK Request Packet Structure",height=50]
293 //////////////////////////////////////////////////////////////////////
294 The llog-origin-handle-next-block-request.png diagram resembles this
297 LLOG_ORIGIN_HANDLE_NEXT_BLOCK:
298 --request-------------------
299 | ptlrpc_body | llogd_body |
300 ----------------------------
301 //////////////////////////////////////////////////////////////////////
303 The client asks for the configuration record.
305 *14 - The MGS sends an LLOG_ORIGIN_HANDLE_NEXT_BLOCK reply to the client.*
307 .LLOG_ORIGIN_HANDLE_NEXT_BLOCK Reply Packet Structure
308 image::llog-origin-handle-next-block-reply.png["LLOG_ORIGIN_HANDLE_NEXT_BLOCK Reply Packet Structure",height=50]
310 //////////////////////////////////////////////////////////////////////
311 The llog-origin-handle-next-block-reply.png diagram resembles this
314 LLOG_ORIGIN_HANDLE_NEXT_BLOCK:
315 --reply------------------------------
316 | ptlrpc_body | llogd_body | eadata |
317 -------------------------------------
318 //////////////////////////////////////////////////////////////////////
320 The MGS replies with the details of where in the configuration data to
321 find the desired details.
323 Knowing the specific configuration records it wants, the client then
324 proceeds to retrieve them. This requires another LDLM_ENQUEUE
325 operation, followed this time by the MGS_CONFIG_READ operation, which
326 get the UUIDs for the servers and targets from the configuration log
329 *15 - The Client sends an LDLM_ENQUEUE to the MGS.*
331 The client again seeks a 'concurrent read' lock on the MGS.
333 *16 - The MGS sends an LDLM_ENQUEUE reply to the client.*
335 The MGS grants that lock.
337 *17 - The Client sends an MGS_CONFIG_READ to the MGS.*
339 The client identifies the desired record in the 'lfs-cliir' file,
340 which contains the current details of the configuration for this file
343 *18 - The MGS sends an MGS_CONFIG_READ reply to the client.*
345 The MGS responds with the actual configuration data. This gives the
346 client the list of all the servers and targets it will need to
349 Finally, the client reads the cluster wide configuration data (the
350 sort that might be set at the client command line with a 'lctl
351 conf_param' command). The following paragraphs go into these four
352 stages in more detail.
354 *19 - The Client sends an LDLM_ENQUEUE to the MGS.*
356 The client again seeks a 'concurrent read' lock on the MGS.
358 *20 - The MGS sends an LDLM_ENQUEUE reply to the client.*
360 The MGS grants that lock.
362 *21 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*
364 The client asks for the 'params' log file.
366 *22 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*
368 The MGS responds that the log file is available.
370 *23 - The Client sends an LLOG_ORIGIN_HANDLE_READ_HEADER to the MGS.*
372 The client asks for that log file's header.
374 *24 - The MGS sends an LLOG_ORIGIN_HANDLE_READ_HEADER reply to the client.*
376 The MGs responds with the header, which holds the actual bitmap. In
377 this case there are no 'params' to report.
379 Messages Between the Client and the MDSs
380 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
382 After the foregoing interaction with the MGS the client has a list of
383 the MDSs and MDTs in the file system. Next, the client invokes four
384 Lustre operations with each MDT on the list.
386 .Connect RPCs Between the Client and each MDT
387 [[client-mdt-connect-rpcs]
388 image::client_mdt_connect_rpcs.png["Connect RPCs Between the Client and each MDT", height=200]
390 //////////////////////////////////////////////////////////////////////
391 The client_mdt_connect_rpcs.png diagram resembles this text art:
394 Step Client MGS MDT OST
395 ------- ------- -------- ------
396 1 MDS_CONNECT-------------------->
397 2 <--------------------MDS_CONNECT
398 3 MDS_STATFS--------------------->
399 4 <---------------------MDS_STATFS
400 5 MDS_GETSTATUS------------------>
401 6 <------------------MDS_GETSTATUS
402 7 MDS_GETATTR-------------------->
403 8 <--------------------MDS_GETATTR
404 //////////////////////////////////////////////////////////////////////
406 The MDS_CONNECT operation establishes a connection between the client
407 and a specific target (MDT) on an MDS. Thus, if an MDS has multiple
408 targets, there is a separate MDS_CONNECT operation for each. This
409 creates an import for the target on the client and an export for the
410 client and target on the MDS.
412 *1 - Client issues an MDS_CONNECT to each MDT.*
414 .MDS_CONNECT Request Packet Structure
415 image::mds-connect-request.png["MDS_CONNECT Request Packet Structure",height=50]
417 //////////////////////////////////////////////////////////////////////
418 The mds-connect-request.png diagram resembles this text art:
421 --request--------------------------------------------
422 | ptlrpc_body | obd_uuid | obd_uuid | lustre_handle |
423 -----------------------------------------------------
425 ---------------------
426 //////////////////////////////////////////////////////////////////////
428 As with the connect operation for the MGS, the connect message from
429 the client includes a UUID to uniquely identify this connection, and
430 subsequent messages to the lock manager on the server will refer to
431 that UUID. The connection data from the client also proposes the set
432 of <<connect-flags,connection flags>> appropriate to connecting to an
433 MDS. The following are the flags always included.
435 .Always included flags for the client connection to an MDS
438 | obd_connect_data->ocd_connect_flags
440 | OBD_CONNECT_VERSION
444 | OBD_CONNECT_NODEVOH
445 | OBD_CONNECT_ATTRFID
446 | OBD_CONNECT_CANCELSET
448 | OBD_CONNECT_RMT_CLIENT
449 | OBD_CONNECT_RMT_CLIENT_FORCE
450 | OBD_CONNECT_BRW_SIZE
451 | OBD_CONNECT_MDS_CAPA
452 | OBD_CONNECT_OSS_CAPA
453 | OBD_CONNECT_MDS_MDS
455 | LRU_RESIZE_CONNECT_FLAG
460 | OBD_CONNECT_64BITHASH
461 | OBD_CONNECT_JOBSTATS
462 | OBD_CONNECT_EINPROGRESS
463 | OBD_CONNECT_LIGHTWEIGHT
465 | OBD_CONNECT_LVB_TYPE
466 | OBD_CONNECT_LAYOUTLOCK
467 | OBD_CONNECT_PINGLESS
468 | OBD_CONNECT_MAX_EASIZE
469 | OBD_CONNECT_FLOCK_DEAD
470 | OBD_CONNECT_DISP_STRIPE
472 | OBD_CONNECT_OPEN_BY_FID
473 | OBD_CONNECT_DIR_STRIPE
476 .Optional flags for the client connection to an MDS
479 | obd_connect_data->ocd_connect_flags
481 | OBD_CONNECT_LRU_RESIZE
487 | OBD_CONNECT_RMT_CLIENT_FORCE
490 *2 - The MDS sends an MDS_CONNECT reply to the client.*
492 .MDS_CONNECT Reply Packet Structure
493 image::mds-connect-reply.png["MDS_CONNECT Reply Packet Structure",height=50]
495 //////////////////////////////////////////////////////////////////////
496 The mds-connect-reply.png diagram resembles this text art:
499 --reply---------------------------
500 | ptlrpc_body | obd_connect_data |
501 ----------------------------------
502 //////////////////////////////////////////////////////////////////////
504 The MDS replies to the connect message with a subset of the flags
505 proposed by the client, and the client notes those values in its
506 import. The MDS's reply to the connection request will include a UUID
507 that the server and client will both use to identify this connection
508 in subsequent messages.
510 *3 - The Client sends an MDS_STATFS to the MDS.*
512 The client next uses an MDS_STATFS operation to request 'statfs'
513 information from the target, and that data is returned in the reply
514 message. The actual fields closely resemble the results of a 'statfs'
515 system call. See the 'obd_statfs' structure in the <<data-structs,Data
516 Structures and Defines Section>>.
518 *4 - The MDS sends an MDS_STATFS reply to the client.*
520 The MDS replies witht eh 'statfs' information.
522 *5 - The Client sends an MDS_GETSTATUS to the MDS.*
524 The client uses the MDS_GETSTATUS operation to request information
525 about the mount point of the file system.
527 *6 - The MDS sends an MDS_GETSTATUS reply to the client.*
529 The server reply contains the 'fid' of the root directory of the file
530 system being mounted. If there is a security policy the capabilities
531 of that security policy are included in the reply.
533 *7 - The Client sends an MDS_GETATTR to the MDS.*
535 The client then uses the MDS_GETATTR operation to get get further
536 information about the root directory of the file system. The request
537 message includes the above fid.
539 *8 - The MDS sends an MDS_GETATTR reply to the client.*
541 It will also include the security capability (if appropriate). The
542 reply also holds the same fid, and in this case the 'mdt_body' has
543 several additional fields filled in. These include the mtime, atime,
544 ctime, mode, uid, and gid. It also includes the size of the extended
545 attributes and the size of the ACL information. The reply message also
546 includes the extended attributes and the ACL. From the extended
547 attributes the client can find out about striping information for the
550 Messages Between the Client and the OSSs
551 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
553 Additional CONNECT messages flow between the client and each OST
554 enumerated by the MGS.
556 .Connect RPCs Between the Client and each OST
557 [[client-ost-connect-rpcs]
558 image::client_ost_connect_rpcs.png["Connect RPCs Between the Client and each OST", height=75]
560 //////////////////////////////////////////////////////////////////////
561 The client_ost_connect_rpcs.png diagram resembles this text art:
564 Step Client MGS MDT OST
565 ------- ------- -------- ------
566 1 OST_CONNECT----------------------------------->
567 2 <-----------------------------------OST_CONNECT
568 //////////////////////////////////////////////////////////////////////
570 The OST_CONNECT operation establishes a connection between the client
571 and a specific target (OST) on an OST. Thus, if an OST has multiple
572 targets, there is a separate OST_CONNECT operation for each. This
573 creates an import for the target on the client and an export for the
574 client and target on the OST.
576 *1 - Client issues an OST_CONNECT to each OST.*
578 .OST_CONNECT Request Packet Structure
579 image::ost-connect-request.png["OST_CONNECT Request Packet Structure",height=50]
581 //////////////////////////////////////////////////////////////////////
582 The ost-connect-request.png diagram resembles this text art:
585 --request--------------------------------------------
586 | ptlrpc_body | obd_uuid | obd_uuid | lustre_handle |
587 -----------------------------------------------------
589 ---------------------
590 //////////////////////////////////////////////////////////////////////
592 As with the connect operations for the MGS and MDTs, the connect
593 message from the client includes a UUID to uniquely identify this
594 connection, and subsequent messages to the lock manager on the server
595 will refer to that UUID. The connection data from the client also
596 proposes the set of <<connect-flags,connection flags>> appropriate to
597 connecting to an OST. The following are the flags always included.
599 .Flags for the client connection to an OST
602 | obd_connect_data->ocd_connect_flags
604 | OBD_CONNECT_SRVLOCK
605 | OBD_CONNECT_VERSION
606 | OBD_CONNECT_REQPORTAL
607 | OBD_CONNECT_TRUNCLOCK
608 | OBD_CONNECT_RMT_CLIENT
609 | OBD_CONNECT_BRW_SIZE
610 | OBD_CONNECT_OSS_CAPA
611 | OBD_CONNECT_CANCELSET
613 | OBD_CONNECT_LRU_RESIZE
618 | OBD_CONNECT_LAYOUTLOCK
619 | OBD_CONNECT_64BITHASH
620 | OBD_CONNECT_MAXBYTES
621 | OBD_CONNECT_JOBSTATS
622 | OBD_CONNECT_EINPROGRESS
623 | OBD_CONNECT_LVB_TYPE
624 | OBD_CONNECT_PINGLESS
627 *2 - The OST sends an OST_CONNECT reply to the client.*
629 .OST_CONNECT Reply Packet Structure
630 image::ost-connect-reply.png["OST_CONNECT Reply Packet Structure",height=50]
632 //////////////////////////////////////////////////////////////////////
633 The ost-connect-reply.png diagram resembles this text art:
636 --reply---------------------------
637 | ptlrpc_body | obd_connect_data |
638 ----------------------------------
639 //////////////////////////////////////////////////////////////////////
641 The OST replies to the connect message with a subset of the flags
642 proposed by the client, and the client notes those values in its
643 import. The OST's reply to the connection request will include a UUID
644 that the server and client will both use to identify this connection
645 in subsequent messages. The flags that the server doesn't include are:
646 OBD_CONNECT_RMT_CLIENT, OBD_CONNECT_OSS_CAPA, and
647 OBD_CONNECT_PINGLESS.