1 #LyX 1.2 created this file. For more info see http://www.lyx.org/
6 \renewenvironment{comment}%
7 {\begin{quote}\textbf{Discussion}: \slshape}%
17 \papersize letterpaper
22 \use_numerical_citations 0
23 \paperorientation portrait
26 \paragraph_separation indent
28 \quotes_language english
32 \paperpagestyle headings
36 The Portals 3.2 Message Passing Interface
49 Riesen are with the Scalable Computing Systems Department, Sandia National
51 Box 5800, Albuquerque, NM\SpecialChar ~
53 87111-1110, bright@cs.sandia.gov, rolf@cs.sandia.gov.
65 Maccabe is with the Computer Science Department, University of New Mexico,
66 Albuquerque, NM\SpecialChar ~
68 87131-1386, maccabe@cs.unm.edu.
71 , Rolf Riesen and Trammell Hudson
74 This report presents a specification for the Portals 3.2 message passing
76 Portals 3.2 is intended to allow scalable, high-performance network communicatio
77 n between nodes of a parallel computing system.
78 Specifically, it is designed to support a parallel computing platform composed
79 of clusters of commodity workstations connected by a commodity system area
81 In addition, Portals 3.2 is well suited to massively parallel processing
83 Portals 3.2 represents an adaption of the data movement layer developed
84 for massively parallel processing platforms, such as the 4500-node Intel
107 \begin_inset LatexCommand \tableofcontents{}
128 \begin_inset FloatList figure
149 \begin_inset FloatList table
169 Summary of Changes for Revision 1.1
172 Updated version number to 3.2 throughout the document
176 \begin_inset LatexCommand \ref{sub:PtlGetId}
192 \begin_inset LatexCommand \ref{sec:meattach}
208 \begin_inset LatexCommand \ref{sec:meunlink}
212 : removed text referring to a list of associated memory descriptors.
216 \begin_inset LatexCommand \ref{sec:mdfree}
220 : added text to describe unlinking a free-floating memory descriptor.
224 \begin_inset LatexCommand \ref{tab:types}
236 \begin_inset LatexCommand \ref{sec:md-type}
251 added text to clarify
260 \begin_inset LatexCommand \ref{sec:mdattach}
272 \begin_inset LatexCommand \ref{sec:niinit}
276 : added text to clarify multiple calls to
284 \begin_inset LatexCommand \ref{sec:mdattach}
288 : added text to clarify
296 \begin_inset LatexCommand \ref{sec:receiving}
300 : removed text indicating that an MD will reject a message if the associated
305 \begin_inset LatexCommand \ref{sec:mdfree}
313 error code and text to indicate that only MDs with no pending operations
318 \begin_inset LatexCommand \ref{tab:retcodes}
330 \begin_inset LatexCommand \ref{sec:event-type}
334 : added user id field, MD handle field, and NI specific failure field to
343 \begin_inset LatexCommand \ref{tab:types}
355 \begin_inset LatexCommand \ref{sec:event-type}
367 \begin_inset LatexCommand \ref{tab:func}
379 \begin_inset LatexCommand \ref{sec:meattach}
384 \begin_inset LatexCommand \ref{sec:meinsert}
389 \begin_inset LatexCommand \ref{sec:put}
393 : listed allowable constants with relevant fields.
397 \begin_inset LatexCommand \ref{tab:func}
409 \begin_inset LatexCommand \ref{tab:retcodes}
425 \begin_inset LatexCommand \ref{tab:oconsts}
429 : updated to reflect new event types.
433 \begin_inset LatexCommand \ref{sec:id-type}
452 Summary of Changes for Version 3.1
458 The most significant change to the interface from version 3.0 to 3.1 involves
459 the clarification of how the interface interacts with multi-threaded applicatio
461 We adopted a generic thread model in which processes define an address
462 space and threads share the address space.
463 Consideration of the API in the light of threads lead to several clarifications
464 throughout the document:
471 added a definition for
478 reworded the definition for
487 Section\SpecialChar ~
489 \begin_inset LatexCommand \ref{sec:apiover}
493 : added section\SpecialChar ~
495 \begin_inset LatexCommand \ref{sec:threads}
499 to describe the multi-threading model used by the Portals API.
503 Section\SpecialChar ~
505 \begin_inset LatexCommand \ref{sec:ptlinit}
513 must be called at least once and may be called any number of times.
517 Section\SpecialChar ~
519 \begin_inset LatexCommand \ref{sec:ptlfini}
527 should be called once as the process is terminating and not as each thread
532 Section\SpecialChar ~
534 \begin_inset LatexCommand \ref{sec:pid}
538 : Portals does not define thread ids.
542 Section\SpecialChar ~
544 \begin_inset LatexCommand \ref{sec:ni}
548 : network interfaces are associated with processes, not threads.
552 Section\SpecialChar ~
554 \begin_inset LatexCommand \ref{sec:niinit}
562 must be called at least once and may be called any number of times.
566 Section\SpecialChar ~
568 \begin_inset LatexCommand \ref{sec:eqget}
580 if a thread is blocked on
588 Section\SpecialChar ~
590 \begin_inset LatexCommand \ref{sec:eqwait}
594 : waiting threads are awakened in FIFO order.
606 were removed from the API.
611 was defined to block the calling process until all of the processes in
612 the application group had invoked
617 We now consider this functionality, along with the concept of groups (see
619 \begin_inset Quotes eld
623 \begin_inset Quotes erd
626 ), to be part of the runtime system, not part of the Portals API.
631 was defined to return the number of events in an event queue.
632 Because external operations may lead to new events being added and other
633 threads may remove events, the value returned by
637 would have to be a hint about the number of events in the event queue.
640 Handling small, unexpected messages
643 Another set of changes relates to handling small unexpected messages in
645 In designing version 3.0, we assumed that each unexpected message would
646 be placed in a unique memory descriptor.
647 To avoid the need to process a long list of memory descriptors, we moved
648 the memory descriptors out of the match list and hung them off of a single
650 In this way, large unexpected messages would only encounter a single
651 \begin_inset Quotes eld
655 \begin_inset Quotes erd
658 match list entry before encountering the
659 \begin_inset Quotes eld
663 \begin_inset Quotes erd
667 Experience with this strategy identified resource management problems with
669 In particular, a long sequence of very short (or zero length) messages
670 could quickly exhaust the memory descriptors constructed for handling unexpecte
672 Our new strategy involves the use of several very large memory descriptors
673 for small unexpected messages.
674 Consecutive unexpected messages will be written into the first of these
675 memory descriptors until the memory descriptor fills up.
676 When the first of the
677 \begin_inset Quotes eld
681 \begin_inset Quotes erd
684 descriptors fills up, it will be unlinked and subsequent short messages
685 will be written into the next
686 \begin_inset Quotes eld
690 \begin_inset Quotes erd
695 \begin_inset Quotes eld
699 \begin_inset Quotes erd
702 memory descriptor will be declared full when it does not have sufficient
703 space for the largest small unexpected message.
706 This lead to two significant changes.
707 First, each match list entry now has a single memory descriptor rather
708 than a list of memory descriptors.
709 Second, in addition to exceeding the operation threshold, a memory descriptor
710 can be unlinked when the local offset exceeds a specified value.
711 These changes have lead to several changes in this document:
714 Section\SpecialChar ~
716 \begin_inset LatexCommand \ref{subsec:paddress}
724 removed references to the memory descriptor list,
727 changed the portals address translation description to indicate that unlinking
728 a memory descriptor implies unlinking the associated match list entry--match
729 list entries can no longer be unlinked independently from the memory descriptor.
734 Section\SpecialChar ~
736 \begin_inset LatexCommand \ref{sec:meattach}
744 removed unlink from argument list,
747 removed description of
754 changed wording of the error condition when the Portal table index already
755 has an associated match list.
760 Section\SpecialChar ~
762 \begin_inset LatexCommand \ref{sec:meinsert}
766 : removed unlink from argument list.
770 Section\SpecialChar ~
772 \begin_inset LatexCommand \ref{sec:md-type}
784 Section\SpecialChar ~
786 \begin_inset LatexCommand \ref{sec:mdattach}
801 removed reference to memory descriptor lists,
804 changed wording of the error condition when match list entry already has
805 an associated memory descriptor,
808 changed the description of the
817 Section\SpecialChar ~
819 \begin_inset LatexCommand \ref{sec:md}
831 Section\SpecialChar ~
833 \begin_inset LatexCommand \ref{sec:mdbind}
837 : removed references to memory descriptor list.
841 Section\SpecialChar ~
843 \begin_inset LatexCommand \ref{sec:mdfree}
847 : removed reference to memory descriptor list.
851 Section\SpecialChar ~
853 \begin_inset LatexCommand \ref{sec:summary}
857 : removed references to PtlMDInsert.
861 Section\SpecialChar ~
863 \begin_inset LatexCommand \ref{sec:semantics}
867 : removed reference to memory descriptor list.
871 Section\SpecialChar ~
873 \begin_inset LatexCommand \ref{sec:exmpi}
877 : revised the MPI example to reflect the changes to the interface.
881 Several changes have been made to improve the general documentation of the
886 Section\SpecialChar ~
888 \begin_inset LatexCommand \ref{sec:handle-type}
892 : documented the special value
900 Section\SpecialChar ~
902 \begin_inset LatexCommand \ref{sec:id-type}
906 : documented the special value
914 Section\SpecialChar ~
916 \begin_inset LatexCommand \ref{sec:mdbind}
920 : documented the return value
925 Section\SpecialChar ~
927 \begin_inset LatexCommand \ref{sec:mdupdate}
931 : clarified the description of the
939 Section\SpecialChar ~
941 \begin_inset LatexCommand \ref{sec:implvals}
945 : introduced a new section to document the implementation defined values.
949 Section\SpecialChar ~
951 \begin_inset LatexCommand \ref{sec:summary}
955 : modified Table\SpecialChar ~
957 \begin_inset LatexCommand \ref{tab:oconsts}
961 to indicate where each constant is introduced and where it is used.
968 Implementation defined limits (Section
969 \begin_inset LatexCommand \ref{sec:niinit}
976 The earlier version provided implementation defined limits for the maximum
977 number of match entries, the maximum number of memory descriptors, etc.
978 Rather than spanning the entire implementation, these limits are now associated
979 with individual network interfaces.
982 Added User Ids (Section
983 \begin_inset LatexCommand \ref{sec:uid}
990 Group Ids had been used to simplify access control entries.
991 In particular, a process could allow access for all of the processes in
993 User Ids have been introduced to regain this functionality.
994 We use user ids to fill this role.
997 Removed Group Ids and Rank Ids (Section
998 \begin_inset LatexCommand \ref{sec:pid}
1005 The earlier version of Portals had two forms for addressing processes: <node
1006 id, process id> and <group id, rank id>.
1007 A process group was defined as the collection processes created during
1009 Each process in the group was given a unique rank id in the range 0 to
1011 \begin_inset Formula $n-1$
1015 \begin_inset Formula $n$
1018 was the number of processes in the group.
1019 We removed groups because they are better handled in the runtime system.
1022 Match lists (Section
1023 \begin_inset LatexCommand \ref{sec:meattach}
1030 It is no longer illegal to have an existing match entry when calling PtlMEAttach.
1031 A position argument was added to the list of arguments supplied to
1035 to specify whether the new match entry is prepended or appended to the
1037 If there is no existing match list, the position argument is ignored.
1040 Unlinking Memory Descriptors (Section
1041 \begin_inset LatexCommand \ref{sec:md}
1048 Previously, a memory descriptor could be unlinked if the offset exceeded
1049 a threshold upon the completion of an operation.
1050 In this version, the unlinking is delayed until there is a matching operation
1051 which requires more memory than is currently available in the descriptor.
1052 In addition to changes in section, this lead to a revision of Figure\SpecialChar ~
1054 \begin_inset LatexCommand \ref{fig:flow}
1061 Split Phase Operations and Events (Section
1062 \begin_inset LatexCommand \ref{sec:eq}
1069 Previously, there were five types of events:
1090 The first four of these reflected the completion of potentially long operations.
1091 We have introduced new event types to reflect the fact that long operations
1092 have a distinct starting point and a distinct completion point.
1093 Moreover, the completion may be successful or unsuccessful.
1096 In addition to providing a mechanism for reporting failure to higher levels
1097 of software, this split provides an opportunity for for improved ordering
1099 Previously, if one process intiated two operations (e.g., two put operations)
1100 on a remote process, these operations were guaranteed to complete in the
1101 same order that they were initiated.
1102 Now, we only guarantee that the initiation events are delivered in the
1104 In particular, the operations do not need to complete in the order that
1108 Well known proces ids (Section
1109 \begin_inset LatexCommand \ref{sec:niinit}
1116 To support the notion of
1117 \begin_inset Quotes eld
1120 well known process ids,
1121 \begin_inset Quotes erd
1124 we added a process id argument to the arguments for PtlNIInit.
1130 API Application Programming Interface.
1131 A definition of the functions and semantics provided by library of functions.
1139 that initiates a message operation.
1143 Message An application-defined unit of data that is exchanged between
1151 Message\SpecialChar ~
1152 Operation Either a put operation, which writes data, or a get operation,
1157 Network A network provides point-to-point communication between
1162 Internally, a network may provide multiple routes between endpoints (to
1163 improve fault tolerance or to improve performance characteristics); however,
1164 multiple paths will not be exposed outside of the network.
1168 Node A node is an endpoint in a
1173 Nodes provide processing capabilities and memory.
1174 A node may provide multiple processors (an SMP node) or it may act as a
1183 Process A context of execution.
1184 A process defines a virtual memory (VM) context.
1185 This context is not shared with other processes.
1186 Several threads may share the VM context defined by a process.
1194 that is acted upon by a message operation.
1198 Thread A context of execution that shares a VM context with other threads.
1215 pagenumbering{arabic}
1222 \begin_inset LatexCommand \label{sec:intro}
1232 This document describes an application programming interface for message
1233 passing between nodes in a system area network.
1234 The goal of this interface is to improve the scalability and performance
1235 of network communication by defining the functions and semantics of message
1236 passing required for scaling a parallel computing system to ten thousand
1238 This goal is achieved by providing an interface that will allow a quality
1239 implementation to take advantage of the inherently scalable design of Portals.
1242 This document is divided into several sections:
1245 Section\SpecialChar ~
1247 \begin_inset LatexCommand \ref{sec:intro}
1251 ---Introduction This section describes the purpose and scope of the Portals
1256 Section\SpecialChar ~
1258 \begin_inset LatexCommand \ref{sec:apiover}
1263 Overview\SpecialChar ~
1266 Portals\SpecialChar ~
1268 API This section gives a brief overview of the
1270 The goal is to introduce the key concepts and terminology used in the descripti
1275 Section\SpecialChar ~
1277 \begin_inset LatexCommand \ref{sec:api}
1281 ---The\SpecialChar ~
1282 Portals\SpecialChar ~
1284 API This section describes the functions and semantics of
1285 the Portals application programming interface.
1289 Section\SpecialChar ~
1291 \begin_inset LatexCommand \ref{sec:semantics}
1296 Semantics\SpecialChar ~
1298 Message\SpecialChar ~
1299 Transmission This section describes the semantics
1300 of message transmission.
1301 In particular, the information transmitted in each type of message and
1302 the processing of incoming messages.
1306 Section\SpecialChar ~
1308 \begin_inset LatexCommand \ref{sec:examples}
1312 ---Examples This section presents several examples intended to illustrates
1313 the use of the Portals API.
1320 Existing message passing technologies available for commodity cluster networking
1321 hardware do not meet the scalability goals required by the Cplant\SpecialChar ~
1323 \begin_inset LatexCommand \cite{Cplant}
1327 project at Sandia National Laboratories.
1328 The goal of the Cplant project is to construct a commodity cluster that
1329 can scale to the order of ten thousand nodes.
1330 This number greatly exceeds the capacity for which existing message passing
1331 technologies have been designed and implemented.
1334 In addition to the scalability requirements of the network, these technologies
1335 must also be able to support a scalable implementation of the Message Passing
1336 Interface (MPI)\SpecialChar ~
1338 \begin_inset LatexCommand \cite{MPIstandard}
1342 standard, which has become the
1346 standard for parallel scientific computing.
1347 While MPI does not impose any scalability limitations, existing message
1348 passing technologies do not provide the functionality needed to allow implement
1349 ations of MPI to meet the scalability requirements of Cplant.
1352 The following are properties of a network architecture that do not impose
1353 any inherent scalability limitations:
1356 Connectionless - Many connection-oriented architectures, such as VIA\SpecialChar ~
1358 \begin_inset LatexCommand \cite{VIA}
1362 and TCP/IP sockets, have limitations on the number of peer connections
1363 that can be established.
1367 Network independence - Many communication systems depend on the host processor
1368 to perform operations in order for messages in the network to be consumed.
1369 Message consumption from the network should not be dependent on host processor
1370 activity, such as the operating system scheduler or user-level thread scheduler.
1374 User-level flow control - Many communication systems manage flow control
1375 internally to avoid depleting resources, which can significantly impact
1376 performance as the number of communicating processes increases.
1380 OS Bypass - High performance network communication should not involve memory
1381 copies into or out of a kernel-managed protocol stack.
1385 The following are properties of a network architecture that do not impose
1386 scalability limitations for an implementation of MPI:
1389 Receiver-managed - Sender-managed message passing implementations require
1390 a persistent block of memory to be available for every process, requiring
1391 memory resources to increase with job size and requiring user-level flow
1392 control mechanisms to manage these resources.
1396 User-level Bypass - While OS Bypass is necessary for high-performance, it
1397 alone is not sufficient to support the Progress Rule of MPI asynchronous
1402 Unexpected messages - Few communication systems have support for receiving
1403 messages for which there is no prior notification.
1404 Support for these types of messages is necessary to avoid flow control
1405 and protocol overhead.
1412 Portals was originally designed for and implemented on the nCube machine
1413 as part of the SUNMOS (Sandia/UNM OS)\SpecialChar ~
1415 \begin_inset LatexCommand \cite{SUNMOS}
1419 and Puma\SpecialChar ~
1421 \begin_inset LatexCommand \cite{PumaOS}
1425 lightweight kernel development projects.
1426 Portals went through two design phases, the latter of which is used on
1427 the 4500-node Intel TeraFLOPS machine\SpecialChar ~
1429 \begin_inset LatexCommand \cite{TFLOPS}
1434 Portals have been very successful in meeting the needs of such a large
1435 machine, not only as a layer for a high-performance MPI implementation\SpecialChar ~
1437 \begin_inset LatexCommand \cite{PumaMPI}
1441 , but also for implementing the scalable run-time environment and parallel
1442 I/O capabilities of the machine.
1445 The second generation Portals implementation was designed to take full advantage
1446 of the hardware architecture of large MPP machines.
1447 However, efforts to implement this same design on commodity cluster technology
1448 identified several limitations, due to the differences in network hardware
1449 as well as to shortcomings in the design of Portals.
1455 The primary goal in the design of Portals is scalability.
1456 Portals are designed specifically for an implementation capable of supporting
1457 a parallel job running on tens of thousands of nodes.
1458 Performance is critical only in terms of scalability.
1459 That is, the level of message passing performance is characterized by how
1460 far it allows an application to scale and not by how it performs in micro-bench
1461 marks (e.g., a two node bandwidth or latency test).
1464 The Portals API is designed to allow for scalability, not to guarantee it.
1465 Portals cannot overcome the shortcomings of a poorly designed application
1467 Applications that have inherent scalability limitations, either through
1468 design or implementation, will not be transformed by Portals into scalable
1470 Scalability must be addressed at all levels.
1471 Portals do not inhibit scalability, but do not guarantee it either.
1474 To support scalability, the Portals interface maintains a minimal amount
1476 Portals provide reliable, ordered delivery of messages between pairs of
1478 They are connectionless: a process is not required to explicitly establish
1479 a point-to-point connection with another process in order to communicate.
1480 Moreover, all buffers used in the transmission of messages are maintained
1482 The target process determines how to respond to incoming messages, and
1483 messages for which there are no buffers are discarded.
1489 Portals combine the characteristics of both one-side and two-sided communication.
1491 \begin_inset Quotes eld
1495 \begin_inset Quotes erd
1499 \begin_inset Quotes eld
1503 \begin_inset Quotes erd
1507 The destination of a put (or send) is not an explicit address; instead,
1508 each message contains a set of match bits that allow the receiver to determine
1509 where incoming messages should be placed.
1510 This flexibility allows Portals to support both traditional one-sided operation
1511 s and two-sided send/receive operations.
1514 Portals allows the target to determine whether incoming messages are acceptable.
1515 A target process can choose to accept message operations from any specific
1516 process or can choose to ignore message operations from any specific process.
1519 Zero Copy, OS Bypass and Application Bypass
1522 In traditional system architectures, network packets arrive at the network
1523 interface card (NIC), are passed through one or more protocol layers in
1524 the operating system, and eventually copied into the address space of the
1526 As network bandwidth began to approach memory copy rates, reduction of
1527 memory copies became a critical concern.
1528 This concern lead to the development of zero-copy message passing protocols
1529 in which message copies are eliminated or pipelined to avoid the loss of
1533 A typical zero-copy protocol has the NIC generate an interrupt for the CPU
1534 when a message arrives from the network.
1535 The interrupt handler then controls the transfer of the incoming message
1536 into the address space of the appropriate application.
1537 The interrupt latency, the time from the initiation of an interrupt until
1538 the interrupt handler is running, is fairly significant.
1539 To avoid this cost, some modern NICs have processors that can be programmed
1540 to implement part of a message passing protocol.
1541 Given a properly designed protocol, it is possible to program the NIC to
1542 control the transfer of incoming messages, without needing to interrupt
1544 Because this strategy does not need to involve the OS on every message
1545 transfer, it is frequently called
1546 \begin_inset Quotes eld
1550 \begin_inset Quotes erd
1555 \begin_inset LatexCommand \cite{ST}
1561 \begin_inset LatexCommand \cite{VIA}
1567 \begin_inset LatexCommand \cite{FM2}
1573 \begin_inset LatexCommand \cite{GM}
1577 , and Portals are examples of OS Bypass protocols.
1580 Many protocols that support OS Bypass still require that the application
1581 actively participate in the protocol to ensure progress.
1582 As an example, the long message protocol of PM requires that the application
1583 receive and reply to a request to put or get a long message.
1584 This complicates the runtime environment, requiring a thread to process
1585 incoming requests, and significantly increases the latency required to
1586 initiate a long message protocol.
1587 The Portals message passing protocol does not require activity on the part
1588 of the application to ensure progress.
1590 \begin_inset Quotes eld
1594 \begin_inset Quotes erd
1597 to refer to this aspect of the Portals protocol.
1603 Given the number of components that we are dealing with and the fact that
1604 we are interested in supporting applications that run for very long times,
1605 failures are inevitable.
1606 The Portals API recognizes that the underlying transport may not be able
1607 to successfully complete an operation once it has been initiated.
1608 This is reflected in the fact that the Portals API reports three types
1609 of events: events indicating the initiation of an operation, events indicating
1610 the successful completion of an operation, and events indicating the unsuccessf
1611 ul completion of an operation.
1612 Every initiation event is eventually followed by a successful completion
1613 event or an unsuccessful completion event.
1616 Between the time an operation is started and the time that the operation
1617 completes (successfully or unsuccessfully), any memory associated with
1618 the operation should be considered volatile.
1619 That is, the memory may be changed in unpredictable ways while the operation
1621 Once the operation completes, the memory associated with the operation
1622 will not be subject to further modification (from this operation).
1623 Notice that unsuccessful operations may alter memory in an essentially
1624 unpredictable fashion.
1627 An Overview of the Portals API
1628 \begin_inset LatexCommand \label{sec:apiover}
1635 In this section, we give a conceptual overview of the Portals API.
1636 The goal is to provide a context for understanding the detailed description
1637 of the API presented in the next section.
1641 \begin_inset LatexCommand \label{sec:dmsemantics}
1648 A Portal represents an opening in the address space of a process.
1649 Other processes can use a Portal to read (get) or write (put) the memory
1650 associated with the portal.
1651 Every data movement operation involves two processes, the
1660 The initiator is the process that initiates the data movement operation.
1661 The target is the process that responds to the operation by either accepting
1662 the data for a put operation, or replying with the data for a get operation.
1665 In this discussion, activities attributed to a process may refer to activities
1666 that are actually performed by the process or
1668 on behalf of the process
1671 The inclusiveness of our terminology is important in the context of
1676 In particular, when we note that the target sends a reply in the case of
1677 a get operation, it is possible that reply will be generated by another
1678 component in the system, bypassing the application.
1681 Figures\SpecialChar ~
1683 \begin_inset LatexCommand \ref{fig:put}
1688 \begin_inset LatexCommand \ref{fig:get}
1692 present graphical interpretations of the Portal data movement operations:
1694 In the case of a put operation, the initiator sends a put request message
1695 containing the data to the target.
1696 The target translates the Portal addressing information in the request
1697 using its local Portal structures.
1698 When the request has been processed, the target optionally sends an acknowledge
1703 \begin_inset Float figure
1711 \begin_inset Graphics FormatVersion 1
1725 \begin_inset LatexCommand \label{fig:put}
1735 In the case of a get operation, the initiator sends a get request to the
1737 As with the put operation, the target translates the Portal addressing
1738 information in the request using its local Portal structures.
1739 Once it has translated the Portal addressing information, the target sends
1740 a reply that includes the requested data.
1744 \begin_inset Float figure
1752 \begin_inset Graphics FormatVersion 1
1766 \begin_inset LatexCommand \label{fig:get}
1776 We should note that Portal address translations are only performed on nodes
1777 that respond to operations initiated by other nodes.
1778 Acknowledgements and replies to get operations bypass the portals address
1779 translation structures.
1783 \begin_inset LatexCommand \label{subsec:paddress}
1790 One-sided data movement models (e.g., shmem\SpecialChar ~
1792 \begin_inset LatexCommand \cite{CraySHMEM}
1798 \begin_inset LatexCommand \cite{ST}
1802 , MPI-2\SpecialChar ~
1804 \begin_inset LatexCommand \cite{MPI2}
1808 ) typically use a triple to address memory on a remote node.
1809 This triple consists of a process id, memory buffer id, and offset.
1810 The process id identifies the target process, the memory buffer id specifies
1811 the region of memory to be used for the operation, and the offset specifies
1812 an offset within the memory buffer.
1815 In addition to the standard address components (process id, memory buffer
1816 id, and offset), a Portal address includes a set of match bits.
1817 This addressing model is appropriate for supporting one-sided operations
1818 as well as traditional two-sided message passing operations.
1819 Specifically, the Portals API provides the flexibility needed for an efficient
1820 implementation of MPI-1, which defines two-sided operations with one-sided
1821 completion semantics.
1824 Figure\SpecialChar ~
1826 \begin_inset LatexCommand \ref{fig:portals}
1830 presents a graphical representation of the structures used by a target
1831 in the interpretation of a Portal address.
1832 The process id is used to route the message to the appropriate node and
1833 is not reflected in this diagram.
1834 The memory buffer id, called the
1838 , is used as an index into the Portal table.
1839 Each element of the Portal table identifies a match list.
1840 Each element of the match list specifies two bit patterns: a set of
1841 \begin_inset Quotes eld
1845 \begin_inset Quotes erd
1849 \begin_inset Quotes eld
1853 \begin_inset Quotes erd
1857 In addition to the two sets of match bits, each match list element has
1858 at most one memory descriptor.
1859 Each memory descriptor identifies a memory region and an optional event
1861 The memory region specifies the memory to be used in the operation and
1862 the event queue is used to record information about these operations.
1866 \begin_inset Float figure
1874 \begin_inset Graphics FormatVersion 1
1875 filename portals.eps
1887 Portal Addressing Structures
1888 \begin_inset LatexCommand \label{fig:portals}
1898 Figure\SpecialChar ~
1900 \begin_inset LatexCommand \ref{fig:flow}
1904 illustrates the steps involved in translating a Portal address, starting
1905 from the first element in a match list.
1906 If the match criteria specified in the match list entry are met and the
1907 memory descriptor list accepts the operation
1913 Memory descriptors can reject operations because a threshold has been exceeded
1914 or because the memory region does not have sufficient space, see Section\SpecialChar ~
1916 \begin_inset LatexCommand \ref{sec:md}
1923 , the operation (put or get) is performed using the memory region specified
1924 in the memory descriptor.
1925 If the memory descriptor specifies that it is to be unlinked when a threshold
1926 has been exceeded, the match list entry is removed from the match list
1927 and the resources associated with the memory descriptor and match list
1928 entry are reclaimed.
1929 Finally, if there is an event queue specified in the memory descriptor,
1930 the operation is logged in the event queue.
1934 \begin_inset Float figure
1942 \begin_inset Graphics FormatVersion 1
1943 filename flow_new.eps
1955 Portals Address Translation
1956 \begin_inset LatexCommand \label{fig:flow}
1966 If the match criteria specified in the match list entry are not met, or
1967 there is no memory descriptor associated with the match list entry, or
1968 the memory descriptor associated with the match list entry rejects the
1969 operation, the address translation continues with the next match list entry.
1970 If the end of the match list has been reached, the address translation
1971 is aborted and the incoming requested is discarded.
1977 A process can control access to its portals using an access control list.
1978 Each entry in the access control list specifies a process id and a Portal
1980 The access control list is actually an array of entries.
1981 Each incoming request includes an index into the access control list (i.e.,
1983 \begin_inset Quotes eld
1987 \begin_inset Quotes erd
1991 If the id of the process issuing the request doesn't match the id specified
1992 in the access control list entry or the Portal table index specified in
1993 the request doesn't match the Portal table index specified in the access
1994 control list entry, the request is rejected.
1995 Process identifiers and Portal table indexes may include wild card values
1996 to increase the flexibility of this mechanism.
2000 Two aspects of this design merit further discussion.
2001 First, the model assumes that the information in a message header, the
2002 sender's id in particular, is trustworthy.
2003 In most contexts, we assume that the entity that constructs the header
2004 is trustworthy; however, using cryptographic techniques, we could easily
2005 devise a protocol that would ensure the authenticity of the sender.
2008 Second, because the access check is performed by the receiver, it is possible
2009 that a malicious process will generate thousands of messages that will
2010 be denied by the receiver.
2011 This could saturate the network and/or the receiver, resulting in a
2016 Moving the check to the sender using capabilities, would remove the potential
2017 for this form of attack.
2018 However, the solution introduces the complexities of capability management
2019 (exchange of capabilities, revocation, protections, etc).
2022 Multi-threaded Applications
2023 \begin_inset LatexCommand \label{sec:threads}
2030 The Portals API supports a generic view of multi-threaded applications.
2031 From the perspective of the Portals API, an application program is defined
2032 by a set of processes.
2033 Each process defines a unique address space.
2034 The Portals API defines access to this address space from other processes
2035 (using portals addressing and the data movement operations).
2036 A process may have one or more
2040 executing in its address space.
2044 With the exception of
2048 every function in the Portals API is non-blocking and atomic with respect
2049 to both other threads and external operations that result from data movement
2051 While individual operations are atomic, sequences of these operations may
2052 be interleaved between different threads and with external operations.
2053 The Portals API does not provide any mechanisms to control this interleaving.
2054 It is expected that these mechanisms will be provided by the API used to
2059 \begin_inset LatexCommand \label{sec:api}
2067 \begin_inset LatexCommand \label{sec:conv}
2074 The Portals API defines two types of entities: functions and types.
2075 Function always start with
2079 and use mixed upper and lower case.
2080 When used in the body of this report, function names appear in italic face,
2086 The functions associated with an object type will have names that start
2091 , followed by the two letter object type code shown in Table\SpecialChar ~
2093 \begin_inset LatexCommand \ref{tab:objcodes}
2098 As an example, the function
2102 allocates resources for an event queue.
2106 \begin_inset Float table
2114 \begin_inset LatexCommand \label{tab:objcodes}
2136 \begin_inset Tabular
2137 <lyxtabular version="3" rows="5" columns="3">
2138 <features firstHeadEmpty="true">
2139 <column alignment="left" valignment="top" width="0pt">
2140 <column alignment="left" valignment="top" width="0pt">
2141 <column alignment="left" valignment="top" width="0pt">
2142 <row bottomline="true">
2143 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
2153 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
2161 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
2171 <cell alignment="left" valignment="top" usebox="none">
2179 <cell alignment="left" valignment="top" usebox="none">
2187 <cell alignment="left" valignment="top" usebox="none">
2193 \begin_inset LatexCommand \ref{sec:eq}
2202 <cell alignment="left" valignment="top" usebox="none">
2210 <cell alignment="left" valignment="top" usebox="none">
2218 <cell alignment="left" valignment="top" usebox="none">
2224 \begin_inset LatexCommand \ref{sec:md}
2233 <cell alignment="left" valignment="top" usebox="none">
2241 <cell alignment="left" valignment="top" usebox="none">
2249 <cell alignment="left" valignment="top" usebox="none">
2255 \begin_inset LatexCommand \ref{sec:me}
2264 <cell alignment="left" valignment="top" usebox="none">
2272 <cell alignment="left" valignment="top" usebox="none">
2280 <cell alignment="left" valignment="top" usebox="none">
2286 \begin_inset LatexCommand \ref{sec:ni}
2304 Type names use lower case with underscores to separate words.
2305 Each type name starts with
2314 When used in the body of this report, type names appear in a fixed font,
2322 Names for constants use upper case with underscores to separate words.
2323 Each constant name starts with
2328 When used in the body of this report, type names appear in a fixed font,
2339 The Portals API defines a variety of base types.
2340 These types represent a simple renaming of the base types provided by the
2341 C programming language.
2342 In most cases these new type names have been introduced to improve type
2343 safety and to avoid issues arising from differences in representation sizes
2344 (e.g., 16-bit or 32-bit integers).
2348 \begin_inset LatexCommand \label{sec:size-t}
2359 is an unsigned 64-bit integral type used for representing sizes.
2363 \begin_inset LatexCommand \label{sec:handle-type}
2370 Objects maintained by the API are accessed through handles.
2371 Handle types have names of the form
2383 is one of the two letter object type codes shown in Table\SpecialChar ~
2385 \begin_inset LatexCommand \ref{tab:objcodes}
2390 For example, the type
2394 is used for network interface handles.
2397 Each type of object is given a unique handle type to enhance type checking.
2402 , can be used when a generic handle is needed.
2403 Every handle value can be converted into a value of type
2407 without loss of information.
2410 Handles are not simple values.
2411 Every portals object is associated with a specific network interface and
2412 an identifier for this interface (along with an object identifier) is part
2413 of the handle for the object.
2424 , is used to indicate the absence of an event queue.
2426 \begin_inset LatexCommand \ref{sec:mdfree}
2432 \begin_inset LatexCommand \ref{sec:mdupdate}
2436 for uses of this value.
2440 \begin_inset LatexCommand \label{sec:index-type}
2455 are integral types used for representing Portal table indexes and access
2456 control tables indexes, respectively.
2457 See section\SpecialChar ~
2459 \begin_inset LatexCommand \ref{sec:niinit}
2463 for limits on values of these types.
2467 \begin_inset LatexCommand \label{sec:mb-type}
2478 is capable of holding unsigned 64-bit integer values.
2482 \begin_inset LatexCommand \label{sec:ni-type}
2493 is an integral type used for identifying different network interfaces.
2494 Users will need to consult the local documentation to determine appropriate
2495 values for the interfaces available.
2500 identifies the default interface.
2504 \begin_inset LatexCommand \label{sec:id-type}
2515 is an integral type used for representing node ids
2519 is an integral type for representing process ids, and
2523 is an integral type for representing user ids.
2530 matches any process identifier, PTL_NID_ANY matches any node identifier,
2535 matches any user identifier.
2537 \begin_inset LatexCommand \ref{sec:meattach}
2543 \begin_inset LatexCommand \ref{sec:acentry}
2547 for uses of these values.
2551 \begin_inset LatexCommand \label{sec:stat-type}
2558 Each network interface maintains an array of status registers that can be
2563 function (see Section\SpecialChar ~
2565 \begin_inset LatexCommand \ref{sec:nistatus}
2574 defines the types of indexes that can be used to access the status registers.
2575 The only index defined for all implementations is
2579 which identifies the status register that counts the dropped requests for
2581 Other indexes (and registers) may be defined by the implementation.
2588 defines the types of values held in status registers.
2589 This is a signed integer type.
2590 The size is implementation dependent, but must be at least 32 bits.
2593 Initialization and Cleanup
2594 \begin_inset LatexCommand \label{sec:init}
2601 The Portals API includes a function,
2605 , to initialize the library and a function,
2609 , to cleanup after the application is done using the library.
2613 \begin_inset LatexCommand \label{sec:ptlinit}
2620 int PtlInit( int *max_interfaces );
2627 function initializes the Portals library.
2628 PtlInit must be called at least once by a process before any thread makes
2629 a Portals function call, but may be safely called more than once.
2630 \layout Subsubsection
2635 PTL_OK Indicates success.
2639 PTL_FAIL Indicates an error during initialization.
2643 PTL_SEGV Indicates that
2647 is not a legal address.
2649 \layout Subsubsection
2655 \begin_inset Tabular
2656 <lyxtabular version="3" rows="1" columns="3">
2658 <column alignment="right" valignment="top" width="0pt">
2659 <column alignment="center" valignment="top" width="0pt">
2660 <column alignment="left" valignment="top" width="5in">
2662 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
2672 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
2682 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
2687 On successful return, this location will hold the maximum number of interfaces
2688 that can be initialized.
2700 \begin_inset LatexCommand \label{sec:ptlfini}
2707 void PtlFini( void );
2714 function cleans up after the Portals library is no longer needed by a process.
2715 After this function is called, calls to any of the functions defined by
2716 the Portal API or use of the structures set up by the Portals API will
2717 result in undefined behavior.
2718 This function should be called once and only once during termination by
2720 Typically, this function will be called in the exit sequence of a process.
2721 Individual threads should not call PtlFini when they terminate.
2725 \begin_inset LatexCommand \label{sec:ni}
2732 The Portals API supports the use of multiple network interfaces.
2733 However, each interface is treated as an independent entity.
2734 Combining interfaces (e.g.,
2735 \begin_inset Quotes eld
2739 \begin_inset Quotes erd
2742 to create a higher bandwidth connection) must be implemented by the application
2743 or embedded in the underlying network.
2744 Interfaces are treated as independent entities to make it easier to cache
2745 information on individual network interface cards.
2748 Once initialized, each interface provides a Portal table, an access control
2749 table, and a collection of status registers.
2750 See Section\SpecialChar ~
2752 \begin_inset LatexCommand \ref{sec:me}
2756 for a discussion of updating Portal table entries using the
2761 See Section\SpecialChar ~
2763 \begin_inset LatexCommand \ref{sec:ac}
2767 for a discussion of the initialization and updating of entries in the access
2769 See Section\SpecialChar ~
2771 \begin_inset LatexCommand \ref{sec:nistatus}
2775 for a discussion of the
2779 function which can be used to determine the value of a status register.
2782 Every other type of Portal object (e.g., memory descriptor, event queue, or
2783 match list entry) is associated with a specific network interface.
2784 The association to a network interface is established when the object is
2785 created and is encoded in the handle for the object.
2788 Each network interface is initialized and shutdown independently.
2789 The initialization routine,
2793 , returns a handle for an interface object which is used in all subsequent
2799 function is used to shutdown an interface and release any resources that
2800 are associated with the interface.
2801 Network interface handles are associated with processes, not threads.
2802 All threads in a process share all of the network interface handles.
2805 The Portals API also defines the
2809 function to query the status registers for a network interface, the
2813 function to determine the
2814 \begin_inset Quotes eld
2818 \begin_inset Quotes erd
2821 to another process, and the
2825 function to determine the network interface that an object is associated
2830 \begin_inset LatexCommand \label{sec:niinit}
2839 int max_match_entries;
2841 int max_mem_descriptors;
2843 int max_event_queues;
2845 ptl_ac_index_t max_atable_index;
2847 ptl_pt_index_t max_ptable_index;
2853 int PtlNIInit( ptl_interface_t interface
2857 ptl_ni_limits_t* desired,
2859 ptl_ni_limits_t* actual,
2861 ptl_handle_ni_t* handle );
2868 include the following members:
2871 max_match_entries Maximum number of match entries that can be allocated
2875 max_mem_descriptors Maximum number of memory descriptors that can be allocated
2879 max_event_queues Maximum number of event queues that can be allocated at
2883 max_atable_index Largest access control table index for this interface,
2884 valid indexes range from zero to
2891 max_ptable_index Largest Portal table index for this interface, valid indexes
2903 function is used to initialized the Portals API for a network interface.
2904 This function must be called at least once by each process before any other
2905 operations that apply to the interface by any process or thread.
2906 For subsequent calls to
2910 from within the same process (either by different threads or the same thread),
2911 the desired limits will be ignored and the call will return the existing
2913 \layout Subsubsection
2918 PTL_OK Indicates success.
2922 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
2926 PTL_INIT_DUP Indicates a duplicate initialization of
2934 PTL_INIT_INV Indicates that
2938 is not a valid network interface.
2942 PTL_NOSPACE Indicates that there is insufficient memory to initialize the
2947 PTL_INV_PROC Indicates that
2951 is not a valid process id.
2954 PTL_SEGV Indicates that
2962 is not a legal address.
2964 \layout Subsubsection
2970 \begin_inset Tabular
2971 <lyxtabular version="3" rows="5" columns="3">
2973 <column alignment="right" valignment="top" width="0pt">
2974 <column alignment="center" valignment="top" width="0pt">
2975 <column alignment="left" valignment="top" width="4.7in">
2977 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
2987 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
2997 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3002 Identifies the network interface to be initialized.
3003 (See section\SpecialChar ~
3005 \begin_inset LatexCommand \ref{sec:ni-type}
3009 for a discussion of values used to identify network interfaces.)
3014 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3024 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3034 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3039 Identifies the desired process id (for well known process ids).
3044 may be used to have the process id assigned by the underlying library.
3049 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3059 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3069 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3074 If non-NULL, points to a structure that holds the desired limits.
3079 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3089 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3099 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3104 On successful return, the location pointed to by actual will hold the actual
3110 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3120 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3130 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3135 On successful return, this location will hold a handle for the interface.
3146 The use of desired is implementation dependent.
3147 In particular, an implementation may choose to ignore this argument.
3151 \begin_inset LatexCommand \label{sec:nifini}
3158 int PtlNIFini( ptl_handle_ni_t interface );
3165 function is used to release the resources allocated for a network interface.
3170 operation has been started, the results of pending API operations (e.g.,
3171 operations initiated by another thread) for this interface are undefined.
3172 Similarly, the effects of incoming operations (puts and gets) or return
3173 values (acknowledgements and replies) for this interface are undefined.
3174 \layout Subsubsection
3179 PTL_OK Indicates success.
3183 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
3187 PTL_INV_NI Indicates that
3191 is not a valid network interface handle.
3193 \layout Subsubsection
3199 \begin_inset Tabular
3200 <lyxtabular version="3" rows="1" columns="3">
3202 <column alignment="right" valignment="top" width="0pt">
3203 <column alignment="center" valignment="top" width="0pt">
3204 <column alignment="center" valignment="top" width="0pt">
3206 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3214 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3224 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3229 A handle for the interface to shutdown.
3241 \begin_inset LatexCommand \label{sec:nistatus}
3248 int PtlNIStatus( ptl_handle_ni_t interface,
3250 ptl_sr_index_t status_register,
3252 ptl_sr_value_t* status );
3259 function returns the value of a status register for the specified interface.
3260 (See section\SpecialChar ~
3262 \begin_inset LatexCommand \ref{sec:stat-type}
3266 for more information on status register indexes and status register values.)
3267 \layout Subsubsection
3272 PTL_OK Indicates success.
3276 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
3280 PTL_INV_NI Indicates that
3284 is not a valid network interface handle.
3288 PTL_INV_SR_INDX Indicates that
3292 is not a valid status register.
3296 PTL_SEGV Indicates that
3300 is not a legal address.
3302 \layout Subsubsection
3308 \begin_inset Tabular
3309 <lyxtabular version="3" rows="3" columns="3">
3311 <column alignment="right" valignment="top" width="0pt">
3312 <column alignment="center" valignment="top" width="0pt">
3313 <column alignment="left" valignment="top" width="4.7in">
3315 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3325 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3335 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3340 A handle for the interface to use.
3346 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3356 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3366 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3371 An index for the status register to read.
3376 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3386 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3396 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3401 On successful return, this location will hold the current value of the status
3413 The only status register that must be defined is a drop count register (
3418 Implementations may define additional status registers.
3419 Identifiers for the indexes associated with these registers should start
3430 int PtlNIDist( ptl_handle_ni_t interface,
3432 ptl_process_id_t process,
3434 unsigned long* distance );
3441 function returns the distance to another process using the specified interface.
3442 Distances are only defined relative to an interface.
3443 Distance comparisons between different interfaces on the same process may
3445 \layout Subsubsection
3450 PTL_OK Indicates success.
3454 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
3458 PTL_INV_NI Indicates that
3462 is not a valid network interface handle.
3466 PTL_INV_PROC Indicates that
3470 is not a valid process identifier.
3474 PTL_SEGV Indicates that
3478 is not a legal address.
3480 \layout Subsubsection
3486 \begin_inset Tabular
3487 <lyxtabular version="3" rows="3" columns="3">
3489 <column alignment="right" valignment="top" width="0pt">
3490 <column alignment="center" valignment="top" width="0pt">
3491 <column alignment="left" valignment="top" width="4.7in">
3493 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3503 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3513 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3518 A handle for the interface to use.
3524 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3534 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3544 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3549 An identifier for the process whose distance is being requested.
3555 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3565 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3575 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3580 On successful return, this location will hold the distance to the remote
3592 This function should return a static measure of distance.
3593 Examples include minimum latency, the inverse of available bandwidth, or
3594 the number of switches between the two endpoints.
3600 int PtlNIHandle( ptl_handle_any_t handle,
3602 ptl_handle_ni_t* interface );
3609 function returns a handle for the network interface with which the object
3615 If the object identified by
3619 is a network interface, this function returns the same value it is passed.
3620 \layout Subsubsection
3625 PTL_OK Indicates success.
3629 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
3633 PTL_INV_HANDLE Indicates that
3637 is not a valid handle.
3641 PTL_SEGV Indicates that
3645 is not a legal address.
3647 \layout Subsubsection
3653 \begin_inset Tabular
3654 <lyxtabular version="3" rows="2" columns="3">
3656 <column alignment="right" valignment="top" width="0pt">
3657 <column alignment="center" valignment="top" width="0pt">
3658 <column alignment="left" valignment="top" width="4.7in">
3660 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3670 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3680 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3685 A handle for the object.
3690 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3700 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3710 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3715 On successful return, this location will hold a handle for the network interface
3731 Every handle should encode the network interface and the object id relative
3733 Both are presumably encoded using integer values.
3737 \begin_inset LatexCommand \label{sec:uid}
3744 Every process runs on behalf of a user.
3751 int PtlGetUid( ptl_handle_ni_t ni_handle,
3754 \layout Subsubsection
3759 PTL_OK Indicates success.
3763 PTL_INV_NI Indicates that
3767 is not a valid network interface handle.
3771 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
3775 PTL_SEGV Indicates that
3779 is not a legal address.
3781 \layout Subsubsection
3787 \begin_inset Tabular
3788 <lyxtabular version="3" rows="2" columns="3">
3790 <column alignment="right" valignment="top" width="0pt">
3791 <column alignment="center" valignment="top" width="0pt">
3792 <column alignment="left" valignment="top" width="5in">
3794 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3804 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3814 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3819 A network interface handle.
3824 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3834 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3844 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
3849 On successful return, this location will hold the user id for the calling
3861 Note that user identifiers are dependent on the network interface(s).
3862 In particular, if a node has multiple interfaces, a process may have multiple
3866 Process Identification
3867 \begin_inset LatexCommand \label{sec:pid}
3874 Processes that use the Portals API, can be identified using a node id and
3876 Every node accessible through a network interface has a unique node identifier
3877 and every process running on a node has a unique process identifier.
3878 As such, any process in the computing system can be identified by its node
3883 The Portals API defines a type,
3887 for representing process ids and a function,
3891 , which can be used to obtain the id of the current process.
3894 The portals API does not include thread identifiers.
3895 Messages are delivered to processes (address spaces) not threads (contexts
3900 \begin_inset LatexCommand \label{sec:pid-type}
3909 ptl_nid_t nid; /* node id */
3911 ptl_pid_t pid; /* process id */
3920 type uses two identifiers to represent a process id: a node id and a process
3926 \begin_inset LatexCommand \label{sub:PtlGetId}
3933 int PtlGetId( ptl_handle_ni_t ni_handle,
3935 ptl_process_id_t* id );
3936 \layout Subsubsection
3941 PTL_OK Indicates success.
3945 PTL_INV_NI Indicates that
3949 is not a valid network interface handle.
3953 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
3957 PTL_SEGV Indicates that
3961 is not a legal address.
3963 \layout Subsubsection
3969 \begin_inset Tabular
3970 <lyxtabular version="3" rows="2" columns="3">
3972 <column alignment="right" valignment="top" width="0pt">
3973 <column alignment="center" valignment="top" width="0pt">
3974 <column alignment="left" valignment="top" width="5in">
3976 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3986 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
3996 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4001 A network interface handle.
4006 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4016 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4026 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4031 On successful return, this location will hold the id for the calling process.
4042 Note that process identifiers are dependent on the network interface(s).
4043 In particular, if a node has multiple interfaces, it may have multiple
4047 Match List Entries and Match Lists
4048 \begin_inset LatexCommand \label{sec:me}
4055 A match list is a chain of match list entries.
4056 Each match list entry includes a memory descriptor and a set of match criteria.
4057 The match criteria can be used to reject incoming requests based on process
4058 id or the match bits provided in the request.
4059 A match list is created using the
4067 functions, which create a match list consisting of a single match list
4068 entry, attaches the match list to the specified Portal index, and returns
4069 a handle for the match list entry.
4070 Match entries can be dynamically inserted and removed from a match list
4083 \begin_inset LatexCommand \label{sec:meattach}
4090 typedef enum { PTL_RETAIN, PTL_UNLINK } ptl_unlink_t;
4095 typedef enum { PTL_INS_BEFORE, PTL_INS_AFTER } ptl_ins_pos_t;
4100 int PtlMEAttach( ptl_handle_ni_t interface,
4102 ptl_pt_index_t index,
4104 ptl_process_id_t matchid,
4106 ptl_match_bits_t match_bits,
4108 ptl_match_bits_t ignorebits,
4110 ptl_unlink_t unlink,
4112 ptl_ins_pos_t position,
4114 ptl_handle_me_t* handle );
4121 are used to control where a new item is inserted.
4126 is used to insert the new item before the current item or before the head
4132 is used to insert the new item after the current item or after the last
4141 function creates a match list consisting of a single entry and attaches
4142 this list to the Portal table for
4147 \layout Subsubsection
4152 PTL_OK Indicates success.
4156 PTL_INV_NI Indicates that
4160 is not a valid network interface handle.
4164 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
4168 PTL_INV_PTINDEX Indicates that
4172 is not a valid Portal table index.
4176 PTL_INV_PROC Indicates that
4180 is not a valid process identifier.
4184 PTL_NOSPACE Indicates that there is insufficient memory to allocate the
4189 PTL_ML_TOOLONG Indicates that the resulting match list is too long.
4190 The maximum length for a match list is defined by the interface.
4192 \layout Subsubsection
4198 \begin_inset Tabular
4199 <lyxtabular version="3" rows="7" columns="3">
4201 <column alignment="left" valignment="top" width="0.8in">
4202 <column alignment="center" valignment="top" width="0pt">
4203 <column alignment="left" valignment="top" width="4.75in">
4205 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4215 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4225 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4230 A handle for the interface to use.
4236 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4246 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4256 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4261 The Portal table index where the match list should be attached.
4266 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4276 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4286 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4291 Specifies the match criteria for the process id of the requestor.
4300 can be used to wildcard either of the ids in the
4310 <cell alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
4317 match_bits, ignorebits
4320 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4330 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4335 Specify the match criteria to apply to the match bits in the incoming request.
4340 are used to mask out insignificant bits in the incoming match bits.
4341 The resulting bits are then compared to the match list entry's match
4342 bits to determine if the incoming request meets the match criteria.
4347 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4357 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4367 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4372 Indicates the match list entry should be unlinked when the last memory descripto
4373 r associated with this match list entry is unlinked.
4374 (Note, the check for unlinking a match entry only occurs when a memory
4375 descriptor is unlinked.)
4380 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4390 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4400 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4405 Indicates whether the new match entry should be prepended or appended to
4406 the existing match list.
4407 If there is no existing list, this argument is ignored and the new match
4408 entry becomes the only entry in the list.
4422 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4432 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4442 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4447 On successful return, this location will hold a handle for the newly created
4460 \begin_inset LatexCommand \label{sec:attachany}
4467 int PtlMEAttachAny( ptl_handle_ni_t interface,
4469 ptl_pt_index_t *index,
4471 ptl_process_id_t matchid,
4473 ptl_match_bits_t match_bits,
4475 ptl_match_bits_t ignorebits,
4477 ptl_unlink_t unlink,
4479 ptl_handle_me_t* handle );
4486 function creates a match list consisting of a single entry and attaches
4487 this list to an unused Portal table entry for
4492 \layout Subsubsection
4497 PTL_OK Indicates success.
4501 PTL_INV_NI Indicates that
4505 is not a valid network interface handle.
4509 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
4513 PTL_INV_PROC Indicates that
4517 is not a valid process identifier.
4521 PTL_NOSPACE Indicates that there is insufficient memory to allocate the
4526 PTL_PT_FULL Indicates that there are no free entries in the Portal table.
4527 \layout Subsubsection
4533 \begin_inset Tabular
4534 <lyxtabular version="3" rows="4" columns="3">
4536 <column alignment="left" valignment="top" width="0.8in">
4537 <column alignment="center" valignment="top" width="0pt">
4538 <column alignment="left" valignment="top" width="4.75in">
4540 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4550 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4560 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4565 A handle for the interface to use.
4571 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4581 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4591 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4596 On succesfful return, this location will hold the Portal index where the
4597 match list has been attached.
4602 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4609 matchid, match_bits, ignorebits, unlink
4612 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4622 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4627 See the discussion for
4636 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4646 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4656 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4661 On successful return, this location will hold a handle for the newly created
4674 \begin_inset LatexCommand \label{sec:meinsert}
4681 int PtlMEInsert( ptl_handle_me_t current,
4683 ptl_process_id_t matchid,
4685 ptl_match_bits_t match_bits,
4687 ptl_match_bits_t ignorebits,
4689 ptl_ins_pos_t position,
4691 ptl_handle_me_t* handle );
4698 function creates a new match list entry and inserts this entry into the
4699 match list containing
4704 \layout Subsubsection
4709 PTL_OK Indicates success.
4713 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
4717 PTL_INV_PROC Indicates that
4721 is not a valid process identifier.
4725 PTL_INV_ME Indicates that
4729 is not a valid match entry handle.
4733 PTL_ML_TOOLONG Indicates that the resulting match list is too long.
4734 The maximum length for a match list is defined by the interface.
4738 PTL_NOSPACE Indicates that there is insufficient memory to allocate the
4741 \layout Subsubsection
4747 \begin_inset Tabular
4748 <lyxtabular version="3" rows="4" columns="3">
4750 <column alignment="left" valignment="top" width="0.8in">
4751 <column alignment="center" valignment="top" width="0pt">
4752 <column alignment="left" valignment="top" width="4.7in">
4754 <cell alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
4764 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4774 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4779 A handle for a match entry.
4780 The new match entry will be inserted immediately before or immediately
4781 after this match entry.
4786 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4810 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4820 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4825 See the discussion for
4834 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4844 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4854 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4859 Indicates whether the new match entry should be inserted before or after
4878 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4888 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4898 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4903 See the discussion for
4919 \begin_inset LatexCommand \label{sec:meunlink}
4926 int PtlMEUnlink( ptl_handle_me_t entry );
4933 function can be used to unlink a match entry from a match list.
4934 This operation also releases any resources associated with the match entry
4935 (including the associated memory descriptor).
4936 It is an error to use the match entry handle after calling
4941 \layout Subsubsection
4946 PTL_OK Indicates success.
4950 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
4954 PTL_INV_ME Indicates that
4958 is not a valid match entry handle.
4960 \layout Subsubsection
4966 \begin_inset Tabular
4967 <lyxtabular version="3" rows="1" columns="3">
4969 <column alignment="right" valignment="top" width="0pt">
4970 <column alignment="center" valignment="top" width="0pt">
4971 <column alignment="center" valignment="top" width="0pt">
4973 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4983 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
4993 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
4998 A handle for the match entry to be unlinked.
5010 \begin_inset LatexCommand \label{sec:md}
5017 A memory descriptor contains information about a region of an application
5018 process' memory and an event queue where information about the operations
5019 performed on the memory descriptor are recorded.
5020 The Portals API provides two operations to create memory descriptors:
5028 ; an operation to update a memory descriptor,
5032 ; and an operation to unlink and release the resources associated with a
5040 The Memory Descriptor Type
5041 \begin_inset LatexCommand \label{sec:md-type}
5056 unsigned int max_offset;
5058 unsigned int options;
5062 ptl_handle_eq_t eventq;
5071 type defines the application view of a memory descriptor.
5072 Values of this type are used to initialize and update the memory descriptors.
5073 \layout Subsubsection
5078 start,\SpecialChar ~
5079 length Specify the memory region associated with the memory descriptor.
5084 member specifies the starting address for the memory region and the
5088 member specifies the length of the region.
5093 can be NULL provided that the
5098 (Zero length buffers are useful to record events.) There are no alignment
5099 restrictions on the starting address or the length of the region; although,
5100 unaligned messages may be slower (i.e., lower bandwidth and/or longer latency)
5101 on some implementations.
5105 threshold Specifies the maximum number of operations that can be performed
5106 on the memory descriptor.
5107 An operation is any action that could possibly generate an event (see Section\SpecialChar ~
5109 \begin_inset LatexCommand \ref{sec:ek-type}
5113 for the different types of events).
5114 In the usual case, the threshold value is decremented for each operation
5115 on the memory descriptor.
5116 When the threshold value is zero, the memory descriptor is
5120 , and does not respond to operations.
5121 A memory descriptor can have an initial threshold value of zero to allow
5122 for manipulation of an inactive memory descriptor by the local process.
5123 A threshold value of
5127 indicates that there is no bound on the number of operations that may be
5128 applied to a memory descriptor.
5129 Note that local operations (e.g.,
5133 ) are not applied to the threshold count.
5137 max_offset Specifies the maximum local offset of a memory descriptor.
5138 When the local offset of a memory descriptor exceeds this maximum, the
5139 memory descriptor becomes
5143 and does not respond to further operations.
5146 options Specifies the behavior of the memory descriptor.
5147 There are five options that can be selected: enable put operations (yes
5148 or no), enable get operations (yes or no), offset management (local or
5149 remote), message truncation (yes or no), and acknowledgement (yes or no).
5150 Values for this argument can be constructed using a bitwise or of the following
5156 PTL_MD_OP_PUT Specifies that the memory descriptor will respond to
5161 By default, memory descriptors reject
5169 PTL_MD_OP_GET Specifies that the memory descriptor will respond to
5174 By default, memory descriptors reject
5182 PTL_MD_MANAGE_REMOTE Specifies that the offset used in accessing the memory
5183 region is provided by the incoming request.
5184 By default, the offset is maintained locally.
5185 When the offset is maintained locally, the offset is incremented by the
5186 length of the request so that the next operation (put and/or get) will
5187 access the next part of the memory region.
5190 PTL_MD_TRUNCATE Specifies that the length provided in the incoming request
5191 can be reduced to match the memory available in the region.
5192 (The memory available in a memory region is determined by subtracting the
5193 offset from the length of the memory region.) By default, if the length
5194 in the incoming operation is greater than the amount of memory available,
5195 the operation is rejected.
5199 PTL_MD_ACK_DISABLE Specifies that an acknowledgement should
5203 be sent for incoming
5207 operations, even if requested.
5208 By default, acknowledgements are sent for
5212 operations that request an acknowledgement.
5213 Acknowledgements are never sent for
5218 The value sent in the reply serves as an implicit acknowledgement.
5227 : It is not considered an error to have a memory descriptor that does not
5236 operations: Every memory descriptor responds to
5241 Nor is it considered an error to have a memory descriptor that responds
5255 user_ptr A user-specified value that is associated with the memory descriptor.
5256 The value does not need to be a pointer, but must fit in the space used
5258 This value (along with other values) is recorded in events associated with
5259 operations on this memory descriptor.
5265 Tying the memory descriptor to a user-defined value can be useful when multiple
5266 memory descriptor share the same event queue or when the memory descriptor
5267 needs to be associated with a data structure maintained by the application.
5268 For example, an MPI implementation can set the
5272 argument to the value of an MPI Request.
5273 This direct association allows for processing of memory descriptor's by
5274 the MPI implementation without a table lookup or a search for the appropriate
5281 eventq A handle for the event queue used to log the operations performed
5282 on the memory region.
5287 , operations performed on this memory descriptor are not logged.
5292 \begin_inset LatexCommand \label{sec:mdattach}
5299 int PtlMDAttach( ptl_handle_me_t match,
5303 ptl_unlink_t unlink_op,
5305 ptl_unlink_t unlink_nofit,
5307 ptl_handle_md_t* handle );
5314 are used to control whether an item is unlinked from a list.
5331 operation is used to create a memory descriptor and attach it to a match
5333 An error code is returned if this match list entry already has an associated
5335 \layout Subsubsection
5340 PTL_OK Indicates success.
5344 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
5348 PTL_INUSE Indicates that
5352 already has a memory descriptor attached.
5356 PTL_INV_ME Indicates that
5360 is not a valid match entry handle.
5364 PTL_ILL_MD Indicates that
5368 is not a legal memory descriptor.
5369 This may happen because the memory region defined in
5373 is invalid or because the network interface associated with the
5381 is not the same as the network interface associated with
5389 PTL_NOSPACE Indicates that there is insufficient memory to allocate the
5392 \layout Subsubsection
5398 \begin_inset Tabular
5399 <lyxtabular version="3" rows="5" columns="3">
5401 <column alignment="right" valignment="top" width="0pt">
5402 <column alignment="center" valignment="top" width="0pt">
5403 <column alignment="left" valignment="top" width="4.7in">
5405 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5415 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5425 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
5430 A handle for the match entry that the memory descriptor will be associated
5436 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5446 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5456 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
5461 Provides initial values for the application visible parts of a memory descriptor.
5462 Other than its use for initialization, there is no linkage between this
5463 structure and the memory descriptor maintained by the API.
5469 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5479 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5489 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
5494 A flag to indicate whether the memory descriptor is unlinked when it becomes
5495 inactive, either because the operation threshold drops to zero or because
5496 the maximum offset has been exceeded.
5497 (Note, the check for unlinking a memory descriptor only occurs after a
5498 the completion of a successful operation.
5499 If the threshold is set to zero during initialization or using
5503 , the memory descriptor is
5512 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5522 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5532 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
5537 A flag to indicate whether the memory descriptor is unlinked when the space
5538 remaining in the memory descriptor is not sufficient for a matching operation.
5539 If an incoming message arrives arrives at a memory descriptor that does
5540 not have sufficient space and the
5544 operation is not specified, the memory descriptor will be unlinked.
5549 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5559 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5569 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
5574 On successful return, this location will hold a handle for the newly created
5580 argument can be NULL, in which case the handle will not be returned.
5592 \begin_inset LatexCommand \label{sec:mdbind}
5599 int PtlMDBind( ptl_handle_ni_t interface,
5603 ptl_handle_md_t* handle );
5610 operation is used to create a
5611 \begin_inset Quotes eld
5615 \begin_inset Quotes erd
5618 memory descriptor, i.e., a memory descriptor that is not associated with
5620 \layout Subsubsection
5625 PTL_OK Indicates success.
5629 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
5633 PTL_INV_NI Indicates that
5637 is not a valid match entry handle.
5641 PTL_ILL_MD Indicates that
5645 is not a legal memory descriptor.
5646 This may happen because the memory region defined in
5650 is invalid or because the network interface associated with the
5658 is not the same as the network interface,
5666 PTL_INV_EQ Indicates that the event queue associated with
5674 PTL_NOSPACE Indicates that there is insufficient memory to allocate the
5679 PTL_SEGV Indicates that
5683 is not a legal address.
5685 \layout Subsubsection
5691 \begin_inset Tabular
5692 <lyxtabular version="3" rows="3" columns="3">
5694 <column alignment="right" valignment="top" width="0pt">
5695 <column alignment="center" valignment="top" width="0pt">
5696 <column alignment="left" valignment="top" width="4.7in">
5698 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5708 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5718 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
5723 A handle for the network interface with which the memory descriptor will
5729 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5739 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5749 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
5754 Provides initial values for the application visible parts of a memory descriptor.
5755 Other than its use for initialization, there is no linkage between this
5756 structure and the memory descriptor maintained by the API.
5762 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5772 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5782 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
5787 On successful return, this location will hold a handle for the newly created
5793 argument must be a valid address and cannot be NULL.
5805 \begin_inset LatexCommand \label{sec:mdfree}
5812 int PtlMDUnlink( ptl_handle_md_t mem_desc );
5819 function unlinks the memory descriptor from any match list entry it may
5820 be linked to and releases the resources associated with a memory descriptor.
5821 (This function does not free the memory region associated with the memory
5822 descriptor.) This function also releases the resources associated with a
5823 floating memory descriptor.
5824 Only memory descriptors with no pending operations may be unlinked.
5825 \layout Subsubsection
5830 PTL_OK Indicates success.
5834 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
5838 PTL_INV_MD Indicates that
5842 is not a valid memory descriptor handle.
5845 PTL_MD_INUSE Indicates that
5849 has pending operations and cannot be unlinked.
5850 \layout Subsubsection
5856 \begin_inset Tabular
5857 <lyxtabular version="3" rows="1" columns="3">
5859 <column alignment="right" valignment="top" width="0pt">
5860 <column alignment="center" valignment="top" width="0pt">
5861 <column alignment="left" valignment="top" width="4.7in">
5863 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5873 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
5883 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
5888 A handle for the memory descriptor to be released.
5900 \begin_inset LatexCommand \label{sec:mdupdate}
5907 int PtlMDUpdate( ptl_handle_md_t mem_desc,
5913 ptl_handle_eq_t testq );
5920 function provides a conditional, atomic update operation for memory descriptors.
5921 The memory descriptor identified by
5925 is only updated if the event queue identified by
5930 The intent is to only enable updates to the memory descriptor when no new
5931 messages have arrived since the last time the queue was checked.
5932 See section\SpecialChar ~
5934 \begin_inset LatexCommand \ref{sec:exmpi}
5938 for an example of how this function can be used.
5945 is not NULL the memory descriptor identified by handle will be updated
5946 to reflect the values in the structure pointed to by
5958 or if the event queue identified by
5967 is not NULL, the current value of the memory descriptor identified by
5971 is recorded in the location identified by
5976 \layout Subsubsection
5981 PTL_OK Indicates success.
5985 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
5989 PTL_NOUPDATE Indicates that the update was not performed because
5997 PTL_INV_MD Indicates that
6001 is not a valid memory descriptor handle.
6005 PTL_ILL_MD Indicates that the value pointed to by
6009 is not a legal memory descriptor (e.g., the memory region specified by the
6010 memory descriptor may be invalid).
6014 PTL_INV_EQ Indicates that
6018 is not a valid event queue handle.
6022 PTL_SEGV Indicates that
6030 is not a legal address.
6032 \layout Subsubsection
6038 \begin_inset Tabular
6039 <lyxtabular version="3" rows="4" columns="3">
6041 <column alignment="right" valignment="top" width="0pt">
6042 <column alignment="center" valignment="top" width="0pt">
6043 <column alignment="left" valignment="top" width="4.7in">
6045 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6055 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6065 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
6070 A handle for the memory descriptor to update.
6075 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6085 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6095 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
6108 , the current value of the memory descriptor will be stored in the location
6118 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6128 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6138 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
6151 , this argument provides the new values for the memory descriptor, if the
6152 update is performed.
6157 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6167 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6177 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
6182 A handle for an event queue used to predicate the update.
6191 , the update is performed unconditionally.
6192 Otherwise, the update is performed if and only if
6197 If the update is not performed, the function returns the value
6206 argument does not need to be the same as the event queue associated with
6207 the memory descriptor.)
6218 The conditional update can be used to ensure that the memory descriptor
6219 has not changed between the time it was examined and the time it is updated.
6220 In particular, it is needed to support an MPI implementation where the
6221 activity of searching an unexpected message queue and posting a receive
6225 Events and Event Queues
6226 \begin_inset LatexCommand \label{sec:eq}
6233 Event queues are used to log operations performed on memory descriptors.
6234 They can also be used to hold acknowledgements for completed
6238 operations and to note when the data specified in a
6242 operation has been sent (i.e., when it is safe to reuse the buffer that holds
6244 Multiple memory descriptors can share a single event queue.
6251 type, the Portals API defines two types associated with events: The
6257 type defines the kinds of events that can be stored in an event queue.
6262 type defines a structure that holds the information associated with an
6266 The Portals API also provides four functions for dealing with event queues:
6271 function is used to allocate the API resources needed for an event queue,
6276 function is used to release these resources, the
6280 function can be used to get the next event from an event queue, and the
6285 function can be used to block a process (or thread) until an event queue
6286 has at least one event.
6290 \begin_inset LatexCommand \label{sec:ek-type}
6299 PTL_EVENT_GET_START, PTL_EVENT_GET_END, PTL_EVENT_GET_FAIL,
6301 PTL_EVENT_PUT_START, PTL_EVENT_PUT_END, PTL_EVENT_PUT_FAIL,
6303 PTL_EVENT_REPLY_START, PTL_EVENT_REPLY_END, PTL_EVENT_REPLY_FAIL,
6305 PTL_EVENT_SEND_START, PTL_EVENT_SEND_END, PTL_EVENT_SEND_FAIL,
6314 The Portals API defines fourteen types of events that can be logged in an
6318 PTL_EVENT_GET_START A remote
6322 operation has been started on the memory descriptor.
6323 The memory region associated with this descriptor should not be altered
6324 until the corresponding END or FAIL event is logged.
6327 PTL_EVENT_GET_END A previously initiated
6331 operation completed successfully.
6332 This event is logged after the reply has been sent by the local node.
6333 As such, the process could free the memory descriptor once it sees this
6338 PTL_EVENT_GET_FAIL A previously initiated
6342 operation completed unsuccessfully.
6343 This event is logged after the reply has been sent by the local node.
6344 As such, the process could free the memory descriptor once it sees this
6349 PTL_EVENT_PUT_START A remote
6353 operation has been started on the memory descriptor.
6354 The memory region associated with this descriptor should should be considered
6355 volatile until the corresponding END or FAIL event is logged.
6358 PTL_EVENT_PUT_END A previously initiated
6362 operation completed successfully.
6363 The underlying layers will not alter the memory (on behalf of this operation)
6364 once this event has been logged.
6368 PTL_EVENT_PUT_FAIL A previously initiated
6372 operation completed unsuccessfully.
6373 The underlying layers will not alter the memory (on behalf of this operation)
6374 once this event has been logged.
6378 PTL_EVENT_REPLY_START A
6382 operation has been started on the memory descriptor.
6386 PTL_EVENT_REPLY_END A previously initiated
6390 operation has completed successfully .
6391 This event is logged after the data (if any) from the reply has been written
6392 into the memory descriptor.
6396 PTL_EVENT_REPLY_FAIL A previously initiated
6400 operation has completed unsuccessfully.
6401 This event is logged after the data (if any) from the reply has been written
6402 into the memory descriptor.
6411 This event is logged when the acknowledgement is received
6414 PTL_EVENT_SEND_START An outgoing
6418 operation has been started.
6419 The memory region associated with this descriptor should not be altered
6420 until the corresponding END or FAIL event is logged.
6423 PTL_EVENT_SEND_END A previously initiated
6427 operation has completed successfully.
6428 This event is logged after the entire buffer has been sent and it is safe
6429 for the application to reuse the buffer.
6433 PTL_EVENT_SEND_FAIL A previously initiated
6437 operation has completed unsuccessfully.
6438 The process can safely manipulate the memory or free the memory descriptor
6439 once it sees this event.
6442 PTL_EVENT_UNLINK A memory descriptor associated with this event queue has
6443 been automatically unlinked.
6444 This event is not generated when a memory descriptor is explicitly unlinked
6450 This event does not decrement the threshold count.
6456 The Portals API guarantees that a when a process initiates two operations
6457 on a remote process, the operations will be initiated on the remote process
6458 in the same order that they were initiated on the original process.
6459 As an example, if process A intitates two
6471 , on process B, the Portals API guarantees that process A will receive the
6474 PTL_EVENT_SEND_START
6484 in the same order that process B receives the
6497 Notice that the API does not guarantee that the start events will be delivered
6498 in the same order that process A initiated the
6507 If process A needs to ensure the ordering of these operations, it should
6508 include code to wait for the initiation of
6519 Failure Notification
6522 Operations may fail to complete successfully; however, unless the node itself
6523 fails, every operation that is started will eventually complete.
6524 While an operation is in progress, the memory associated with the operation
6525 should not be viewed (in the case of a put or a reply) or altered (in the
6526 case of a send or get).
6527 Operation completion, whether successful or unsuccessful, is final.
6528 That is, when an operation completes, the memory associated with the operation
6529 will no longer be read or altered by the operation.
6530 A network interface can use the
6534 to define more specific information regarding the failure of the operation
6535 and record this information in the
6543 \begin_inset LatexCommand \label{sec:event-type}
6552 ptl_event_kind_t type;
6554 ptl_process_id_t initiator;
6559 ptl_pt_index_t portal;
6561 ptl_match_bits_t match_bits;
6569 ptl_handle_md_t md_handle;
6573 ptl_hdr_data_t hdr_data;
6577 ptl_ni_fail_t ni_fail_type;
6579 volatile ptl_seq_t sequence;
6584 An event structure includes the following members:
6587 type Indicates the type of the event.
6591 initiator The id of the initiator.
6595 portal The Portal table index specified in the request.
6599 match_bits A copy of the match bits specified in the request.
6600 See section\SpecialChar ~
6602 \begin_inset LatexCommand \ref{sec:me}
6606 for more information on match bits.
6610 rlength The length (in bytes) specified in the request.
6614 mlength The length (in bytes) of the data that was manipulated by the operation.
6615 For truncated operations, the manipulated length will be the number of
6616 bytes specified by the memory descriptor (possibly with an offset) operation.
6617 For all other operations, the manipulated length will be the length of
6618 the requested operation.
6622 offset Is the displacement (in bytes) into the memory region that the operation
6624 The offset can be determined by the operation (see Section\SpecialChar ~
6626 \begin_inset LatexCommand \ref{sec:datamovement}
6630 ) for a remote managed memory descriptor, or by the local memory descriptor
6631 (see Section\SpecialChar ~
6633 \begin_inset LatexCommand \ref{sec:md}
6641 md_handle Is the handle to the memory descriptor associated with the event.
6644 mem_desc Is the state of the memory descriptor immediately after the event
6649 hdr_data 64 bits of out-of-band user data (see Section\SpecialChar ~
6651 \begin_inset LatexCommand \ref{sec:put}
6663 member is used to link
6675 event that signifies completion of the operation.
6680 member will be the same for the two events associated with an operation.
6681 The link member is also used to link an
6685 event with the event that caused the memory descriptor to be unlinked.
6688 sequence The sequence number for this event.
6689 Sequence numbers are unique to each event.
6696 member is the last member and is volatile to support SMP implementations.
6697 When an event structure is filled in, the
6701 member should be written after all other members have been updated.
6702 Moreover, a memory barrier should be inserted between the updating of other
6703 members and the updating of the
6711 \begin_inset LatexCommand \label{sec:eqalloc}
6718 int PtlEQAlloc( ptl_handle_ni_t interface,
6722 ptl_handle_eq_t* handle );
6729 function is used to build an event queue.
6731 \layout Subsubsection
6736 PTL_OK Indicates success.
6740 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
6744 PTL_INV_NI Indicates that
6748 is not a valid network interface handle.
6752 PTL_NOSPACE Indicates that there is insufficient memory to allocate the
6757 PTL_SEGV Indicates that
6761 is not a legal address.
6763 \layout Subsubsection
6769 \begin_inset Tabular
6770 <lyxtabular version="3" rows="3" columns="3">
6772 <column alignment="right" valignment="top" width="0pt">
6773 <column alignment="center" valignment="top" width="0pt">
6774 <column alignment="left" valignment="top" width="4.7in">
6776 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6786 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6796 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
6801 A handle for the interface with which the event queue will be associated.
6806 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6816 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6826 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
6831 The number of events that can be stored in the event queue.
6836 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6846 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6856 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
6861 On successful return, this location will hold a handle for the newly created
6874 \begin_inset LatexCommand \label{sec:eqfree}
6881 int PtlEQFree( ptl_handle_eq_t eventq );
6888 function releases the resources associated with an event queue.
6889 It is up to the user to insure that no memory descriptors are associated
6890 with the event queue once it is freed.
6892 \layout Subsubsection
6897 PTL_OK Indicates success.
6901 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
6905 PTL_INV_EQ Indicates that
6909 is not a valid event queue handle.
6911 \layout Subsubsection
6917 \begin_inset Tabular
6918 <lyxtabular version="3" rows="1" columns="3">
6920 <column alignment="right" valignment="top" width="0pt">
6921 <column alignment="center" valignment="top" width="0pt">
6922 <column alignment="left" valignment="top" width="4.7in">
6924 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6934 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
6944 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
6949 A handle for the event queue to be released.
6961 \begin_inset LatexCommand \label{sec:eqget}
6968 int PtlEQGet( ptl_handle_eq_t eventq,
6970 ptl_event_t* event );
6977 function is a nonblocking function that can be used to get the next event
6979 The event is removed from the queue.
6980 \layout Subsubsection
6985 PTL_OK Indicates success.
6989 PTL_EQ_DROPPED Indicates success (i.e., an event is returned) and that at
6990 least one event between this event and the last event obtained (using
6998 ) from this event queue has been dropped due to limited space in the event
7003 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
7007 PTL_EQ_EMPTY Indicates that
7011 is empty or another thread is waiting on
7019 PTL_INV_EQ Indicates that
7023 is not a valid event queue handle.
7027 PTL_SEGV Indicates that
7031 is not a legal address.
7033 \layout Subsubsection
7039 \begin_inset Tabular
7040 <lyxtabular version="3" rows="2" columns="3">
7042 <column alignment="right" valignment="top" width="0pt">
7043 <column alignment="center" valignment="top" width="0pt">
7044 <column alignment="left" valignment="top" width="4.5in">
7046 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7056 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7066 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7071 A handle for the event queue.
7076 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7086 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7096 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7101 On successful return, this location will hold the values associated with
7102 the next event in the event queue.
7114 \begin_inset LatexCommand \label{sec:eqwait}
7121 int PtlEQWait( ptl_handle_eq_t eventq,
7123 ptl_event_t* event );
7130 function can be used to block the calling process (thread) until there
7131 is an event in an event queue.
7132 This function also returns the next event in the event queue and removes
7133 this event from the queue.
7134 This is the only blocking operation in the Portals 3.2 API.
7135 In the event that multiple threads are waiting on the same event queue,
7136 PtlEQWait is guaranteed to wake exactly one thread, but the order in which
7137 they are awakened is not specified.
7138 \layout Subsubsection
7143 PTL_OK Indicates success.
7147 PTL_EQ_DROPPED Indicates success (i.e., an event is returned) and that at
7148 least one event between this event and the last event obtained (using
7156 ) from this event queue has been dropped due to limited space in the event
7161 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
7165 PTL_INV_EQ Indicates that
7169 is not a valid event queue handle.
7173 PTL_SEGV Indicates that
7177 is not a legal address.
7180 \layout Subsubsection
7186 \begin_inset Tabular
7187 <lyxtabular version="3" rows="2" columns="3">
7189 <column alignment="right" valignment="top" width="0pt">
7190 <column alignment="center" valignment="top" width="0pt">
7191 <column alignment="left" valignment="top" width="4.7in">
7193 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7203 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7213 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7218 A handle for the event queue to wait on.
7219 The calling process (thread) will be blocked until
7228 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7238 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7248 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7253 On successful return, this location will hold the values associated with
7254 the next event in the event queue.
7265 The Access Control Table
7266 \begin_inset LatexCommand \label{sec:ac}
7273 Processes can use the access control table to control which processes are
7274 allowed to perform operations on Portal table entries.
7275 Each communication interface has a Portal table and an access control table.
7276 The access control table for the default interface contains an entry at
7277 index zero that allows all processes with the same user id to communicate.
7278 Entries in the access control table can be manipulated using the
7286 \begin_inset LatexCommand \label{sec:acentry}
7293 int PtlACEntry( ptl_handle_ni_t interface,
7295 ptl_ac_index_t index,
7297 ptl_process_id_t matchid,
7301 ptl_pt_index_t portal );
7308 function can be used to update an entry in the access control table for
7310 \layout Subsubsection
7315 PTL_OK Indicates success.
7319 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
7323 PTL_INV_NI Indicates that
7327 is not a valid network interface handle.
7331 PTL_AC_INV_INDEX Indicates that
7335 is not a valid access control table index.
7339 PTL_INV_PROC Indicates that
7343 is not a valid process identifier.
7347 PTL_PT_INV_INDEX Indicates that
7351 is not a valid Portal table index.
7353 \layout Subsubsection
7359 \begin_inset Tabular
7360 <lyxtabular version="3" rows="5" columns="3">
7362 <column alignment="right" valignment="top" width="0pt">
7363 <column alignment="center" valignment="top" width="0pt">
7364 <column alignment="left" valignment="top" width="4.7in">
7366 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7376 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7386 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7391 Identifies the interface to use.
7396 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7406 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7416 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7421 The index of the entry in the access control table to update.
7426 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7436 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7446 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7451 Identifies the process(es) that are allowed to perform operations.
7460 can be used to wildcard either of the ids in the
7470 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7480 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7490 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7495 Identifies the user that is allowed to perform operations.
7500 can be used to wildcard the user.
7505 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7515 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7525 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7530 Identifies the Portal index(es) that can be used.
7535 can be used to wildcard the Portal index.
7546 Data Movement Operations
7547 \begin_inset LatexCommand \label{sec:datamovement}
7554 The Portals API provides two data movement operations:
7566 \begin_inset LatexCommand \label{sec:put}
7573 typedef enum { PTL_ACK_REQ, PTL_NOACK_REQ } ptl_ack_req_t;
7577 int PtlPut( ptl_handle_md_t mem_desc,
7579 ptl_ack_req_t ack_req,
7581 ptl_process_id_t target,
7583 ptl_pt_index_t portal,
7585 ptl_ac_index_t cookie,
7587 ptl_match_bits_t match_bits,
7591 ptl_hdr_data_t hdr_data );
7598 are used to control whether an acknowledgement should be sent when the
7599 operation completes (i.e., when the data has been written to a memory descriptor
7609 requests an acknowledgement, the value
7613 requests that no acknowledgement should be generated.
7620 function initiates an asynchronous put operation.
7621 There are several events associated with a put operation: initiation of
7622 the send on the local node (
7624 PTL_EVENT_SEND_START
7626 ), completion of the send on the local node (
7634 ), and, when the send completes successfully, the receipt of an acknowledgement
7639 ) indicating that the operation was accepted by the target.
7640 These events will be logged in the event queue associated with the memory
7645 ) used in the put operation.
7646 Using a memory descriptor that does not have an associated event queue
7647 results in these events being discarded.
7648 In this case, the application must have another mechanism (e.g., a higher
7649 level protocol) for determining when it is safe to modify the memory region
7650 associated with the memory descriptor.
7651 \layout Subsubsection
7656 PTL_OK Indicates success.
7660 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
7664 PTL_INV_MD Indicates that
7668 is not a valid memory descriptor.
7672 PTL_INV_PROC Indicates that
7676 is not a valid process id.
7678 \layout Subsubsection
7684 \begin_inset Tabular
7685 <lyxtabular version="3" rows="8" columns="3">
7687 <column alignment="center" valignment="top" width="0pt">
7688 <column alignment="center" valignment="top" width="0pt">
7689 <column alignment="left" valignment="top" width="4.7in">
7691 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7701 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7711 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7716 A handle for the memory descriptor that describes the memory to be sent.
7717 If the memory descriptor has an event queue associated with it, it will
7718 be used to record events when the message has been sent (PTL_EVENT_SEND_START,
7719 PTL_EVENT_SEND_END).
7725 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7735 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7745 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7750 Controls whether an acknowledgement event is requested.
7751 Acknowledgements are only sent when they are requested by the initiating
7756 the memory descriptor has an event queue
7760 the target memory descriptor enables them.
7774 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7784 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7794 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7799 A process id for the target process.
7804 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7814 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7824 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7829 The index in the remote Portal table.
7834 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7844 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7854 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7859 The index into the access control table of the target process.
7864 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7874 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7884 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7889 The match bits to use for message selection at the target process.
7894 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7904 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7914 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7919 The offset into the target memory descriptor (only used when the target
7920 memory descriptor has the
7922 PTL_MD_MANAGE_REMOTE
7929 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7939 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
7949 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
7954 64 bits of user data that can be included in message header.
7955 This data is written to an event queue entry at the target if an event
7956 queue is present on the matching memory descriptor.
7968 \begin_inset LatexCommand \label{sec:get}
7975 int PtlGet( ptl_handle_md_t mem_desc,
7977 ptl_process_id_t target,
7979 ptl_pt_index_t portal,
7981 ptl_ac_index_t cookie,
7983 ptl_match_bits_t match_bits,
7985 ptl_size_t offset );
7992 function initiates a remote read operation.
7993 There are two event pairs associated with a get operation , when the data
7994 is sent from the remote node, a
7996 PTL_EVENT_GET{START|END}
7998 event pair is registered on the remote node; and when the data is returned
7999 from the remote node a
8001 PTL_EVENT_REPLY{START|END}
8003 event pair is registered on the local node.
8004 \layout Subsubsection
8009 PTL_OK Indicates success.
8013 PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
8017 PTL_INV_MD Indicates that
8021 is not a valid memory descriptor.
8025 PTL_INV_PROC Indicates that
8029 is not a valid process id.
8031 \layout Subsubsection
8037 \begin_inset Tabular
8038 <lyxtabular version="3" rows="6" columns="3">
8040 <column alignment="right" valignment="top" width="0pt">
8041 <column alignment="center" valignment="top" width="0pt">
8042 <column alignment="left" valignment="top" width="4.7in">
8044 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8054 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8064 <cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
8069 A handle for the memory descriptor that describes the memory into which
8070 the requested data will be received.
8071 The memory descriptor can have an event queue associated with it to record
8072 events, such as when the message receive has started (
8085 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8095 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8105 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
8110 A process id for the target process.
8115 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8125 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8135 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
8140 The index in the remote Portal table.
8145 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8155 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8165 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
8170 The index into the access control table of the target process.
8175 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8185 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8195 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
8200 The match bits to use for message selection at the target process.
8205 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8215 <cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
8225 <cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
8230 The offset into the target memory descriptor (only used when the target
8231 memory descriptor has the
8233 PTL_MD_MANAGE_REMOTE
8250 \begin_inset LatexCommand \label{sec:summary}
8254 We conclude this section by summarizing the names introduced by the Portals
8256 We start by summarizing the names of the types introduced by the API.
8257 This is followed by a summary of the functions introduced by the API.
8258 Which is followed by a summary of the function return codes.
8259 Finally, we conclude with a summary of the other constant values introduced
8265 \begin_inset LatexCommand \ref{tab:types}
8269 presents a summary of the types defined by the Portals API.
8270 The first column in this table gives the type name, the second column gives
8271 a brief description of the type, the third column identifies the section
8272 where the type is defined, and the fourth column lists the functions that
8273 have arguments of this type.
8277 \begin_inset Float table
8284 Types Defined by the Portals 3.2 API
8285 \begin_inset LatexCommand \label{tab:types}
8308 \begin_inset Tabular
8309 <lyxtabular version="3" rows="25" columns="4">
8310 <features firstHeadEmpty="true">
8311 <column alignment="left" valignment="top" width="0pt">
8312 <column alignment="left" valignment="top" width="2in">
8313 <column alignment="left" valignment="top" width="0pt">
8314 <column alignment="left" valignment="top" width="2.2in">
8315 <row bottomline="true">
8316 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8326 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8336 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8346 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8358 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8368 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8373 indexes for an access control table
8376 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8382 \begin_inset LatexCommand \ref{sec:index-type}
8389 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8394 PtlACEntry, PtlPut, PtlGet
8399 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8409 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8414 acknowledgement request types
8417 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8423 \begin_inset LatexCommand \ref{sec:put}
8430 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8440 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8452 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8460 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8466 \begin_inset LatexCommand \ref{sec:ek-type}
8473 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8483 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8493 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8498 information about events
8501 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8507 \begin_inset LatexCommand \ref{sec:event-type}
8514 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8524 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8534 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8539 event sequence number
8542 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8548 \begin_inset LatexCommand \ref{sec:event-type}
8555 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8565 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8575 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8580 handles for any object
8583 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
8589 \begin_inset LatexCommand \ref{sec:handle-type}
8596 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
8606 <cell alignment="right" valignment="top" usebox="none">
8616 <cell alignment="left" valignment="top" usebox="none">
8621 handles for event queues
8624 <cell alignment="right" valignment="top" usebox="none">
8630 \begin_inset LatexCommand \ref{sec:handle-type}
8637 <cell alignment="left" valignment="top" usebox="none">
8642 PtlEQAlloc, PtlEQFree, PtlEQGet, PtlEQWait, PtlMDUpdate
8647 <cell alignment="right" valignment="top" usebox="none">
8659 <cell alignment="left" valignment="top" usebox="none">
8664 handles for memory descriptors
8667 <cell alignment="right" valignment="top" usebox="none">
8673 \begin_inset LatexCommand \ref{sec:handle-type}
8680 <cell alignment="left" valignment="top" usebox="none">
8685 PtlMDAlloc, PtlMDUnlink, PtlMDUpdate, PtlMEAttach, PtlMEAttachAny, PtlMEInsert,
8691 <cell alignment="right" valignment="top" usebox="none">
8703 <cell alignment="left" valignment="top" usebox="none">
8708 handles for match entries
8711 <cell alignment="right" valignment="top" usebox="none">
8717 \begin_inset LatexCommand \ref{sec:handle-type}
8724 <cell alignment="left" valignment="top" usebox="none">
8729 PtlMEAttach, PtlMEAttachAny, PtlMEInsert, PtlMEUnlink
8734 <cell alignment="right" valignment="top" usebox="none">
8744 <cell alignment="left" valignment="top" usebox="none">
8749 handles for network interfaces
8752 <cell alignment="right" valignment="top" usebox="none">
8758 \begin_inset LatexCommand \ref{sec:handle-type}
8765 <cell alignment="left" valignment="top" usebox="none">
8770 PtlNIInit, PtlNIFini, PtlNIStatus, PtlNIDist, PtlEQAlloc, PtlACEntry, PtlPut,
8776 <cell alignment="right" valignment="top" usebox="none">
8788 <cell alignment="left" valignment="top" usebox="none">
8796 <cell alignment="right" valignment="top" usebox="none">
8802 \begin_inset LatexCommand \ref{sec:id-type}
8809 <cell alignment="left" valignment="top" usebox="none">
8819 <cell alignment="right" valignment="top" usebox="none">
8829 <cell alignment="left" valignment="top" usebox="none">
8837 <cell alignment="right" valignment="top" usebox="none">
8843 \begin_inset LatexCommand \ref{sec:id-type}
8850 <cell alignment="left" valignment="top" usebox="none">
8855 PtlGetId, PtlACEntry
8860 <cell alignment="right" valignment="top" usebox="none">
8870 <cell alignment="left" valignment="top" usebox="none">
8878 <cell alignment="right" valignment="top" usebox="none">
8884 \begin_inset LatexCommand \ref{sec:id-type}
8891 <cell alignment="left" valignment="top" usebox="none">
8896 PtlGetUid, PtlACEntry
8901 <cell alignment="right" valignment="top" usebox="none">
8911 <cell alignment="left" valignment="top" usebox="none">
8916 insertion position (before or after)
8919 <cell alignment="right" valignment="top" usebox="none">
8925 \begin_inset LatexCommand \ref{sec:meattach}
8932 <cell alignment="left" valignment="top" usebox="none">
8937 PtlMEAttach, PtlMEAttachAny, PtlMEInsert
8942 <cell alignment="right" valignment="top" usebox="none">
8952 <cell alignment="left" valignment="top" usebox="none">
8957 identifiers for network interfaces
8960 <cell alignment="right" valignment="top" usebox="none">
8966 \begin_inset LatexCommand \ref{sec:ni-type}
8973 <cell alignment="left" valignment="top" usebox="none">
8983 <cell alignment="right" valignment="top" usebox="none">
8995 <cell alignment="left" valignment="top" usebox="none">
9000 match (and ignore) bits
9003 <cell alignment="right" valignment="top" usebox="none">
9009 \begin_inset LatexCommand \ref{sec:mb-type}
9016 <cell alignment="left" valignment="top" usebox="none">
9021 PtlMEAttach, PtlMEAttachAny, PtlMEInsert, PtlPut, PtlGet
9026 <cell alignment="right" valignment="top" usebox="none">
9038 <cell alignment="left" valignment="top" usebox="none">
9046 <cell alignment="right" valignment="top" usebox="none">
9052 \begin_inset LatexCommand \ref{sec:md-type}
9059 <cell alignment="left" valignment="top" usebox="none">
9064 PtlMDAttach, PtlMDUpdate
9069 <cell alignment="right" valignment="top" usebox="none">
9079 <cell alignment="left" valignment="top" usebox="none">
9084 network interface-specific failures
9087 <cell alignment="right" valignment="top" usebox="none">
9093 \begin_inset LatexCommand \ref{sec:eq}
9100 <cell alignment="left" valignment="top" usebox="none">
9110 <cell alignment="right" valignment="top" usebox="none">
9122 <cell alignment="left" valignment="top" usebox="none">
9130 <cell alignment="right" valignment="top" usebox="none">
9136 \begin_inset LatexCommand \ref{sec:pid-type}
9143 <cell alignment="left" valignment="top" usebox="none">
9148 PtlGetId, PtlNIDist, PtlMEAttach, PtlMEAttachAny, PtlACEntry, PtlPut, PtlGet
9154 <cell alignment="right" valignment="top" usebox="none">
9164 <cell alignment="left" valignment="top" usebox="none">
9169 indexes for Portal tables
9172 <cell alignment="right" valignment="top" usebox="none">
9178 \begin_inset LatexCommand \ref{sec:index-type}
9185 <cell alignment="left" valignment="top" usebox="none">
9190 PtlMEAttach, PtlMEAttachAny, PtlACEntry
9195 <cell alignment="right" valignment="top" usebox="none">
9207 <cell alignment="left" valignment="top" usebox="none">
9215 <cell alignment="right" valignment="top" usebox="none">
9221 \begin_inset LatexCommand \ref{sec:size-t}
9228 <cell alignment="left" valignment="top" usebox="none">
9233 PtlEQAlloc, PtlPut, PtlGet
9238 <cell alignment="right" valignment="top" usebox="none">
9248 <cell alignment="left" valignment="top" usebox="none">
9253 indexes for status registers
9256 <cell alignment="right" valignment="top" usebox="none">
9262 \begin_inset LatexCommand \ref{sec:stat-type}
9269 <cell alignment="left" valignment="top" usebox="none">
9279 <cell alignment="right" valignment="top" usebox="none">
9291 <cell alignment="left" valignment="top" usebox="none">
9296 values in status registers
9299 <cell alignment="right" valignment="top" usebox="none">
9305 \begin_inset LatexCommand \ref{sec:stat-type}
9312 <cell alignment="left" valignment="top" usebox="none">
9322 <cell alignment="right" valignment="top" usebox="none">
9332 <cell alignment="left" valignment="top" usebox="none">
9340 <cell alignment="right" valignment="top" usebox="none">
9346 \begin_inset LatexCommand \ref{sec:meattach}
9353 <cell alignment="left" valignment="top" usebox="none">
9358 PtlMEAttach, PtlMEAttachAny, PtlMEInsert, PtlMDAttach
9374 \begin_inset LatexCommand \ref{tab:func}
9378 presents a summary of the functions defined by the Portals API.
9379 The first column in this table gives the name for the function, the second
9380 column gives a brief description of the operation implemented by the function,
9381 and the third column identifies the section where the function is defined.
9385 \begin_inset Float table
9392 Functions Defined by the Portals 3.2 API
9393 \begin_inset LatexCommand \label{tab:func}
9416 \begin_inset Tabular
9417 <lyxtabular version="3" rows="24" columns="3">
9418 <features firstHeadEmpty="true">
9419 <column alignment="left" valignment="top" width="0pt">
9420 <column alignment="left" valignment="top" width="0pt">
9421 <column alignment="left" valignment="top" width="0pt">
9422 <row bottomline="true">
9423 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
9431 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
9439 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
9449 <cell alignment="left" valignment="top" usebox="none">
9457 <cell alignment="left" valignment="top" usebox="none">
9462 update an entry in an access control table
9465 <cell alignment="left" valignment="top" usebox="none">
9471 \begin_inset LatexCommand \ref{sec:ac}
9480 <cell alignment="left" valignment="top" usebox="none">
9488 <cell alignment="left" valignment="top" usebox="none">
9493 create an event queue
9496 <cell alignment="left" valignment="top" usebox="none">
9502 \begin_inset LatexCommand \ref{sec:eq}
9511 <cell alignment="left" valignment="top" usebox="none">
9519 <cell alignment="left" valignment="top" usebox="none">
9524 get the next event from an event queue
9527 <cell alignment="left" valignment="top" usebox="none">
9533 \begin_inset LatexCommand \ref{sec:eq}
9542 <cell alignment="left" valignment="top" usebox="none">
9550 <cell alignment="left" valignment="top" usebox="none">
9555 release the resources for an event queue
9558 <cell alignment="left" valignment="top" usebox="none">
9564 \begin_inset LatexCommand \ref{sec:eq}
9573 <cell alignment="left" valignment="top" usebox="none">
9581 <cell alignment="left" valignment="top" usebox="none">
9586 wait for a new event in an event queue
9589 <cell alignment="left" valignment="top" usebox="none">
9595 \begin_inset LatexCommand \ref{sec:eq}
9604 <cell alignment="left" valignment="top" usebox="none">
9612 <cell alignment="left" valignment="top" usebox="none">
9617 shutdown the Portals API
9620 <cell alignment="left" valignment="top" usebox="none">
9626 \begin_inset LatexCommand \ref{sec:init}
9635 <cell alignment="left" valignment="top" usebox="none">
9643 <cell alignment="left" valignment="top" usebox="none">
9648 perform a get operation
9651 <cell alignment="left" valignment="top" usebox="none">
9657 \begin_inset LatexCommand \ref{sec:datamovement}
9666 <cell alignment="left" valignment="top" usebox="none">
9674 <cell alignment="left" valignment="top" usebox="none">
9679 get the id for the current process
9682 <cell alignment="left" valignment="top" usebox="none">
9688 \begin_inset LatexCommand \ref{sec:pid}
9697 <cell alignment="left" valignment="top" usebox="none">
9705 <cell alignment="left" valignment="top" usebox="none">
9710 initialize the Portals API
9713 <cell alignment="left" valignment="top" usebox="none">
9719 \begin_inset LatexCommand \ref{sec:init}
9728 <cell alignment="left" valignment="top" usebox="none">
9736 <cell alignment="left" valignment="top" usebox="none">
9741 create a memory descriptor and attach it to a match entry
9744 <cell alignment="left" valignment="top" usebox="none">
9750 \begin_inset LatexCommand \ref{sec:md}
9759 <cell alignment="left" valignment="top" usebox="none">
9767 <cell alignment="left" valignment="top" usebox="none">
9772 create a free-floating memory descriptor
9775 <cell alignment="left" valignment="top" usebox="none">
9781 \begin_inset LatexCommand \ref{sec:mdbind}
9790 <cell alignment="left" valignment="top" usebox="none">
9798 <cell alignment="left" valignment="top" usebox="none">
9803 remove a memory descriptor from a list and release its resources
9806 <cell alignment="left" valignment="top" usebox="none">
9812 \begin_inset LatexCommand \ref{sec:md}
9821 <cell alignment="left" valignment="top" usebox="none">
9829 <cell alignment="left" valignment="top" usebox="none">
9834 update a memory descriptor
9837 <cell alignment="left" valignment="top" usebox="none">
9843 \begin_inset LatexCommand \ref{sec:md}
9852 <cell alignment="left" valignment="top" usebox="none">
9860 <cell alignment="left" valignment="top" usebox="none">
9865 create a match entry and attach it to a Portal table
9868 <cell alignment="left" valignment="top" usebox="none">
9874 \begin_inset LatexCommand \ref{sec:me}
9883 <cell alignment="left" valignment="top" usebox="none">
9891 <cell alignment="left" valignment="top" usebox="none">
9896 create a match entry and attach it to a free Portal table entry
9899 <cell alignment="left" valignment="top" usebox="none">
9905 \begin_inset LatexCommand \ref{sec:attachany}
9914 <cell alignment="left" valignment="top" usebox="none">
9922 <cell alignment="left" valignment="top" usebox="none">
9927 create a match entry and insert it in a list
9930 <cell alignment="left" valignment="top" usebox="none">
9936 \begin_inset LatexCommand \ref{sec:me}
9945 <cell alignment="left" valignment="top" usebox="none">
9953 <cell alignment="left" valignment="top" usebox="none">
9958 remove a match entry from a list and release its resources
9961 <cell alignment="left" valignment="top" usebox="none">
9967 \begin_inset LatexCommand \ref{sec:me}
9976 <cell alignment="left" valignment="top" usebox="none">
9984 <cell alignment="left" valignment="top" usebox="none">
9989 get the distance to another process
9992 <cell alignment="left" valignment="top" usebox="none">
9998 \begin_inset LatexCommand \ref{sec:ni}
10007 <cell alignment="left" valignment="top" usebox="none">
10015 <cell alignment="left" valignment="top" usebox="none">
10020 shutdown a network interface
10023 <cell alignment="left" valignment="top" usebox="none">
10029 \begin_inset LatexCommand \ref{sec:ni}
10038 <cell alignment="left" valignment="top" usebox="none">
10046 <cell alignment="left" valignment="top" usebox="none">
10051 get the network interface handle for an object
10054 <cell alignment="left" valignment="top" usebox="none">
10060 \begin_inset LatexCommand \ref{sec:ni}
10069 <cell alignment="left" valignment="top" usebox="none">
10077 <cell alignment="left" valignment="top" usebox="none">
10082 initialize a network interface
10085 <cell alignment="left" valignment="top" usebox="none">
10091 \begin_inset LatexCommand \ref{sec:ni}
10100 <cell alignment="left" valignment="top" usebox="none">
10108 <cell alignment="left" valignment="top" usebox="none">
10113 read a network interface status register
10116 <cell alignment="left" valignment="top" usebox="none">
10122 \begin_inset LatexCommand \ref{sec:ni}
10131 <cell alignment="left" valignment="top" usebox="none">
10139 <cell alignment="left" valignment="top" usebox="none">
10144 perform a put operation
10147 <cell alignment="left" valignment="top" usebox="none">
10153 \begin_inset LatexCommand \ref{sec:datamovement}
10171 Table\SpecialChar ~
10173 \begin_inset LatexCommand \ref{tab:retcodes}
10177 summarizes the return codes used by functions defined by the Portals API.
10178 All of these constants are integer values.
10179 The first column of this table gives the symbolic name for the constant,
10180 the second column gives a brief description of the value, and the third
10181 column identifies the functions that can return this value.
10185 \begin_inset Float table
10192 Function Return Codes for the Portals 3.2 API
10193 \begin_inset LatexCommand \label{tab:retcodes}
10216 \begin_inset Tabular
10217 <lyxtabular version="3" rows="27" columns="3">
10219 <column alignment="left" valignment="top" width="0pt">
10220 <column alignment="left" valignment="top" width="0pt">
10221 <column alignment="left" valignment="top" width="2.6in">
10222 <row bottomline="true">
10223 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
10233 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
10243 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
10257 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
10269 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
10274 invalid access control table index
10277 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
10287 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
10299 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
10304 at least one event has been dropped
10307 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
10317 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
10327 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
10332 no events available in an event queue
10335 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
10345 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
10355 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
10360 error during initialization or cleanup
10363 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
10373 <cell alignment="right" valignment="top" usebox="none">
10385 <cell alignment="left" valignment="top" usebox="none">
10390 illegal memory descriptor values
10393 <cell alignment="right" valignment="top" usebox="none">
10398 PtlMDAttach, PtlMDBind, PtlMDUpdate
10403 <cell alignment="right" valignment="top" usebox="none">
10413 <cell alignment="left" valignment="top" usebox="none">
10418 duplicate initialization of an interface
10421 <cell alignment="right" valignment="top" usebox="none">
10431 <cell alignment="right" valignment="top" usebox="none">
10443 <cell alignment="left" valignment="top" usebox="none">
10448 initialization of an invalid interface
10451 <cell alignment="right" valignment="top" usebox="none">
10461 <cell alignment="right" valignment="top" usebox="none">
10471 <cell alignment="left" valignment="top" usebox="none">
10476 the ME already has an MD
10479 <cell alignment="right" valignment="top" usebox="none">
10489 <cell alignment="right" valignment="top" usebox="none">
10501 <cell alignment="left" valignment="top" usebox="none">
10506 invalid access control table size
10509 <cell alignment="right" valignment="top" usebox="none">
10519 <cell alignment="right" valignment="top" usebox="none">
10531 <cell alignment="left" valignment="top" usebox="none">
10536 invalid event queue handle
10539 <cell alignment="right" valignment="top" usebox="none">
10544 PtlMDUpdate, PtlEQFree, PtlEQGet
10549 <cell alignment="right" valignment="top" usebox="none">
10559 <cell alignment="left" valignment="top" usebox="none">
10567 <cell alignment="right" valignment="top" usebox="none">
10577 <cell alignment="right" valignment="top" usebox="none">
10587 <cell alignment="left" valignment="top" usebox="none">
10592 invalid memory descriptor handle
10595 <cell alignment="right" valignment="top" usebox="none">
10600 PtlMDUnlink, PtlMDUpdate
10605 <cell alignment="right" valignment="top" usebox="none">
10617 <cell alignment="left" valignment="top" usebox="none">
10622 invalid match entry handle
10625 <cell alignment="right" valignment="top" usebox="none">
10635 <cell alignment="right" valignment="top" usebox="none">
10645 <cell alignment="left" valignment="top" usebox="none">
10650 invalid network interface handle
10653 <cell alignment="right" valignment="top" usebox="none">
10658 PtlNIDist, PtlNIFini, PtlMDBind, PtlEQAlloc
10663 <cell alignment="right" valignment="top" usebox="none">
10673 <cell alignment="left" valignment="top" usebox="none">
10678 invalid process identifier
10681 <cell alignment="right" valignment="top" usebox="none">
10686 PtlNIInit, PtlNIDist, PtlMEAttach, PtlMEInsert, PtlACEntry, PtlPut, PtlGet
10692 <cell alignment="right" valignment="top" usebox="none">
10704 <cell alignment="left" valignment="top" usebox="none">
10709 invalid Portal table index
10712 <cell alignment="right" valignment="top" usebox="none">
10722 <cell alignment="right" valignment="top" usebox="none">
10732 <cell alignment="left" valignment="top" usebox="none">
10737 invalid status register
10740 <cell alignment="right" valignment="top" usebox="none">
10750 <cell alignment="right" valignment="top" usebox="none">
10760 <cell alignment="left" valignment="top" usebox="none">
10765 invalid status register index
10768 <cell alignment="right" valignment="top" usebox="none">
10778 <cell alignment="right" valignment="top" usebox="none">
10788 <cell alignment="left" valignment="top" usebox="none">
10793 match list too long
10796 <cell alignment="right" valignment="top" usebox="none">
10801 PtlMEAttach, PtlMEInsert
10806 <cell alignment="right" valignment="top" usebox="none">
10816 <cell alignment="left" valignment="top" usebox="none">
10821 MD has pending operations
10824 <cell alignment="right" valignment="top" usebox="none">
10834 <cell alignment="right" valignment="top" usebox="none">
10844 <cell alignment="left" valignment="top" usebox="none">
10852 <cell alignment="right" valignment="top" usebox="none">
10866 <cell alignment="right" valignment="top" usebox="none">
10878 <cell alignment="left" valignment="top" usebox="none">
10883 insufficient memory
10886 <cell alignment="right" valignment="top" usebox="none">
10891 PtlNIInit, PtlMDAttach, PtlMDBind, PtlEQAlloc, PtlMEAttach, PtlMEInsert
10897 <cell alignment="right" valignment="top" usebox="none">
10909 <cell alignment="left" valignment="top" usebox="none">
10914 no update was performed
10917 <cell alignment="right" valignment="top" usebox="none">
10927 <cell alignment="right" valignment="top" usebox="none">
10937 <cell alignment="left" valignment="top" usebox="none">
10942 Portal table is full
10945 <cell alignment="right" valignment="top" usebox="none">
10955 <cell alignment="right" valignment="top" usebox="none">
10965 <cell alignment="left" valignment="top" usebox="none">
10973 <cell alignment="right" valignment="top" usebox="none">
10985 <cell alignment="right" valignment="top" usebox="none">
10995 <cell alignment="left" valignment="top" usebox="none">
11000 addressing violation
11003 <cell alignment="left" valignment="top" usebox="none">
11008 PtlNIInit, PtlNIStatus, PtlNIDist, PtlNIHandle, PtlMDBind, PtlMDUpdate,
11009 PtlEQAlloc, PtlEQGet, PtlEQWait
11023 Table\SpecialChar ~
11025 \begin_inset LatexCommand \ref{tab:oconsts}
11029 summarizes the remaining constant values introduced by the Portals API.
11030 The first column in this table presents the symbolic name for the constant,
11031 the second column gives a brief description of the value, the third column
11032 identifies the type for the value, and the fourth column identifies the
11033 sections in which the value is mentioned.
11037 \begin_inset Float table
11044 Other Constants Defined by the Portals 3.2 API
11045 \begin_inset LatexCommand \label{tab:oconsts}
11068 \begin_inset Tabular
11069 <lyxtabular version="3" rows="36" columns="5">
11071 <column alignment="left" valignment="top" width="0pt">
11072 <column alignment="left" valignment="top" width="0pt">
11073 <column alignment="left" valignment="top" width="0pt">
11074 <column alignment="left" valignment="top" width="0pt">
11075 <column alignment="left" valignment="top" width="0pt">
11076 <row bottomline="true">
11077 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
11087 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
11097 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
11107 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
11117 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
11129 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
11141 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
11146 request an acknowledgement
11149 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
11159 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
11165 \begin_inset LatexCommand \ref{sec:put}
11172 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
11181 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
11191 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
11196 a NULL event queue handle
11199 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
11209 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
11215 \begin_inset LatexCommand \ref{sec:handle-type}
11222 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
11228 \begin_inset LatexCommand \ref{sec:md}
11233 \begin_inset LatexCommand \ref{sec:mdupdate}
11242 <cell alignment="right" valignment="top" usebox="none">
11249 PTL_EVENT_GET_START
11252 <cell alignment="left" valignment="top" usebox="none">
11260 <cell alignment="right" valignment="top" usebox="none">
11270 <cell alignment="left" valignment="top" usebox="none">
11276 \begin_inset LatexCommand \ref{sec:ek-type}
11283 <cell alignment="left" valignment="top" usebox="none">
11289 \begin_inset LatexCommand \ref{sec:get}
11298 <cell alignment="right" valignment="top" usebox="none">
11308 <cell alignment="left" valignment="top" usebox="none">
11316 <cell alignment="right" valignment="top" usebox="none">
11326 <cell alignment="left" valignment="top" usebox="none">
11332 \begin_inset LatexCommand \ref{sec:ek-type}
11339 <cell alignment="left" valignment="top" usebox="none">
11345 \begin_inset LatexCommand \ref{sec:get}
11354 <cell alignment="right" valignment="top" usebox="none">
11364 <cell alignment="left" valignment="top" usebox="none">
11372 <cell alignment="right" valignment="top" usebox="none">
11382 <cell alignment="left" valignment="top" usebox="none">
11388 \begin_inset LatexCommand \ref{sec:ek-type}
11395 <cell alignment="left" valignment="top" usebox="none">
11401 \begin_inset LatexCommand \ref{sec:get}
11410 <cell alignment="right" valignment="top" usebox="none">
11417 PTL_EVENT_PUT_START
11420 <cell alignment="left" valignment="top" usebox="none">
11428 <cell alignment="right" valignment="top" usebox="none">
11438 <cell alignment="left" valignment="top" usebox="none">
11444 \begin_inset LatexCommand \ref{sec:ek-type}
11451 <cell alignment="left" valignment="top" usebox="none">
11457 \begin_inset LatexCommand \ref{sec:put}
11466 <cell alignment="right" valignment="top" usebox="none">
11476 <cell alignment="left" valignment="top" usebox="none">
11484 <cell alignment="right" valignment="top" usebox="none">
11494 <cell alignment="left" valignment="top" usebox="none">
11500 \begin_inset LatexCommand \ref{sec:ek-type}
11507 <cell alignment="left" valignment="top" usebox="none">
11513 \begin_inset LatexCommand \ref{sec:put}
11522 <cell alignment="right" valignment="top" usebox="none">
11532 <cell alignment="left" valignment="top" usebox="none">
11540 <cell alignment="right" valignment="top" usebox="none">
11550 <cell alignment="left" valignment="top" usebox="none">
11556 \begin_inset LatexCommand \ref{sec:ek-type}
11563 <cell alignment="left" valignment="top" usebox="none">
11569 \begin_inset LatexCommand \ref{sec:put}
11578 <cell alignment="right" valignment="top" usebox="none">
11585 PTL_EVENT_REPLY_START
11588 <cell alignment="left" valignment="top" usebox="none">
11596 <cell alignment="right" valignment="top" usebox="none">
11606 <cell alignment="left" valignment="top" usebox="none">
11612 \begin_inset LatexCommand \ref{sec:ek-type}
11619 <cell alignment="left" valignment="top" usebox="none">
11625 \begin_inset LatexCommand \ref{sec:get}
11634 <cell alignment="right" valignment="top" usebox="none">
11641 PTL_EVENT_REPLY_END
11644 <cell alignment="left" valignment="top" usebox="none">
11652 <cell alignment="right" valignment="top" usebox="none">
11662 <cell alignment="left" valignment="top" usebox="none">
11668 \begin_inset LatexCommand \ref{sec:ek-type}
11675 <cell alignment="left" valignment="top" usebox="none">
11681 \begin_inset LatexCommand \ref{sec:get}
11690 <cell alignment="right" valignment="top" usebox="none">
11697 PTL_EVENT_REPLY_FAIL
11700 <cell alignment="left" valignment="top" usebox="none">
11708 <cell alignment="right" valignment="top" usebox="none">
11718 <cell alignment="left" valignment="top" usebox="none">
11724 \begin_inset LatexCommand \ref{sec:ek-type}
11731 <cell alignment="left" valignment="top" usebox="none">
11737 \begin_inset LatexCommand \ref{sec:get}
11746 <cell alignment="right" valignment="top" usebox="none">
11753 PTL_EVENT_ACK_START
11756 <cell alignment="left" valignment="top" usebox="none">
11761 acknowledgement event start
11764 <cell alignment="right" valignment="top" usebox="none">
11774 <cell alignment="left" valignment="top" usebox="none">
11780 \begin_inset LatexCommand \ref{sec:ek-type}
11787 <cell alignment="left" valignment="top" usebox="none">
11793 \begin_inset LatexCommand \ref{sec:put}
11802 <cell alignment="right" valignment="top" usebox="none">
11812 <cell alignment="left" valignment="top" usebox="none">
11817 acknowledgement event end
11820 <cell alignment="right" valignment="top" usebox="none">
11830 <cell alignment="left" valignment="top" usebox="none">
11836 \begin_inset LatexCommand \ref{sec:ek-type}
11843 <cell alignment="left" valignment="top" usebox="none">
11849 \begin_inset LatexCommand \ref{sec:put}
11858 <cell alignment="right" valignment="top" usebox="none">
11868 <cell alignment="left" valignment="top" usebox="none">
11873 acknowledgement event fail
11876 <cell alignment="right" valignment="top" usebox="none">
11886 <cell alignment="left" valignment="top" usebox="none">
11892 \begin_inset LatexCommand \ref{sec:ek-type}
11899 <cell alignment="left" valignment="top" usebox="none">
11905 \begin_inset LatexCommand \ref{sec:put}
11914 <cell alignment="right" valignment="top" usebox="none">
11921 PTL_EVENT_SEND_START
11924 <cell alignment="left" valignment="top" usebox="none">
11932 <cell alignment="right" valignment="top" usebox="none">
11942 <cell alignment="left" valignment="top" usebox="none">
11948 \begin_inset LatexCommand \ref{sec:ek-type}
11955 <cell alignment="left" valignment="top" usebox="none">
11961 \begin_inset LatexCommand \ref{sec:put}
11970 <cell alignment="right" valignment="top" usebox="none">
11980 <cell alignment="left" valignment="top" usebox="none">
11988 <cell alignment="right" valignment="top" usebox="none">
11998 <cell alignment="left" valignment="top" usebox="none">
12004 \begin_inset LatexCommand \ref{sec:ek-type}
12011 <cell alignment="left" valignment="top" usebox="none">
12017 \begin_inset LatexCommand \ref{sec:put}
12026 <cell alignment="right" valignment="top" usebox="none">
12033 PTL_EVENT_SEND_FAIL
12036 <cell alignment="left" valignment="top" usebox="none">
12044 <cell alignment="right" valignment="top" usebox="none">
12054 <cell alignment="left" valignment="top" usebox="none">
12060 \begin_inset LatexCommand \ref{sec:ek-type}
12067 <cell alignment="left" valignment="top" usebox="none">
12073 \begin_inset LatexCommand \ref{sec:put}
12082 <cell alignment="right" valignment="top" usebox="none">
12092 <cell alignment="left" valignment="top" usebox="none">
12100 <cell alignment="right" valignment="top" usebox="none">
12110 <cell alignment="left" valignment="top" usebox="none">
12116 \begin_inset LatexCommand \ref{sec:ek-type}
12123 <cell alignment="left" valignment="top" usebox="none">
12129 \begin_inset LatexCommand \ref{sec:md-type}
12138 <cell alignment="right" valignment="top" usebox="none">
12148 <cell alignment="left" valignment="top" usebox="none">
12153 wildcard for process id fields
12156 <cell alignment="right" valignment="top" usebox="none">
12166 <cell alignment="left" valignment="top" usebox="none">
12172 \begin_inset LatexCommand \ref{sec:id-type}
12179 <cell alignment="left" valignment="top" usebox="none">
12185 \begin_inset LatexCommand \ref{sec:meattach}
12190 \begin_inset LatexCommand \ref{sec:acentry}
12199 <cell alignment="right" valignment="top" usebox="none">
12209 <cell alignment="left" valignment="top" usebox="none">
12214 wildcard for node id fields
12217 <cell alignment="right" valignment="top" usebox="none">
12227 <cell alignment="left" valignment="top" usebox="none">
12233 \begin_inset LatexCommand \ref{sec:id-type}
12240 <cell alignment="left" valignment="top" usebox="none">
12246 \begin_inset LatexCommand \ref{sec:meattach}
12251 \begin_inset LatexCommand \ref{sec:acentry}
12260 <cell alignment="right" valignment="top" usebox="none">
12270 <cell alignment="left" valignment="top" usebox="none">
12275 wildcard for user id
12278 <cell alignment="right" valignment="top" usebox="none">
12288 <cell alignment="left" valignment="top" usebox="none">
12294 \begin_inset LatexCommand \ref{sec:id-type}
12301 <cell alignment="left" valignment="top" usebox="none">
12307 \begin_inset LatexCommand \ref{sec:meattach}
12312 \begin_inset LatexCommand \ref{sec:acentry}
12321 <cell alignment="right" valignment="top" usebox="none">
12331 <cell alignment="left" valignment="top" usebox="none">
12339 <cell alignment="right" valignment="top" usebox="none">
12349 <cell alignment="left" valignment="top" usebox="none">
12355 \begin_inset LatexCommand \ref{sec:ni-type}
12362 <cell alignment="left" valignment="top" usebox="none">
12371 <cell alignment="right" valignment="top" usebox="none">
12381 <cell alignment="left" valignment="top" usebox="none">
12389 <cell alignment="right" valignment="top" usebox="none">
12401 <cell alignment="left" valignment="top" usebox="none">
12407 \begin_inset LatexCommand \ref{sec:meinsert}
12414 <cell alignment="left" valignment="top" usebox="none">
12423 <cell alignment="right" valignment="top" usebox="none">
12433 <cell alignment="left" valignment="top" usebox="none">
12441 <cell alignment="right" valignment="top" usebox="none">
12453 <cell alignment="left" valignment="top" usebox="none">
12459 \begin_inset LatexCommand \ref{sec:meinsert}
12466 <cell alignment="left" valignment="top" usebox="none">
12475 <cell alignment="right" valignment="top" usebox="none">
12485 <cell alignment="left" valignment="top" usebox="none">
12490 a flag to disable acknowledgements
12493 <cell alignment="right" valignment="top" usebox="none">
12505 <cell alignment="left" valignment="top" usebox="none">
12511 \begin_inset LatexCommand \ref{sec:md-type}
12518 <cell alignment="left" valignment="top" usebox="none">
12527 <cell alignment="right" valignment="top" usebox="none">
12534 PTL_MD_MANAGE_REMOTE
12537 <cell alignment="left" valignment="top" usebox="none">
12542 a flag to enable the use of remote offsets
12545 <cell alignment="right" valignment="top" usebox="none">
12555 <cell alignment="left" valignment="top" usebox="none">
12561 \begin_inset LatexCommand \ref{sec:md-type}
12568 <cell alignment="left" valignment="top" usebox="none">
12574 \begin_inset LatexCommand \ref{sec:put}
12579 \begin_inset LatexCommand \ref{sec:get}
12588 <cell alignment="right" valignment="top" usebox="none">
12598 <cell alignment="left" valignment="top" usebox="none">
12603 a flag to enable get operations
12606 <cell alignment="right" valignment="top" usebox="none">
12616 <cell alignment="left" valignment="top" usebox="none">
12622 \begin_inset LatexCommand \ref{sec:md-type}
12629 <cell alignment="left" valignment="top" usebox="none">
12638 <cell alignment="right" valignment="top" usebox="none">
12648 <cell alignment="left" valignment="top" usebox="none">
12653 a flag to enable put operations
12656 <cell alignment="right" valignment="top" usebox="none">
12668 <cell alignment="left" valignment="top" usebox="none">
12674 \begin_inset LatexCommand \ref{sec:md-type}
12681 <cell alignment="left" valignment="top" usebox="none">
12690 <cell alignment="right" valignment="top" usebox="none">
12700 <cell alignment="left" valignment="top" usebox="none">
12705 infinite threshold for a memory descriptor
12708 <cell alignment="right" valignment="top" usebox="none">
12718 <cell alignment="left" valignment="top" usebox="none">
12724 \begin_inset LatexCommand \ref{sec:md-type}
12731 <cell alignment="left" valignment="top" usebox="none">
12740 <cell alignment="right" valignment="top" usebox="none">
12750 <cell alignment="left" valignment="top" usebox="none">
12755 a flag to enable truncation of a request
12758 <cell alignment="right" valignment="top" usebox="none">
12768 <cell alignment="left" valignment="top" usebox="none">
12774 \begin_inset LatexCommand \ref{sec:md-type}
12781 <cell alignment="left" valignment="top" usebox="none">
12790 <cell alignment="right" valignment="top" usebox="none">
12800 <cell alignment="left" valignment="top" usebox="none">
12805 request no acknowledgement
12808 <cell alignment="right" valignment="top" usebox="none">
12818 <cell alignment="left" valignment="top" usebox="none">
12824 \begin_inset LatexCommand \ref{sec:put}
12831 <cell alignment="left" valignment="top" usebox="none">
12840 <cell alignment="right" valignment="top" usebox="none">
12850 <cell alignment="left" valignment="top" usebox="none">
12855 wildcard for Portal indexes
12858 <cell alignment="right" valignment="top" usebox="none">
12870 <cell alignment="left" valignment="top" usebox="none">
12876 \begin_inset LatexCommand \ref{sec:acentry}
12883 <cell alignment="left" valignment="top" usebox="none">
12892 <cell alignment="right" valignment="top" usebox="none">
12902 <cell alignment="left" valignment="top" usebox="none">
12910 <cell alignment="right" valignment="top" usebox="none">
12920 <cell alignment="left" valignment="top" usebox="none">
12926 \begin_inset LatexCommand \ref{sec:mdattach}
12933 <cell alignment="left" valignment="top" usebox="none">
12942 <cell alignment="right" valignment="top" usebox="none">
12952 <cell alignment="left" valignment="top" usebox="none">
12957 index for the dropped count register
12960 <cell alignment="right" valignment="top" usebox="none">
12970 <cell alignment="left" valignment="top" usebox="none">
12976 \begin_inset LatexCommand \ref{sec:stat-type}
12983 <cell alignment="left" valignment="top" usebox="none">
12989 \begin_inset LatexCommand \ref{sec:nistatus}
12998 <cell alignment="right" valignment="top" usebox="none">
13008 <cell alignment="left" valignment="top" usebox="none">
13016 <cell alignment="right" valignment="top" usebox="none">
13026 <cell alignment="left" valignment="top" usebox="none">
13032 \begin_inset LatexCommand \ref{sec:mdattach}
13039 <cell alignment="left" valignment="top" usebox="none">
13057 The Semantics of Message Transmission
13058 \begin_inset LatexCommand \label{sec:semantics}
13065 The portals API uses four types of messages: put requests, acknowledgements,
13066 get requests, and replies.
13067 In this section, we describe the information passed on the wire for each
13069 We also describe how this information is used to process incoming messages.
13075 Table\SpecialChar ~
13077 \begin_inset LatexCommand \ref{tab:put-wire}
13081 summarizes the information that is transmitted for a put request.
13082 The first column provides a descriptive name for the information, the second
13083 column provides the type for this information, the third column identifies
13084 the source of the information, and the fourth column provides additional
13086 Most information that is transmitted is obtained directly from the
13091 Notice that the handle for the memory descriptor used in the
13095 operation is transmitted even though this value cannot be interpreted by
13097 A value of anything other than
13101 , is interpreted as a request for an acknowledgement.
13105 \begin_inset Float table
13112 Information Passed in a Put Request
13113 \begin_inset LatexCommand \label{tab:put-wire}
13136 \begin_inset Tabular
13137 <lyxtabular version="3" rows="12" columns="4">
13138 <features firstHeadEmpty="true">
13139 <column alignment="left" valignment="top" width="0pt">
13140 <column alignment="left" valignment="top" width="0pt">
13141 <column alignment="left" valignment="top" width="0pt">
13142 <column alignment="left" valignment="top" width="0pt">
13143 <row bottomline="true">
13144 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13154 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13164 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13177 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13189 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13197 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13207 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13214 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13219 indicates a put request
13224 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13232 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13242 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13249 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13259 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13267 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13277 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13284 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13294 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13302 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13312 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13324 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13333 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13341 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13353 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13363 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13372 <cell alignment="left" valignment="top" usebox="none">
13380 <cell alignment="right" valignment="top" usebox="none">
13390 <cell alignment="left" valignment="top" usebox="none">
13402 <cell alignment="right" valignment="top" usebox="none">
13411 <cell alignment="left" valignment="top" usebox="none">
13419 <cell alignment="right" valignment="top" usebox="none">
13429 <cell alignment="left" valignment="top" usebox="none">
13439 <cell alignment="right" valignment="top" usebox="none">
13448 <cell alignment="left" valignment="top" usebox="none">
13456 <cell alignment="right" valignment="top" usebox="none">
13466 <cell alignment="left" valignment="top" usebox="none">
13478 <cell alignment="right" valignment="top" usebox="none">
13487 <cell alignment="left" valignment="top" usebox="none">
13495 <cell alignment="right" valignment="top" usebox="none">
13505 <cell alignment="left" valignment="top" usebox="none">
13517 <cell alignment="right" valignment="top" usebox="none">
13529 <cell alignment="left" valignment="top" usebox="none">
13537 <cell alignment="right" valignment="top" usebox="none">
13547 <cell alignment="left" valignment="top" usebox="none">
13557 <cell alignment="right" valignment="top" usebox="none">
13571 <cell alignment="left" valignment="top" usebox="none">
13579 <cell alignment="right" valignment="top" usebox="none">
13590 <cell alignment="left" valignment="top" usebox="none">
13600 <cell alignment="right" valignment="top" usebox="none">
13627 Table\SpecialChar ~
13629 \begin_inset LatexCommand \ref{tab:ack-wire}
13633 summarizes the information transmitted in an acknowledgement.
13634 Most of the information is simply echoed from the put request.
13635 Notice that the initiator and target are obtained directly from the put
13636 request, but are swapped in generating the acknowledgement.
13637 The only new piece of information in the acknowledgement is the manipulated
13638 length which is determined as the put request is satisfied.
13642 \begin_inset Float table
13649 Information Passed in an Acknowledgement
13650 \begin_inset LatexCommand \label{tab:ack-wire}
13673 \begin_inset Tabular
13674 <lyxtabular version="3" rows="10" columns="4">
13675 <features firstHeadEmpty="true">
13676 <column alignment="left" valignment="top" width="0pt">
13677 <column alignment="left" valignment="top" width="0pt">
13678 <column alignment="left" valignment="top" width="0pt">
13679 <column alignment="left" valignment="top" width="0pt">
13680 <row bottomline="true">
13681 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13693 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13703 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13713 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13725 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13733 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13743 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13750 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13755 indicates an acknowledgement
13760 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13768 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13778 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13786 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13795 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13803 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13813 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
13821 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
13830 <cell alignment="left" valignment="top" usebox="none">
13838 <cell alignment="right" valignment="top" usebox="none">
13848 <cell alignment="left" valignment="top" usebox="none">
13856 <cell alignment="right" valignment="top" usebox="none">
13866 <cell alignment="left" valignment="top" usebox="none">
13874 <cell alignment="right" valignment="top" usebox="none">
13884 <cell alignment="left" valignment="top" usebox="none">
13892 <cell alignment="right" valignment="top" usebox="none">
13902 <cell alignment="left" valignment="top" usebox="none">
13910 <cell alignment="right" valignment="top" usebox="none">
13920 <cell alignment="left" valignment="top" usebox="none">
13928 <cell alignment="right" valignment="top" usebox="none">
13938 <cell alignment="left" valignment="top" usebox="none">
13946 <cell alignment="right" valignment="top" usebox="none">
13956 <cell alignment="left" valignment="top" usebox="none">
13964 <cell alignment="right" valignment="top" usebox="none">
13974 <cell alignment="left" valignment="top" usebox="none">
13982 <cell alignment="right" valignment="top" usebox="none">
13994 <cell alignment="left" valignment="top" usebox="none">
14002 <cell alignment="right" valignment="top" usebox="none">
14012 <cell alignment="left" valignment="top" usebox="none">
14020 <cell alignment="right" valignment="top" usebox="none">
14032 <cell alignment="left" valignment="top" usebox="none">
14039 <cell alignment="right" valignment="top" usebox="none">
14044 obtained from the operation
14058 Table\SpecialChar ~
14060 \begin_inset LatexCommand \ref{tab:get-wire}
14064 summarizes the information that is transmitted for a get request.
14065 Like the information transmitted in a put request, most of the information
14066 transmitted in a get request is obtained directly from the
14071 Unlike put requests, get requests do not include the event queue handle.
14072 In this case, the reply is generated whenever the operation succeeds and
14073 the memory descriptor must not be unlinked until the reply is received.
14074 As such, there is no advantage to explicitly sending the event queue handle.
14078 \begin_inset Float table
14085 Information Passed in a Get Request
14086 \begin_inset LatexCommand \label{tab:get-wire}
14109 \begin_inset Tabular
14110 <lyxtabular version="3" rows="11" columns="4">
14111 <features firstHeadEmpty="true">
14112 <column alignment="left" valignment="top" width="0pt">
14113 <column alignment="left" valignment="top" width="0pt">
14114 <column alignment="left" valignment="top" width="0pt">
14115 <column alignment="left" valignment="top" width="0pt">
14116 <row bottomline="true">
14117 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14129 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14139 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14152 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14164 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14172 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14182 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14189 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14194 indicates a get operation
14199 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14207 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14217 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14224 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14234 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14242 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14252 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14259 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14269 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14277 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14287 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14297 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14306 <cell alignment="left" valignment="top" usebox="none">
14314 <cell alignment="right" valignment="top" usebox="none">
14324 <cell alignment="left" valignment="top" usebox="none">
14336 <cell alignment="right" valignment="top" usebox="none">
14345 <cell alignment="left" valignment="top" usebox="none">
14353 <cell alignment="right" valignment="top" usebox="none">
14363 <cell alignment="left" valignment="top" usebox="none">
14373 <cell alignment="right" valignment="top" usebox="none">
14382 <cell alignment="left" valignment="top" usebox="none">
14390 <cell alignment="right" valignment="top" usebox="none">
14400 <cell alignment="left" valignment="top" usebox="none">
14412 <cell alignment="right" valignment="top" usebox="none">
14421 <cell alignment="left" valignment="top" usebox="none">
14429 <cell alignment="right" valignment="top" usebox="none">
14439 <cell alignment="left" valignment="top" usebox="none">
14449 <cell alignment="right" valignment="top" usebox="none">
14458 <cell alignment="left" valignment="top" usebox="none">
14466 <cell alignment="right" valignment="top" usebox="none">
14476 <cell alignment="left" valignment="top" usebox="none">
14488 <cell alignment="right" valignment="top" usebox="none">
14497 <cell alignment="left" valignment="top" usebox="none">
14505 <cell alignment="right" valignment="top" usebox="none">
14515 <cell alignment="left" valignment="top" usebox="none">
14525 <cell alignment="right" valignment="top" usebox="none">
14548 Table\SpecialChar ~
14550 \begin_inset LatexCommand \ref{tab:reply-wire}
14554 summarizes the information transmitted in a reply.
14555 Like an acknowledgement, most of the information is simply echoed from
14557 The initiator and target are obtained directly from the get request, but
14558 are swapped in generating the acknowledgement.
14559 The only new information in the acknowledgement are the manipulated length
14560 and the data, which are determined as the get request is satisfied.
14564 \begin_inset Float table
14571 Information Passed in a Reply
14572 \begin_inset LatexCommand \label{tab:reply-wire}
14595 \begin_inset Tabular
14596 <lyxtabular version="3" rows="11" columns="4">
14597 <features firstHeadEmpty="true">
14598 <column alignment="left" valignment="top" width="0pt">
14599 <column alignment="left" valignment="top" width="0pt">
14600 <column alignment="left" valignment="top" width="0pt">
14601 <column alignment="left" valignment="top" width="0pt">
14602 <row bottomline="true">
14603 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14615 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14625 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14635 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14647 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14655 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14667 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14674 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14679 indicates an acknowledgement
14684 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14692 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14702 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14710 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14719 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14727 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14737 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14745 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14754 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14762 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14772 <cell alignment="left" valignment="top" bottomline="true" usebox="none">
14780 <cell alignment="right" valignment="top" bottomline="true" usebox="none">
14790 <cell alignment="left" valignment="top" usebox="none">
14798 <cell alignment="right" valignment="top" usebox="none">
14808 <cell alignment="left" valignment="top" usebox="none">
14816 <cell alignment="right" valignment="top" usebox="none">
14826 <cell alignment="left" valignment="top" usebox="none">
14834 <cell alignment="right" valignment="top" usebox="none">
14846 <cell alignment="left" valignment="top" usebox="none">
14854 <cell alignment="right" valignment="top" usebox="none">
14864 <cell alignment="left" valignment="top" usebox="none">
14872 <cell alignment="right" valignment="top" usebox="none">
14884 <cell alignment="left" valignment="top" usebox="none">
14892 <cell alignment="right" valignment="top" usebox="none">
14902 <cell alignment="left" valignment="top" usebox="none">
14910 <cell alignment="right" valignment="top" usebox="none">
14922 <cell alignment="left" valignment="top" usebox="none">
14930 <cell alignment="right" valignment="top" usebox="none">
14940 <cell alignment="left" valignment="top" usebox="none">
14948 <cell alignment="right" valignment="top" usebox="none">
14960 <cell alignment="left" valignment="top" usebox="none">
14967 <cell alignment="right" valignment="top" usebox="none">
14972 obtained from the operation
14977 <cell alignment="left" valignment="top" usebox="none">
14985 <cell alignment="right" valignment="top" usebox="none">
14995 <cell alignment="left" valignment="top" usebox="none">
15002 <cell alignment="right" valignment="top" usebox="none">
15007 obtained from the operation
15022 \begin_inset LatexCommand \label{sec:receiving}
15029 When an incoming message arrives on a network interface, the communication
15030 system first checks that the target process identified in the request is
15031 a valid process that has initialized the network interface (i.e., that the
15032 target process has a valid Portal table).
15033 If this test fails, the communication system discards the message and increment
15034 s the dropped message count for the interface.
15035 The remainder of the processing depends on the type of the incoming message.
15036 Put and get messages are subject to access control checks and translation
15037 (searching a match list), while acknowledgement and reply messages bypass
15038 the access control checks and the translation step.
15041 Acknowledgement messages include a handle for the memory descriptor used
15047 This memory descriptor will identify the event queue where the event should
15049 Upon receipt of an acknowledgement, the runtime system only needs to confirm
15050 that the memory descriptor and event queue still exist and that there is
15051 space for another event.
15052 Should the any of these conditions fail, the message is simply discarded
15053 and the dropped message count for the interface is incremented.
15054 Otherwise, the system builds an acknowledgement event from the information
15055 in the acknowledgement message and adds it to the event queue.
15058 Reception of reply messages is also relatively straightforward.
15059 Each reply message includes a handle for a memory descriptor.
15060 If this descriptor exists, it is used to receive the message.
15061 A reply message will be dropped if the memory descriptor identified in
15062 the request doesn't exist.
15063 In either of this case, the dropped message count for the interface is
15065 These are the only reasons for dropping reply messages.
15066 Every memory descriptor accepts and truncates incoming reply messages,
15067 eliminating the other potential reasons for rejecting a reply message.
15070 The critical step in processing an incoming put or get request involves
15071 mapping the request to a memory descriptor.
15072 This step starts by using the Portal index in the incoming request to identify
15073 a list of match entries.
15074 This list of match entries is searched in order until a match entry is
15075 found whose match criteria matches the match bits in the incoming request
15076 and whose memory descriptor accepts the request.
15079 Because acknowledge and reply messages are generated in response to requests
15080 made by the process receiving these messages, the checks performed by the
15081 runtime system for acknowledgements and replies are minimal.
15082 In contrast, put and get messages are generated by remote processes and
15083 the checks performed for these messages are more extensive.
15084 Incoming put or get messages may be rejected because:
15087 the Portal index supplied in the request is not valid;
15090 the cookie supplied in the request is not a valid access control entry;
15094 the access control entry identified by the cookie does not match the identifier
15095 of the requesting process;
15098 the access control entry identified by the access control entry does not
15099 match the Portal index supplied in the request; or
15102 the match bits supplied in the request do not match any of the match entries
15103 with a memory descriptor that accepts the request.
15107 In all cases, if the message is rejected, the incoming message is discarded
15108 and the dropped message count for the interface is incremented.
15111 A memory descriptor may reject an incoming request for any of the following
15123 option has not been enabled and the operation is put or get, respectively;
15127 the length specified in the request is too long for the memory descriptor
15132 option has not been enabled.
15136 \begin_inset LatexCommand \label{sec:examples}
15143 The examples presented in this chapter have not been updated to reflect
15147 In this section we present several example to illustrate expected usage
15148 patterns for the Portals 3.2 API.
15149 The first example describes how to implement parallel servers using the
15150 features of the Portals 3.2 API.
15151 This example covers the access control list and the use of remote managed
15153 The second example presents an approach to dealing with dropped requests.
15154 This example covers aspects of match lists and memory descriptors.
15155 The final example covers message reception in MPI.
15156 This example illustrates more sophisticated uses of matching and a procedure
15157 to update a memory descriptor.
15160 Parallel File Servers
15161 \begin_inset LatexCommand \label{sec:expfs}
15168 Figure\SpecialChar ~
15170 \begin_inset LatexCommand \ref{fig:file}
15174 illustrates the logical structure of a parallel file server.
15175 In this case, the parallel server consists of four servers that stripe
15176 application data across four disks.
15177 We would like to present applications with the illusion that the file server
15178 is a single entity.
15179 We will assume that all of the processes that constitute the parallel server
15180 have the same user id.
15184 \begin_inset Float figure
15192 \begin_inset Graphics FormatVersion 1
15196 rotateOrigin center
15205 Parallel File Server
15206 \begin_inset LatexCommand \label{fig:file}
15216 When an application establishes a connection to the parallel file server,
15217 it will allocate a Portal and access control list entry for communicating
15219 The access control list entry will include the Portal and match any process
15220 in the parallel file server's, so all of the file server processes will
15221 have access to the portal.
15222 The Portal information and access control entry will be sent to the file
15223 server at this time.
15224 If the application and server need to have multiple, concurrent I/O operations,
15225 they can use additional portals or match entries to keep the operations
15226 from interfering with one another.
15229 When an application initiates an I/O operation, it first builds a memory
15230 descriptor that describes the memory region involved in the operation.
15231 This memory descriptor will enable the appropriate operation (put for read
15232 operations and get for write operations) and enable the use of remote offsets
15233 (this lets the servers decide where their data should be placed in the
15235 After creating the memory descriptor and linking it into the appropriate
15236 Portal entry, the application sends a read or write request (using
15240 ) to one of the file server processes.
15241 The file server processes can then use put or get operations with the appropria
15242 te offsets to fill or retrieve the contents of the application's buffer.
15243 To know when the operation has completed, the application can add an event
15244 queue to the memory descriptor and add up the lengths of the remote operations
15245 until the sum is the size of the requested I/O operation.
15248 Dealing with Dropped Requests
15249 \begin_inset LatexCommand \label{sec:exdrop}
15256 If a process does not anticipate unexpected requests, they will be discarded.
15257 Applications using the Portals API can query the dropped count for the
15258 interface to determine the number of requests that have been dropped (see
15259 Section\SpecialChar ~
15261 \begin_inset LatexCommand \ref{sec:nistatus}
15266 While this approach minimizes resource consumption, it does not provide
15267 information that might be critical in debugging the implementation of a
15268 higher level protocol.
15271 To keep track of more information about dropped requests, we use a memory
15272 descriptor that truncates each incoming request to zero bytes and logs
15274 \begin_inset Quotes eld
15278 \begin_inset Quotes erd
15281 operations in an event queue.
15282 Note that the operations are not dropped in the Portals sense, because
15283 the operation succeeds.
15286 The following code fragment illustrates an implementation of this approach.
15287 In this case, we assume that a thread is launched to execute the function
15293 This code starts by building an event queue to log truncated operations
15294 and a memory descriptor to truncate the incoming requests.
15295 This example only captures
15296 \begin_inset Quotes eld
15300 \begin_inset Quotes erd
15303 requests for a single portal.
15304 In a more realistic situation, the memory descriptor would be appended
15305 to the match list for every portal.
15306 We also assume that the thread is capable of keeping up with the
15307 \begin_inset Quotes eld
15311 \begin_inset Quotes erd
15315 If this is not the case, we could use a finite threshold on the memory
15316 descriptor to capture the first few dropped requests.
15323 #include <stdlib.h>
15325 #include <portals.h>
15329 #define DROP_SIZE 32 /* number of dropped requests to track */
15333 int watch_drop( ptl_handle_ni_t ni, ptl_pt_index_t index ) {
15335 ptl_handle_eq_t drop_events;
15339 ptl_handle_md_t drop_em;
15341 ptl_md_t drop_desc;
15343 ptl_process_id_t any_proc;
15345 ptl_handle_me_t match_any;
15349 /* create the event queue */
15351 if( PtlEQAlloc(ni, DROP_SIZE, &drop_events) != PTL_OK ) {
15353 fprintf( stderr, "Couldn't create the event queue
15363 /* build a match entry */
15365 any_proc.nid = PTL_ID_ANY;
15367 any_proc.pid = PTL_ID_ANY;
15369 PtlMEAttach( index, any_proc, 0, ~(ptl_match_bits_t)0, PTL_RETAIN,
15375 /* create the memory descriptor */
15377 drop_desc.start = NULL;
15379 drop_desc.length = 0;
15381 drop_desc.threshold = PTL_MD_THRESH_INF;
15383 drop_desc.options = PTL_MD_OP_PUT | PTL_MD_OP_GET | PTL_MD_TRUNCATE;
15385 drop_desc.user_ptr = NULL;
15387 drop_desc.eventq = drop_events;
15389 if( PtlMDAttach(match_any, drop_desc, &drop_em) != PTL_OK ) {
15391 fprintf( stderr, "Couldn't create the memory descriptor
15401 /* watch for "dropped" requests */
15405 if( PtlEQWait( drop_events, &event ) != PTL_OK ) break;
15407 fprintf( stderr, "Dropped request from gid = event.initiator.gid,
15408 event.initiator.rid );
15415 Message Transmission in MPI
15416 \begin_inset LatexCommand \label{sec:exmpi}
15423 We conclude this section with a fairly extensive example that describes
15424 an approach to implementing message transmission for MPI.
15425 Like many MPI implementations, we distinguish two message transmission
15426 protocols: a short message protocol and a long message protocol.
15427 We use the constant
15431 to determine the size of a long message.
15434 For small messages, the sender simply sends the message and presumes that
15435 the message will be received (i.e., the receiver has allocated a memory region
15436 to receive the message body).
15437 For large messages, the sender also sends the message, but does not presume
15438 that the message body will be saved.
15439 Instead, the sender builds a memory descriptor for the message and enables
15440 get operations on this descriptor.
15441 If the target does not save the body of the message, it will record an
15442 event for the put operation.
15443 When the process later issues a matching MPI receive, it will perform a
15444 get operation to retrieve the body of the message.
15447 To facilitate receive side matching based on the protocol, we use the most
15448 significant bit in the match bits to indicate the protocol: 1 for long
15449 messages and 0 for short messages.
15452 The following code presents a function that implements the send side of
15454 The global variable
15458 is the last match entry attached to the Portal index used for posting long
15460 This entry does not match any incoming requests (i.e., the memory descriptor
15461 rejects all get operations) and is built during initialization of the MPI
15463 The other global variable,
15467 , is a handle for the network interface used by the MPI implementation.
15472 extern ptl_handle_me_t EndGet;
15474 extern ptl_handle_ni_t MPI_NI;
15478 void MPIsend( void *buf, ptl_size_t len, void *data, ptl_handle_eq_t eventq,
15480 ptl_process_id target, ptl_match_bits_t match )
15484 ptl_handle_md_t send_handle;
15488 ptl_ack_req_t want_ack;
15492 mem_desc.start = buf;
15494 mem_desc.length = len;
15496 mem_desc.threshold = 1;
15498 mem_desc.options = PTL_MD_GET_OP;
15500 mem_desc.user_ptr = data;
15502 mem_desc.eventq = eventq;
15506 if( len >= MPI_LONG_LENGTH ) {
15508 ptl_handle_me_t me_handle;
15512 /* add a match entry to the end of the get list */
15514 PtlMEInsert( target, match, 0, PTL_UNLINK, PTL_INS_BEFORE, EndGet,
15517 PtlMDAttach( me_handle, mem_desc, PTL_UNLINK, NULL );
15521 /* we want an ack for long messages */
15523 want_ack = PTL_ACK_REQ;
15527 /* set the protocol bit to indicate that this is a long message
15534 /* we don't want an ack for short messages */
15536 want_ack = PTL_ACK_REQ;
15540 /* set the protocol bit to indicate that this is a short message
15549 /* create a memory descriptor and send it */
15551 PtlMDBind( MPI_NI, mem_desc, &send_handle );
15553 PtlPut( send_handle, want_ack, target, MPI_SEND_PINDEX, MPI_AINDEX, match,
15563 function returns as soon as the message has been scheduled for transmission.
15564 The event queue argument,
15568 , can be used to determine the disposition of the message.
15581 event will be recorded for each message as the message is transmitted.
15582 For small messages, this is the only event that will be recorded in
15587 In contrast, long messages include an explicit request for an acknowledgement.
15592 process has posted a matching receive, the acknowledgement will be sent
15593 as the message is received.
15594 If a matching receive has not been posted, the message will be discarded
15595 and no acknowledgement will be sent.
15600 process later issues a matching receive, the receive will be translated
15601 into a get operation and a
15605 event will be recorded in
15612 Figure\SpecialChar ~
15614 \begin_inset LatexCommand \ref{fig:mpi}
15618 illustrates the organization of the match list used for receiving MPI messages.
15619 The initial entries (not shown in this figure) would be used to match the
15620 MPI receives that have been preposted by the application.
15621 The preposted receives are followed by a match entry,
15625 , that marks the boundary between preposted receives and the memory descriptors
15627 \begin_inset Quotes eld
15631 \begin_inset Quotes erd
15639 entry is followed by a small collection of match entries that match unexpected
15641 \begin_inset Quotes eld
15645 \begin_inset Quotes erd
15648 messages, i.e., messages that have a 0 in the most significant bit of their
15650 The memory descriptors associated with these match entries will append
15651 the incoming message to the associated memory descriptor and record an
15652 event in an event queue for unexpected messages.
15653 The unexpected short message matching entries are followed by a match entry
15654 that will match messages that were not matched by the preceding match entries,
15655 i.e., the unexpected long messages.
15656 The memory descriptor associated with this match entry truncates the message
15657 body and records an event in the event queue for unexpected messages.
15658 Note that of the memory descriptors used for unexpected messages share
15659 a common event queue.
15660 This makes it possible to process the unexpected messages in the order
15661 in which they arrived, regardless of.
15665 \begin_inset Float figure
15673 \begin_inset Graphics FormatVersion 1
15677 rotateOrigin center
15686 Message Reception in MPI
15687 \begin_inset LatexCommand \label{fig:mpi}
15697 When the local MPI process posts an MPI receive, we must first search the
15698 events unexpected message queue to see if a matching message has already
15700 If no matching message is found, a match entry for the receive is inserted
15705 entry--after the match entries for all of the previously posted receives
15706 and before the match entries for the unexpected messages.
15707 This ensures that preposted receives are matched in the order that they
15708 were posted (a requirement of MPI).
15712 While this strategy respects the temporal semantics of MPI, it introduces
15713 a race condition: a matching message might arrive after the events in the
15714 unexpected message queue have been searched, but before the match entry
15715 for the receive has been inserted in the match list.
15719 To avoid this race condition we start by setting the
15723 of the memory descriptor to 0, making the descriptor inactive.
15724 We then insert the match entry into the match list and proceed to search
15725 the events in the unexpected message queue.
15726 A matching message that arrives as we are searching the unexpected message
15727 queue will not be accepted by the memory descriptor and, if not matched
15728 by an earlier match list element, will add an event to the unexpected message
15730 After searching the events in the unexpected message queue, we update the
15731 memory descriptor, setting the threshold to 1 to activate the memory descriptor.
15732 This update is predicated by the condition that the unexpected message
15734 We repeat the process of searching the unexpected message queue until the
15738 The following code fragment illustrates this approach.
15739 Because events must be removed from the unexpected message queue to be
15740 examined, this code fragment assumes the existence of a user managed event
15745 , for the events that have already been removed from the unexpected message
15747 In an effort to keep the example focused on the basic protocol, we have
15748 omitted the code that would be needed to manage the memory descriptors
15749 used for unexpected short messages.
15750 In particular, we simply leave messages in these descriptors until they
15751 are received by the application.
15752 In a robust implementation, we would introduce code to ensure that short
15753 unexpected messages are removed from these memory descriptors so that they
15759 extern ptl_handle_eq_t UnexpQueue;
15761 extern ptl_handle_me_t RcvMark;
15763 extern ptl_handle_me_t ShortMatch;
15767 typedef struct event_list_tag {
15771 struct event_list_tag* next;
15777 extern event_list Rcvd;
15781 void AppendRcvd( ptl_event_t event )
15785 /* append an event onto the Rcvd list */
15791 int SearchRcvd( void *buf, ptl_size_t len, ptl_process_id_t sender, ptl_match_bi
15794 ptl_match_bits_t ignore, ptl_event_t *event )
15798 /* Search the Rcvd event queue, looking for a message that matches the
15801 * If one is found, remove the event from the Rcvd list and return it.
15808 typedef enum { RECEIVED, POSTED } receive_state;
15812 receive_state CopyMsg( void *buf, ptl_size_t &length, ptl_event_t event,
15819 ptl_handle_me_t me_handle;
15823 if( event.rlength >= MPI_LONG_LENGTH ) {
15825 PtlMDBind( MPI_NI, md_buf, &md_handle );
15827 PtlGet( event.initiator, MPI_GET_PINDEX, 0, event.match_bits, MPI_AINDEX,
15834 /* copy the message */
15836 if( event.mlength < *length ) *length = event.mlength;
15838 memcpy( buf, (char*)event.md_desc.start+event.offset, *length );
15848 receive_state MPIreceive( void *buf, ptl_size_t &len, void *MPI_data, ptl_handle
15851 ptl_process_id_t sender, ptl_match_bits_t match,
15852 ptl_match_bits_t ignore )
15858 ptl_handle_md_t md_handle;
15860 ptl_handle_me_t me_handle;
15866 /* build a memory descriptor for the receive */
15868 md_buf.start = buf;
15870 md_buf.length = *len;
15872 md_buf.threshold = 0; /* temporarily disabled */
15874 md_buf.options = PTL_MD_PUT_OP;
15876 md_buf.user_ptr = MPI_data;
15878 md_buf.eventq = eventq;
15882 /* see if we have already received the message */
15884 if( SearchRcvd(buf, len, sender, match, ignore, &event) )
15886 return CopyMsg( buf, len, event, md_buf );
15890 /* create the match entry and attach the memory descriptor */
15892 PtlMEInsert(sender, match, ignore, PTL_UNLINK, PTL_INS_BEFORE, RcvMark,
15895 PtlMDAttach( me_handle, md_buf, PTL_UNLINK, &md_handle );
15899 md_buf.threshold = 1;
15903 if( PtlEQGet( UnexpQueue, &event ) != PTL_EQ_EMPTY ) {
15905 if( MPIMatch(event, match, ignore, sender) ) {
15907 return CopyMsg( buf, len, (char*)event.md_desc.start+event.offset,
15912 AppendRcvd( event );
15918 while( PtlMDUpdate(md_handle, NULL, &md_buf, unexp_queue) == PTL_NOUPDATE
15929 Several people have contributed to the philosophy, design, and implementation
15930 of the Portals message passing architecture as it has evolved.
15931 We acknowledge the following people for their contributions: Al Audette,
15932 Lee Ann Fisk, David Greenberg, Tramm Hudson, Gabi Istrail, Chu Jong, Mike
15933 Levenhagen, Jim Otto, Mark Sears, Lance Shuler, Mack Stallcup, Jeff VanDyke,
15934 Dave van Dresser, Lee Ward, and Stephen Wheat.
15939 \begin_inset LatexCommand \BibTeX[ieee]{portals3}