#LyX 1.2 created this file. For more info see http://www.lyx.org/ \lyxformat 220 \textclass report \begin_preamble \usepackage{fullpage} \renewenvironment{comment}% {\begin{quote}\textbf{Discussion}: \slshape}% {\end{quote}} \pagestyle{myheadings} \end_preamble \language american \inputencoding auto \fontscheme pslatex \graphics default \paperfontsize 10 \spacing single \papersize letterpaper \paperpackage a4 \use_geometry 0 \use_amsmath 0 \use_natbib 0 \use_numerical_citations 0 \paperorientation portrait \secnumdepth 2 \tocdepth 2 \paragraph_separation indent \defskip medskip \quotes_language english \quotes_times 2 \papercolumns 1 \papersides 2 \paperpagestyle headings \layout Title The Portals 3.2 Message Passing Interface \newline Revision 1.1 \layout Author Ron Brightwell \begin_inset Foot collapsed true \layout Standard R. Brightwell and R. Riesen are with the Scalable Computing Systems Department, Sandia National Laboratories, P.O. Box 5800, Albuquerque, NM\SpecialChar ~ \SpecialChar ~ 87111-1110, bright@cs.sandia.gov, rolf@cs.sandia.gov. \end_inset , Arthur B. Maccabe \begin_inset Foot collapsed true \layout Standard A. B. Maccabe is with the Computer Science Department, University of New Mexico, Albuquerque, NM\SpecialChar ~ \SpecialChar ~ 87131-1386, maccabe@cs.unm.edu. \end_inset , Rolf Riesen and Trammell Hudson \layout Abstract This report presents a specification for the Portals 3.2 message passing interface. Portals 3.2 is intended to allow scalable, high-performance network communicatio n between nodes of a parallel computing system. Specifically, it is designed to support a parallel computing platform composed of clusters of commodity workstations connected by a commodity system area network fabric. In addition, Portals 3.2 is well suited to massively parallel processing and embedded systems. Portals 3.2 represents an adaption of the data movement layer developed for massively parallel processing platforms, such as the 4500-node Intel TeraFLOPS machine. \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash clearpage \backslash pagenumbering{roman} \backslash setcounter{page}{3} \end_inset \layout Standard \begin_inset LatexCommand \tableofcontents{} \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash cleardoublepage \end_inset \layout Standard \begin_inset FloatList figure \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash cleardoublepage \end_inset \layout Standard \begin_inset FloatList table \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash cleardoublepage \end_inset \layout Chapter* Summary of Changes for Revision 1.1 \layout Enumerate Updated version number to 3.2 throughout the document \layout Enumerate Section \begin_inset LatexCommand \ref{sub:PtlGetId} \end_inset : added \family typewriter PTL_SEGV \family default to error list for \shape italic PtlGetId \shape default . \layout Enumerate Section \begin_inset LatexCommand \ref{sec:meattach} \end_inset : added \family typewriter PTL_ML_TOOLONG \family default to error list for \shape italic PtlMEAttach \shape default . \layout Enumerate Section \begin_inset LatexCommand \ref{sec:meunlink} \end_inset : removed text referring to a list of associated memory descriptors. \layout Enumerate Section \begin_inset LatexCommand \ref{sec:mdfree} \end_inset : added text to describe unlinking a free-floating memory descriptor. \layout Enumerate Table \begin_inset LatexCommand \ref{tab:types} \end_inset : added entry for \family typewriter ptl_seq_t \family default . \layout Enumerate Section \begin_inset LatexCommand \ref{sec:md-type} \end_inset : \begin_deeper \layout Enumerate added definition of \family typewriter max_offset \family default . \layout Enumerate added text to clarify \family typewriter PTL_MD_MANAGE_REMOTE \family default . \end_deeper \layout Enumerate Section \begin_inset LatexCommand \ref{sec:mdattach} \end_inset : modified text for \family typewriter unlink_op \family default . \layout Enumerate Section \begin_inset LatexCommand \ref{sec:niinit} \end_inset : added text to clarify multiple calls to \shape italic PtlNIInit \shape default . \layout Enumerate Section \begin_inset LatexCommand \ref{sec:mdattach} \end_inset : added text to clarify \family typewriter unlink_nofit \family default . \layout Enumerate Section \begin_inset LatexCommand \ref{sec:receiving} \end_inset : removed text indicating that an MD will reject a message if the associated EQ is full. \layout Enumerate Section \begin_inset LatexCommand \ref{sec:mdfree} \end_inset : added \family typewriter PTL_MD_INUSE \family default error code and text to indicate that only MDs with no pending operations can be unlinked. \layout Enumerate Table \begin_inset LatexCommand \ref{tab:retcodes} \end_inset : added \family typewriter PTL_MD_INUSE \family default return code. \layout Enumerate Section \begin_inset LatexCommand \ref{sec:event-type} \end_inset : added user id field, MD handle field, and NI specific failure field to the \family typewriter ptl_event_t \family default structure. \layout Enumerate Table \begin_inset LatexCommand \ref{tab:types} \end_inset : added \family typewriter ptl_ni_fail_t \family default . \layout Enumerate Section \begin_inset LatexCommand \ref{sec:event-type} \end_inset : added \family typewriter PTL_EVENT_UNLINK \family default event type. \layout Enumerate Table \begin_inset LatexCommand \ref{tab:func} \end_inset : removed \shape slanted PtlTransId \shape default . \layout Enumerate Section \begin_inset LatexCommand \ref{sec:meattach} \end_inset , Section \begin_inset LatexCommand \ref{sec:meinsert} \end_inset , Section \begin_inset LatexCommand \ref{sec:put} \end_inset : listed allowable constants with relevant fields. \layout Enumerate Table \begin_inset LatexCommand \ref{tab:func} \end_inset : added \shape italic PtlMEAttachAny \shape default function. \layout Enumerate Table \begin_inset LatexCommand \ref{tab:retcodes} \end_inset : added \family typewriter PTL_PT_FULL \family default return code for \shape italic PtlMEAttachAny \shape default . \layout Enumerate Table \begin_inset LatexCommand \ref{tab:oconsts} \end_inset : updated to reflect new event types. \layout Enumerate Section \begin_inset LatexCommand \ref{sec:id-type} \end_inset : added \family typewriter ptl_nid_t \family default , \family typewriter ptl_pid_t \family default , and \family typewriter ptl_uid_t \family default . \layout Chapter* Summary of Changes for Version 3.1 \layout Section* Thread Issues \layout Standard The most significant change to the interface from version 3.0 to 3.1 involves the clarification of how the interface interacts with multi-threaded applicatio ns. We adopted a generic thread model in which processes define an address space and threads share the address space. Consideration of the API in the light of threads lead to several clarifications throughout the document: \layout Enumerate Glossary: \begin_deeper \layout Enumerate added a definition for \emph on thread \emph default , \layout Enumerate reworded the definition for \emph on process \emph default . \end_deeper \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:apiover} \end_inset : added section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:threads} \end_inset to describe the multi-threading model used by the Portals API. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:ptlinit} \end_inset : \emph on PtlInit \emph default must be called at least once and may be called any number of times. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:ptlfini} \end_inset : \emph on PtlFini \emph default should be called once as the process is terminating and not as each thread terminates. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:pid} \end_inset : Portals does not define thread ids. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:ni} \end_inset : network interfaces are associated with processes, not threads. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:niinit} \end_inset : \emph on PtlNIInit \emph default must be called at least once and may be called any number of times. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:eqget} \end_inset : \emph on PtlEQGet \emph default returns \family typewriter PTL_EQ_EMPTY \family default if a thread is blocked on \emph on PtlEQWait \emph default . \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:eqwait} \end_inset : waiting threads are awakened in FIFO order. \layout Standard Two functions, \emph on PtlNIBarrier \emph default and \emph on PtlEQCount \emph default were removed from the API. \emph on PtlNIBarrier \emph default was defined to block the calling process until all of the processes in the application group had invoked \emph on PtlNIBarrier \emph default . We now consider this functionality, along with the concept of groups (see the discussion under \begin_inset Quotes eld \end_inset other changes \begin_inset Quotes erd \end_inset ), to be part of the runtime system, not part of the Portals API. \emph on PtlEQCount \emph default was defined to return the number of events in an event queue. Because external operations may lead to new events being added and other threads may remove events, the value returned by \emph on PtlEQCount \emph default would have to be a hint about the number of events in the event queue. \layout Section* Handling small, unexpected messages \layout Standard Another set of changes relates to handling small unexpected messages in MPI. In designing version 3.0, we assumed that each unexpected message would be placed in a unique memory descriptor. To avoid the need to process a long list of memory descriptors, we moved the memory descriptors out of the match list and hung them off of a single match list entry. In this way, large unexpected messages would only encounter a single \begin_inset Quotes eld \end_inset short message \begin_inset Quotes erd \end_inset match list entry before encountering the \begin_inset Quotes eld \end_inset long message \begin_inset Quotes erd \end_inset match list entry. Experience with this strategy identified resource management problems with this approach. In particular, a long sequence of very short (or zero length) messages could quickly exhaust the memory descriptors constructed for handling unexpecte d messages. Our new strategy involves the use of several very large memory descriptors for small unexpected messages. Consecutive unexpected messages will be written into the first of these memory descriptors until the memory descriptor fills up. When the first of the \begin_inset Quotes eld \end_inset small memory \begin_inset Quotes erd \end_inset descriptors fills up, it will be unlinked and subsequent short messages will be written into the next \begin_inset Quotes eld \end_inset short message \begin_inset Quotes erd \end_inset memory descriptor. In this case, a \begin_inset Quotes eld \end_inset short message \begin_inset Quotes erd \end_inset memory descriptor will be declared full when it does not have sufficient space for the largest small unexpected message. \layout Standard This lead to two significant changes. First, each match list entry now has a single memory descriptor rather than a list of memory descriptors. Second, in addition to exceeding the operation threshold, a memory descriptor can be unlinked when the local offset exceeds a specified value. These changes have lead to several changes in this document: \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{subsec:paddress} \end_inset : \begin_deeper \layout Enumerate removed references to the memory descriptor list, \layout Enumerate changed the portals address translation description to indicate that unlinking a memory descriptor implies unlinking the associated match list entry--match list entries can no longer be unlinked independently from the memory descriptor. \end_deeper \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:meattach} \end_inset : \begin_deeper \layout Enumerate removed unlink from argument list, \layout Enumerate removed description of \family typewriter ptl_unlink \family default type, \layout Enumerate changed wording of the error condition when the Portal table index already has an associated match list. \end_deeper \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:meinsert} \end_inset : removed unlink from argument list. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:md-type} \end_inset : added \family typewriter max_offset \family default . \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:mdattach} \end_inset : \begin_deeper \layout Enumerate added description of \family typewriter ptl_unlink \family default type, \layout Enumerate removed reference to memory descriptor lists, \layout Enumerate changed wording of the error condition when match list entry already has an associated memory descriptor, \layout Enumerate changed the description of the \family typewriter unlink \family default argument. \end_deeper \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:md} \end_inset : removed \family typewriter PtlMDInsert \family default operation. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:mdbind} \end_inset : removed references to memory descriptor list. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:mdfree} \end_inset : removed reference to memory descriptor list. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:summary} \end_inset : removed references to PtlMDInsert. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:semantics} \end_inset : removed reference to memory descriptor list. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:exmpi} \end_inset : revised the MPI example to reflect the changes to the interface. \layout Standard Several changes have been made to improve the general documentation of the interface. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:handle-type} \end_inset : documented the special value \family typewriter PTL_EQ_NONE \family default . \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:id-type} \end_inset : documented the special value \family typewriter PTL_ID_ANY \family default . \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:mdbind} \end_inset : documented the return value \family typewriter PTL_INV_EQ \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:mdupdate} \end_inset : clarified the description of the \emph on PtlMDUpdate \emph default function. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:implvals} \end_inset : introduced a new section to document the implementation defined values. \layout Enumerate Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:summary} \end_inset : modified Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:oconsts} \end_inset to indicate where each constant is introduced and where it is used. \layout Section* Other changes \layout Subsection* Implementation defined limits (Section \begin_inset LatexCommand \ref{sec:niinit} \end_inset ) \layout Standard The earlier version provided implementation defined limits for the maximum number of match entries, the maximum number of memory descriptors, etc. Rather than spanning the entire implementation, these limits are now associated with individual network interfaces. \layout Subsection* Added User Ids (Section \begin_inset LatexCommand \ref{sec:uid} \end_inset ) \layout Standard Group Ids had been used to simplify access control entries. In particular, a process could allow access for all of the processes in a group. User Ids have been introduced to regain this functionality. We use user ids to fill this role. \layout Subsection* Removed Group Ids and Rank Ids (Section \begin_inset LatexCommand \ref{sec:pid} \end_inset ) \layout Standard The earlier version of Portals had two forms for addressing processes: and . A process group was defined as the collection processes created during application launch. Each process in the group was given a unique rank id in the range 0 to \begin_inset Formula $n-1$ \end_inset where \begin_inset Formula $n$ \end_inset was the number of processes in the group. We removed groups because they are better handled in the runtime system. \layout Subsection* Match lists (Section \begin_inset LatexCommand \ref{sec:meattach} \end_inset ) \layout Standard It is no longer illegal to have an existing match entry when calling PtlMEAttach. A position argument was added to the list of arguments supplied to \emph on PtlMEAttach \emph default to specify whether the new match entry is prepended or appended to the existing list. If there is no existing match list, the position argument is ignored. \layout Subsection* Unlinking Memory Descriptors (Section \begin_inset LatexCommand \ref{sec:md} \end_inset ) \layout Standard Previously, a memory descriptor could be unlinked if the offset exceeded a threshold upon the completion of an operation. In this version, the unlinking is delayed until there is a matching operation which requires more memory than is currently available in the descriptor. In addition to changes in section, this lead to a revision of Figure\SpecialChar ~ \begin_inset LatexCommand \ref{fig:flow} \end_inset . \layout Subsection* Split Phase Operations and Events (Section \begin_inset LatexCommand \ref{sec:eq} \end_inset ) \layout Standard Previously, there were five types of events: \family typewriter PTL_EVENT_PUT \family default , \family typewriter PTL_EVENT_GET \family default , \family typewriter PTL_EVENT_REPLY \family default , \family typewriter PTL_EVENT_SENT \family default , and \family typewriter PTL_EVENT_ACK. \family default The first four of these reflected the completion of potentially long operations. We have introduced new event types to reflect the fact that long operations have a distinct starting point and a distinct completion point. Moreover, the completion may be successful or unsuccessful. \layout Standard In addition to providing a mechanism for reporting failure to higher levels of software, this split provides an opportunity for for improved ordering semantics. Previously, if one process intiated two operations (e.g., two put operations) on a remote process, these operations were guaranteed to complete in the same order that they were initiated. Now, we only guarantee that the initiation events are delivered in the same order. In particular, the operations do not need to complete in the order that they were intiated. \layout Subsection* Well known proces ids (Section \begin_inset LatexCommand \ref{sec:niinit} \end_inset ) \layout Standard To support the notion of \begin_inset Quotes eld \end_inset well known process ids, \begin_inset Quotes erd \end_inset we added a process id argument to the arguments for PtlNIInit. \layout Chapter* Glossary \layout Description API Application Programming Interface. A definition of the functions and semantics provided by library of functions. \layout Description Initiator A \emph on process \emph default that initiates a message operation. \layout Description Message An application-defined unit of data that is exchanged between \emph on processes \emph default . \layout Description Message\SpecialChar ~ Operation Either a put operation, which writes data, or a get operation, which reads data. \layout Description Network A network provides point-to-point communication between \emph on nodes \emph default . Internally, a network may provide multiple routes between endpoints (to improve fault tolerance or to improve performance characteristics); however, multiple paths will not be exposed outside of the network. \layout Description Node A node is an endpoint in a \emph on network \emph default . Nodes provide processing capabilities and memory. A node may provide multiple processors (an SMP node) or it may act as a \emph on gateway \emph default between networks. \layout Description Process A context of execution. A process defines a virtual memory (VM) context. This context is not shared with other processes. Several threads may share the VM context defined by a process. \layout Description Target A \emph on process \emph default that is acted upon by a message operation. \layout Description Thread A context of execution that shares a VM context with other threads. \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash cleardoublepage \layout Standard \backslash setcounter{page}{1} \backslash pagenumbering{arabic} \end_inset \layout Chapter Introduction \begin_inset LatexCommand \label{sec:intro} \end_inset \layout Section Overview \layout Standard This document describes an application programming interface for message passing between nodes in a system area network. The goal of this interface is to improve the scalability and performance of network communication by defining the functions and semantics of message passing required for scaling a parallel computing system to ten thousand nodes. This goal is achieved by providing an interface that will allow a quality implementation to take advantage of the inherently scalable design of Portals. \layout Standard This document is divided into several sections: \layout Description Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:intro} \end_inset ---Introduction This section describes the purpose and scope of the Portals API. \layout Description Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:apiover} \end_inset ---An\SpecialChar ~ Overview\SpecialChar ~ of\SpecialChar ~ the\SpecialChar ~ Portals\SpecialChar ~ 3.1\SpecialChar ~ API This section gives a brief overview of the Portals API. The goal is to introduce the key concepts and terminology used in the descripti on of the API. \layout Description Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:api} \end_inset ---The\SpecialChar ~ Portals\SpecialChar ~ 3.2\SpecialChar ~ API This section describes the functions and semantics of the Portals application programming interface. \layout Description Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:semantics} \end_inset --The\SpecialChar ~ Semantics\SpecialChar ~ of\SpecialChar ~ Message\SpecialChar ~ Transmission This section describes the semantics of message transmission. In particular, the information transmitted in each type of message and the processing of incoming messages. \layout Description Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:examples} \end_inset ---Examples This section presents several examples intended to illustrates the use of the Portals API. \layout Section Purpose \layout Standard Existing message passing technologies available for commodity cluster networking hardware do not meet the scalability goals required by the Cplant\SpecialChar ~ \begin_inset LatexCommand \cite{Cplant} \end_inset project at Sandia National Laboratories. The goal of the Cplant project is to construct a commodity cluster that can scale to the order of ten thousand nodes. This number greatly exceeds the capacity for which existing message passing technologies have been designed and implemented. \layout Standard In addition to the scalability requirements of the network, these technologies must also be able to support a scalable implementation of the Message Passing Interface (MPI)\SpecialChar ~ \begin_inset LatexCommand \cite{MPIstandard} \end_inset standard, which has become the \shape italic de facto \shape default standard for parallel scientific computing. While MPI does not impose any scalability limitations, existing message passing technologies do not provide the functionality needed to allow implement ations of MPI to meet the scalability requirements of Cplant. \layout Standard The following are properties of a network architecture that do not impose any inherent scalability limitations: \layout Itemize Connectionless - Many connection-oriented architectures, such as VIA\SpecialChar ~ \begin_inset LatexCommand \cite{VIA} \end_inset and TCP/IP sockets, have limitations on the number of peer connections that can be established. \layout Itemize Network independence - Many communication systems depend on the host processor to perform operations in order for messages in the network to be consumed. Message consumption from the network should not be dependent on host processor activity, such as the operating system scheduler or user-level thread scheduler. \layout Itemize User-level flow control - Many communication systems manage flow control internally to avoid depleting resources, which can significantly impact performance as the number of communicating processes increases. \layout Itemize OS Bypass - High performance network communication should not involve memory copies into or out of a kernel-managed protocol stack. \layout Standard The following are properties of a network architecture that do not impose scalability limitations for an implementation of MPI: \layout Itemize Receiver-managed - Sender-managed message passing implementations require a persistent block of memory to be available for every process, requiring memory resources to increase with job size and requiring user-level flow control mechanisms to manage these resources. \layout Itemize User-level Bypass - While OS Bypass is necessary for high-performance, it alone is not sufficient to support the Progress Rule of MPI asynchronous operations. \layout Itemize Unexpected messages - Few communication systems have support for receiving messages for which there is no prior notification. Support for these types of messages is necessary to avoid flow control and protocol overhead. \layout Section Background \layout Standard Portals was originally designed for and implemented on the nCube machine as part of the SUNMOS (Sandia/UNM OS)\SpecialChar ~ \begin_inset LatexCommand \cite{SUNMOS} \end_inset and Puma\SpecialChar ~ \begin_inset LatexCommand \cite{PumaOS} \end_inset lightweight kernel development projects. Portals went through two design phases, the latter of which is used on the 4500-node Intel TeraFLOPS machine\SpecialChar ~ \begin_inset LatexCommand \cite{TFLOPS} \end_inset . Portals have been very successful in meeting the needs of such a large machine, not only as a layer for a high-performance MPI implementation\SpecialChar ~ \begin_inset LatexCommand \cite{PumaMPI} \end_inset , but also for implementing the scalable run-time environment and parallel I/O capabilities of the machine. \layout Standard The second generation Portals implementation was designed to take full advantage of the hardware architecture of large MPP machines. However, efforts to implement this same design on commodity cluster technology identified several limitations, due to the differences in network hardware as well as to shortcomings in the design of Portals. \layout Section Scalability \layout Standard The primary goal in the design of Portals is scalability. Portals are designed specifically for an implementation capable of supporting a parallel job running on tens of thousands of nodes. Performance is critical only in terms of scalability. That is, the level of message passing performance is characterized by how far it allows an application to scale and not by how it performs in micro-bench marks (e.g., a two node bandwidth or latency test). \layout Standard The Portals API is designed to allow for scalability, not to guarantee it. Portals cannot overcome the shortcomings of a poorly designed application program. Applications that have inherent scalability limitations, either through design or implementation, will not be transformed by Portals into scalable applications. Scalability must be addressed at all levels. Portals do not inhibit scalability, but do not guarantee it either. \layout Standard To support scalability, the Portals interface maintains a minimal amount of state. Portals provide reliable, ordered delivery of messages between pairs of processes. They are connectionless: a process is not required to explicitly establish a point-to-point connection with another process in order to communicate. Moreover, all buffers used in the transmission of messages are maintained in user space. The target process determines how to respond to incoming messages, and messages for which there are no buffers are discarded. \layout Section Communication Model \layout Standard Portals combine the characteristics of both one-side and two-sided communication. They define a \begin_inset Quotes eld \end_inset matching put \begin_inset Quotes erd \end_inset operation and a \begin_inset Quotes eld \end_inset matching get \begin_inset Quotes erd \end_inset operation. The destination of a put (or send) is not an explicit address; instead, each message contains a set of match bits that allow the receiver to determine where incoming messages should be placed. This flexibility allows Portals to support both traditional one-sided operation s and two-sided send/receive operations. \layout Standard Portals allows the target to determine whether incoming messages are acceptable. A target process can choose to accept message operations from any specific process or can choose to ignore message operations from any specific process. \layout Section Zero Copy, OS Bypass and Application Bypass \layout Standard In traditional system architectures, network packets arrive at the network interface card (NIC), are passed through one or more protocol layers in the operating system, and eventually copied into the address space of the application. As network bandwidth began to approach memory copy rates, reduction of memory copies became a critical concern. This concern lead to the development of zero-copy message passing protocols in which message copies are eliminated or pipelined to avoid the loss of bandwidth. \layout Standard A typical zero-copy protocol has the NIC generate an interrupt for the CPU when a message arrives from the network. The interrupt handler then controls the transfer of the incoming message into the address space of the appropriate application. The interrupt latency, the time from the initiation of an interrupt until the interrupt handler is running, is fairly significant. To avoid this cost, some modern NICs have processors that can be programmed to implement part of a message passing protocol. Given a properly designed protocol, it is possible to program the NIC to control the transfer of incoming messages, without needing to interrupt the CPU. Because this strategy does not need to involve the OS on every message transfer, it is frequently called \begin_inset Quotes eld \end_inset OS Bypass. \begin_inset Quotes erd \end_inset ST\SpecialChar ~ \begin_inset LatexCommand \cite{ST} \end_inset , VIA\SpecialChar ~ \begin_inset LatexCommand \cite{VIA} \end_inset , FM\SpecialChar ~ \begin_inset LatexCommand \cite{FM2} \end_inset , GM\SpecialChar ~ \begin_inset LatexCommand \cite{GM} \end_inset , and Portals are examples of OS Bypass protocols. \layout Standard Many protocols that support OS Bypass still require that the application actively participate in the protocol to ensure progress. As an example, the long message protocol of PM requires that the application receive and reply to a request to put or get a long message. This complicates the runtime environment, requiring a thread to process incoming requests, and significantly increases the latency required to initiate a long message protocol. The Portals message passing protocol does not require activity on the part of the application to ensure progress. We use the term \begin_inset Quotes eld \end_inset Application Bypass \begin_inset Quotes erd \end_inset to refer to this aspect of the Portals protocol. \layout Section Faults \layout Standard Given the number of components that we are dealing with and the fact that we are interested in supporting applications that run for very long times, failures are inevitable. The Portals API recognizes that the underlying transport may not be able to successfully complete an operation once it has been initiated. This is reflected in the fact that the Portals API reports three types of events: events indicating the initiation of an operation, events indicating the successful completion of an operation, and events indicating the unsuccessf ul completion of an operation. Every initiation event is eventually followed by a successful completion event or an unsuccessful completion event. \layout Standard Between the time an operation is started and the time that the operation completes (successfully or unsuccessfully), any memory associated with the operation should be considered volatile. That is, the memory may be changed in unpredictable ways while the operation is progressing. Once the operation completes, the memory associated with the operation will not be subject to further modification (from this operation). Notice that unsuccessful operations may alter memory in an essentially unpredictable fashion. \layout Chapter An Overview of the Portals API \begin_inset LatexCommand \label{sec:apiover} \end_inset \layout Standard In this section, we give a conceptual overview of the Portals API. The goal is to provide a context for understanding the detailed description of the API presented in the next section. \layout Section Data Movement \begin_inset LatexCommand \label{sec:dmsemantics} \end_inset \layout Standard A Portal represents an opening in the address space of a process. Other processes can use a Portal to read (get) or write (put) the memory associated with the portal. Every data movement operation involves two processes, the \series bold initiator \series default and the \series bold target \series default . The initiator is the process that initiates the data movement operation. The target is the process that responds to the operation by either accepting the data for a put operation, or replying with the data for a get operation. \layout Standard In this discussion, activities attributed to a process may refer to activities that are actually performed by the process or \emph on on behalf of the process \emph default . The inclusiveness of our terminology is important in the context of \emph on application bypass \emph default . In particular, when we note that the target sends a reply in the case of a get operation, it is possible that reply will be generated by another component in the system, bypassing the application. \layout Standard Figures\SpecialChar ~ \begin_inset LatexCommand \ref{fig:put} \end_inset and \begin_inset LatexCommand \ref{fig:get} \end_inset present graphical interpretations of the Portal data movement operations: put and get. In the case of a put operation, the initiator sends a put request message containing the data to the target. The target translates the Portal addressing information in the request using its local Portal structures. When the request has been processed, the target optionally sends an acknowledge ment message. \layout Standard \begin_inset Float figure placement htbp wide false collapsed false \layout Standard \align center \begin_inset Graphics FormatVersion 1 filename put.eps display color size_type 0 rotateOrigin center lyxsize_type 1 lyxwidth 218pt lyxheight 119pt \end_inset \layout Caption Portal Put (Send) \begin_inset LatexCommand \label{fig:put} \end_inset \end_inset \layout Standard In the case of a get operation, the initiator sends a get request to the target. As with the put operation, the target translates the Portal addressing information in the request using its local Portal structures. Once it has translated the Portal addressing information, the target sends a reply that includes the requested data. \layout Standard \begin_inset Float figure placement htbp wide false collapsed false \layout Standard \align center \begin_inset Graphics FormatVersion 1 filename get.eps display color size_type 0 rotateOrigin center lyxsize_type 1 lyxwidth 218pt lyxheight 119pt \end_inset \layout Caption Portal Get \begin_inset LatexCommand \label{fig:get} \end_inset \end_inset \layout Standard We should note that Portal address translations are only performed on nodes that respond to operations initiated by other nodes. Acknowledgements and replies to get operations bypass the portals address translation structures. \layout Section Portal Addressing \begin_inset LatexCommand \label{subsec:paddress} \end_inset \layout Standard One-sided data movement models (e.g., shmem\SpecialChar ~ \begin_inset LatexCommand \cite{CraySHMEM} \end_inset , ST\SpecialChar ~ \begin_inset LatexCommand \cite{ST} \end_inset , MPI-2\SpecialChar ~ \begin_inset LatexCommand \cite{MPI2} \end_inset ) typically use a triple to address memory on a remote node. This triple consists of a process id, memory buffer id, and offset. The process id identifies the target process, the memory buffer id specifies the region of memory to be used for the operation, and the offset specifies an offset within the memory buffer. \layout Standard In addition to the standard address components (process id, memory buffer id, and offset), a Portal address includes a set of match bits. This addressing model is appropriate for supporting one-sided operations as well as traditional two-sided message passing operations. Specifically, the Portals API provides the flexibility needed for an efficient implementation of MPI-1, which defines two-sided operations with one-sided completion semantics. \layout Standard Figure\SpecialChar ~ \begin_inset LatexCommand \ref{fig:portals} \end_inset presents a graphical representation of the structures used by a target in the interpretation of a Portal address. The process id is used to route the message to the appropriate node and is not reflected in this diagram. The memory buffer id, called the \series bold portal id \series default , is used as an index into the Portal table. Each element of the Portal table identifies a match list. Each element of the match list specifies two bit patterns: a set of \begin_inset Quotes eld \end_inset don't care \begin_inset Quotes erd \end_inset bits, and a set of \begin_inset Quotes eld \end_inset must match \begin_inset Quotes erd \end_inset bits. In addition to the two sets of match bits, each match list element has at most one memory descriptor. Each memory descriptor identifies a memory region and an optional event queue. The memory region specifies the memory to be used in the operation and the event queue is used to record information about these operations. \layout Standard \begin_inset Float figure placement htbp wide false collapsed false \layout Standard \align center \begin_inset Graphics FormatVersion 1 filename portals.eps display color size_type 0 rotateOrigin center lyxsize_type 1 lyxwidth 305pt lyxheight 106pt \end_inset \layout Caption Portal Addressing Structures \begin_inset LatexCommand \label{fig:portals} \end_inset \end_inset \layout Standard Figure\SpecialChar ~ \begin_inset LatexCommand \ref{fig:flow} \end_inset illustrates the steps involved in translating a Portal address, starting from the first element in a match list. If the match criteria specified in the match list entry are met and the memory descriptor list accepts the operation \begin_inset Foot collapsed true \layout Standard Memory descriptors can reject operations because a threshold has been exceeded or because the memory region does not have sufficient space, see Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:md} \end_inset \end_inset , the operation (put or get) is performed using the memory region specified in the memory descriptor. If the memory descriptor specifies that it is to be unlinked when a threshold has been exceeded, the match list entry is removed from the match list and the resources associated with the memory descriptor and match list entry are reclaimed. Finally, if there is an event queue specified in the memory descriptor, the operation is logged in the event queue. \layout Standard \begin_inset Float figure placement htbp wide false collapsed false \layout Standard \align center \begin_inset Graphics FormatVersion 1 filename flow_new.eps display color size_type 0 rotateOrigin center lyxsize_type 1 lyxwidth 447pt lyxheight 282pt \end_inset \layout Caption Portals Address Translation \begin_inset LatexCommand \label{fig:flow} \end_inset \end_inset \layout Standard If the match criteria specified in the match list entry are not met, or there is no memory descriptor associated with the match list entry, or the memory descriptor associated with the match list entry rejects the operation, the address translation continues with the next match list entry. If the end of the match list has been reached, the address translation is aborted and the incoming requested is discarded. \layout Section Access Control \layout Standard A process can control access to its portals using an access control list. Each entry in the access control list specifies a process id and a Portal table index. The access control list is actually an array of entries. Each incoming request includes an index into the access control list (i.e., a \begin_inset Quotes eld \end_inset cookie \begin_inset Quotes erd \end_inset or hint). If the id of the process issuing the request doesn't match the id specified in the access control list entry or the Portal table index specified in the request doesn't match the Portal table index specified in the access control list entry, the request is rejected. Process identifiers and Portal table indexes may include wild card values to increase the flexibility of this mechanism. \layout Standard Two aspects of this design merit further discussion. First, the model assumes that the information in a message header, the sender's id in particular, is trustworthy. In most contexts, we assume that the entity that constructs the header is trustworthy; however, using cryptographic techniques, we could easily devise a protocol that would ensure the authenticity of the sender. \layout Standard Second, because the access check is performed by the receiver, it is possible that a malicious process will generate thousands of messages that will be denied by the receiver. This could saturate the network and/or the receiver, resulting in a \emph on denial of service \emph default attack. Moving the check to the sender using capabilities, would remove the potential for this form of attack. However, the solution introduces the complexities of capability management (exchange of capabilities, revocation, protections, etc). \layout Section Multi-threaded Applications \begin_inset LatexCommand \label{sec:threads} \end_inset \layout Standard The Portals API supports a generic view of multi-threaded applications. From the perspective of the Portals API, an application program is defined by a set of processes. Each process defines a unique address space. The Portals API defines access to this address space from other processes (using portals addressing and the data movement operations). A process may have one or more \emph on threads \emph default executing in its address space. \layout Standard With the exception of \emph on PtlEQWait \emph default every function in the Portals API is non-blocking and atomic with respect to both other threads and external operations that result from data movement operations. While individual operations are atomic, sequences of these operations may be interleaved between different threads and with external operations. The Portals API does not provide any mechanisms to control this interleaving. It is expected that these mechanisms will be provided by the API used to create threads. \layout Chapter The Portals API \begin_inset LatexCommand \label{sec:api} \end_inset \layout Section Naming Conventions \begin_inset LatexCommand \label{sec:conv} \end_inset \layout Standard The Portals API defines two types of entities: functions and types. Function always start with \emph on Ptl \emph default and use mixed upper and lower case. When used in the body of this report, function names appear in italic face, e.g., \emph on PtlInit \emph default . The functions associated with an object type will have names that start with \emph on Ptl \emph default , followed by the two letter object type code shown in Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:objcodes} \end_inset . As an example, the function \emph on PtlEQAlloc \emph default allocates resources for an event queue. \layout Standard \begin_inset Float table placement htbp wide false collapsed false \layout Caption Object Type Codes \begin_inset LatexCommand \label{tab:objcodes} \end_inset \begin_inset ERT status Collapsed \layout Standard \backslash medskip \newline \end_inset \layout Standard \align center \size small \begin_inset Tabular \begin_inset Text \layout Standard \emph on xx \end_inset \begin_inset Text \layout Standard Name \end_inset \begin_inset Text \layout Standard Section \end_inset \begin_inset Text \layout Standard EQ \end_inset \begin_inset Text \layout Standard Event Queue \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:eq} \end_inset \end_inset \begin_inset Text \layout Standard MD \end_inset \begin_inset Text \layout Standard Memory Descriptor \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md} \end_inset \end_inset \begin_inset Text \layout Standard ME \end_inset \begin_inset Text \layout Standard Match list Entry \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:me} \end_inset \end_inset \begin_inset Text \layout Standard NI \end_inset \begin_inset Text \layout Standard Network Interface \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ni} \end_inset \end_inset \end_inset \end_inset \layout Standard Type names use lower case with underscores to separate words. Each type name starts with \family typewriter ptl \family default _ and ends with \family typewriter _t \family default . When used in the body of this report, type names appear in a fixed font, e.g., \family typewriter ptl_match_bits_t \family default . \layout Standard Names for constants use upper case with underscores to separate words. Each constant name starts with \family typewriter PTL_ \family default . When used in the body of this report, type names appear in a fixed font, e.g., \family typewriter PTL_OK \family default . \layout Section Base Types \layout Standard The Portals API defines a variety of base types. These types represent a simple renaming of the base types provided by the C programming language. In most cases these new type names have been introduced to improve type safety and to avoid issues arising from differences in representation sizes (e.g., 16-bit or 32-bit integers). \layout Subsection Sizes \begin_inset LatexCommand \label{sec:size-t} \end_inset \layout Standard The type \family typewriter ptl_size_t \family default is an unsigned 64-bit integral type used for representing sizes. \layout Subsection Handles \begin_inset LatexCommand \label{sec:handle-type} \end_inset \layout Standard Objects maintained by the API are accessed through handles. Handle types have names of the form \family typewriter ptl_handle_ \emph on xx \emph default _t \family default , where \emph on xx \emph default is one of the two letter object type codes shown in Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:objcodes} \end_inset . For example, the type \family typewriter ptl_handle_ni_t \family default is used for network interface handles. \layout Standard Each type of object is given a unique handle type to enhance type checking. The type, \family typewriter ptl_handle_any_t \family default , can be used when a generic handle is needed. Every handle value can be converted into a value of type \family typewriter ptl_handle_any_t \family default without loss of information. \layout Standard Handles are not simple values. Every portals object is associated with a specific network interface and an identifier for this interface (along with an object identifier) is part of the handle for the object. \layout Standard The special value \family typewriter PTL_EQ_NONE \family default , of type \family typewriter ptl_handle_eq_t \family default , is used to indicate the absence of an event queue. See sections \begin_inset LatexCommand \ref{sec:mdfree} \end_inset and\SpecialChar ~ \begin_inset LatexCommand \ref{sec:mdupdate} \end_inset for uses of this value. \layout Subsection Indexes \begin_inset LatexCommand \label{sec:index-type} \end_inset \layout Standard The types \family typewriter ptl_pt_index_t \family default and \family typewriter ptl_ac_index_t \family default are integral types used for representing Portal table indexes and access control tables indexes, respectively. See section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:niinit} \end_inset for limits on values of these types. \layout Subsection Match Bits \begin_inset LatexCommand \label{sec:mb-type} \end_inset \layout Standard The type \family typewriter ptl_match_bits_t \family default is capable of holding unsigned 64-bit integer values. \layout Subsection Network Interfaces \begin_inset LatexCommand \label{sec:ni-type} \end_inset \layout Standard The type \family typewriter ptl_interface_t \family default is an integral type used for identifying different network interfaces. Users will need to consult the local documentation to determine appropriate values for the interfaces available. The special value \family typewriter PTL_IFACE_DEFAULT \family default identifies the default interface. \layout Subsection Identifiers \begin_inset LatexCommand \label{sec:id-type} \end_inset \layout Standard The type \family typewriter ptl_nid_t \family default is an integral type used for representing node ids \family typewriter , ptl_pid_t \family default is an integral type for representing process ids, and \family typewriter ptl_uid_t \family default is an integral type for representing user ids. \layout Standard The special values \family typewriter PTL_PID_ANY \family default matches any process identifier, PTL_NID_ANY matches any node identifier, and \family typewriter PTL_UID_ANY \family default matches any user identifier. See sections \begin_inset LatexCommand \ref{sec:meattach} \end_inset and\SpecialChar ~ \begin_inset LatexCommand \ref{sec:acentry} \end_inset for uses of these values. \layout Subsection Status Registers \begin_inset LatexCommand \label{sec:stat-type} \end_inset \layout Standard Each network interface maintains an array of status registers that can be accessed using the \family typewriter PtlNIStatus \family default function (see Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:nistatus} \end_inset ). The type \family typewriter ptl_sr_index_t \family default defines the types of indexes that can be used to access the status registers. The only index defined for all implementations is \family typewriter PTL_SR_DROP_COUNT \family default which identifies the status register that counts the dropped requests for the interface. Other indexes (and registers) may be defined by the implementation. \layout Standard The type \family typewriter ptl_sr_value_t \family default defines the types of values held in status registers. This is a signed integer type. The size is implementation dependent, but must be at least 32 bits. \layout Section Initialization and Cleanup \begin_inset LatexCommand \label{sec:init} \end_inset \layout Standard The Portals API includes a function, \emph on PtlInit \emph default , to initialize the library and a function, \emph on PtlFini \emph default , to cleanup after the application is done using the library. \layout Subsection PtlInit \begin_inset LatexCommand \label{sec:ptlinit} \end_inset \layout LyX-Code int PtlInit( int *max_interfaces ); \layout Standard \noindent The \emph on PtlInit \emph default function initializes the Portals library. PtlInit must be called at least once by a process before any thread makes a Portals function call, but may be safely called more than once. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_FAIL Indicates an error during initialization. \layout Description PTL_SEGV Indicates that \family typewriter max_interfaces \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter max_interfaces \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold the maximum number of interfaces that can be initialized. \end_inset \end_inset \layout Subsection PtlFini \begin_inset LatexCommand \label{sec:ptlfini} \end_inset \layout LyX-Code void PtlFini( void ); \layout Standard \noindent The \emph on PtlFini \emph default function cleans up after the Portals library is no longer needed by a process. After this function is called, calls to any of the functions defined by the Portal API or use of the structures set up by the Portals API will result in undefined behavior. This function should be called once and only once during termination by a process. Typically, this function will be called in the exit sequence of a process. Individual threads should not call PtlFini when they terminate. \layout Section Network Interfaces \begin_inset LatexCommand \label{sec:ni} \end_inset \layout Standard The Portals API supports the use of multiple network interfaces. However, each interface is treated as an independent entity. Combining interfaces (e.g., \begin_inset Quotes eld \end_inset bonding \begin_inset Quotes erd \end_inset to create a higher bandwidth connection) must be implemented by the application or embedded in the underlying network. Interfaces are treated as independent entities to make it easier to cache information on individual network interface cards. \layout Standard Once initialized, each interface provides a Portal table, an access control table, and a collection of status registers. See Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:me} \end_inset for a discussion of updating Portal table entries using the \emph on PtlMEAttach \emph default function. See Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:ac} \end_inset for a discussion of the initialization and updating of entries in the access control table. See Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:nistatus} \end_inset for a discussion of the \emph on PtlNIStatus \emph default function which can be used to determine the value of a status register. \layout Standard Every other type of Portal object (e.g., memory descriptor, event queue, or match list entry) is associated with a specific network interface. The association to a network interface is established when the object is created and is encoded in the handle for the object. \layout Standard Each network interface is initialized and shutdown independently. The initialization routine, \emph on PtlNIInit \emph default , returns a handle for an interface object which is used in all subsequent Portal operations. The \emph on PtlNIFini \emph default function is used to shutdown an interface and release any resources that are associated with the interface. Network interface handles are associated with processes, not threads. All threads in a process share all of the network interface handles. \layout Standard The Portals API also defines the \emph on PtlNIStatus \emph default function to query the status registers for a network interface, the \emph on PtlNIDist \emph default function to determine the \begin_inset Quotes eld \end_inset distance \begin_inset Quotes erd \end_inset to another process, and the \emph on PtlNIHandle \emph default function to determine the network interface that an object is associated with. \layout Subsection PtlNIInit \begin_inset LatexCommand \label{sec:niinit} \end_inset \layout LyX-Code typedef struct { \newline int max_match_entries; \newline int max_mem_descriptors; \newline int max_event_queues; \newline ptl_ac_index_t max_atable_index; \newline ptl_pt_index_t max_ptable_index; \newline } ptl_ni_limits_t; \newline \newline int PtlNIInit( ptl_interface_t interface \newline ptl_pid_t pid, \newline ptl_ni_limits_t* desired, \newline ptl_ni_limits_t* actual, \newline ptl_handle_ni_t* handle ); \layout Standard Values of type \family typewriter ptl_ni_limits_t \family default include the following members: \layout Description max_match_entries Maximum number of match entries that can be allocated at any one time. \layout Description max_mem_descriptors Maximum number of memory descriptors that can be allocated at any one time. \layout Description max_event_queues Maximum number of event queues that can be allocated at any one time. \layout Description max_atable_index Largest access control table index for this interface, valid indexes range from zero to \family typewriter max_atable_index \family default , inclusive. \layout Description max_ptable_index Largest Portal table index for this interface, valid indexes range from zero to \family typewriter max_ptable_index \family default , inclusive. \layout Standard \noindent The \emph on PtlNIInit \emph default function is used to initialized the Portals API for a network interface. This function must be called at least once by each process before any other operations that apply to the interface by any process or thread. For subsequent calls to \shape italic PtlNIInit \shape default from within the same process (either by different threads or the same thread), the desired limits will be ignored and the call will return the existing NI handle. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INIT_DUP Indicates a duplicate initialization of \family typewriter interface \family default . \layout Description PTL_INIT_INV Indicates that \family typewriter interface \family default is not a valid network interface. \layout Description PTL_NOSPACE Indicates that there is insufficient memory to initialize the interface. \layout Description PTL_INV_PROC Indicates that \family typewriter pid \family default is not a valid process id. \layout Description PTL_SEGV Indicates that \family typewriter actual \family default or \family typewriter handle \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter interface \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Identifies the network interface to be initialized. (See section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:ni-type} \end_inset for a discussion of values used to identify network interfaces.) \end_inset \begin_inset Text \layout Standard \family typewriter pid \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Identifies the desired process id (for well known process ids). The value \family typewriter PTL_PID_ANY \family default may be used to have the process id assigned by the underlying library. \end_inset \begin_inset Text \layout Standard \family typewriter desired \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent If non-NULL, points to a structure that holds the desired limits. \end_inset \begin_inset Text \layout Standard \family typewriter actual \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, the location pointed to by actual will hold the actual limits. \end_inset \begin_inset Text \layout Standard \family typewriter handle \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold a handle for the interface. \end_inset \end_inset \layout Comment The use of desired is implementation dependent. In particular, an implementation may choose to ignore this argument. \layout Subsection PtlNIFini \begin_inset LatexCommand \label{sec:nifini} \end_inset \layout LyX-Code int PtlNIFini( ptl_handle_ni_t interface ); \layout Standard \noindent The \emph on PtlNIFini \emph default function is used to release the resources allocated for a network interface. Once the \emph on PtlNIFini \emph default operation has been started, the results of pending API operations (e.g., operations initiated by another thread) for this interface are undefined. Similarly, the effects of incoming operations (puts and gets) or return values (acknowledgements and replies) for this interface are undefined. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_NI Indicates that \family typewriter interface \family default is not a valid network interface handle. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard interface \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard A handle for the interface to shutdown. \end_inset \end_inset \layout Subsection PtlNIStatus \begin_inset LatexCommand \label{sec:nistatus} \end_inset \layout LyX-Code int PtlNIStatus( ptl_handle_ni_t interface, \newline ptl_sr_index_t status_register, \newline ptl_sr_value_t* status ); \layout Standard \noindent The \emph on PtlNIStatus \emph default function returns the value of a status register for the specified interface. (See section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:stat-type} \end_inset for more information on status register indexes and status register values.) \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_NI Indicates that \family typewriter interface \family default is not a valid network interface handle. \layout Description PTL_INV_SR_INDX Indicates that \family typewriter status_register \family default is not a valid status register. \layout Description PTL_SEGV Indicates that \family typewriter status \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter interface \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the interface to use. \end_inset \begin_inset Text \layout Standard \family typewriter status_register \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent An index for the status register to read. \end_inset \begin_inset Text \layout Standard \family typewriter status \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold the current value of the status register. \end_inset \end_inset \layout Comment The only status register that must be defined is a drop count register ( \family typewriter PTL_SR_DROP_COUNT \family default ). Implementations may define additional status registers. Identifiers for the indexes associated with these registers should start with the prefix \family typewriter PTL_SR_ \family default . \layout Subsection PtlNIDist \layout LyX-Code int PtlNIDist( ptl_handle_ni_t interface, \newline ptl_process_id_t process, \newline unsigned long* distance ); \layout Standard \noindent The \emph on PtlNIDist \emph default function returns the distance to another process using the specified interface. Distances are only defined relative to an interface. Distance comparisons between different interfaces on the same process may be meaningless. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_NI Indicates that \family typewriter interface \family default is not a valid network interface handle. \layout Description PTL_INV_PROC Indicates that \family typewriter process \family default is not a valid process identifier. \layout Description PTL_SEGV Indicates that \family typewriter distance \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter interface \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the interface to use. \end_inset \begin_inset Text \layout Standard \family typewriter process \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent An identifier for the process whose distance is being requested. \end_inset \begin_inset Text \layout Standard \family typewriter distance \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold the distance to the remote process. \end_inset \end_inset \layout Comment This function should return a static measure of distance. Examples include minimum latency, the inverse of available bandwidth, or the number of switches between the two endpoints. \layout Subsection PtlNIHandle \layout LyX-Code int PtlNIHandle( ptl_handle_any_t handle, \newline ptl_handle_ni_t* interface ); \layout Standard \noindent The \emph on PtlNIHandle \emph default function returns a handle for the network interface with which the object identified by \family typewriter handle \family default is associated. If the object identified by \family typewriter handle \family default is a network interface, this function returns the same value it is passed. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_HANDLE Indicates that \family typewriter handle \family default is not a valid handle. \layout Description PTL_SEGV Indicates that \family typewriter interface \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter handle \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the object. \end_inset \begin_inset Text \layout Standard \family typewriter interface \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold a handle for the network interface associated with \family typewriter handle \family default . \end_inset \end_inset \layout Comment Every handle should encode the network interface and the object id relative to this handle. Both are presumably encoded using integer values. \layout Section User Identification \begin_inset LatexCommand \label{sec:uid} \end_inset \layout Standard Every process runs on behalf of a user. \layout Subsection PtlGetUid \layout LyX-Code int PtlGetUid( ptl_handle_ni_t ni_handle, \newline ptl_uid_t* uid ); \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_INV_NI Indicates that \family typewriter ni_handle \family default is not a valid network interface handle. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_SEGV Indicates that \family typewriter interface \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter handle \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A network interface handle. \end_inset \begin_inset Text \layout Standard \family typewriter id \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold the user id for the calling process. \end_inset \end_inset \layout Comment Note that user identifiers are dependent on the network interface(s). In particular, if a node has multiple interfaces, a process may have multiple user identifiers. \layout Section Process Identification \begin_inset LatexCommand \label{sec:pid} \end_inset \layout Standard Processes that use the Portals API, can be identified using a node id and process id. Every node accessible through a network interface has a unique node identifier and every process running on a node has a unique process identifier. As such, any process in the computing system can be identified by its node id and process id. \layout Standard The Portals API defines a type, \family typewriter ptl_process_id_t \family default for representing process ids and a function, \emph on PtlGetId \emph default , which can be used to obtain the id of the current process. \layout Comment The portals API does not include thread identifiers. Messages are delivered to processes (address spaces) not threads (contexts of execution). \layout Subsection The Process Id Type \begin_inset LatexCommand \label{sec:pid-type} \end_inset \layout LyX-Code typedef struct { \newline ptl_nid_t nid; /* node id */ \newline ptl_pid_t pid; /* process id */ \newline } ptl_process_id_t; \layout Standard \noindent The \family typewriter ptl_process_id_t \family default type uses two identifiers to represent a process id: a node id and a process id. \layout Subsection PtlGetId \begin_inset LatexCommand \label{sub:PtlGetId} \end_inset \layout LyX-Code int PtlGetId( ptl_handle_ni_t ni_handle, \newline ptl_process_id_t* id ); \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_INV_NI Indicates that \family typewriter ni_handle \family default is not a valid network interface handle. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_SEGV Indicates that \family typewriter id \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter handle \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A network interface handle. \end_inset \begin_inset Text \layout Standard \family typewriter id \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold the id for the calling process. \end_inset \end_inset \layout Comment Note that process identifiers are dependent on the network interface(s). In particular, if a node has multiple interfaces, it may have multiple node identifiers. \layout Section Match List Entries and Match Lists \begin_inset LatexCommand \label{sec:me} \end_inset \layout Standard A match list is a chain of match list entries. Each match list entry includes a memory descriptor and a set of match criteria. The match criteria can be used to reject incoming requests based on process id or the match bits provided in the request. A match list is created using the \emph on PtlMEAttach \emph default or \shape italic PtlMEAttachAny \shape default functions, which create a match list consisting of a single match list entry, attaches the match list to the specified Portal index, and returns a handle for the match list entry. Match entries can be dynamically inserted and removed from a match list using the \emph on PtlMEInsert \emph default and \emph on PtlMEUnlink \emph default functions. \layout Subsection PtlMEAttach \begin_inset LatexCommand \label{sec:meattach} \end_inset \layout LyX-Code typedef enum { PTL_RETAIN, PTL_UNLINK } ptl_unlink_t; \newline \layout LyX-Code typedef enum { PTL_INS_BEFORE, PTL_INS_AFTER } ptl_ins_pos_t; \newline \layout LyX-Code int PtlMEAttach( ptl_handle_ni_t interface, \newline ptl_pt_index_t index, \newline ptl_process_id_t matchid, \newline ptl_match_bits_t match_bits, \newline ptl_match_bits_t ignorebits, \newline ptl_unlink_t unlink, \newline ptl_ins_pos_t position, \newline ptl_handle_me_t* handle ); \layout Standard \noindent Values of the type \family typewriter ptl_ins_pos_t \family default are used to control where a new item is inserted. The value \family typewriter PTL_INS_BEFORE \family default is used to insert the new item before the current item or before the head of the list. The value \family typewriter PTL_INS_AFTER \family default is used to insert the new item after the current item or after the last item in the list. \layout Standard The \emph on PtlMEAttach \emph default function creates a match list consisting of a single entry and attaches this list to the Portal table for \family typewriter interface \family default . \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_INV_NI Indicates that \family typewriter interface \family default is not a valid network interface handle. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_PTINDEX Indicates that \family typewriter index \family default is not a valid Portal table index. \layout Description PTL_INV_PROC Indicates that \family typewriter matchid \family default is not a valid process identifier. \layout Description PTL_NOSPACE Indicates that there is insufficient memory to allocate the match list entry. \layout Description PTL_ML_TOOLONG Indicates that the resulting match list is too long. The maximum length for a match list is defined by the interface. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \noindent \family typewriter interface \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the interface to use. \end_inset \begin_inset Text \layout Standard \noindent \family typewriter index \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The Portal table index where the match list should be attached. \end_inset \begin_inset Text \layout Standard \noindent \family typewriter matchid \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Specifies the match criteria for the process id of the requestor. The constants \family typewriter PTL_PID_ANY \family default and \family typewriter PTL_NID_ANY \family default can be used to wildcard either of the ids in the \family typewriter ptl_process_id_t \family default structure. \end_inset \begin_inset Text \layout Standard \noindent \family typewriter match_bits, ignorebits \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Specify the match criteria to apply to the match bits in the incoming request. The \family typewriter ignorebits \family default are used to mask out insignificant bits in the incoming match bits. The resulting bits are then compared to the match list entry's match bits to determine if the incoming request meets the match criteria. \end_inset \begin_inset Text \layout Standard \noindent \family typewriter unlink \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Indicates the match list entry should be unlinked when the last memory descripto r associated with this match list entry is unlinked. (Note, the check for unlinking a match entry only occurs when a memory descriptor is unlinked.) \end_inset \begin_inset Text \layout Standard \noindent \family typewriter position \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Indicates whether the new match entry should be prepended or appended to the existing match list. If there is no existing list, this argument is ignored and the new match entry becomes the only entry in the list. Allowed constants: \family typewriter PTL_INS_BEFORE \family default , \family typewriter PTL_INS_AFTER \family default . \end_inset \begin_inset Text \layout Standard \noindent \family typewriter handle \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold a handle for the newly created match list entry. \end_inset \end_inset \layout Subsection PtlMEAttachAny \begin_inset LatexCommand \label{sec:attachany} \end_inset \layout LyX-Code int PtlMEAttachAny( ptl_handle_ni_t interface, \newline ptl_pt_index_t *index, \newline ptl_process_id_t matchid, \newline ptl_match_bits_t match_bits, \newline ptl_match_bits_t ignorebits, \newline ptl_unlink_t unlink, \newline ptl_handle_me_t* handle ); \layout Standard The \emph on PtlMEAttachAny \emph default function creates a match list consisting of a single entry and attaches this list to an unused Portal table entry for \family typewriter interface \family default . \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_INV_NI Indicates that \family typewriter interface \family default is not a valid network interface handle. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_PROC Indicates that \family typewriter matchid \family default is not a valid process identifier. \layout Description PTL_NOSPACE Indicates that there is insufficient memory to allocate the match list entry. \layout Description PTL_PT_FULL Indicates that there are no free entries in the Portal table. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \noindent \family typewriter interface \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the interface to use. \end_inset \begin_inset Text \layout Standard \noindent \family typewriter index \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold the Portal index where the match list has been attached. \end_inset \begin_inset Text \layout Standard \noindent \family typewriter matchid, match_bits, ignorebits, unlink \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent See the discussion for \shape italic PtlMEAttach \shape default . \end_inset \begin_inset Text \layout Standard \noindent \family typewriter handle \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold a handle for the newly created match list entry. \end_inset \end_inset \layout Subsection PtlMEInsert \begin_inset LatexCommand \label{sec:meinsert} \end_inset \layout LyX-Code int PtlMEInsert( ptl_handle_me_t current, \newline ptl_process_id_t matchid, \newline ptl_match_bits_t match_bits, \newline ptl_match_bits_t ignorebits, \newline ptl_ins_pos_t position, \newline ptl_handle_me_t* handle ); \layout Standard The \emph on PtlMEInsert \emph default function creates a new match list entry and inserts this entry into the match list containing \family typewriter current \family default . \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_PROC Indicates that \family typewriter matchid \family default is not a valid process identifier. \layout Description PTL_INV_ME Indicates that \family typewriter current \family default is not a valid match entry handle. \layout Description PTL_ML_TOOLONG Indicates that the resulting match list is too long. The maximum length for a match list is defined by the interface. \layout Description PTL_NOSPACE Indicates that there is insufficient memory to allocate the match entry. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \noindent \family typewriter current \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for a match entry. The new match entry will be inserted immediately before or immediately after this match entry. \end_inset \begin_inset Text \layout Standard \noindent \family typewriter matchid \family default , \family typewriter match_bits \family default , \family typewriter ignorebits \family default , \family typewriter unlink \family default \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent See the discussion for \emph on PtlMEAttach \emph default \end_inset \begin_inset Text \layout Standard \noindent \family typewriter position \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Indicates whether the new match entry should be inserted before or after the \family typewriter current \family default entry. Allowed constants: \family typewriter PTL_INS_BEFORE \family default , \family typewriter PTL_INS_AFTER \family default . \end_inset \begin_inset Text \layout Standard \noindent \family typewriter handle \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent See the discussion for \emph on PtlMEAttach \emph default . \end_inset \end_inset \layout Subsection PtlMEUnlink \begin_inset LatexCommand \label{sec:meunlink} \end_inset \layout LyX-Code int PtlMEUnlink( ptl_handle_me_t entry ); \layout Standard \noindent The \emph on PtlMEUnlink \emph default function can be used to unlink a match entry from a match list. This operation also releases any resources associated with the match entry (including the associated memory descriptor). It is an error to use the match entry handle after calling \emph on PtlMEUnlink \emph default . \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_ME Indicates that \family typewriter entry \family default is not a valid match entry handle. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter entry \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard A handle for the match entry to be unlinked. \end_inset \end_inset \layout Section Memory Descriptors \begin_inset LatexCommand \label{sec:md} \end_inset \layout Standard A memory descriptor contains information about a region of an application process' memory and an event queue where information about the operations performed on the memory descriptor are recorded. The Portals API provides two operations to create memory descriptors: \emph on PtlMDAttach \emph default , and \emph on PtlMDBind \emph default ; an operation to update a memory descriptor, \emph on PtlMDUpdate \emph default ; and an operation to unlink and release the resources associated with a memory descriptor, \emph on PtlMDUnlink \emph default . \layout Subsection The Memory Descriptor Type \begin_inset LatexCommand \label{sec:md-type} \end_inset \layout LyX-Code typedef struct { \newline void* start; \newline ptl_size_t length; \newline int threshold; \newline unsigned int max_offset; \newline unsigned int options; \newline void* user_ptr; \newline ptl_handle_eq_t eventq; \newline } ptl_md_t; \layout Standard \noindent The \family typewriter ptl_md_t \family default type defines the application view of a memory descriptor. Values of this type are used to initialize and update the memory descriptors. \layout Subsubsection Members \layout Description start,\SpecialChar ~ length Specify the memory region associated with the memory descriptor. The \family typewriter start \family default member specifies the starting address for the memory region and the \family typewriter length \family default member specifies the length of the region. The \family typewriter start member \family default can be NULL provided that the \family typewriter length \family default member is zero. (Zero length buffers are useful to record events.) There are no alignment restrictions on the starting address or the length of the region; although, unaligned messages may be slower (i.e., lower bandwidth and/or longer latency) on some implementations. \layout Description threshold Specifies the maximum number of operations that can be performed on the memory descriptor. An operation is any action that could possibly generate an event (see Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:ek-type} \end_inset for the different types of events). In the usual case, the threshold value is decremented for each operation on the memory descriptor. When the threshold value is zero, the memory descriptor is \emph on inactive \emph default , and does not respond to operations. A memory descriptor can have an initial threshold value of zero to allow for manipulation of an inactive memory descriptor by the local process. A threshold value of \family typewriter PTL_MD_THRESH_INF \family default indicates that there is no bound on the number of operations that may be applied to a memory descriptor. Note that local operations (e.g., \emph on PtlMDUpdate \emph default ) are not applied to the threshold count. \layout Description max_offset Specifies the maximum local offset of a memory descriptor. When the local offset of a memory descriptor exceeds this maximum, the memory descriptor becomes \shape italic inactive \shape default and does not respond to further operations. \layout Description options Specifies the behavior of the memory descriptor. There are five options that can be selected: enable put operations (yes or no), enable get operations (yes or no), offset management (local or remote), message truncation (yes or no), and acknowledgement (yes or no). Values for this argument can be constructed using a bitwise or of the following values: \begin_deeper \begin_deeper \layout Description PTL_MD_OP_PUT Specifies that the memory descriptor will respond to \emph on put \emph default operations. By default, memory descriptors reject \emph on put \emph default operations. \layout Description PTL_MD_OP_GET Specifies that the memory descriptor will respond to \emph on get \emph default operations. By default, memory descriptors reject \emph on get \emph default operations. \layout Description PTL_MD_MANAGE_REMOTE Specifies that the offset used in accessing the memory region is provided by the incoming request. By default, the offset is maintained locally. When the offset is maintained locally, the offset is incremented by the length of the request so that the next operation (put and/or get) will access the next part of the memory region. \layout Description PTL_MD_TRUNCATE Specifies that the length provided in the incoming request can be reduced to match the memory available in the region. (The memory available in a memory region is determined by subtracting the offset from the length of the memory region.) By default, if the length in the incoming operation is greater than the amount of memory available, the operation is rejected. \layout Description PTL_MD_ACK_DISABLE Specifies that an acknowledgement should \emph on not \emph default be sent for incoming \emph on put \emph default operations, even if requested. By default, acknowledgements are sent for \emph on put \emph default operations that request an acknowledgement. Acknowledgements are never sent for \emph on get \emph default operations. The value sent in the reply serves as an implicit acknowledgement. \end_deeper \layout Standard \series bold Note \series default : It is not considered an error to have a memory descriptor that does not respond to either \emph on put \emph default or \emph on get \emph default operations: Every memory descriptor responds to \emph on reply \emph default operations. Nor is it considered an error to have a memory descriptor that responds to both \emph on put \emph default and \emph on get \emph default operations. \end_deeper \layout Description user_ptr A user-specified value that is associated with the memory descriptor. The value does not need to be a pointer, but must fit in the space used by a pointer. This value (along with other values) is recorded in events associated with operations on this memory descriptor. \begin_inset Foot collapsed true \layout Standard Tying the memory descriptor to a user-defined value can be useful when multiple memory descriptor share the same event queue or when the memory descriptor needs to be associated with a data structure maintained by the application. For example, an MPI implementation can set the \family typewriter user_ptr \family default argument to the value of an MPI Request. This direct association allows for processing of memory descriptor's by the MPI implementation without a table lookup or a search for the appropriate MPI Request. \end_inset \layout Description eventq A handle for the event queue used to log the operations performed on the memory region. If this argument is \family typewriter PTl_EQ_NONE \family default , operations performed on this memory descriptor are not logged. \layout Subsection PtlMDAttach \begin_inset LatexCommand \label{sec:mdattach} \end_inset \layout LyX-Code int PtlMDAttach( ptl_handle_me_t match, \newline ptl_md_t mem_desc, \newline ptl_unlink_t unlink_op, \newline ptl_unlink_t unlink_nofit, \newline ptl_handle_md_t* handle ); \layout Standard \noindent Values of the type \family typewriter ptl_unlink_t \family default are used to control whether an item is unlinked from a list. The value \family typewriter PTL_UNLINK \family default enables unlinking. The value \family typewriter PTL_RETAIN \family default disables unlinking. \layout Standard The \emph on PtlMDAttach \emph default operation is used to create a memory descriptor and attach it to a match list entry. An error code is returned if this match list entry already has an associated memory descriptor. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INUSE Indicates that \family typewriter match \family default already has a memory descriptor attached. \layout Description PTL_INV_ME Indicates that \family typewriter match \family default is not a valid match entry handle. \layout Description PTL_ILL_MD Indicates that \family typewriter mem_desc \family default is not a legal memory descriptor. This may happen because the memory region defined in \family typewriter mem_desc \family default is invalid or because the network interface associated with the \family typewriter eventq \family default in \family typewriter mem_desc \family default is not the same as the network interface associated with \family typewriter match \family default . \layout Description PTL_NOSPACE Indicates that there is insufficient memory to allocate the memory descriptor. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter match \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the match entry that the memory descriptor will be associated with. \end_inset \begin_inset Text \layout Standard \family typewriter mem_desc \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Provides initial values for the application visible parts of a memory descriptor. Other than its use for initialization, there is no linkage between this structure and the memory descriptor maintained by the API. \end_inset \begin_inset Text \layout Standard \family typewriter unlink_op \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A flag to indicate whether the memory descriptor is unlinked when it becomes inactive, either because the operation threshold drops to zero or because the maximum offset has been exceeded. (Note, the check for unlinking a memory descriptor only occurs after a the completion of a successful operation. If the threshold is set to zero during initialization or using \emph on PtlMDUpdate \emph default , the memory descriptor is \series bold not \series default unlinked.) \end_inset \begin_inset Text \layout Standard \family typewriter unlink_nofit \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A flag to indicate whether the memory descriptor is unlinked when the space remaining in the memory descriptor is not sufficient for a matching operation. If an incoming message arrives arrives at a memory descriptor that does not have sufficient space and the \series bold PTL_MD_TRUNCATE \series default operation is not specified, the memory descriptor will be unlinked. \end_inset \begin_inset Text \layout Standard \family typewriter handle \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold a handle for the newly created memory descriptor. The \family typewriter handle \family default argument can be NULL, in which case the handle will not be returned. \end_inset \end_inset \layout Subsection PtlMDBind \begin_inset LatexCommand \label{sec:mdbind} \end_inset \layout LyX-Code int PtlMDBind( ptl_handle_ni_t interface, \newline ptl_md_t mem_desc, \newline ptl_handle_md_t* handle ); \layout Standard \noindent The \emph on PtlMDBind \emph default operation is used to create a \begin_inset Quotes eld \end_inset free floating \begin_inset Quotes erd \end_inset memory descriptor, i.e., a memory descriptor that is not associated with a match list entry. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_NI Indicates that \family typewriter interface \family default is not a valid match entry handle. \layout Description PTL_ILL_MD Indicates that \family typewriter mem_desc \family default is not a legal memory descriptor. This may happen because the memory region defined in \family typewriter mem_desc \family default is invalid or because the network interface associated with the \family typewriter eventq \family default in \family typewriter mem_desc \family default is not the same as the network interface, \family typewriter interface \family default . \layout Description PTL_INV_EQ Indicates that the event queue associated with \family typewriter mem_desc \family default is not valid. \layout Description PTL_NOSPACE Indicates that there is insufficient memory to allocate the memory descriptor. \layout Description PTL_SEGV Indicates that \family typewriter handle \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter interface \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the network interface with which the memory descriptor will be associated. \end_inset \begin_inset Text \layout Standard \family typewriter mem_desc \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Provides initial values for the application visible parts of a memory descriptor. Other than its use for initialization, there is no linkage between this structure and the memory descriptor maintained by the API. \end_inset \begin_inset Text \layout Standard \family typewriter handle \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold a handle for the newly created memory descriptor. The \family typewriter handle \family default argument must be a valid address and cannot be NULL. \end_inset \end_inset \layout Subsection PtlMDUnlink \begin_inset LatexCommand \label{sec:mdfree} \end_inset \layout LyX-Code int PtlMDUnlink( ptl_handle_md_t mem_desc ); \layout Standard \noindent The \emph on PtlMDUnlink \emph default function unlinks the memory descriptor from any match list entry it may be linked to and releases the resources associated with a memory descriptor. (This function does not free the memory region associated with the memory descriptor.) This function also releases the resources associated with a floating memory descriptor. Only memory descriptors with no pending operations may be unlinked. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_MD Indicates that \family typewriter mem_desc \family default is not a valid memory descriptor handle. \layout Description PTL_MD_INUSE Indicates that \family typewriter mem_desc \family default has pending operations and cannot be unlinked. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter mem_desc \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the memory descriptor to be released. \end_inset \end_inset \layout Subsection PtlMDUpdate \begin_inset LatexCommand \label{sec:mdupdate} \end_inset \layout LyX-Code int PtlMDUpdate( ptl_handle_md_t mem_desc, \newline ptl_md_t* old_md, \newline ptl_md_t* new_md, \newline ptl_handle_eq_t testq ); \layout Standard \noindent The \emph on PtlMDUpdate \emph default function provides a conditional, atomic update operation for memory descriptors. The memory descriptor identified by \family typewriter mem_desc \family default is only updated if the event queue identified by \family typewriter testq \family default is empty. The intent is to only enable updates to the memory descriptor when no new messages have arrived since the last time the queue was checked. See section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:exmpi} \end_inset for an example of how this function can be used. \layout Standard If \family typewriter new \family default is not NULL the memory descriptor identified by handle will be updated to reflect the values in the structure pointed to by \family typewriter new \family default if \family typewriter testq \family default has the value \family typewriter PTL_EQ_NONE \family default or if the event queue identified by \family typewriter testq \family default is empty. If \family typewriter old \family default is not NULL, the current value of the memory descriptor identified by \family typewriter mem_desc \family default is recorded in the location identified by \family typewriter old \family default . \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_NOUPDATE Indicates that the update was not performed because \family typewriter testq \family default was not empty. \layout Description PTL_INV_MD Indicates that \family typewriter mem_desc \family default is not a valid memory descriptor handle. \layout Description PTL_ILL_MD Indicates that the value pointed to by \family typewriter new \family default is not a legal memory descriptor (e.g., the memory region specified by the memory descriptor may be invalid). \layout Description PTL_INV_EQ Indicates that \family typewriter testq \family default is not a valid event queue handle. \layout Description PTL_SEGV Indicates that \family typewriter new \family default or \family typewriter old \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter mem_desc \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the memory descriptor to update. \end_inset \begin_inset Text \layout Standard \family typewriter old_md \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent If \family typewriter old_md \family default is not the value \family typewriter NULL \family default , the current value of the memory descriptor will be stored in the location identified by \family typewriter old \family default _md. \end_inset \begin_inset Text \layout Standard \family typewriter new_md \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent If \family typewriter new_md \family default is not the value \family typewriter NULL \family default , this argument provides the new values for the memory descriptor, if the update is performed. \end_inset \begin_inset Text \layout Standard \family typewriter testq \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for an event queue used to predicate the update. If \family typewriter testq \family default is equal to \family typewriter PTL_EQ_NONE \family default , the update is performed unconditionally. Otherwise, the update is performed if and only if \family typewriter testq \family default is empty. If the update is not performed, the function returns the value \family typewriter PTL_NOUPDATE \family default . (Note, the \family typewriter testq \family default argument does not need to be the same as the event queue associated with the memory descriptor.) \end_inset \end_inset \layout Standard The conditional update can be used to ensure that the memory descriptor has not changed between the time it was examined and the time it is updated. In particular, it is needed to support an MPI implementation where the activity of searching an unexpected message queue and posting a receive must be atomic. \layout Section Events and Event Queues \begin_inset LatexCommand \label{sec:eq} \end_inset \layout Standard Event queues are used to log operations performed on memory descriptors. They can also be used to hold acknowledgements for completed \emph on put \emph default operations and to note when the data specified in a \emph on put \emph default operation has been sent (i.e., when it is safe to reuse the buffer that holds this data). Multiple memory descriptors can share a single event queue. \layout Standard In addition to the \family typewriter ptl_handle_eq_t \family default type, the Portals API defines two types associated with events: The \family typewriter \newline ptl_event_kind_t \family default type defines the kinds of events that can be stored in an event queue. The \family typewriter ptl_event_t \family default type defines a structure that holds the information associated with an event. \layout Standard The Portals API also provides four functions for dealing with event queues: The \emph on PtlEQAlloc \emph default function is used to allocate the API resources needed for an event queue, the \emph on PtlEQFree \emph default function is used to release these resources, the \emph on PtlEQGet \emph default function can be used to get the next event from an event queue, and the \emph on PtlEQWait \emph default function can be used to block a process (or thread) until an event queue has at least one event. \layout Subsection Kinds of Events \begin_inset LatexCommand \label{sec:ek-type} \end_inset \layout LyX-Code typedef enum { \newline PTL_EVENT_GET_START, PTL_EVENT_GET_END, PTL_EVENT_GET_FAIL, \newline PTL_EVENT_PUT_START, PTL_EVENT_PUT_END, PTL_EVENT_PUT_FAIL, \newline PTL_EVENT_REPLY_START, PTL_EVENT_REPLY_END, PTL_EVENT_REPLY_FAIL, \newline PTL_EVENT_SEND_START, PTL_EVENT_SEND_END, PTL_EVENT_SEND_FAIL, \newline PTL_EVENT_ACK, \newline PTL_EVENT_UNLINK \newline } ptl_event_kind_t; \layout Standard \noindent The Portals API defines fourteen types of events that can be logged in an event queue: \layout Description PTL_EVENT_GET_START A remote \emph on get \emph default operation has been started on the memory descriptor. The memory region associated with this descriptor should not be altered until the corresponding END or FAIL event is logged. \layout Description PTL_EVENT_GET_END A previously initiated \emph on get \emph default operation completed successfully. This event is logged after the reply has been sent by the local node. As such, the process could free the memory descriptor once it sees this event. \layout Description PTL_EVENT_GET_FAIL A previously initiated \emph on get \emph default operation completed unsuccessfully. This event is logged after the reply has been sent by the local node. As such, the process could free the memory descriptor once it sees this event. \layout Description PTL_EVENT_PUT_START A remote \emph on put \emph default operation has been started on the memory descriptor. The memory region associated with this descriptor should should be considered volatile until the corresponding END or FAIL event is logged. \layout Description PTL_EVENT_PUT_END A previously initiated \emph on put \emph default operation completed successfully. The underlying layers will not alter the memory (on behalf of this operation) once this event has been logged. \layout Description PTL_EVENT_PUT_FAIL A previously initiated \emph on put \emph default operation completed unsuccessfully. The underlying layers will not alter the memory (on behalf of this operation) once this event has been logged. \layout Description PTL_EVENT_REPLY_START A \emph on reply \emph default operation has been started on the memory descriptor. \layout Description PTL_EVENT_REPLY_END A previously initiated \emph on reply \emph default operation has completed successfully . This event is logged after the data (if any) from the reply has been written into the memory descriptor. \layout Description PTL_EVENT_REPLY_FAIL A previously initiated \emph on reply \emph default operation has completed unsuccessfully. This event is logged after the data (if any) from the reply has been written into the memory descriptor. \layout Description PTL_EVENT_ACK An \emph on acknowledgement \emph default was received. This event is logged when the acknowledgement is received \layout Description PTL_EVENT_SEND_START An outgoing \emph on send \emph default operation has been started. The memory region associated with this descriptor should not be altered until the corresponding END or FAIL event is logged. \layout Description PTL_EVENT_SEND_END A previously initiated \emph on send \emph default operation has completed successfully. This event is logged after the entire buffer has been sent and it is safe for the application to reuse the buffer. \layout Description PTL_EVENT_SEND_FAIL A previously initiated \emph on send \emph default operation has completed unsuccessfully. The process can safely manipulate the memory or free the memory descriptor once it sees this event. \layout Description PTL_EVENT_UNLINK A memory descriptor associated with this event queue has been automatically unlinked. This event is not generated when a memory descriptor is explicitly unlinked by calling \shape italic PtlMDUnlink \shape default . This event does not decrement the threshold count. \layout Subsection Event Ordering \layout Standard The Portals API guarantees that a when a process initiates two operations on a remote process, the operations will be initiated on the remote process in the same order that they were initiated on the original process. As an example, if process A intitates two \emph on put \emph default operations, \emph on x \emph default and \emph on y \emph default , on process B, the Portals API guarantees that process A will receive the \family typewriter PTL_EVENT_SEND_START \family default events for \emph on x \emph default and \emph on y \emph default in the same order that process B receives the \family typewriter PTL_EVENT_PUT_START \family default events for \emph on x \emph default and \emph on y \emph default . Notice that the API does not guarantee that the start events will be delivered in the same order that process A initiated the \emph on x \emph default and \emph on y \emph default operations. If process A needs to ensure the ordering of these operations, it should include code to wait for the initiation of \emph on x \emph default before it initiates \emph on y \emph default . \layout Subsection Failure Notification \layout Standard Operations may fail to complete successfully; however, unless the node itself fails, every operation that is started will eventually complete. While an operation is in progress, the memory associated with the operation should not be viewed (in the case of a put or a reply) or altered (in the case of a send or get). Operation completion, whether successful or unsuccessful, is final. That is, when an operation completes, the memory associated with the operation will no longer be read or altered by the operation. A network interface can use the \family typewriter ptl_ni_fail_t \family default to define more specific information regarding the failure of the operation and record this information in the \family typewriter ni_fail_type \family default field of the event. \layout Subsection The Event Type \begin_inset LatexCommand \label{sec:event-type} \end_inset \layout LyX-Code typedef struct { \newline ptl_event_kind_t type; \newline ptl_process_id_t initiator; \newline ptl_uid_t uid; \layout LyX-Code ptl_pt_index_t portal; \newline ptl_match_bits_t match_bits; \newline ptl_size_t rlength; \newline ptl_size_t mlength; \newline ptl_size_t offset; \newline ptl_handle_md_t md_handle; \newline ptl_md_t mem_desc; \newline ptl_hdr_data_t hdr_data; \newline ptl_seq_t link; \newline ptl_ni_fail_t ni_fail_type; \newline volatile ptl_seq_t sequence; \newline } ptl_event_t; \layout Standard \noindent An event structure includes the following members: \layout Description type Indicates the type of the event. \layout Description initiator The id of the initiator. \layout Description portal The Portal table index specified in the request. \layout Description match_bits A copy of the match bits specified in the request. See section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:me} \end_inset for more information on match bits. \layout Description rlength The length (in bytes) specified in the request. \layout Description mlength The length (in bytes) of the data that was manipulated by the operation. For truncated operations, the manipulated length will be the number of bytes specified by the memory descriptor (possibly with an offset) operation. For all other operations, the manipulated length will be the length of the requested operation. \layout Description offset Is the displacement (in bytes) into the memory region that the operation used. The offset can be determined by the operation (see Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:datamovement} \end_inset ) for a remote managed memory descriptor, or by the local memory descriptor (see Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:md} \end_inset ). \layout Description md_handle Is the handle to the memory descriptor associated with the event. \layout Description mem_desc Is the state of the memory descriptor immediately after the event has been processed. \layout Description hdr_data 64 bits of out-of-band user data (see Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:put} \end_inset ). \layout Description link The \emph on link \emph default member is used to link \family typewriter START \family default events with the \family typewriter END \family default or \family typewriter FAIL \family default event that signifies completion of the operation. The \emph on link \emph default member will be the same for the two events associated with an operation. The link member is also used to link an \family typewriter UNLINK \family default event with the event that caused the memory descriptor to be unlinked. \layout Description sequence The sequence number for this event. Sequence numbers are unique to each event. \layout Comment The \emph on sequence \emph default member is the last member and is volatile to support SMP implementations. When an event structure is filled in, the \emph on sequence \emph default member should be written after all other members have been updated. Moreover, a memory barrier should be inserted between the updating of other members and the updating of the \emph on sequence \emph default member. \layout Subsection PtlEQAlloc \begin_inset LatexCommand \label{sec:eqalloc} \end_inset \layout LyX-Code int PtlEQAlloc( ptl_handle_ni_t interface, \newline ptl_size_t count, \newline ptl_handle_eq_t* handle ); \layout Standard \noindent The \emph on PtlEQAlloc \emph default function is used to build an event queue. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_NI Indicates that \family typewriter interface \family default is not a valid network interface handle. \layout Description PTL_NOSPACE Indicates that there is insufficient memory to allocate the event queue. \layout Description PTL_SEGV Indicates that \family typewriter handle \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter interface \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the interface with which the event queue will be associated. \end_inset \begin_inset Text \layout Standard \family typewriter count \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The number of events that can be stored in the event queue. \end_inset \begin_inset Text \layout Standard \family typewriter handle \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold a handle for the newly created event queue. \end_inset \end_inset \layout Subsection PtlEQFree \begin_inset LatexCommand \label{sec:eqfree} \end_inset \layout LyX-Code int PtlEQFree( ptl_handle_eq_t eventq ); \layout Standard \noindent The \emph on PtlEQFree \emph default function releases the resources associated with an event queue. It is up to the user to insure that no memory descriptors are associated with the event queue once it is freed. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_EQ Indicates that \family typewriter eventq \family default is not a valid event queue handle. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter eventq \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard A handle for the event queue to be released. \end_inset \end_inset \layout Subsection PtlEQGet \begin_inset LatexCommand \label{sec:eqget} \end_inset \layout LyX-Code int PtlEQGet( ptl_handle_eq_t eventq, \newline ptl_event_t* event ); \layout Standard \noindent The \emph on PTLEQGet \emph default function is a nonblocking function that can be used to get the next event in an event queue. The event is removed from the queue. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_EQ_DROPPED Indicates success (i.e., an event is returned) and that at least one event between this event and the last event obtained (using \emph on PtlEQGet \emph default or \emph on PtlEQWait \emph default ) from this event queue has been dropped due to limited space in the event queue. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_EQ_EMPTY Indicates that \family typewriter eventq \family default is empty or another thread is waiting on \emph on PtlEQWait \emph default . \layout Description PTL_INV_EQ Indicates that \family typewriter eventq \family default is not a valid event queue handle. \layout Description PTL_SEGV Indicates that \family typewriter event \family default is not a legal address. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter eventq \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the event queue. \end_inset \begin_inset Text \layout Standard \family typewriter event \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold the values associated with the next event in the event queue. \end_inset \end_inset \layout Subsection PtlEQWait \begin_inset LatexCommand \label{sec:eqwait} \end_inset \layout LyX-Code int PtlEQWait( ptl_handle_eq_t eventq, \newline ptl_event_t* event ); \layout Standard \noindent The \emph on PTLEQWait \emph default function can be used to block the calling process (thread) until there is an event in an event queue. This function also returns the next event in the event queue and removes this event from the queue. This is the only blocking operation in the Portals 3.2 API. In the event that multiple threads are waiting on the same event queue, PtlEQWait is guaranteed to wake exactly one thread, but the order in which they are awakened is not specified. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_EQ_DROPPED Indicates success (i.e., an event is returned) and that at least one event between this event and the last event obtained (using \emph on PtlEQGet \emph default or \emph on PtlEQWait \emph default ) from this event queue has been dropped due to limited space in the event queue. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_EQ Indicates that \family typewriter eventq \family default is not a valid event queue handle. \layout Description PTL_SEGV Indicates that \family typewriter event \family default is not a legal address. queue handle. \layout Subsubsection Arguments \layout Standard \noindent \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter eventq \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the event queue to wait on. The calling process (thread) will be blocked until \family typewriter eventq \family default is not empty. \end_inset \begin_inset Text \layout Standard \family typewriter event \end_inset \begin_inset Text \layout Standard \series bold output \end_inset \begin_inset Text \layout Standard \noindent On successful return, this location will hold the values associated with the next event in the event queue. \end_inset \end_inset \layout Section The Access Control Table \begin_inset LatexCommand \label{sec:ac} \end_inset \layout Standard Processes can use the access control table to control which processes are allowed to perform operations on Portal table entries. Each communication interface has a Portal table and an access control table. The access control table for the default interface contains an entry at index zero that allows all processes with the same user id to communicate. Entries in the access control table can be manipulated using the \emph on PtlACEntry \emph default function. \layout Subsection PtlACEntry \begin_inset LatexCommand \label{sec:acentry} \end_inset \layout LyX-Code int PtlACEntry( ptl_handle_ni_t interface, \newline ptl_ac_index_t index, \newline ptl_process_id_t matchid, \newline ptl_uid_t user_id, \newline ptl_pt_index_t portal ); \layout Standard \noindent The \emph on PtlACEntry \emph default function can be used to update an entry in the access control table for an interface. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_NI Indicates that \family typewriter interface \family default is not a valid network interface handle. \layout Description PTL_AC_INV_INDEX Indicates that \family typewriter index \family default is not a valid access control table index. \layout Description PTL_INV_PROC Indicates that \family typewriter matchid \family default is not a valid process identifier. \layout Description PTL_PT_INV_INDEX Indicates that \family typewriter portal \family default is not a valid Portal table index. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter interface \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Identifies the interface to use. \end_inset \begin_inset Text \layout Standard \family typewriter index \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The index of the entry in the access control table to update. \end_inset \begin_inset Text \layout Standard \family typewriter matchid \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Identifies the process(es) that are allowed to perform operations. The constants \family typewriter PTL_PID_ANY \family default and \family typewriter PTL_NID_ANY \family default can be used to wildcard either of the ids in the \family typewriter ptl_process_id_t \family default structure. \end_inset \begin_inset Text \layout Standard \family typewriter user_id \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Identifies the user that is allowed to perform operations. The value \family typewriter PTL_UID_ANY \family default can be used to wildcard the user. \end_inset \begin_inset Text \layout Standard \family typewriter portal \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Identifies the Portal index(es) that can be used. The value \family typewriter PTL_PT_INDEX_ANY \family default can be used to wildcard the Portal index. \end_inset \end_inset \layout Section Data Movement Operations \begin_inset LatexCommand \label{sec:datamovement} \end_inset \layout Standard The Portals API provides two data movement operations: \emph on PtlPut \emph default and \emph on PtlGet \emph default . \layout Subsection PtlPut \begin_inset LatexCommand \label{sec:put} \end_inset \layout LyX-Code typedef enum { PTL_ACK_REQ, PTL_NOACK_REQ } ptl_ack_req_t; \newline \newline int PtlPut( ptl_handle_md_t mem_desc, \newline ptl_ack_req_t ack_req, \newline ptl_process_id_t target, \newline ptl_pt_index_t portal, \newline ptl_ac_index_t cookie, \newline ptl_match_bits_t match_bits, \newline ptl_size_t offset, \newline ptl_hdr_data_t hdr_data ); \layout Standard \noindent Values of the type \family typewriter ptl_ack_req_t \family default are used to control whether an acknowledgement should be sent when the operation completes (i.e., when the data has been written to a memory descriptor of the \family typewriter target \family default process). The value \family typewriter PTL_ACK_REQ \family default requests an acknowledgement, the value \family typewriter PTL_NOACK_REQ \family default requests that no acknowledgement should be generated. \layout Standard The \emph on PtlPut \emph default function initiates an asynchronous put operation. There are several events associated with a put operation: initiation of the send on the local node ( \family typewriter PTL_EVENT_SEND_START \family default ), completion of the send on the local node ( \family typewriter PTL_EVENT_SEND_END \family default or \family typewriter PTL_EVENT_SEND_FAIL \family default ), and, when the send completes successfully, the receipt of an acknowledgement ( \family typewriter PTL_EVENT_ACK \family default ) indicating that the operation was accepted by the target. These events will be logged in the event queue associated with the memory descriptor ( \family typewriter mem_desc \family default ) used in the put operation. Using a memory descriptor that does not have an associated event queue results in these events being discarded. In this case, the application must have another mechanism (e.g., a higher level protocol) for determining when it is safe to modify the memory region associated with the memory descriptor. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_MD Indicates that \family typewriter mem_desc \family default is not a valid memory descriptor. \layout Description PTL_INV_PROC Indicates that \family typewriter target \family default is not a valid process id. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter mem_desc \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the memory descriptor that describes the memory to be sent. If the memory descriptor has an event queue associated with it, it will be used to record events when the message has been sent (PTL_EVENT_SEND_START, PTL_EVENT_SEND_END). \end_inset \begin_inset Text \layout Standard \family typewriter ack_req \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent Controls whether an acknowledgement event is requested. Acknowledgements are only sent when they are requested by the initiating process \series bold and \series default the memory descriptor has an event queue \series bold and \series default the target memory descriptor enables them. Allowed constants: \family typewriter PTL_ACK_REQ \family default , \family typewriter PTL_NOACK_REQ \family default . \end_inset \begin_inset Text \layout Standard \family typewriter target \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A process id for the target process. \end_inset \begin_inset Text \layout Standard \family typewriter portal \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The index in the remote Portal table. \end_inset \begin_inset Text \layout Standard \family typewriter cookie \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The index into the access control table of the target process. \end_inset \begin_inset Text \layout Standard \family typewriter match_bits \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The match bits to use for message selection at the target process. \end_inset \begin_inset Text \layout Standard \family typewriter offset \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The offset into the target memory descriptor (only used when the target memory descriptor has the \family typewriter PTL_MD_MANAGE_REMOTE \family default option set). \end_inset \begin_inset Text \layout Standard \family typewriter hdr_data \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent 64 bits of user data that can be included in message header. This data is written to an event queue entry at the target if an event queue is present on the matching memory descriptor. \end_inset \end_inset \layout Subsection PtlGet \begin_inset LatexCommand \label{sec:get} \end_inset \layout LyX-Code int PtlGet( ptl_handle_md_t mem_desc, \newline ptl_process_id_t target, \newline ptl_pt_index_t portal, \newline ptl_ac_index_t cookie, \newline ptl_match_bits_t match_bits, \newline ptl_size_t offset ); \layout Standard \noindent The \emph on PtlGet \emph default function initiates a remote read operation. There are two event pairs associated with a get operation , when the data is sent from the remote node, a \family typewriter PTL_EVENT_GET{START|END} \family default event pair is registered on the remote node; and when the data is returned from the remote node a \family typewriter PTL_EVENT_REPLY{START|END} \family default event pair is registered on the local node. \layout Subsubsection Return Codes \layout Description PTL_OK Indicates success. \layout Description PTL_NOINIT Indicates that the Portals API has not been successfully initialized. \layout Description PTL_INV_MD Indicates that \family typewriter mem_desc \family default is not a valid memory descriptor. \layout Description PTL_INV_PROC Indicates that \family typewriter target \family default is not a valid process id. \layout Subsubsection Arguments \layout Standard \begin_inset Tabular \begin_inset Text \layout Standard \family typewriter mem_desc \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A handle for the memory descriptor that describes the memory into which the requested data will be received. The memory descriptor can have an event queue associated with it to record events, such as when the message receive has started ( \family typewriter PTL_EVENT_REPLY \family default _ \family typewriter START \family default ). \end_inset \begin_inset Text \layout Standard \family typewriter target \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent A process id for the target process. \end_inset \begin_inset Text \layout Standard \family typewriter portal \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The index in the remote Portal table. \end_inset \begin_inset Text \layout Standard \family typewriter cookie \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The index into the access control table of the target process. \end_inset \begin_inset Text \layout Standard \family typewriter match_bits \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The match bits to use for message selection at the target process. \end_inset \begin_inset Text \layout Standard \family typewriter offset \end_inset \begin_inset Text \layout Standard \series bold input \end_inset \begin_inset Text \layout Standard \noindent The offset into the target memory descriptor (only used when the target memory descriptor has the \family typewriter PTL_MD_MANAGE_REMOTE \family default option set). \end_inset \end_inset \layout Section Summary \layout Standard \begin_inset LatexCommand \label{sec:summary} \end_inset We conclude this section by summarizing the names introduced by the Portals 3.2 API. We start by summarizing the names of the types introduced by the API. This is followed by a summary of the functions introduced by the API. Which is followed by a summary of the function return codes. Finally, we conclude with a summary of the other constant values introduced by the API. \layout Standard Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:types} \end_inset presents a summary of the types defined by the Portals API. The first column in this table gives the type name, the second column gives a brief description of the type, the third column identifies the section where the type is defined, and the fourth column lists the functions that have arguments of this type. \layout Standard \begin_inset Float table placement htbp wide false collapsed false \layout Caption Types Defined by the Portals 3.2 API \begin_inset LatexCommand \label{tab:types} \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash medskip \end_inset \layout Standard \noindent \size small \begin_inset Tabular \begin_inset Text \layout Standard \series bold Name \end_inset \begin_inset Text \layout Standard \series bold Meaning \end_inset \begin_inset Text \layout Standard \series bold Sect \end_inset \begin_inset Text \layout Standard \series bold Functions \end_inset \begin_inset Text \layout Standard \family typewriter ptl_ac_index_t \end_inset \begin_inset Text \layout Standard \noindent indexes for an access control table \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:index-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlACEntry, PtlPut, PtlGet \end_inset \begin_inset Text \layout Standard \family typewriter ptl_ack_req_t \end_inset \begin_inset Text \layout Standard \noindent acknowledgement request types \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlPut \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \family default \end_inset \begin_inset Text \layout Standard \noindent kinds of events \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlGet \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_t \end_inset \begin_inset Text \layout Standard \noindent information about events \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:event-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlEQGet \end_inset \begin_inset Text \layout Standard \family typewriter plt_seq_t \end_inset \begin_inset Text \layout Standard event sequence number \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:event-type} \end_inset \end_inset \begin_inset Text \layout Standard PtlEQGet, PtlEQWait \end_inset \begin_inset Text \layout Standard \family typewriter ptl_handle_any_t \end_inset \begin_inset Text \layout Standard \noindent handles for any object \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:handle-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlNIHandle \end_inset \begin_inset Text \layout Standard \family typewriter ptl_handle_eq_t \end_inset \begin_inset Text \layout Standard \noindent handles for event queues \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:handle-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlEQAlloc, PtlEQFree, PtlEQGet, PtlEQWait, PtlMDUpdate \end_inset \begin_inset Text \layout Standard \family typewriter ptl_handle_md_t \family default \end_inset \begin_inset Text \layout Standard \noindent handles for memory descriptors \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:handle-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlMDAlloc, PtlMDUnlink, PtlMDUpdate, PtlMEAttach, PtlMEAttachAny, PtlMEInsert, PtlPut, PtlGet \end_inset \begin_inset Text \layout Standard \family typewriter ptl_handle_me_t \family default \end_inset \begin_inset Text \layout Standard \noindent handles for match entries \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:handle-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlMEAttach, PtlMEAttachAny, PtlMEInsert, PtlMEUnlink \end_inset \begin_inset Text \layout Standard \family typewriter ptl_handle_ni_t \end_inset \begin_inset Text \layout Standard \noindent handles for network interfaces \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:handle-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlNIInit, PtlNIFini, PtlNIStatus, PtlNIDist, PtlEQAlloc, PtlACEntry, PtlPut, PtlGet \end_inset \begin_inset Text \layout Standard \family typewriter ptl_nid_t \family default \end_inset \begin_inset Text \layout Standard \noindent node identifiers \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:id-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlGetId,PtlACEntry \end_inset \begin_inset Text \layout Standard \family typewriter ptl_pid_t \end_inset \begin_inset Text \layout Standard process identifier \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:id-type} \end_inset \end_inset \begin_inset Text \layout Standard PtlGetId, PtlACEntry \end_inset \begin_inset Text \layout Standard \family typewriter ptl_uid_t \end_inset \begin_inset Text \layout Standard user indentifier \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:id-type} \end_inset \end_inset \begin_inset Text \layout Standard PtlGetUid, PtlACEntry \end_inset \begin_inset Text \layout Standard \family typewriter ptl_ins_pos_t \end_inset \begin_inset Text \layout Standard \noindent insertion position (before or after) \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:meattach} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlMEAttach, PtlMEAttachAny, PtlMEInsert \end_inset \begin_inset Text \layout Standard \family typewriter ptl_interface_t \end_inset \begin_inset Text \layout Standard \noindent identifiers for network interfaces \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ni-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlNIInit \end_inset \begin_inset Text \layout Standard \family typewriter ptl_match_bits_t \family default \end_inset \begin_inset Text \layout Standard \noindent match (and ignore) bits \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:mb-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlMEAttach, PtlMEAttachAny, PtlMEInsert, PtlPut, PtlGet \end_inset \begin_inset Text \layout Standard \family typewriter ptl_md_t \family default \end_inset \begin_inset Text \layout Standard \noindent memory descriptors \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlMDAttach, PtlMDUpdate \end_inset \begin_inset Text \layout Standard \family typewriter ptl_ni_fail_t \end_inset \begin_inset Text \layout Standard network interface-specific failures \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:eq} \end_inset \end_inset \begin_inset Text \layout Standard PtlEQGet, PtlEQWait \end_inset \begin_inset Text \layout Standard \family typewriter ptl_process_id_t \family default \end_inset \begin_inset Text \layout Standard \noindent process identifiers \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:pid-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlGetId, PtlNIDist, PtlMEAttach, PtlMEAttachAny, PtlACEntry, PtlPut, PtlGet \end_inset \begin_inset Text \layout Standard \family typewriter ptl_pt_index_t \end_inset \begin_inset Text \layout Standard \noindent indexes for Portal tables \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:index-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlMEAttach, PtlMEAttachAny, PtlACEntry \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \family default \end_inset \begin_inset Text \layout Standard \noindent sizes \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:size-t} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlEQAlloc, PtlPut, PtlGet \end_inset \begin_inset Text \layout Standard \family typewriter ptl_sr_index_t \end_inset \begin_inset Text \layout Standard \noindent indexes for status registers \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:stat-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlNIStatus \end_inset \begin_inset Text \layout Standard \family typewriter ptl_sr_value_t \family default \end_inset \begin_inset Text \layout Standard \noindent values in status registers \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:stat-type} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlNIStatus \end_inset \begin_inset Text \layout Standard \family typewriter ptl_unlink_t \end_inset \begin_inset Text \layout Standard \noindent unlink options \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:meattach} \end_inset \end_inset \begin_inset Text \layout Standard \noindent PtlMEAttach, PtlMEAttachAny, PtlMEInsert, PtlMDAttach \end_inset \end_inset \end_inset \layout Standard Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:func} \end_inset presents a summary of the functions defined by the Portals API. The first column in this table gives the name for the function, the second column gives a brief description of the operation implemented by the function, and the third column identifies the section where the function is defined. \layout Standard \begin_inset Float table placement htbp wide false collapsed false \layout Caption Functions Defined by the Portals 3.2 API \begin_inset LatexCommand \label{tab:func} \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash medskip \end_inset \layout Standard \align center \size small \begin_inset Tabular \begin_inset Text \layout Standard Name \end_inset \begin_inset Text \layout Standard Operation \end_inset \begin_inset Text \layout Standard Section \end_inset \begin_inset Text \layout Standard PtlACEntry \end_inset \begin_inset Text \layout Standard update an entry in an access control table \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ac} \end_inset \end_inset \begin_inset Text \layout Standard PtlEQAlloc \end_inset \begin_inset Text \layout Standard create an event queue \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:eq} \end_inset \end_inset \begin_inset Text \layout Standard PtlEQGet \end_inset \begin_inset Text \layout Standard get the next event from an event queue \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:eq} \end_inset \end_inset \begin_inset Text \layout Standard PtlEQFree \end_inset \begin_inset Text \layout Standard release the resources for an event queue \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:eq} \end_inset \end_inset \begin_inset Text \layout Standard PtlEQWait \end_inset \begin_inset Text \layout Standard wait for a new event in an event queue \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:eq} \end_inset \end_inset \begin_inset Text \layout Standard PtlFini \end_inset \begin_inset Text \layout Standard shutdown the Portals API \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:init} \end_inset \end_inset \begin_inset Text \layout Standard PtlGet \end_inset \begin_inset Text \layout Standard perform a get operation \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:datamovement} \end_inset \end_inset \begin_inset Text \layout Standard PtlGetId \end_inset \begin_inset Text \layout Standard get the id for the current process \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:pid} \end_inset \end_inset \begin_inset Text \layout Standard PtlInit \end_inset \begin_inset Text \layout Standard initialize the Portals API \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:init} \end_inset \end_inset \begin_inset Text \layout Standard PtlMDAttach \end_inset \begin_inset Text \layout Standard create a memory descriptor and attach it to a match entry \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md} \end_inset \end_inset \begin_inset Text \layout Standard PtlMDBind \end_inset \begin_inset Text \layout Standard create a free-floating memory descriptor \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:mdbind} \end_inset \end_inset \begin_inset Text \layout Standard PtlMDUnlink \end_inset \begin_inset Text \layout Standard remove a memory descriptor from a list and release its resources \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md} \end_inset \end_inset \begin_inset Text \layout Standard PtlMDUpdate \end_inset \begin_inset Text \layout Standard update a memory descriptor \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md} \end_inset \end_inset \begin_inset Text \layout Standard PtlMEAttach \end_inset \begin_inset Text \layout Standard create a match entry and attach it to a Portal table \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:me} \end_inset \end_inset \begin_inset Text \layout Standard PtlMEAttachAny \end_inset \begin_inset Text \layout Standard create a match entry and attach it to a free Portal table entry \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:attachany} \end_inset \end_inset \begin_inset Text \layout Standard PtlMEInsert \end_inset \begin_inset Text \layout Standard create a match entry and insert it in a list \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:me} \end_inset \end_inset \begin_inset Text \layout Standard PtlMEUnlink \end_inset \begin_inset Text \layout Standard remove a match entry from a list and release its resources \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:me} \end_inset \end_inset \begin_inset Text \layout Standard PtlNIDist \end_inset \begin_inset Text \layout Standard get the distance to another process \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ni} \end_inset \end_inset \begin_inset Text \layout Standard PtlNIFini \end_inset \begin_inset Text \layout Standard shutdown a network interface \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ni} \end_inset \end_inset \begin_inset Text \layout Standard PtlNIHandle \end_inset \begin_inset Text \layout Standard get the network interface handle for an object \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ni} \end_inset \end_inset \begin_inset Text \layout Standard PtlNIInit \end_inset \begin_inset Text \layout Standard initialize a network interface \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ni} \end_inset \end_inset \begin_inset Text \layout Standard PtlNIStatus \end_inset \begin_inset Text \layout Standard read a network interface status register \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ni} \end_inset \end_inset \begin_inset Text \layout Standard PtlPut \end_inset \begin_inset Text \layout Standard perform a put operation \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:datamovement} \end_inset \end_inset \end_inset \end_inset \layout Standard Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:retcodes} \end_inset summarizes the return codes used by functions defined by the Portals API. All of these constants are integer values. The first column of this table gives the symbolic name for the constant, the second column gives a brief description of the value, and the third column identifies the functions that can return this value. \layout Standard \begin_inset Float table placement htbp wide false collapsed false \layout Caption Function Return Codes for the Portals 3.2 API \begin_inset LatexCommand \label{tab:retcodes} \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash medskip \end_inset \layout Standard \align center \size small \begin_inset Tabular \begin_inset Text \layout Standard \series bold Name \end_inset \begin_inset Text \layout Standard \series bold Meaning \end_inset \begin_inset Text \layout Standard \series bold Functions \series default \end_inset \begin_inset Text \layout Standard \family typewriter PTL_AC_INV_INDEX \family default \end_inset \begin_inset Text \layout Standard invalid access control table index \end_inset \begin_inset Text \layout Standard \noindent PtlACEntry \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EQ_DROPPED \family default \end_inset \begin_inset Text \layout Standard at least one event has been dropped \end_inset \begin_inset Text \layout Standard \noindent PtlEQGet, PtlWait \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EQ_EMPTY \end_inset \begin_inset Text \layout Standard no events available in an event queue \end_inset \begin_inset Text \layout Standard \noindent PtlEQGet \end_inset \begin_inset Text \layout Standard \family typewriter PTL_FAIL \end_inset \begin_inset Text \layout Standard error during initialization or cleanup \end_inset \begin_inset Text \layout Standard \noindent PtlInit, PtlFini \end_inset \begin_inset Text \layout Standard \family typewriter PTL_ILL_MD \family default \end_inset \begin_inset Text \layout Standard illegal memory descriptor values \end_inset \begin_inset Text \layout Standard \noindent PtlMDAttach, PtlMDBind, PtlMDUpdate \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INIT_DUP \end_inset \begin_inset Text \layout Standard duplicate initialization of an interface \end_inset \begin_inset Text \layout Standard \noindent PtlNIInit \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INIT_INV \family default \end_inset \begin_inset Text \layout Standard initialization of an invalid interface \end_inset \begin_inset Text \layout Standard \noindent PtlNIInit \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INUSE \end_inset \begin_inset Text \layout Standard the ME already has an MD \end_inset \begin_inset Text \layout Standard \noindent PtlMDAttach \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INV_ASIZE \family default \end_inset \begin_inset Text \layout Standard invalid access control table size \end_inset \begin_inset Text \layout Standard \noindent PtlNIInit \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INV_EQ \family default \end_inset \begin_inset Text \layout Standard invalid event queue handle \end_inset \begin_inset Text \layout Standard \noindent PtlMDUpdate, PtlEQFree, PtlEQGet \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INV_HANDLE \end_inset \begin_inset Text \layout Standard invalid handle \end_inset \begin_inset Text \layout Standard \noindent PtlNIHandle \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INV_MD \end_inset \begin_inset Text \layout Standard invalid memory descriptor handle \end_inset \begin_inset Text \layout Standard \noindent PtlMDUnlink, PtlMDUpdate \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INV_ME \family default \end_inset \begin_inset Text \layout Standard invalid match entry handle \end_inset \begin_inset Text \layout Standard \noindent PtlMDAttach \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INV_NI \end_inset \begin_inset Text \layout Standard invalid network interface handle \end_inset \begin_inset Text \layout Standard \noindent PtlNIDist, PtlNIFini, PtlMDBind, PtlEQAlloc \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INV_PROC \end_inset \begin_inset Text \layout Standard invalid process identifier \end_inset \begin_inset Text \layout Standard \noindent PtlNIInit, PtlNIDist, PtlMEAttach, PtlMEInsert, PtlACEntry, PtlPut, PtlGet \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INV_PTINDEX \family default \end_inset \begin_inset Text \layout Standard invalid Portal table index \end_inset \begin_inset Text \layout Standard \noindent PtlMEAttach \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INV_REG \end_inset \begin_inset Text \layout Standard invalid status register \end_inset \begin_inset Text \layout Standard \noindent PtlNIStatus \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INV_SR_INDX \end_inset \begin_inset Text \layout Standard invalid status register index \end_inset \begin_inset Text \layout Standard \noindent PtlNIStatus \end_inset \begin_inset Text \layout Standard \family typewriter PTL_ML_TOOLONG \end_inset \begin_inset Text \layout Standard match list too long \end_inset \begin_inset Text \layout Standard \noindent PtlMEAttach, PtlMEInsert \end_inset \begin_inset Text \layout Standard \family typewriter PTL_MD_INUSE \end_inset \begin_inset Text \layout Standard MD has pending operations \end_inset \begin_inset Text \layout Standard PtlMDUnlink \end_inset \begin_inset Text \layout Standard \family typewriter PTL_NOINIT \end_inset \begin_inset Text \layout Standard uninitialized API \end_inset \begin_inset Text \layout Standard \noindent \emph on all \emph default , except PtlInit \end_inset \begin_inset Text \layout Standard \family typewriter PTL_NOSPACE \family default \end_inset \begin_inset Text \layout Standard insufficient memory \end_inset \begin_inset Text \layout Standard \noindent PtlNIInit, PtlMDAttach, PtlMDBind, PtlEQAlloc, PtlMEAttach, PtlMEInsert \end_inset \begin_inset Text \layout Standard \family typewriter PTL_NOUPDATE \family default \end_inset \begin_inset Text \layout Standard no update was performed \end_inset \begin_inset Text \layout Standard \noindent PtlMDUpdate \end_inset \begin_inset Text \layout Standard \family typewriter PTL_PT_FULL \end_inset \begin_inset Text \layout Standard Portal table is full \end_inset \begin_inset Text \layout Standard PtlMEAttachAny \end_inset \begin_inset Text \layout Standard \family typewriter PTL_OK \end_inset \begin_inset Text \layout Standard success \end_inset \begin_inset Text \layout Standard \noindent \emph on all \end_inset \begin_inset Text \layout Standard \family typewriter PTL_SEGV \end_inset \begin_inset Text \layout Standard addressing violation \end_inset \begin_inset Text \layout Standard \noindent PtlNIInit, PtlNIStatus, PtlNIDist, PtlNIHandle, PtlMDBind, PtlMDUpdate, PtlEQAlloc, PtlEQGet, PtlEQWait \end_inset \end_inset \end_inset \layout Standard Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:oconsts} \end_inset summarizes the remaining constant values introduced by the Portals API. The first column in this table presents the symbolic name for the constant, the second column gives a brief description of the value, the third column identifies the type for the value, and the fourth column identifies the sections in which the value is mentioned. \layout Standard \begin_inset Float table placement htbp wide false collapsed false \layout Caption Other Constants Defined by the Portals 3.2 API \begin_inset LatexCommand \label{tab:oconsts} \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash medskip \end_inset \layout Standard \align center \size small \begin_inset Tabular \begin_inset Text \layout Standard \series bold Name \end_inset \begin_inset Text \layout Standard \series bold Meaning \end_inset \begin_inset Text \layout Standard \series bold Base type \end_inset \begin_inset Text \layout Standard \series bold Intr. \end_inset \begin_inset Text \layout Standard \series bold Ref. \end_inset \begin_inset Text \layout Standard \family typewriter PTL_ACK_REQ \family default \end_inset \begin_inset Text \layout Standard request an acknowledgement \end_inset \begin_inset Text \layout Standard \family typewriter ptl_ack_req_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EQ_NONE \end_inset \begin_inset Text \layout Standard a NULL event queue handle \end_inset \begin_inset Text \layout Standard \family typewriter ptl_handle_eq_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:handle-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md} \end_inset , \begin_inset LatexCommand \ref{sec:mdupdate} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_GET_START \end_inset \begin_inset Text \layout Standard get event start \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:get} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_GET_END \end_inset \begin_inset Text \layout Standard get event end \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:get} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_GET_FAIL \end_inset \begin_inset Text \layout Standard get event fail \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:get} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_PUT_START \end_inset \begin_inset Text \layout Standard put event start \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_PUT_END \end_inset \begin_inset Text \layout Standard put event end \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_PUT_FAIL \end_inset \begin_inset Text \layout Standard put event fail \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_REPLY_START \end_inset \begin_inset Text \layout Standard reply event start \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:get} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_REPLY_END \end_inset \begin_inset Text \layout Standard reply event end \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:get} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_REPLY_FAIL \end_inset \begin_inset Text \layout Standard reply event fail \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:get} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_ACK_START \end_inset \begin_inset Text \layout Standard acknowledgement event start \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_ACK_END \end_inset \begin_inset Text \layout Standard acknowledgement event end \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_ACK_FAIL \end_inset \begin_inset Text \layout Standard acknowledgement event fail \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_SEND_START \end_inset \begin_inset Text \layout Standard send event start \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_SEND_END \end_inset \begin_inset Text \layout Standard send event end \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_SEND_FAIL \end_inset \begin_inset Text \layout Standard send event fail \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_EVENT_UNLINK \end_inset \begin_inset Text \layout Standard unlink event \end_inset \begin_inset Text \layout Standard \family typewriter ptl_event_kind_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ek-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md-type} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_PID_ANY \end_inset \begin_inset Text \layout Standard wildcard for process id fields \end_inset \begin_inset Text \layout Standard \family typewriter ptl_pid_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:id-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:meattach} \end_inset , \begin_inset LatexCommand \ref{sec:acentry} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_NID_ANY \end_inset \begin_inset Text \layout Standard wildcard for node id fields \end_inset \begin_inset Text \layout Standard \family typewriter ptl_nid_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:id-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:meattach} \end_inset , \begin_inset LatexCommand \ref{sec:acentry} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_UID_ANY \end_inset \begin_inset Text \layout Standard wildcard for user id \end_inset \begin_inset Text \layout Standard \family typewriter ptl_uid_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:id-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:meattach} \end_inset , \begin_inset LatexCommand \ref{sec:acentry} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_IFACE_DEFAULT \end_inset \begin_inset Text \layout Standard default interface \end_inset \begin_inset Text \layout Standard \family typewriter ptl_interface_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:ni-type} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INS_AFTER \end_inset \begin_inset Text \layout Standard insert after \end_inset \begin_inset Text \layout Standard \family typewriter ptl_ins_pos_t \family default \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:meinsert} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_INS_BEFORE \end_inset \begin_inset Text \layout Standard insert before \end_inset \begin_inset Text \layout Standard \family typewriter ptl_ins_pos_t \family default \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:meinsert} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_MD_ACK_DISABLE \end_inset \begin_inset Text \layout Standard a flag to disable acknowledgements \end_inset \begin_inset Text \layout Standard \family typewriter int \family default \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md-type} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_MD_MANAGE_REMOTE \end_inset \begin_inset Text \layout Standard a flag to enable the use of remote offsets \end_inset \begin_inset Text \layout Standard \family typewriter int \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset , \begin_inset LatexCommand \ref{sec:get} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_MD_OP_GET \end_inset \begin_inset Text \layout Standard a flag to enable get operations \end_inset \begin_inset Text \layout Standard \family typewriter int \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md-type} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_MD_OP_PUT \end_inset \begin_inset Text \layout Standard a flag to enable put operations \end_inset \begin_inset Text \layout Standard \family typewriter int \family default \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md-type} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_MD_THRESH_INF \end_inset \begin_inset Text \layout Standard infinite threshold for a memory descriptor \end_inset \begin_inset Text \layout Standard \family typewriter int \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md-type} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_MD_TRUNCATE \end_inset \begin_inset Text \layout Standard a flag to enable truncation of a request \end_inset \begin_inset Text \layout Standard \family typewriter int \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:md-type} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_NOACK_REQ \end_inset \begin_inset Text \layout Standard request no acknowledgement \end_inset \begin_inset Text \layout Standard \family typewriter ptl_ack_req_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:put} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_PT_INDEX_ANY \end_inset \begin_inset Text \layout Standard wildcard for Portal indexes \end_inset \begin_inset Text \layout Standard \family typewriter ptl_pt_index_t \family default \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:acentry} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_RETAIN \end_inset \begin_inset Text \layout Standard disable unlinking \end_inset \begin_inset Text \layout Standard \family typewriter ptl_unlink_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:mdattach} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard \family typewriter PTL_SR_DROP_COUNT \end_inset \begin_inset Text \layout Standard index for the dropped count register \end_inset \begin_inset Text \layout Standard \family typewriter ptl_sr_index_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:stat-type} \end_inset \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:nistatus} \end_inset \end_inset \begin_inset Text \layout Standard \family typewriter PTL_UNLINK \end_inset \begin_inset Text \layout Standard enable unlinking \end_inset \begin_inset Text \layout Standard \family typewriter ptl_unlink_t \end_inset \begin_inset Text \layout Standard \begin_inset LatexCommand \ref{sec:mdattach} \end_inset \end_inset \begin_inset Text \layout Standard \end_inset \end_inset \end_inset \layout Chapter The Semantics of Message Transmission \begin_inset LatexCommand \label{sec:semantics} \end_inset \layout Standard The portals API uses four types of messages: put requests, acknowledgements, get requests, and replies. In this section, we describe the information passed on the wire for each type of message. We also describe how this information is used to process incoming messages. \layout Section Sending Messages \layout Standard Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:put-wire} \end_inset summarizes the information that is transmitted for a put request. The first column provides a descriptive name for the information, the second column provides the type for this information, the third column identifies the source of the information, and the fourth column provides additional notes. Most information that is transmitted is obtained directly from the \emph on PtlPut \emph default operation. Notice that the handle for the memory descriptor used in the \emph on PtlPut \emph default operation is transmitted even though this value cannot be interpreted by the target. A value of anything other than \family typewriter PTL_MD_NONE \family default , is interpreted as a request for an acknowledgement. \layout Standard \begin_inset Float table placement htbp wide false collapsed false \layout Caption Information Passed in a Put Request \begin_inset LatexCommand \label{tab:put-wire} \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash medskip \end_inset \layout Standard \align center \size small \begin_inset Tabular \begin_inset Text \layout Standard \series bold Information \end_inset \begin_inset Text \layout Standard \series bold Type \end_inset \begin_inset Text \layout Standard \series bold \emph on PtlPut \emph default arg \end_inset \begin_inset Text \layout Standard \series bold Notes \end_inset \begin_inset Text \layout Standard operation \end_inset \begin_inset Text \layout Standard \family typewriter int \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard indicates a put request \end_inset \begin_inset Text \layout Standard initiator \end_inset \begin_inset Text \layout Standard \family typewriter ptl_process_id_t \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard local information \end_inset \begin_inset Text \layout Standard user \end_inset \begin_inset Text \layout Standard \family typewriter ptl_uid_t \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard local information \end_inset \begin_inset Text \layout Standard target \end_inset \begin_inset Text \layout Standard \family typewriter ptl_process_id_t \end_inset \begin_inset Text \layout Standard \family typewriter target \family default \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard portal index \end_inset \begin_inset Text \layout Standard \family typewriter ptl_pt_index_t \family default \end_inset \begin_inset Text \layout Standard \family typewriter portal \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard cookie \end_inset \begin_inset Text \layout Standard \family typewriter ptl_ac_index_t \end_inset \begin_inset Text \layout Standard \family typewriter cookie \family default \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard match bits \end_inset \begin_inset Text \layout Standard \family typewriter ptl_match_bits_t \end_inset \begin_inset Text \layout Standard \family typewriter match_bits \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard offset \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \end_inset \begin_inset Text \layout Standard \family typewriter offset \family default \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard memory desc \end_inset \begin_inset Text \layout Standard \family typewriter ptl_handle_md_t \end_inset \begin_inset Text \layout Standard \family typewriter mem_desc \family default \end_inset \begin_inset Text \layout Standard no ack if \family typewriter PTL_MD_NONE \end_inset \begin_inset Text \layout Standard length \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \end_inset \begin_inset Text \layout Standard \family typewriter mem_desc \end_inset \begin_inset Text \layout Standard \family typewriter length \family default member \end_inset \begin_inset Text \layout Standard data \end_inset \begin_inset Text \layout Standard \family roman \emph on bytes \end_inset \begin_inset Text \layout Standard \family typewriter mem_desc \end_inset \begin_inset Text \layout Standard \family typewriter start \family default and \family typewriter length \family default members \end_inset \end_inset \end_inset \layout Standard Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:ack-wire} \end_inset summarizes the information transmitted in an acknowledgement. Most of the information is simply echoed from the put request. Notice that the initiator and target are obtained directly from the put request, but are swapped in generating the acknowledgement. The only new piece of information in the acknowledgement is the manipulated length which is determined as the put request is satisfied. \layout Standard \begin_inset Float table placement htbp wide false collapsed false \layout Caption Information Passed in an Acknowledgement \begin_inset LatexCommand \label{tab:ack-wire} \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash medskip \end_inset \layout Standard \align center \size small \begin_inset Tabular \begin_inset Text \layout Standard \series bold Information \series default \end_inset \begin_inset Text \layout Standard \series bold Type \end_inset \begin_inset Text \layout Standard \series bold Put Information \end_inset \begin_inset Text \layout Standard \series bold Notes \end_inset \begin_inset Text \layout Standard operation \end_inset \begin_inset Text \layout Standard \family typewriter int \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard indicates an acknowledgement \end_inset \begin_inset Text \layout Standard initiator \end_inset \begin_inset Text \layout Standard \family typewriter ptl_process_id_t \end_inset \begin_inset Text \layout Standard target \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard target \end_inset \begin_inset Text \layout Standard \family typewriter ptl_process_id_t \end_inset \begin_inset Text \layout Standard initiator \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard portal index \end_inset \begin_inset Text \layout Standard \family typewriter ptl_pt_index_t \end_inset \begin_inset Text \layout Standard portal index \end_inset \begin_inset Text \layout Standard echo \end_inset \begin_inset Text \layout Standard match bits \end_inset \begin_inset Text \layout Standard \family typewriter ptl_match_bits_t \end_inset \begin_inset Text \layout Standard match bits \end_inset \begin_inset Text \layout Standard echo \end_inset \begin_inset Text \layout Standard offset \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \end_inset \begin_inset Text \layout Standard offset \end_inset \begin_inset Text \layout Standard echo \end_inset \begin_inset Text \layout Standard memory desc \end_inset \begin_inset Text \layout Standard \family typewriter ptl_handle_md_t \end_inset \begin_inset Text \layout Standard memory desc \end_inset \begin_inset Text \layout Standard echo \end_inset \begin_inset Text \layout Standard requested length \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \family default \end_inset \begin_inset Text \layout Standard length \end_inset \begin_inset Text \layout Standard echo \end_inset \begin_inset Text \layout Standard manipulated length \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \family default \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard obtained from the operation \end_inset \end_inset \end_inset \layout Standard Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:get-wire} \end_inset summarizes the information that is transmitted for a get request. Like the information transmitted in a put request, most of the information transmitted in a get request is obtained directly from the \emph on PtlGet \emph default operation. Unlike put requests, get requests do not include the event queue handle. In this case, the reply is generated whenever the operation succeeds and the memory descriptor must not be unlinked until the reply is received. As such, there is no advantage to explicitly sending the event queue handle. \layout Standard \begin_inset Float table placement htbp wide false collapsed false \layout Caption Information Passed in a Get Request \begin_inset LatexCommand \label{tab:get-wire} \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash medskip \end_inset \layout Standard \align center \size small \begin_inset Tabular \begin_inset Text \layout Standard \series bold Information \series default \end_inset \begin_inset Text \layout Standard \series bold Type \end_inset \begin_inset Text \layout Standard \series bold \emph on PtlGet \emph default argument \end_inset \begin_inset Text \layout Standard \series bold Notes \end_inset \begin_inset Text \layout Standard operation \end_inset \begin_inset Text \layout Standard \family typewriter int \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard indicates a get operation \end_inset \begin_inset Text \layout Standard initiator \end_inset \begin_inset Text \layout Standard \family typewriter ptl_process_id_t \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard local information \end_inset \begin_inset Text \layout Standard user \end_inset \begin_inset Text \layout Standard \family typewriter ptl_uid_t \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard local information \end_inset \begin_inset Text \layout Standard target \end_inset \begin_inset Text \layout Standard \family typewriter ptl_process_id_t \end_inset \begin_inset Text \layout Standard \family typewriter target \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard portal index \end_inset \begin_inset Text \layout Standard \family typewriter ptl_pt_index_t \end_inset \begin_inset Text \layout Standard \family typewriter portal \family default \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard cookie \end_inset \begin_inset Text \layout Standard \family typewriter ptl_ac_index_t \end_inset \begin_inset Text \layout Standard \family typewriter cookie \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard match bits \end_inset \begin_inset Text \layout Standard \family typewriter ptl_match_bits_t \end_inset \begin_inset Text \layout Standard \family typewriter match_bits \family default \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard offset \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \end_inset \begin_inset Text \layout Standard \family typewriter offset \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard memory desc \end_inset \begin_inset Text \layout Standard \family typewriter ptl_handle_md_t \end_inset \begin_inset Text \layout Standard \family typewriter mem_desc \family default \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard length \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \end_inset \begin_inset Text \layout Standard \family typewriter mem_desc \end_inset \begin_inset Text \layout Standard \family typewriter length \family default member \end_inset \end_inset \end_inset \layout Standard Table\SpecialChar ~ \begin_inset LatexCommand \ref{tab:reply-wire} \end_inset summarizes the information transmitted in a reply. Like an acknowledgement, most of the information is simply echoed from the get request. The initiator and target are obtained directly from the get request, but are swapped in generating the acknowledgement. The only new information in the acknowledgement are the manipulated length and the data, which are determined as the get request is satisfied. \layout Standard \begin_inset Float table placement htbp wide false collapsed false \layout Caption Information Passed in a Reply \begin_inset LatexCommand \label{tab:reply-wire} \end_inset \layout Standard \begin_inset ERT status Collapsed \layout Standard \backslash medskip \end_inset \layout Standard \align center \size small \begin_inset Tabular \begin_inset Text \layout Standard \series bold Information \series default \end_inset \begin_inset Text \layout Standard \series bold Type \end_inset \begin_inset Text \layout Standard \series bold Put Information \end_inset \begin_inset Text \layout Standard \series bold Notes \end_inset \begin_inset Text \layout Standard operation \end_inset \begin_inset Text \layout Standard \family typewriter int \family default \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard indicates an acknowledgement \end_inset \begin_inset Text \layout Standard initiator \end_inset \begin_inset Text \layout Standard \family typewriter ptl_process_id_t \end_inset \begin_inset Text \layout Standard target \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard target \end_inset \begin_inset Text \layout Standard \family typewriter ptl_process_id_t \end_inset \begin_inset Text \layout Standard initiator \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard portal index \end_inset \begin_inset Text \layout Standard \family typewriter ptl_pt_index_t \end_inset \begin_inset Text \layout Standard portal index \end_inset \begin_inset Text \layout Standard echo \end_inset \begin_inset Text \layout Standard match bits \end_inset \begin_inset Text \layout Standard \family typewriter ptl_match_bits_t \end_inset \begin_inset Text \layout Standard match bits \end_inset \begin_inset Text \layout Standard echo \end_inset \begin_inset Text \layout Standard offset \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \family default \end_inset \begin_inset Text \layout Standard offset \end_inset \begin_inset Text \layout Standard echo \end_inset \begin_inset Text \layout Standard memory desc \end_inset \begin_inset Text \layout Standard \family typewriter ptl_handle_md_t \family default \end_inset \begin_inset Text \layout Standard memory desc \end_inset \begin_inset Text \layout Standard echo \end_inset \begin_inset Text \layout Standard requested length \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \family default \end_inset \begin_inset Text \layout Standard length \end_inset \begin_inset Text \layout Standard echo \end_inset \begin_inset Text \layout Standard manipulated length \end_inset \begin_inset Text \layout Standard \family typewriter ptl_size_t \family default \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard obtained from the operation \end_inset \begin_inset Text \layout Standard data \end_inset \begin_inset Text \layout Standard \emph on bytes \end_inset \begin_inset Text \layout Standard \end_inset \begin_inset Text \layout Standard obtained from the operation \end_inset \end_inset \end_inset \layout Section Receiving Messages \begin_inset LatexCommand \label{sec:receiving} \end_inset \layout Standard When an incoming message arrives on a network interface, the communication system first checks that the target process identified in the request is a valid process that has initialized the network interface (i.e., that the target process has a valid Portal table). If this test fails, the communication system discards the message and increment s the dropped message count for the interface. The remainder of the processing depends on the type of the incoming message. Put and get messages are subject to access control checks and translation (searching a match list), while acknowledgement and reply messages bypass the access control checks and the translation step. \layout Standard Acknowledgement messages include a handle for the memory descriptor used in the original \emph on PtlPut \emph default operation. This memory descriptor will identify the event queue where the event should be recorded. Upon receipt of an acknowledgement, the runtime system only needs to confirm that the memory descriptor and event queue still exist and that there is space for another event. Should the any of these conditions fail, the message is simply discarded and the dropped message count for the interface is incremented. Otherwise, the system builds an acknowledgement event from the information in the acknowledgement message and adds it to the event queue. \layout Standard Reception of reply messages is also relatively straightforward. Each reply message includes a handle for a memory descriptor. If this descriptor exists, it is used to receive the message. A reply message will be dropped if the memory descriptor identified in the request doesn't exist. In either of this case, the dropped message count for the interface is incremented. These are the only reasons for dropping reply messages. Every memory descriptor accepts and truncates incoming reply messages, eliminating the other potential reasons for rejecting a reply message. \layout Standard The critical step in processing an incoming put or get request involves mapping the request to a memory descriptor. This step starts by using the Portal index in the incoming request to identify a list of match entries. This list of match entries is searched in order until a match entry is found whose match criteria matches the match bits in the incoming request and whose memory descriptor accepts the request. \layout Standard Because acknowledge and reply messages are generated in response to requests made by the process receiving these messages, the checks performed by the runtime system for acknowledgements and replies are minimal. In contrast, put and get messages are generated by remote processes and the checks performed for these messages are more extensive. Incoming put or get messages may be rejected because: \layout Itemize the Portal index supplied in the request is not valid; \layout Itemize the cookie supplied in the request is not a valid access control entry; \layout Itemize the access control entry identified by the cookie does not match the identifier of the requesting process; \layout Itemize the access control entry identified by the access control entry does not match the Portal index supplied in the request; or \layout Itemize the match bits supplied in the request do not match any of the match entries with a memory descriptor that accepts the request. \layout Standard In all cases, if the message is rejected, the incoming message is discarded and the dropped message count for the interface is incremented. \layout Standard A memory descriptor may reject an incoming request for any of the following reasons: \layout Itemize the \family typewriter PTL_MD_PUT \family default or \family typewriter PTL_MD_GET \family default option has not been enabled and the operation is put or get, respectively; \layout Itemize the length specified in the request is too long for the memory descriptor and the \family typewriter PTL_MD_TRUNCATE \family default option has not been enabled. \layout Chapter Examples \begin_inset LatexCommand \label{sec:examples} \end_inset \layout Comment The examples presented in this chapter have not been updated to reflect the current API. \layout Standard In this section we present several example to illustrate expected usage patterns for the Portals 3.2 API. The first example describes how to implement parallel servers using the features of the Portals 3.2 API. This example covers the access control list and the use of remote managed offsets. The second example presents an approach to dealing with dropped requests. This example covers aspects of match lists and memory descriptors. The final example covers message reception in MPI. This example illustrates more sophisticated uses of matching and a procedure to update a memory descriptor. \layout Section Parallel File Servers \begin_inset LatexCommand \label{sec:expfs} \end_inset \layout Standard Figure\SpecialChar ~ \begin_inset LatexCommand \ref{fig:file} \end_inset illustrates the logical structure of a parallel file server. In this case, the parallel server consists of four servers that stripe application data across four disks. We would like to present applications with the illusion that the file server is a single entity. We will assume that all of the processes that constitute the parallel server have the same user id. \layout Standard \begin_inset Float figure placement htbp wide false collapsed false \layout Standard \align center \begin_inset Graphics FormatVersion 1 filename file.eps display color size_type 0 rotateOrigin center lyxsize_type 1 lyxwidth 196pt lyxheight 147pt \end_inset \layout Caption Parallel File Server \begin_inset LatexCommand \label{fig:file} \end_inset \end_inset \layout Standard When an application establishes a connection to the parallel file server, it will allocate a Portal and access control list entry for communicating with the server. The access control list entry will include the Portal and match any process in the parallel file server's, so all of the file server processes will have access to the portal. The Portal information and access control entry will be sent to the file server at this time. If the application and server need to have multiple, concurrent I/O operations, they can use additional portals or match entries to keep the operations from interfering with one another. \layout Standard When an application initiates an I/O operation, it first builds a memory descriptor that describes the memory region involved in the operation. This memory descriptor will enable the appropriate operation (put for read operations and get for write operations) and enable the use of remote offsets (this lets the servers decide where their data should be placed in the memory region). After creating the memory descriptor and linking it into the appropriate Portal entry, the application sends a read or write request (using \emph on PtlPut \emph default ) to one of the file server processes. The file server processes can then use put or get operations with the appropria te offsets to fill or retrieve the contents of the application's buffer. To know when the operation has completed, the application can add an event queue to the memory descriptor and add up the lengths of the remote operations until the sum is the size of the requested I/O operation. \layout Section Dealing with Dropped Requests \begin_inset LatexCommand \label{sec:exdrop} \end_inset \layout Standard If a process does not anticipate unexpected requests, they will be discarded. Applications using the Portals API can query the dropped count for the interface to determine the number of requests that have been dropped (see Section\SpecialChar ~ \begin_inset LatexCommand \ref{sec:nistatus} \end_inset ). While this approach minimizes resource consumption, it does not provide information that might be critical in debugging the implementation of a higher level protocol. \layout Standard To keep track of more information about dropped requests, we use a memory descriptor that truncates each incoming request to zero bytes and logs the \begin_inset Quotes eld \end_inset dropped \begin_inset Quotes erd \end_inset operations in an event queue. Note that the operations are not dropped in the Portals sense, because the operation succeeds. \layout Standard The following code fragment illustrates an implementation of this approach. In this case, we assume that a thread is launched to execute the function \family typewriter watch_drop \family default . This code starts by building an event queue to log truncated operations and a memory descriptor to truncate the incoming requests. This example only captures \begin_inset Quotes eld \end_inset dropped \begin_inset Quotes erd \end_inset requests for a single portal. In a more realistic situation, the memory descriptor would be appended to the match list for every portal. We also assume that the thread is capable of keeping up with the \begin_inset Quotes eld \end_inset dropped \begin_inset Quotes erd \end_inset requests. If this is not the case, we could use a finite threshold on the memory descriptor to capture the first few dropped requests. \layout LyX-Code \size small #include \newline #include \newline #include \newline \newline #define DROP_SIZE 32 /* number of dropped requests to track */ \newline \newline int watch_drop( ptl_handle_ni_t ni, ptl_pt_index_t index ) { \newline ptl_handle_eq_t drop_events; \newline ptl_event_t event; \newline ptl_handle_md_t drop_em; \newline ptl_md_t drop_desc; \newline ptl_process_id_t any_proc; \newline ptl_handle_me_t match_any; \newline \newline /* create the event queue */ \newline if( PtlEQAlloc(ni, DROP_SIZE, &drop_events) != PTL_OK ) { \newline fprintf( stderr, "Couldn't create the event queue \backslash n" ); \newline exit( 1 ); \newline } \newline \newline /* build a match entry */ \newline any_proc.nid = PTL_ID_ANY; \newline any_proc.pid = PTL_ID_ANY; \newline PtlMEAttach( index, any_proc, 0, ~(ptl_match_bits_t)0, PTL_RETAIN, \newline &match_any ); \newline \newline /* create the memory descriptor */ \newline drop_desc.start = NULL; \newline drop_desc.length = 0; \newline drop_desc.threshold = PTL_MD_THRESH_INF; \newline drop_desc.options = PTL_MD_OP_PUT | PTL_MD_OP_GET | PTL_MD_TRUNCATE; \newline drop_desc.user_ptr = NULL; \newline drop_desc.eventq = drop_events; \newline if( PtlMDAttach(match_any, drop_desc, &drop_em) != PTL_OK ) { \newline fprintf( stderr, "Couldn't create the memory descriptor \backslash n" ); \newline exit( 1 ); \newline } \newline \newline /* watch for "dropped" requests */ \newline while( 1 ) { \newline if( PtlEQWait( drop_events, &event ) != PTL_OK ) break; \newline fprintf( stderr, "Dropped request from gid = event.initiator.gid, event.initiator.rid ); \newline } \newline } \layout Section Message Transmission in MPI \begin_inset LatexCommand \label{sec:exmpi} \end_inset \layout Standard We conclude this section with a fairly extensive example that describes an approach to implementing message transmission for MPI. Like many MPI implementations, we distinguish two message transmission protocols: a short message protocol and a long message protocol. We use the constant \family typewriter MPI_LONG_LENGTH \family default to determine the size of a long message. \layout Standard For small messages, the sender simply sends the message and presumes that the message will be received (i.e., the receiver has allocated a memory region to receive the message body). For large messages, the sender also sends the message, but does not presume that the message body will be saved. Instead, the sender builds a memory descriptor for the message and enables get operations on this descriptor. If the target does not save the body of the message, it will record an event for the put operation. When the process later issues a matching MPI receive, it will perform a get operation to retrieve the body of the message. \layout Standard To facilitate receive side matching based on the protocol, we use the most significant bit in the match bits to indicate the protocol: 1 for long messages and 0 for short messages. \layout Standard The following code presents a function that implements the send side of the protocol. The global variable \family typewriter EndGet \family default is the last match entry attached to the Portal index used for posting long messages. This entry does not match any incoming requests (i.e., the memory descriptor rejects all get operations) and is built during initialization of the MPI library. The other global variable, \family typewriter MPI_NI \family default , is a handle for the network interface used by the MPI implementation. \layout LyX-Code \size small extern ptl_handle_me_t EndGet; \newline extern ptl_handle_ni_t MPI_NI; \newline \newline void MPIsend( void *buf, ptl_size_t len, void *data, ptl_handle_eq_t eventq, \newline ptl_process_id target, ptl_match_bits_t match ) \newline { \newline ptl_handle_md_t send_handle; \newline ptl_md_t mem_desc; \newline ptl_ack_req_t want_ack; \newline \newline mem_desc.start = buf; \newline mem_desc.length = len; \newline mem_desc.threshold = 1; \newline mem_desc.options = PTL_MD_GET_OP; \newline mem_desc.user_ptr = data; \newline mem_desc.eventq = eventq; \newline \newline if( len >= MPI_LONG_LENGTH ) { \newline ptl_handle_me_t me_handle; \newline \newline /* add a match entry to the end of the get list */ \newline PtlMEInsert( target, match, 0, PTL_UNLINK, PTL_INS_BEFORE, EndGet, &me_handle ); \newline PtlMDAttach( me_handle, mem_desc, PTL_UNLINK, NULL ); \newline \newline /* we want an ack for long messages */ \newline want_ack = PTL_ACK_REQ; \newline \newline /* set the protocol bit to indicate that this is a long message */ \newline match |= 1<<63; \newline } else { \newline /* we don't want an ack for short messages */ \newline want_ack = PTL_ACK_REQ; \newline \newline /* set the protocol bit to indicate that this is a short message */ \newline match &= ~(1<<63); \newline } \newline \newline /* create a memory descriptor and send it */ \newline PtlMDBind( MPI_NI, mem_desc, &send_handle ); \newline PtlPut( send_handle, want_ack, target, MPI_SEND_PINDEX, MPI_AINDEX, match, 0 ); \newline } \layout Standard The \emph on MPISend \emph default function returns as soon as the message has been scheduled for transmission. The event queue argument, \family typewriter eventq \family default , can be used to determine the disposition of the message. Assuming that \family typewriter eventq \family default is not \family typewriter PTL_EQ_NONE \family default , a \family typewriter PTL_EVENT_SENT \family default event will be recorded for each message as the message is transmitted. For small messages, this is the only event that will be recorded in \family typewriter eventq \family default . In contrast, long messages include an explicit request for an acknowledgement. If the \family typewriter target \family default process has posted a matching receive, the acknowledgement will be sent as the message is received. If a matching receive has not been posted, the message will be discarded and no acknowledgement will be sent. When the \family typewriter target \family default process later issues a matching receive, the receive will be translated into a get operation and a \family typewriter PTL_EVENT_GET \family default event will be recorded in \family typewriter eventq \family default . \layout Standard Figure\SpecialChar ~ \begin_inset LatexCommand \ref{fig:mpi} \end_inset illustrates the organization of the match list used for receiving MPI messages. The initial entries (not shown in this figure) would be used to match the MPI receives that have been preposted by the application. The preposted receives are followed by a match entry, \emph on RcvMark \emph default , that marks the boundary between preposted receives and the memory descriptors used for \begin_inset Quotes eld \end_inset unexpected \begin_inset Quotes erd \end_inset messages. The \emph on RcvMark \emph default entry is followed by a small collection of match entries that match unexpected \begin_inset Quotes eld \end_inset short \begin_inset Quotes erd \end_inset messages, i.e., messages that have a 0 in the most significant bit of their match bits. The memory descriptors associated with these match entries will append the incoming message to the associated memory descriptor and record an event in an event queue for unexpected messages. The unexpected short message matching entries are followed by a match entry that will match messages that were not matched by the preceding match entries, i.e., the unexpected long messages. The memory descriptor associated with this match entry truncates the message body and records an event in the event queue for unexpected messages. Note that of the memory descriptors used for unexpected messages share a common event queue. This makes it possible to process the unexpected messages in the order in which they arrived, regardless of. \layout Standard \begin_inset Float figure placement htbp wide false collapsed false \layout Standard \align center \begin_inset Graphics FormatVersion 1 filename mpi.eps display color size_type 0 rotateOrigin center lyxsize_type 1 lyxwidth 389pt lyxheight 284pt \end_inset \layout Caption Message Reception in MPI \begin_inset LatexCommand \label{fig:mpi} \end_inset \end_inset \layout Standard When the local MPI process posts an MPI receive, we must first search the events unexpected message queue to see if a matching message has already arrived. If no matching message is found, a match entry for the receive is inserted before the \emph on RcvMark \emph default entry--after the match entries for all of the previously posted receives and before the match entries for the unexpected messages. This ensures that preposted receives are matched in the order that they were posted (a requirement of MPI). \layout Standard While this strategy respects the temporal semantics of MPI, it introduces a race condition: a matching message might arrive after the events in the unexpected message queue have been searched, but before the match entry for the receive has been inserted in the match list. \layout Standard To avoid this race condition we start by setting the \family typewriter threshold \family default of the memory descriptor to 0, making the descriptor inactive. We then insert the match entry into the match list and proceed to search the events in the unexpected message queue. A matching message that arrives as we are searching the unexpected message queue will not be accepted by the memory descriptor and, if not matched by an earlier match list element, will add an event to the unexpected message queue. After searching the events in the unexpected message queue, we update the memory descriptor, setting the threshold to 1 to activate the memory descriptor. This update is predicated by the condition that the unexpected message queue is empty. We repeat the process of searching the unexpected message queue until the update succeeds. \layout Standard The following code fragment illustrates this approach. Because events must be removed from the unexpected message queue to be examined, this code fragment assumes the existence of a user managed event list, \family typewriter Rcvd \family default , for the events that have already been removed from the unexpected message queue. In an effort to keep the example focused on the basic protocol, we have omitted the code that would be needed to manage the memory descriptors used for unexpected short messages. In particular, we simply leave messages in these descriptors until they are received by the application. In a robust implementation, we would introduce code to ensure that short unexpected messages are removed from these memory descriptors so that they can be re-used. \layout LyX-Code \size small extern ptl_handle_eq_t UnexpQueue; \newline extern ptl_handle_me_t RcvMark; \newline extern ptl_handle_me_t ShortMatch; \newline \newline typedef struct event_list_tag { \newline ptl_event_t event; \newline struct event_list_tag* next; \newline } event_list; \newline \newline extern event_list Rcvd; \newline \newline void AppendRcvd( ptl_event_t event ) \newline { \newline /* append an event onto the Rcvd list */ \newline } \newline \newline int SearchRcvd( void *buf, ptl_size_t len, ptl_process_id_t sender, ptl_match_bi ts_t match, \newline ptl_match_bits_t ignore, ptl_event_t *event ) \newline { \newline /* Search the Rcvd event queue, looking for a message that matches the requested message. \newline * If one is found, remove the event from the Rcvd list and return it. */ \newline } \newline \newline typedef enum { RECEIVED, POSTED } receive_state; \newline \newline receive_state CopyMsg( void *buf, ptl_size_t &length, ptl_event_t event, ptl_md_t md_buf ) \newline { \newline ptl_md_t md_buf; \newline ptl_handle_me_t me_handle; \newline \newline if( event.rlength >= MPI_LONG_LENGTH ) { \newline PtlMDBind( MPI_NI, md_buf, &md_handle ); \newline PtlGet( event.initiator, MPI_GET_PINDEX, 0, event.match_bits, MPI_AINDEX, md_handle ); \newline return POSTED; \newline } else { \newline /* copy the message */ \newline if( event.mlength < *length ) *length = event.mlength; \newline memcpy( buf, (char*)event.md_desc.start+event.offset, *length ); \newline return RECEIVED; \newline } \newline } \newline \newline receive_state MPIreceive( void *buf, ptl_size_t &len, void *MPI_data, ptl_handle _eq_t eventq, \newline ptl_process_id_t sender, ptl_match_bits_t match, ptl_match_bits_t ignore ) \newline { \newline ptl_md_t md_buf; \newline ptl_handle_md_t md_handle; \newline ptl_handle_me_t me_handle; \newline ptl_event_t event; \newline \newline /* build a memory descriptor for the receive */ \newline md_buf.start = buf; \newline md_buf.length = *len; \newline md_buf.threshold = 0; /* temporarily disabled */ \newline md_buf.options = PTL_MD_PUT_OP; \newline md_buf.user_ptr = MPI_data; \newline md_buf.eventq = eventq; \newline \newline /* see if we have already received the message */ \newline if( SearchRcvd(buf, len, sender, match, ignore, &event) ) \newline return CopyMsg( buf, len, event, md_buf ); \newline \newline /* create the match entry and attach the memory descriptor */ \newline PtlMEInsert(sender, match, ignore, PTL_UNLINK, PTL_INS_BEFORE, RcvMark, &me_handle); \newline PtlMDAttach( me_handle, md_buf, PTL_UNLINK, &md_handle ); \newline \newline md_buf.threshold = 1; \newline do \newline if( PtlEQGet( UnexpQueue, &event ) != PTL_EQ_EMPTY ) { \newline if( MPIMatch(event, match, ignore, sender) ) { \newline return CopyMsg( buf, len, (char*)event.md_desc.start+event.offset, md_buf ); \newline } else { \newline AppendRcvd( event ); \newline } \newline } \newline while( PtlMDUpdate(md_handle, NULL, &md_buf, unexp_queue) == PTL_NOUPDATE ); \newline return POSTED; \newline } \layout Chapter* Acknowledgments \layout Standard Several people have contributed to the philosophy, design, and implementation of the Portals message passing architecture as it has evolved. We acknowledge the following people for their contributions: Al Audette, Lee Ann Fisk, David Greenberg, Tramm Hudson, Gabi Istrail, Chu Jong, Mike Levenhagen, Jim Otto, Mark Sears, Lance Shuler, Mack Stallcup, Jeff VanDyke, Dave van Dresser, Lee Ward, and Stephen Wheat. \layout Standard \begin_inset LatexCommand \BibTeX[ieee]{portals3} \end_inset \the_end