LU-9078 lnet: Fix route hops print

[fs/lustre-release.git] / lnet / doc / portals3.lyx

#LyX 1.2 created this file. For more info see http://www.lyx.org/
\lyxformat 220
\textclass report
\begin_preamble
\usepackage{fullpage}
\renewenvironment{comment}%
{\begin{quote}\textbf{Discussion}: \slshape}%
{\end{quote}}
\pagestyle{myheadings}
\end_preamble
\language american
\inputencoding auto
\fontscheme pslatex
\graphics default
\paperfontsize 10
\spacing single 
\papersize letterpaper
\paperpackage a4
\use_geometry 0
\use_amsmath 0
\use_natbib 0
\use_numerical_citations 0
\paperorientation portrait
\secnumdepth 2
\tocdepth 2
\paragraph_separation indent
\defskip medskip
\quotes_language english
\quotes_times 2
\papercolumns 1
\papersides 2
\paperpagestyle headings

\layout Title

The Portals 3.2 Message Passing Interface 
\newline 
 Revision 1.1
\layout Author

Ron Brightwell
\begin_inset Foot
collapsed true

\layout Standard

R.
 Brightwell and R.
 Riesen are with the Scalable Computing Systems Department, Sandia National
 Laboratories, P.O.
 Box 5800, Albuquerque, NM\SpecialChar ~
\SpecialChar ~
87111-1110, bright@cs.sandia.gov, rolf@cs.sandia.gov.
\end_inset 

, Arthur B.
 Maccabe
\begin_inset Foot
collapsed true

\layout Standard

A.
 B.
 Maccabe is with the Computer Science Department, University of New Mexico,
 Albuquerque, NM\SpecialChar ~
\SpecialChar ~
87131-1386, maccabe@cs.unm.edu.
\end_inset 

, Rolf Riesen and Trammell Hudson
\layout Abstract

This report presents a specification for the Portals 3.2 message passing
 interface.
 Portals 3.2 is intended to allow scalable, high-performance network communicatio
n between nodes of a parallel computing system.
 Specifically, it is designed to support a parallel computing platform composed
 of clusters of commodity workstations connected by a commodity system area
 network fabric.
 In addition, Portals 3.2 is well suited to massively parallel processing
 and embedded systems.
 Portals 3.2 represents an adaption of the data movement layer developed
 for massively parallel processing platforms, such as the 4500-node Intel
 TeraFLOPS machine.
 
\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
clearpage
\backslash 
pagenumbering{roman}
\backslash 
setcounter{page}{3}
\end_inset 


\layout Standard


\begin_inset LatexCommand \tableofcontents{}

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
cleardoublepage
\end_inset 


\layout Standard


\begin_inset FloatList figure

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
cleardoublepage
\end_inset 


\layout Standard


\begin_inset FloatList table

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
cleardoublepage
\end_inset 


\layout Chapter*

Summary of Changes for Revision 1.1
\layout Enumerate

Updated version number to 3.2 throughout the document
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sub:PtlGetId}

\end_inset 

: added 
\family typewriter 
PTL_SEGV
\family default 
 to error list for 
\shape italic 
PtlGetId
\shape default 
.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:meattach}

\end_inset 

: added 
\family typewriter 
PTL_ML_TOOLONG
\family default 
 to error list for 
\shape italic 
PtlMEAttach
\shape default 
.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:meunlink}

\end_inset 

: removed text referring to a list of associated memory descriptors.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:mdfree}

\end_inset 

: added text to describe unlinking a free-floating memory descriptor.
\layout Enumerate

Table 
\begin_inset LatexCommand \ref{tab:types}

\end_inset 

: added entry for 
\family typewriter 
ptl_seq_t
\family default 
.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:md-type}

\end_inset 

:
\begin_deeper 
\layout Enumerate

added definition of 
\family typewriter 
max_offset
\family default 
.
\layout Enumerate

added text to clarify 
\family typewriter 
PTL_MD_MANAGE_REMOTE
\family default 
.
\end_deeper 
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:mdattach}

\end_inset 

: modified text for 
\family typewriter 
unlink_op
\family default 
.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:niinit}

\end_inset 

: added text to clarify multiple calls to 
\shape italic 
PtlNIInit
\shape default 
.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:mdattach}

\end_inset 

: added text to clarify 
\family typewriter 
unlink_nofit
\family default 
.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:receiving}

\end_inset 

: removed text indicating that an MD will reject a message if the associated
 EQ is full.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:mdfree}

\end_inset 

: added 
\family typewriter 
PTL_MD_INUSE
\family default 
 error code and text to indicate that only MDs with no pending operations
 can be unlinked.
\layout Enumerate

Table 
\begin_inset LatexCommand \ref{tab:retcodes}

\end_inset 

: added 
\family typewriter 
PTL_MD_INUSE
\family default 
 return code.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:event-type}

\end_inset 

: added user id field, MD handle field, and NI specific failure field to
 the 
\family typewriter 
ptl_event_t
\family default 
 structure.
\layout Enumerate

Table 
\begin_inset LatexCommand \ref{tab:types}

\end_inset 

: added 
\family typewriter 
ptl_ni_fail_t
\family default 
.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:event-type}

\end_inset 

: added 
\family typewriter 
PTL_EVENT_UNLINK
\family default 
 event type.
\layout Enumerate

Table 
\begin_inset LatexCommand \ref{tab:func}

\end_inset 

: removed 
\shape slanted 
PtlTransId
\shape default 
.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:meattach}

\end_inset 

, Section 
\begin_inset LatexCommand \ref{sec:meinsert}

\end_inset 

, Section 
\begin_inset LatexCommand \ref{sec:put}

\end_inset 

: listed allowable constants with relevant fields.
\layout Enumerate

Table 
\begin_inset LatexCommand \ref{tab:func}

\end_inset 

: added 
\shape italic 
PtlMEAttachAny
\shape default 
 function.
\layout Enumerate

Table 
\begin_inset LatexCommand \ref{tab:retcodes}

\end_inset 

: added 
\family typewriter 
PTL_PT_FULL
\family default 
 return code for 
\shape italic 
PtlMEAttachAny
\shape default 
.
\layout Enumerate

Table 
\begin_inset LatexCommand \ref{tab:oconsts}

\end_inset 

: updated to reflect new event types.
\layout Enumerate

Section 
\begin_inset LatexCommand \ref{sec:id-type}

\end_inset 

: added 
\family typewriter 
ptl_nid_t
\family default 
, 
\family typewriter 
ptl_pid_t
\family default 
, and 
\family typewriter 
ptl_uid_t
\family default 
.
\layout Chapter*

Summary of Changes for Version 3.1
\layout Section*

Thread Issues
\layout Standard

The most significant change to the interface from version 3.0 to 3.1 involves
 the clarification of how the interface interacts with multi-threaded applicatio
ns.
 We adopted a generic thread model in which processes define an address
 space and threads share the address space.
 Consideration of the API in the light of threads lead to several clarifications
 throughout the document: 
\layout Enumerate

Glossary: 
\begin_deeper 
\layout Enumerate

added a definition for 
\emph on 
thread
\emph default 
, 
\layout Enumerate

reworded the definition for 
\emph on 
process
\emph default 
.
 
\end_deeper 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:apiover}

\end_inset 

: added section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:threads}

\end_inset 

 to describe the multi-threading model used by the Portals API.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:ptlinit}

\end_inset 

: 
\emph on 
PtlInit
\emph default 
 must be called at least once and may be called any number of times.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:ptlfini}

\end_inset 

: 
\emph on 
PtlFini
\emph default 
 should be called once as the process is terminating and not as each thread
 terminates.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:pid}

\end_inset 

: Portals does not define thread ids.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:ni}

\end_inset 

: network interfaces are associated with processes, not threads.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:niinit}

\end_inset 

: 
\emph on 
PtlNIInit
\emph default 
 must be called at least once and may be called any number of times.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:eqget}

\end_inset 

: 
\emph on 
PtlEQGet
\emph default 
 returns 
\family typewriter 
PTL_EQ_EMPTY
\family default 
 if a thread is blocked on 
\emph on 
PtlEQWait
\emph default 
.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:eqwait}

\end_inset 

: waiting threads are awakened in FIFO order.
 
\layout Standard

Two functions, 
\emph on 
PtlNIBarrier
\emph default 
 and 
\emph on 
PtlEQCount
\emph default 
 were removed from the API.
 
\emph on 
PtlNIBarrier
\emph default 
 was defined to block the calling process until all of the processes in
 the application group had invoked 
\emph on 
PtlNIBarrier
\emph default 
.
 We now consider this functionality, along with the concept of groups (see
 the discussion under 
\begin_inset Quotes eld
\end_inset 

other changes
\begin_inset Quotes erd
\end_inset 

), to be part of the runtime system, not part of the Portals API.
 
\emph on 
PtlEQCount
\emph default 
 was defined to return the number of events in an event queue.
 Because external operations may lead to new events being added and other
 threads may remove events, the value returned by 
\emph on 
PtlEQCount
\emph default 
 would have to be a hint about the number of events in the event queue.
\layout Section*

Handling small, unexpected messages
\layout Standard

Another set of changes relates to handling small unexpected messages in
 MPI.
 In designing version 3.0, we assumed that each unexpected message would
 be placed in a unique memory descriptor.
 To avoid the need to process a long list of memory descriptors, we moved
 the memory descriptors out of the match list and hung them off of a single
 match list entry.
 In this way, large unexpected messages would only encounter a single 
\begin_inset Quotes eld
\end_inset 

short message
\begin_inset Quotes erd
\end_inset 

 match list entry before encountering the 
\begin_inset Quotes eld
\end_inset 

long message
\begin_inset Quotes erd
\end_inset 

 match list entry.
 Experience with this strategy identified resource management problems with
 this approach.
 In particular, a long sequence of very short (or zero length) messages
 could quickly exhaust the memory descriptors constructed for handling unexpecte
d messages.
 Our new strategy involves the use of several very large memory descriptors
 for small unexpected messages.
 Consecutive unexpected messages will be written into the first of these
 memory descriptors until the memory descriptor fills up.
 When the first of the 
\begin_inset Quotes eld
\end_inset 

small memory
\begin_inset Quotes erd
\end_inset 

 descriptors fills up, it will be unlinked and subsequent short messages
 will be written into the next 
\begin_inset Quotes eld
\end_inset 

short message
\begin_inset Quotes erd
\end_inset 

 memory descriptor.
 In this case, a 
\begin_inset Quotes eld
\end_inset 

short message
\begin_inset Quotes erd
\end_inset 

 memory descriptor will be declared full when it does not have sufficient
 space for the largest small unexpected message.
\layout Standard

This lead to two significant changes.
 First, each match list entry now has a single memory descriptor rather
 than a list of memory descriptors.
 Second, in addition to exceeding the operation threshold, a memory descriptor
 can be unlinked when the local offset exceeds a specified value.
 These changes have lead to several changes in this document: 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{subsec:paddress}

\end_inset 

: 
\begin_deeper 
\layout Enumerate

removed references to the memory descriptor list, 
\layout Enumerate

changed the portals address translation description to indicate that unlinking
 a memory descriptor implies unlinking the associated match list entry--match
 list entries can no longer be unlinked independently from the memory descriptor.
 
\end_deeper 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:meattach}

\end_inset 

: 
\begin_deeper 
\layout Enumerate

removed unlink from argument list, 
\layout Enumerate

removed description of 
\family typewriter 
ptl_unlink
\family default 
 type, 
\layout Enumerate

changed wording of the error condition when the Portal table index already
 has an associated match list.
 
\end_deeper 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:meinsert}

\end_inset 

: removed unlink from argument list.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:md-type}

\end_inset 

: added 
\family typewriter 
max_offset
\family default 
.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:mdattach}

\end_inset 

: 
\begin_deeper 
\layout Enumerate

added description of 
\family typewriter 
ptl_unlink
\family default 
 type, 
\layout Enumerate

removed reference to memory descriptor lists, 
\layout Enumerate

changed wording of the error condition when match list entry already has
 an associated memory descriptor, 
\layout Enumerate

changed the description of the 
\family typewriter 
unlink
\family default 
 argument.
 
\end_deeper 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:md}

\end_inset 

: removed 
\family typewriter 
PtlMDInsert
\family default 
 operation.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:mdbind}

\end_inset 

: removed references to memory descriptor list.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:mdfree}

\end_inset 

: removed reference to memory descriptor list.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:summary}

\end_inset 

: removed references to PtlMDInsert.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:semantics}

\end_inset 

: removed reference to memory descriptor list.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:exmpi}

\end_inset 

: revised the MPI example to reflect the changes to the interface.
 
\layout Standard

Several changes have been made to improve the general documentation of the
 interface.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:handle-type}

\end_inset 

: documented the special value 
\family typewriter 
PTL_EQ_NONE
\family default 
.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:id-type}

\end_inset 

: documented the special value 
\family typewriter 
PTL_ID_ANY
\family default 
.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:mdbind}

\end_inset 

: documented the return value 
\family typewriter 
PTL_INV_EQ
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:mdupdate}

\end_inset 

: clarified the description of the 
\emph on 
PtlMDUpdate
\emph default 
 function.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:implvals}

\end_inset 

: introduced a new section to document the implementation defined values.
 
\layout Enumerate

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:summary}

\end_inset 

: modified Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:oconsts}

\end_inset 

 to indicate where each constant is introduced and where it is used.
 
\layout Section*

Other changes
\layout Subsection*

Implementation defined limits (Section 
\begin_inset LatexCommand \ref{sec:niinit}

\end_inset 

)
\layout Standard

The earlier version provided implementation defined limits for the maximum
 number of match entries, the maximum number of memory descriptors, etc.
 Rather than spanning the entire implementation, these limits are now associated
 with individual network interfaces.
\layout Subsection*

Added User Ids (Section 
\begin_inset LatexCommand \ref{sec:uid}

\end_inset 

)
\layout Standard

Group Ids had been used to simplify access control entries.
 In particular, a process could allow access for all of the processes in
 a group.
 User Ids have been introduced to regain this functionality.
 We use user ids to fill this role.
\layout Subsection*

Removed Group Ids and Rank Ids (Section 
\begin_inset LatexCommand \ref{sec:pid}

\end_inset 

)
\layout Standard

The earlier version of Portals had two forms for addressing processes: <node
 id, process id> and <group id, rank id>.
 A process group was defined as the collection processes created during
 application launch.
 Each process in the group was given a unique rank id in the range 0 to
 
\begin_inset Formula $n-1$
\end_inset 

 where 
\begin_inset Formula $n$
\end_inset 

 was the number of processes in the group.
 We removed groups because they are better handled in the runtime system.
\layout Subsection*

Match lists (Section 
\begin_inset LatexCommand \ref{sec:meattach}

\end_inset 

)
\layout Standard

It is no longer illegal to have an existing match entry when calling PtlMEAttach.
 A position argument was added to the list of arguments supplied to 
\emph on 
PtlMEAttach
\emph default 
 to specify whether the new match entry is prepended or appended to the
 existing list.
 If there is no existing match list, the position argument is ignored.
\layout Subsection*

Unlinking Memory Descriptors (Section 
\begin_inset LatexCommand \ref{sec:md}

\end_inset 

)
\layout Standard

Previously, a memory descriptor could be unlinked if the offset exceeded
 a threshold upon the completion of an operation.
 In this version, the unlinking is delayed until there is a matching operation
 which requires more memory than is currently available in the descriptor.
 In addition to changes in section, this lead to a revision of Figure\SpecialChar ~

\begin_inset LatexCommand \ref{fig:flow}

\end_inset 

.
\layout Subsection*

Split Phase Operations and Events (Section 
\begin_inset LatexCommand \ref{sec:eq}

\end_inset 

)
\layout Standard

Previously, there were five types of events: 
\family typewriter 
PTL_EVENT_PUT
\family default 
, 
\family typewriter 
PTL_EVENT_GET
\family default 
, 
\family typewriter 
PTL_EVENT_REPLY
\family default 
, 
\family typewriter 
PTL_EVENT_SENT
\family default 
, and 
\family typewriter 
PTL_EVENT_ACK.
 
\family default 
The first four of these reflected the completion of potentially long operations.
 We have introduced new event types to reflect the fact that long operations
 have a distinct starting point and a distinct completion point.
 Moreover, the completion may be successful or unsuccessful.
\layout Standard

In addition to providing a mechanism for reporting failure to higher levels
 of software, this split provides an opportunity for for improved ordering
 semantics.
 Previously, if one process intiated two operations (e.g., two put operations)
 on a remote process, these operations were guaranteed to complete in the
 same order that they were initiated.
 Now, we only guarantee that the initiation events are delivered in the
 same order.
 In particular, the operations do not need to complete in the order that
 they were intiated.
\layout Subsection*

Well known proces ids (Section 
\begin_inset LatexCommand \ref{sec:niinit}

\end_inset 

)
\layout Standard

To support the notion of 
\begin_inset Quotes eld
\end_inset 

well known process ids,
\begin_inset Quotes erd
\end_inset 

 we added a process id argument to the arguments for PtlNIInit.
\layout Chapter*

Glossary
\layout Description

API Application Programming Interface.
 A definition of the functions and semantics provided by library of functions.
 
\layout Description

Initiator A 
\emph on 
process
\emph default 
 that initiates a message operation.
 
\layout Description

Message An application-defined unit of data that is exchanged between 
\emph on 
processes
\emph default 
.
 
\layout Description

Message\SpecialChar ~
Operation Either a put operation, which writes data, or a get operation,
 which reads data.
 
\layout Description

Network A network provides point-to-point communication between 
\emph on 
nodes
\emph default 
.
 Internally, a network may provide multiple routes between endpoints (to
 improve fault tolerance or to improve performance characteristics); however,
 multiple paths will not be exposed outside of the network.
 
\layout Description

Node A node is an endpoint in a 
\emph on 
network
\emph default 
.
 Nodes provide processing capabilities and memory.
 A node may provide multiple processors (an SMP node) or it may act as a
 
\emph on 
gateway
\emph default 
 between networks.
 
\layout Description

Process A context of execution.
 A process defines a virtual memory (VM) context.
 This context is not shared with other processes.
 Several threads may share the VM context defined by a process.
 
\layout Description

Target A 
\emph on 
process
\emph default 
 that is acted upon by a message operation.
 
\layout Description

Thread A context of execution that shares a VM context with other threads.
 
\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
cleardoublepage
\layout Standard

\backslash 
setcounter{page}{1}
\backslash 
pagenumbering{arabic}
\end_inset 


\layout Chapter

Introduction
\begin_inset LatexCommand \label{sec:intro}

\end_inset 


\layout Section

Overview
\layout Standard

This document describes an application programming interface for message
 passing between nodes in a system area network.
 The goal of this interface is to improve the scalability and performance
 of network communication by defining the functions and semantics of message
 passing required for scaling a parallel computing system to ten thousand
 nodes.
 This goal is achieved by providing an interface that will allow a quality
 implementation to take advantage of the inherently scalable design of Portals.
\layout Standard

This document is divided into several sections: 
\layout Description

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:intro}

\end_inset 

---Introduction This section describes the purpose and scope of the Portals
 API.
 
\layout Description

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:apiover}

\end_inset 

---An\SpecialChar ~
Overview\SpecialChar ~
of\SpecialChar ~
the\SpecialChar ~
Portals\SpecialChar ~
3.1\SpecialChar ~
API This section gives a brief overview of the
 Portals API.
 The goal is to introduce the key concepts and terminology used in the descripti
on of the API.
 
\layout Description

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:api}

\end_inset 

---The\SpecialChar ~
Portals\SpecialChar ~
3.2\SpecialChar ~
API This section describes the functions and semantics of
 the Portals application programming interface.
 
\layout Description

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:semantics}

\end_inset 

--The\SpecialChar ~
Semantics\SpecialChar ~
of\SpecialChar ~
Message\SpecialChar ~
Transmission This section describes the semantics
 of message transmission.
 In particular, the information transmitted in each type of message and
 the processing of incoming messages.
 
\layout Description

Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:examples}

\end_inset 

---Examples This section presents several examples intended to illustrates
 the use of the Portals API.
 
\layout Section

Purpose
\layout Standard

Existing message passing technologies available for commodity cluster networking
 hardware do not meet the scalability goals required by the Cplant\SpecialChar ~

\begin_inset LatexCommand \cite{Cplant}

\end_inset 

 project at Sandia National Laboratories.
 The goal of the Cplant project is to construct a commodity cluster that
 can scale to the order of ten thousand nodes.
 This number greatly exceeds the capacity for which existing message passing
 technologies have been designed and implemented.
\layout Standard

In addition to the scalability requirements of the network, these technologies
 must also be able to support a scalable implementation of the Message Passing
 Interface (MPI)\SpecialChar ~

\begin_inset LatexCommand \cite{MPIstandard}

\end_inset 

 standard, which has become the 
\shape italic 
de facto
\shape default 
 standard for parallel scientific computing.
 While MPI does not impose any scalability limitations, existing message
 passing technologies do not provide the functionality needed to allow implement
ations of MPI to meet the scalability requirements of Cplant.
\layout Standard

The following are properties of a network architecture that do not impose
 any inherent scalability limitations: 
\layout Itemize

Connectionless - Many connection-oriented architectures, such as VIA\SpecialChar ~

\begin_inset LatexCommand \cite{VIA}

\end_inset 

 and TCP/IP sockets, have limitations on the number of peer connections
 that can be established.
 
\layout Itemize

Network independence - Many communication systems depend on the host processor
 to perform operations in order for messages in the network to be consumed.
 Message consumption from the network should not be dependent on host processor
 activity, such as the operating system scheduler or user-level thread scheduler.
 
\layout Itemize

User-level flow control - Many communication systems manage flow control
 internally to avoid depleting resources, which can significantly impact
 performance as the number of communicating processes increases.
 
\layout Itemize

OS Bypass - High performance network communication should not involve memory
 copies into or out of a kernel-managed protocol stack.
 
\layout Standard

The following are properties of a network architecture that do not impose
 scalability limitations for an implementation of MPI:
\layout Itemize

Receiver-managed - Sender-managed message passing implementations require
 a persistent block of memory to be available for every process, requiring
 memory resources to increase with job size and requiring user-level flow
 control mechanisms to manage these resources.
 
\layout Itemize

User-level Bypass - While OS Bypass is necessary for high-performance, it
 alone is not sufficient to support the Progress Rule of MPI asynchronous
 operations.
 
\layout Itemize

Unexpected messages - Few communication systems have support for receiving
 messages for which there is no prior notification.
 Support for these types of messages is necessary to avoid flow control
 and protocol overhead.
 
\layout Section

Background
\layout Standard

Portals was originally designed for and implemented on the nCube machine
 as part of the SUNMOS (Sandia/UNM OS)\SpecialChar ~

\begin_inset LatexCommand \cite{SUNMOS}

\end_inset 

 and Puma\SpecialChar ~

\begin_inset LatexCommand \cite{PumaOS}

\end_inset 

 lightweight kernel development projects.
 Portals went through two design phases, the latter of which is used on
 the 4500-node Intel TeraFLOPS machine\SpecialChar ~

\begin_inset LatexCommand \cite{TFLOPS}

\end_inset 

.
 Portals have been very successful in meeting the needs of such a large
 machine, not only as a layer for a high-performance MPI implementation\SpecialChar ~

\begin_inset LatexCommand \cite{PumaMPI}

\end_inset 

, but also for implementing the scalable run-time environment and parallel
 I/O capabilities of the machine.
\layout Standard

The second generation Portals implementation was designed to take full advantage
 of the hardware architecture of large MPP machines.
 However, efforts to implement this same design on commodity cluster technology
 identified several limitations, due to the differences in network hardware
 as well as to shortcomings in the design of Portals.
\layout Section

Scalability
\layout Standard

The primary goal in the design of Portals is scalability.
 Portals are designed specifically for an implementation capable of supporting
 a parallel job running on tens of thousands of nodes.
 Performance is critical only in terms of scalability.
 That is, the level of message passing performance is characterized by how
 far it allows an application to scale and not by how it performs in micro-bench
marks (e.g., a two node bandwidth or latency test).
\layout Standard

The Portals API is designed to allow for scalability, not to guarantee it.
 Portals cannot overcome the shortcomings of a poorly designed application
 program.
 Applications that have inherent scalability limitations, either through
 design or implementation, will not be transformed by Portals into scalable
 applications.
 Scalability must be addressed at all levels.
 Portals do not inhibit scalability, but do not guarantee it either.
\layout Standard

To support scalability, the Portals interface maintains a minimal amount
 of state.
 Portals provide reliable, ordered delivery of messages between pairs of
 processes.
 They are connectionless: a process is not required to explicitly establish
 a point-to-point connection with another process in order to communicate.
 Moreover, all buffers used in the transmission of messages are maintained
 in user space.
 The target process determines how to respond to incoming messages, and
 messages for which there are no buffers are discarded.
\layout Section

Communication Model
\layout Standard

Portals combine the characteristics of both one-side and two-sided communication.
 They define a 
\begin_inset Quotes eld
\end_inset 

matching put
\begin_inset Quotes erd
\end_inset 

 operation and a 
\begin_inset Quotes eld
\end_inset 

matching get
\begin_inset Quotes erd
\end_inset 

 operation.
 The destination of a put (or send) is not an explicit address; instead,
 each message contains a set of match bits that allow the receiver to determine
 where incoming messages should be placed.
 This flexibility allows Portals to support both traditional one-sided operation
s and two-sided send/receive operations.
\layout Standard

Portals allows the target to determine whether incoming messages are acceptable.
 A target process can choose to accept message operations from any specific
 process or can choose to ignore message operations from any specific process.
\layout Section

Zero Copy, OS Bypass and Application Bypass
\layout Standard

In traditional system architectures, network packets arrive at the network
 interface card (NIC), are passed through one or more protocol layers in
 the operating system, and eventually copied into the address space of the
 application.
 As network bandwidth began to approach memory copy rates, reduction of
 memory copies became a critical concern.
 This concern lead to the development of zero-copy message passing protocols
 in which message copies are eliminated or pipelined to avoid the loss of
 bandwidth.
\layout Standard

A typical zero-copy protocol has the NIC generate an interrupt for the CPU
 when a message arrives from the network.
 The interrupt handler then controls the transfer of the incoming message
 into the address space of the appropriate application.
 The interrupt latency, the time from the initiation of an interrupt until
 the interrupt handler is running, is fairly significant.
 To avoid this cost, some modern NICs have processors that can be programmed
 to implement part of a message passing protocol.
 Given a properly designed protocol, it is possible to program the NIC to
 control the transfer of incoming messages, without needing to interrupt
 the CPU.
 Because this strategy does not need to involve the OS on every message
 transfer, it is frequently called 
\begin_inset Quotes eld
\end_inset 

OS Bypass.
\begin_inset Quotes erd
\end_inset 

 ST\SpecialChar ~

\begin_inset LatexCommand \cite{ST}

\end_inset 

, VIA\SpecialChar ~

\begin_inset LatexCommand \cite{VIA}

\end_inset 

, FM\SpecialChar ~

\begin_inset LatexCommand \cite{FM2}

\end_inset 

, GM\SpecialChar ~

\begin_inset LatexCommand \cite{GM}

\end_inset 

, and Portals are examples of OS Bypass protocols.
\layout Standard

Many protocols that support OS Bypass still require that the application
 actively participate in the protocol to ensure progress.
 As an example, the long message protocol of PM requires that the application
 receive and reply to a request to put or get a long message.
 This complicates the runtime environment, requiring a thread to process
 incoming requests, and significantly increases the latency required to
 initiate a long message protocol.
 The Portals message passing protocol does not require activity on the part
 of the application to ensure progress.
 We use the term 
\begin_inset Quotes eld
\end_inset 

Application Bypass
\begin_inset Quotes erd
\end_inset 

 to refer to this aspect of the Portals protocol.
\layout Section

Faults 
\layout Standard

Given the number of components that we are dealing with and the fact that
 we are interested in supporting applications that run for very long times,
 failures are inevitable.
 The Portals API recognizes that the underlying transport may not be able
 to successfully complete an operation once it has been initiated.
 This is reflected in the fact that the Portals API reports three types
 of events: events indicating the initiation of an operation, events indicating
 the successful completion of an operation, and events indicating the unsuccessf
ul completion of an operation.
 Every initiation event is eventually followed by a successful completion
 event or an unsuccessful completion event.
\layout Standard

Between the time an operation is started and the time that the operation
 completes (successfully or unsuccessfully), any memory associated with
 the operation should be considered volatile.
 That is, the memory may be changed in unpredictable ways while the operation
 is progressing.
 Once the operation completes, the memory associated with the operation
 will not be subject to further modification (from this operation).
 Notice that unsuccessful operations may alter memory in an essentially
 unpredictable fashion.
\layout Chapter

An Overview of the Portals API
\begin_inset LatexCommand \label{sec:apiover}

\end_inset 


\layout Standard

In this section, we give a conceptual overview of the Portals API.
 The goal is to provide a context for understanding the detailed description
 of the API presented in the next section.
\layout Section

Data Movement
\begin_inset LatexCommand \label{sec:dmsemantics}

\end_inset 


\layout Standard

A Portal represents an opening in the address space of a process.
 Other processes can use a Portal to read (get) or write (put) the memory
 associated with the portal.
 Every data movement operation involves two processes, the 
\series bold 
initiator
\series default 
 and the 
\series bold 
target
\series default 
.
 The initiator is the process that initiates the data movement operation.
 The target is the process that responds to the operation by either accepting
 the data for a put operation, or replying with the data for a get operation.
\layout Standard

In this discussion, activities attributed to a process may refer to activities
 that are actually performed by the process or 
\emph on 
on behalf of the process
\emph default 
.
 The inclusiveness of our terminology is important in the context of 
\emph on 
application bypass
\emph default 
.
 In particular, when we note that the target sends a reply in the case of
 a get operation, it is possible that reply will be generated by another
 component in the system, bypassing the application.
\layout Standard

Figures\SpecialChar ~

\begin_inset LatexCommand \ref{fig:put}

\end_inset 

 and 
\begin_inset LatexCommand \ref{fig:get}

\end_inset 

 present graphical interpretations of the Portal data movement operations:
 put and get.
 In the case of a put operation, the initiator sends a put request message
 containing the data to the target.
 The target translates the Portal addressing information in the request
 using its local Portal structures.
 When the request has been processed, the target optionally sends an acknowledge
ment message.
\layout Standard


\begin_inset Float figure
placement htbp
wide false
collapsed false

\layout Standard
\align center 

\begin_inset Graphics FormatVersion 1
        filename put.eps
        display color
        size_type 0
        rotateOrigin center
        lyxsize_type 1
        lyxwidth 218pt
        lyxheight 119pt
\end_inset 


\layout Caption

Portal Put (Send)
\begin_inset LatexCommand \label{fig:put}

\end_inset 


\end_inset 


\layout Standard

In the case of a get operation, the initiator sends a get request to the
 target.
 As with the put operation, the target translates the Portal addressing
 information in the request using its local Portal structures.
 Once it has translated the Portal addressing information, the target sends
 a reply that includes the requested data.
\layout Standard


\begin_inset Float figure
placement htbp
wide false
collapsed false

\layout Standard
\align center 

\begin_inset Graphics FormatVersion 1
        filename get.eps
        display color
        size_type 0
        rotateOrigin center
        lyxsize_type 1
        lyxwidth 218pt
        lyxheight 119pt
\end_inset 


\layout Caption

Portal Get
\begin_inset LatexCommand \label{fig:get}

\end_inset 


\end_inset 


\layout Standard

We should note that Portal address translations are only performed on nodes
 that respond to operations initiated by other nodes.
 Acknowledgements and replies to get operations bypass the portals address
 translation structures.
\layout Section

Portal Addressing
\begin_inset LatexCommand \label{subsec:paddress}

\end_inset 


\layout Standard

One-sided data movement models (e.g., shmem\SpecialChar ~

\begin_inset LatexCommand \cite{CraySHMEM}

\end_inset 

, ST\SpecialChar ~

\begin_inset LatexCommand \cite{ST}

\end_inset 

, MPI-2\SpecialChar ~

\begin_inset LatexCommand \cite{MPI2}

\end_inset 

) typically use a triple to address memory on a remote node.
 This triple consists of a process id, memory buffer id, and offset.
 The process id identifies the target process, the memory buffer id specifies
 the region of memory to be used for the operation, and the offset specifies
 an offset within the memory buffer.
\layout Standard

In addition to the standard address components (process id, memory buffer
 id, and offset), a Portal address includes a set of match bits.
 This addressing model is appropriate for supporting one-sided operations
 as well as traditional two-sided message passing operations.
 Specifically, the Portals API provides the flexibility needed for an efficient
 implementation of MPI-1, which defines two-sided operations with one-sided
 completion semantics.
\layout Standard

Figure\SpecialChar ~

\begin_inset LatexCommand \ref{fig:portals}

\end_inset 

 presents a graphical representation of the structures used by a target
 in the interpretation of a Portal address.
 The process id is used to route the message to the appropriate node and
 is not reflected in this diagram.
 The memory buffer id, called the 
\series bold 
portal id
\series default 
, is used as an index into the Portal table.
 Each element of the Portal table identifies a match list.
 Each element of the match list specifies two bit patterns: a set of 
\begin_inset Quotes eld
\end_inset 

don't care
\begin_inset Quotes erd
\end_inset 

 bits, and a set of 
\begin_inset Quotes eld
\end_inset 

must match
\begin_inset Quotes erd
\end_inset 

 bits.
 In addition to the two sets of match bits, each match list element has
 at most one memory descriptor.
 Each memory descriptor identifies a memory region and an optional event
 queue.
 The memory region specifies the memory to be used in the operation and
 the event queue is used to record information about these operations.
\layout Standard


\begin_inset Float figure
placement htbp
wide false
collapsed false

\layout Standard
\align center 

\begin_inset Graphics FormatVersion 1
        filename portals.eps
        display color
        size_type 0
        rotateOrigin center
        lyxsize_type 1
        lyxwidth 305pt
        lyxheight 106pt
\end_inset 


\layout Caption

Portal Addressing Structures
\begin_inset LatexCommand \label{fig:portals}

\end_inset 


\end_inset 


\layout Standard

Figure\SpecialChar ~

\begin_inset LatexCommand \ref{fig:flow}

\end_inset 

 illustrates the steps involved in translating a Portal address, starting
 from the first element in a match list.
 If the match criteria specified in the match list entry are met and the
 memory descriptor list accepts the operation
\begin_inset Foot
collapsed true

\layout Standard

Memory descriptors can reject operations because a threshold has been exceeded
 or because the memory region does not have sufficient space, see Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:md}

\end_inset 


\end_inset 

, the operation (put or get) is performed using the memory region specified
 in the memory descriptor.
 If the memory descriptor specifies that it is to be unlinked when a threshold
 has been exceeded, the match list entry is removed from the match list
 and the resources associated with the memory descriptor and match list
 entry are reclaimed.
 Finally, if there is an event queue specified in the memory descriptor,
 the operation is logged in the event queue.
\layout Standard


\begin_inset Float figure
placement htbp
wide false
collapsed false

\layout Standard
\align center 

\begin_inset Graphics FormatVersion 1
        filename flow_new.eps
        display color
        size_type 0
        rotateOrigin center
        lyxsize_type 1
        lyxwidth 447pt
        lyxheight 282pt
\end_inset 


\layout Caption

Portals Address Translation
\begin_inset LatexCommand \label{fig:flow}

\end_inset 


\end_inset 


\layout Standard

If the match criteria specified in the match list entry are not met, or
 there is no memory descriptor associated with the match list entry, or
 the memory descriptor associated with the match list entry rejects the
 operation, the address translation continues with the next match list entry.
 If the end of the match list has been reached, the address translation
 is aborted and the incoming requested is discarded.
\layout Section

Access Control
\layout Standard

A process can control access to its portals using an access control list.
 Each entry in the access control list specifies a process id and a Portal
 table index.
 The access control list is actually an array of entries.
 Each incoming request includes an index into the access control list (i.e.,
 a 
\begin_inset Quotes eld
\end_inset 

cookie
\begin_inset Quotes erd
\end_inset 

 or hint).
 If the id of the process issuing the request doesn't match the id specified
 in the access control list entry or the Portal table index specified in
 the request doesn't match the Portal table index specified in the access
 control list entry, the request is rejected.
 Process identifiers and Portal table indexes may include wild card values
 to increase the flexibility of this mechanism.
 
\layout Standard

Two aspects of this design merit further discussion.
 First, the model assumes that the information in a message header, the
 sender's id in particular, is trustworthy.
 In most contexts, we assume that the entity that constructs the header
 is trustworthy; however, using cryptographic techniques, we could easily
 devise a protocol that would ensure the authenticity of the sender.
\layout Standard

Second, because the access check is performed by the receiver, it is possible
 that a malicious process will generate thousands of messages that will
 be denied by the receiver.
 This could saturate the network and/or the receiver, resulting in a 
\emph on 
denial of service
\emph default 
 attack.
 Moving the check to the sender using capabilities, would remove the potential
 for this form of attack.
 However, the solution introduces the complexities of capability management
 (exchange of capabilities, revocation, protections, etc).
\layout Section

Multi-threaded Applications
\begin_inset LatexCommand \label{sec:threads}

\end_inset 

 
\layout Standard

The Portals API supports a generic view of multi-threaded applications.
 From the perspective of the Portals API, an application program is defined
 by a set of processes.
 Each process defines a unique address space.
 The Portals API defines access to this address space from other processes
 (using portals addressing and the data movement operations).
 A process may have one or more 
\emph on 
threads
\emph default 
 executing in its address space.
 
\layout Standard

With the exception of 
\emph on 
PtlEQWait
\emph default 
 every function in the Portals API is non-blocking and atomic with respect
 to both other threads and external operations that result from data movement
 operations.
 While individual operations are atomic, sequences of these operations may
 be interleaved between different threads and with external operations.
 The Portals API does not provide any mechanisms to control this interleaving.
 It is expected that these mechanisms will be provided by the API used to
 create threads.
\layout Chapter

The Portals API
\begin_inset LatexCommand \label{sec:api}

\end_inset 


\layout Section

Naming Conventions
\begin_inset LatexCommand \label{sec:conv}

\end_inset 


\layout Standard

The Portals API defines two types of entities: functions and types.
 Function always start with 
\emph on 
Ptl
\emph default 
 and use mixed upper and lower case.
 When used in the body of this report, function names appear in italic face,
 e.g., 
\emph on 
PtlInit
\emph default 
.
 The functions associated with an object type will have names that start
 with 
\emph on 
Ptl
\emph default 
, followed by the two letter object type code shown in Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:objcodes}

\end_inset 

.
 As an example, the function 
\emph on 
PtlEQAlloc
\emph default 
 allocates resources for an event queue.
\layout Standard


\begin_inset Float table
placement htbp
wide false
collapsed false

\layout Caption

Object Type Codes
\begin_inset LatexCommand \label{tab:objcodes}

\end_inset 


\begin_inset ERT
status Collapsed

\layout Standard
 
\backslash 
medskip
\newline 
  
\end_inset 


\layout Standard
\align center 

\size small 

\begin_inset  Tabular
<lyxtabular version="3" rows="5" columns="3">
<features firstHeadEmpty="true">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<row bottomline="true">
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\emph on 
xx
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 Name 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 Section 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

EQ 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 Event Queue 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:eq}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 MD 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 Memory Descriptor 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 ME 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 Match list Entry 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:me}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 NI 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 Network Interface 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ni}

\end_inset 


\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\end_inset 


\layout Standard

Type names use lower case with underscores to separate words.
 Each type name starts with 
\family typewriter 
ptl
\family default 
_ and ends with 
\family typewriter 
_t
\family default 
.
 When used in the body of this report, type names appear in a fixed font,
 e.g., 
\family typewriter 
ptl_match_bits_t
\family default 
.
\layout Standard

Names for constants use upper case with underscores to separate words.
 Each constant name starts with 
\family typewriter 
PTL_
\family default 
.
 When used in the body of this report, type names appear in a fixed font,
 e.g., 
\family typewriter 
PTL_OK
\family default 
.
\layout Section

Base Types
\layout Standard

The Portals API defines a variety of base types.
 These types represent a simple renaming of the base types provided by the
 C programming language.
 In most cases these new type names have been introduced to improve type
 safety and to avoid issues arising from differences in representation sizes
 (e.g., 16-bit or 32-bit integers).
\layout Subsection

Sizes
\begin_inset LatexCommand \label{sec:size-t}

\end_inset 

 
\layout Standard

The type 
\family typewriter 
ptl_size_t
\family default 
 is an unsigned 64-bit integral type used for representing sizes.
\layout Subsection

Handles
\begin_inset LatexCommand \label{sec:handle-type}

\end_inset 

 
\layout Standard

Objects maintained by the API are accessed through handles.
 Handle types have names of the form 
\family typewriter 
ptl_handle_
\emph on 
xx
\emph default 
_t
\family default 
, where 
\emph on 
xx
\emph default 
 is one of the two letter object type codes shown in Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:objcodes}

\end_inset 

.
 For example, the type 
\family typewriter 
ptl_handle_ni_t
\family default 
 is used for network interface handles.
\layout Standard

Each type of object is given a unique handle type to enhance type checking.
 The type, 
\family typewriter 
ptl_handle_any_t
\family default 
, can be used when a generic handle is needed.
 Every handle value can be converted into a value of type 
\family typewriter 
ptl_handle_any_t
\family default 
 without loss of information.
\layout Standard

Handles are not simple values.
 Every portals object is associated with a specific network interface and
 an identifier for this interface (along with an object identifier) is part
 of the handle for the object.
\layout Standard

The special value 
\family typewriter 
PTL_EQ_NONE
\family default 
, of type 
\family typewriter 
ptl_handle_eq_t
\family default 
, is used to indicate the absence of an event queue.
 See sections 
\begin_inset LatexCommand \ref{sec:mdfree}

\end_inset 

 and\SpecialChar ~

\begin_inset LatexCommand \ref{sec:mdupdate}

\end_inset 

 for uses of this value.
\layout Subsection

Indexes
\begin_inset LatexCommand \label{sec:index-type}

\end_inset 

 
\layout Standard

The types 
\family typewriter 
ptl_pt_index_t
\family default 
 and 
\family typewriter 
ptl_ac_index_t
\family default 
 are integral types used for representing Portal table indexes and access
 control tables indexes, respectively.
 See section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:niinit}

\end_inset 

 for limits on values of these types.
\layout Subsection

Match Bits
\begin_inset LatexCommand \label{sec:mb-type}

\end_inset 

 
\layout Standard

The type 
\family typewriter 
ptl_match_bits_t
\family default 
 is capable of holding unsigned 64-bit integer values.
\layout Subsection

Network Interfaces
\begin_inset LatexCommand \label{sec:ni-type}

\end_inset 

 
\layout Standard

The type 
\family typewriter 
ptl_interface_t
\family default 
 is an integral type used for identifying different network interfaces.
 Users will need to consult the local documentation to determine appropriate
 values for the interfaces available.
 The special value 
\family typewriter 
PTL_IFACE_DEFAULT
\family default 
 identifies the default interface.
\layout Subsection

Identifiers
\begin_inset LatexCommand \label{sec:id-type}

\end_inset 


\layout Standard

The type 
\family typewriter 
ptl_nid_t
\family default 
 is an integral type used for representing node ids
\family typewriter 
, ptl_pid_t
\family default 
 is an integral type for representing process ids, and 
\family typewriter 
ptl_uid_t 
\family default 
is an integral type for representing user ids.
\layout Standard

The special values 
\family typewriter 
PTL_PID_ANY
\family default 
 matches any process identifier, PTL_NID_ANY matches any node identifier,
 and 
\family typewriter 
PTL_UID_ANY
\family default 
 matches any user identifier.
 See sections 
\begin_inset LatexCommand \ref{sec:meattach}

\end_inset 

 and\SpecialChar ~

\begin_inset LatexCommand \ref{sec:acentry}

\end_inset 

 for uses of these values.
\layout Subsection

Status Registers
\begin_inset LatexCommand \label{sec:stat-type}

\end_inset 


\layout Standard

Each network interface maintains an array of status registers that can be
 accessed using the 
\family typewriter 
PtlNIStatus
\family default 
 function (see Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:nistatus}

\end_inset 

).
 The type 
\family typewriter 
ptl_sr_index_t
\family default 
 defines the types of indexes that can be used to access the status registers.
 The only index defined for all implementations is 
\family typewriter 
PTL_SR_DROP_COUNT
\family default 
 which identifies the status register that counts the dropped requests for
 the interface.
 Other indexes (and registers) may be defined by the implementation.
\layout Standard

The type 
\family typewriter 
ptl_sr_value_t
\family default 
 defines the types of values held in status registers.
 This is a signed integer type.
 The size is implementation dependent, but must be at least 32 bits.
\layout Section

Initialization and Cleanup
\begin_inset LatexCommand \label{sec:init}

\end_inset 


\layout Standard

The Portals API includes a function, 
\emph on 
PtlInit
\emph default 
, to initialize the library and a function, 
\emph on 
PtlFini
\emph default 
, to cleanup after the application is done using the library.
\layout Subsection

PtlInit
\begin_inset LatexCommand \label{sec:ptlinit}

\end_inset 


\layout LyX-Code

int PtlInit( int *max_interfaces );
\layout Standard
\noindent 
The 
\emph on 
PtlInit
\emph default 
 function initializes the Portals library.
 PtlInit must be called at least once by a process before any thread makes
 a Portals function call, but may be safely called more than once.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_FAIL Indicates an error during initialization.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
max_interfaces
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="1" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="5in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
max_interfaces
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold the maximum number of interfaces
 that can be initialized.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlFini
\begin_inset LatexCommand \label{sec:ptlfini}

\end_inset 


\layout LyX-Code

void PtlFini( void );
\layout Standard
\noindent 
The 
\emph on 
PtlFini
\emph default 
 function cleans up after the Portals library is no longer needed by a process.
 After this function is called, calls to any of the functions defined by
 the Portal API or use of the structures set up by the Portals API will
 result in undefined behavior.
 This function should be called once and only once during termination by
 a process.
 Typically, this function will be called in the exit sequence of a process.
 Individual threads should not call PtlFini when they terminate.
\layout Section

Network Interfaces
\begin_inset LatexCommand \label{sec:ni}

\end_inset 

 
\layout Standard

The Portals API supports the use of multiple network interfaces.
 However, each interface is treated as an independent entity.
 Combining interfaces (e.g., 
\begin_inset Quotes eld
\end_inset 

bonding
\begin_inset Quotes erd
\end_inset 

 to create a higher bandwidth connection) must be implemented by the application
 or embedded in the underlying network.
 Interfaces are treated as independent entities to make it easier to cache
 information on individual network interface cards.
\layout Standard

Once initialized, each interface provides a Portal table, an access control
 table, and a collection of status registers.
 See Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:me}

\end_inset 

 for a discussion of updating Portal table entries using the 
\emph on 
PtlMEAttach
\emph default 
 function.
 See Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:ac}

\end_inset 

 for a discussion of the initialization and updating of entries in the access
 control table.
 See Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:nistatus}

\end_inset 

 for a discussion of the 
\emph on 
PtlNIStatus
\emph default 
 function which can be used to determine the value of a status register.
\layout Standard

Every other type of Portal object (e.g., memory descriptor, event queue, or
 match list entry) is associated with a specific network interface.
 The association to a network interface is established when the object is
 created and is encoded in the handle for the object.
\layout Standard

Each network interface is initialized and shutdown independently.
 The initialization routine, 
\emph on 
PtlNIInit
\emph default 
, returns a handle for an interface object which is used in all subsequent
 Portal operations.
 The 
\emph on 
PtlNIFini
\emph default 
 function is used to shutdown an interface and release any resources that
 are associated with the interface.
 Network interface handles are associated with processes, not threads.
 All threads in a process share all of the network interface handles.
\layout Standard

The Portals API also defines the 
\emph on 
PtlNIStatus
\emph default 
 function to query the status registers for a network interface, the 
\emph on 
PtlNIDist
\emph default 
 function to determine the 
\begin_inset Quotes eld
\end_inset 

distance
\begin_inset Quotes erd
\end_inset 

 to another process, and the 
\emph on 
PtlNIHandle
\emph default 
 function to determine the network interface that an object is associated
 with.
\layout Subsection

PtlNIInit
\begin_inset LatexCommand \label{sec:niinit}

\end_inset 


\layout LyX-Code

typedef struct {
\newline 
    int            max_match_entries;
\newline 
    int            max_mem_descriptors;
\newline 
    int            max_event_queues;
\newline 
    ptl_ac_index_t max_atable_index; 
\newline 
    ptl_pt_index_t max_ptable_index;
\newline 
} ptl_ni_limits_t;
\newline 

\newline 
int PtlNIInit( ptl_interface_t  interface
\newline 
               ptl_pid_t        pid,
\newline 
               ptl_ni_limits_t* desired,
\newline 
               ptl_ni_limits_t* actual,
\newline 
               ptl_handle_ni_t* handle );
\layout Standard

Values of type 
\family typewriter 
ptl_ni_limits_t
\family default 
 include the following members:
\layout Description

max_match_entries Maximum number of match entries that can be allocated
 at any one time.
\layout Description

max_mem_descriptors Maximum number of memory descriptors that can be allocated
 at any one time.
\layout Description

max_event_queues Maximum number of event queues that can be allocated at
 any one time.
\layout Description

max_atable_index Largest access control table index for this interface,
 valid indexes range from zero to 
\family typewriter 
max_atable_index
\family default 
, inclusive.
\layout Description

max_ptable_index Largest Portal table index for this interface, valid indexes
 range from zero to 
\family typewriter 
max_ptable_index
\family default 
, inclusive.
\layout Standard
\noindent 
The 
\emph on 
PtlNIInit
\emph default 
 function is used to initialized the Portals API for a network interface.
 This function must be called at least once by each process before any other
 operations that apply to the interface by any process or thread.
 For subsequent calls to 
\shape italic 
PtlNIInit
\shape default 
 from within the same process (either by different threads or the same thread),
 the desired limits will be ignored and the call will return the existing
 NI handle.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INIT_DUP Indicates a duplicate initialization of 
\family typewriter 
interface
\family default 
.
 
\layout Description

PTL_INIT_INV Indicates that 
\family typewriter 
interface
\family default 
 is not a valid network interface.
 
\layout Description

PTL_NOSPACE Indicates that there is insufficient memory to initialize the
 interface.
 
\layout Description

PTL_INV_PROC Indicates that 
\family typewriter 
pid
\family default 
 is not a valid process id.
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
actual 
\family default 
or
\family typewriter 
 handle
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="5" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
interface
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Identifies the network interface to be initialized.
  (See section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:ni-type}

\end_inset 

 for a discussion of  values used to identify network interfaces.)
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
pid
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Identifies the desired process id (for well known process ids).
 The value 
\family typewriter 
PTL_PID_ANY
\family default 
 may be used to have the process id assigned by the underlying library.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
desired
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
If non-NULL, points to a structure that holds the desired limits.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
actual
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, the location pointed to by actual will hold the actual
 limits.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
handle
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold a  handle for the interface.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Comment

The use of desired is implementation dependent.
 In particular, an implementation may choose to ignore this argument.
\layout Subsection

PtlNIFini
\begin_inset LatexCommand \label{sec:nifini}

\end_inset 


\layout LyX-Code

int PtlNIFini( ptl_handle_ni_t interface );
\layout Standard
\noindent 
The 
\emph on 
PtlNIFini
\emph default 
 function is used to release the resources allocated for a network interface.
 Once the 
\emph on 
PtlNIFini
\emph default 
 operation has been started, the results of pending API operations (e.g.,
 operations initiated by another thread) for this interface are undefined.
 Similarly, the effects of incoming operations (puts and gets) or return
 values (acknowledgements and replies) for this interface are undefined.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_NI Indicates that 
\family typewriter 
interface
\family default 
 is not a valid network interface handle.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="1" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard

interface
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard

A handle for the interface to shutdown.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlNIStatus
\begin_inset LatexCommand \label{sec:nistatus}

\end_inset 


\layout LyX-Code

int PtlNIStatus( ptl_handle_ni_t interface,
\newline 
                 ptl_sr_index_t  status_register,
\newline 
                 ptl_sr_value_t* status );
\layout Standard
\noindent 
The 
\emph on 
PtlNIStatus
\emph default 
 function returns the value of a status register for the specified interface.
 (See section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:stat-type}

\end_inset 

 for more information on status register indexes and status register values.)
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_NI Indicates that 
\family typewriter 
interface
\family default 
 is not a valid network interface handle.
 
\layout Description

PTL_INV_SR_INDX Indicates that 
\family typewriter 
status_register
\family default 
 is not a valid status register.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
status
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="3" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
interface
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the interface to use.
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
status_register
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
An index for the status register to read.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
status
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold the current value of the status
 register.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Comment

The only status register that must be defined is a drop count register (
\family typewriter 
PTL_SR_DROP_COUNT
\family default 
).
 Implementations may define additional status registers.
 Identifiers for the indexes associated with these registers should start
 with the prefix 
\family typewriter 
PTL_SR_
\family default 
.
\layout Subsection

PtlNIDist
\layout LyX-Code

int PtlNIDist( ptl_handle_ni_t  interface,
\newline 
               ptl_process_id_t process,
\newline 
               unsigned long*   distance );
\layout Standard
\noindent 
The 
\emph on 
PtlNIDist
\emph default 
 function returns the distance to another process using the specified interface.
 Distances are only defined relative to an interface.
 Distance comparisons between different interfaces on the same process may
 be meaningless.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_NI Indicates that 
\family typewriter 
interface
\family default 
 is not a valid network interface handle.
 
\layout Description

PTL_INV_PROC Indicates that 
\family typewriter 
process
\family default 
 is not a valid process identifier.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
distance
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="3" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
interface
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the interface to use.
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
process
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
An identifier for the process whose distance is being  requested.
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
distance
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold the  distance to the remote
 process.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Comment

This function should return a static measure of distance.
 Examples include minimum latency, the inverse of available bandwidth, or
 the number of switches between the two endpoints.
\layout Subsection

PtlNIHandle
\layout LyX-Code

int PtlNIHandle( ptl_handle_any_t handle,
\newline 
                 ptl_handle_ni_t* interface );
\layout Standard
\noindent 
The 
\emph on 
PtlNIHandle
\emph default 
 function returns a handle for the network interface with which the object
 identified by 
\family typewriter 
handle
\family default 
 is associated.
 If the object identified by 
\family typewriter 
handle
\family default 
 is a network interface, this function returns the same value it is passed.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_HANDLE Indicates that 
\family typewriter 
handle
\family default 
 is not a valid handle.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
interface
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="2" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
handle
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the object.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
interface
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold a handle for the network interface
 associated with 
\family typewriter 
handle
\family default 
.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Comment

Every handle should encode the network interface and the object id relative
 to this handle.
 Both are presumably encoded using integer values.
\layout Section

User Identification
\begin_inset LatexCommand \label{sec:uid}

\end_inset 


\layout Standard

Every process runs on behalf of a user.
 
\layout Subsection

PtlGetUid
\layout LyX-Code

int PtlGetUid( ptl_handle_ni_t   ni_handle,
\newline 
               ptl_uid_t*        uid );
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_INV_NI Indicates that 
\family typewriter 
ni_handle
\family default 
 is not a valid network interface handle.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
interface
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="2" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="5in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
handle
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A network interface handle.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
id
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold the user id for the calling
 process.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Comment

Note that user identifiers are dependent on the network interface(s).
 In particular, if a node has multiple interfaces, a process may have multiple
 user identifiers.
\layout Section

Process Identification
\begin_inset LatexCommand \label{sec:pid}

\end_inset 


\layout Standard

Processes that use the Portals API, can be identified using a node id and
 process id.
 Every node accessible through a network interface has a unique node identifier
 and every process running on a node has a unique process identifier.
 As such, any process in the computing system can be identified by its node
 id and process id.
 
\layout Standard

The Portals API defines a type, 
\family typewriter 
ptl_process_id_t
\family default 
 for representing process ids and a function, 
\emph on 
PtlGetId
\emph default 
, which can be used to obtain the id of the current process.
\layout Comment

The portals API does not include thread identifiers.
  Messages are delivered to processes (address spaces) not threads (contexts
 of  execution).
\layout Subsection

The Process Id Type
\begin_inset LatexCommand \label{sec:pid-type}

\end_inset 


\layout LyX-Code

typedef struct {
\newline 
    ptl_nid_t       nid; /* node id */
\newline 
    ptl_pid_t       pid; /* process id */
\newline 
} ptl_process_id_t;
\layout Standard
\noindent 
The 
\family typewriter 
ptl_process_id_t
\family default 
 type uses two identifiers to represent a process id: a node id and a process
 id.
 
\layout Subsection

PtlGetId
\begin_inset LatexCommand \label{sub:PtlGetId}

\end_inset 


\layout LyX-Code

int PtlGetId( ptl_handle_ni_t   ni_handle,
\newline 
              ptl_process_id_t* id );
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_INV_NI Indicates that 
\family typewriter 
ni_handle
\family default 
 is not a valid network interface handle.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
id
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="2" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="5in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
handle
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A network interface handle.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
id
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold the id for the calling process.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Comment

Note that process identifiers are dependent on the network interface(s).
 In particular, if a node has multiple interfaces, it may have multiple
 node identifiers.
\layout Section

Match List Entries and Match Lists
\begin_inset LatexCommand \label{sec:me}

\end_inset 


\layout Standard

A match list is a chain of match list entries.
 Each match list entry includes a memory descriptor and a set of match criteria.
 The match criteria can be used to reject incoming requests based on process
 id or the match bits provided in the request.
 A match list is created using the 
\emph on 
PtlMEAttach
\emph default 
 or 
\shape italic 
PtlMEAttachAny
\shape default 
 functions, which create a match list consisting of a single match list
 entry, attaches the match list to the specified Portal index, and returns
 a handle for the match list entry.
 Match entries can be dynamically inserted and removed from a match list
 using the 
\emph on 
PtlMEInsert
\emph default 
 and 
\emph on 
PtlMEUnlink
\emph default 
 functions.
\layout Subsection

PtlMEAttach
\begin_inset LatexCommand \label{sec:meattach}

\end_inset 


\layout LyX-Code

typedef enum { PTL_RETAIN, PTL_UNLINK } ptl_unlink_t;
\newline 

\layout LyX-Code

typedef enum { PTL_INS_BEFORE, PTL_INS_AFTER } ptl_ins_pos_t;
\newline 

\layout LyX-Code

int PtlMEAttach( ptl_handle_ni_t  interface,
\newline 
                 ptl_pt_index_t   index,
\newline 
                 ptl_process_id_t matchid,
\newline 
                 ptl_match_bits_t match_bits,
\newline 
                 ptl_match_bits_t ignorebits,
\newline 
                 ptl_unlink_t     unlink,
\newline 
                 ptl_ins_pos_t    position,
\newline 
                 ptl_handle_me_t* handle );
\layout Standard
\noindent 
Values of the type 
\family typewriter 
ptl_ins_pos_t
\family default 
 are used to control where a new item is inserted.
 The value 
\family typewriter 
PTL_INS_BEFORE
\family default 
 is used to insert the new item before the current item or before the head
 of the list.
 The value 
\family typewriter 
PTL_INS_AFTER
\family default 
 is used to insert the new item after the current item or after the last
 item in the list.
 
\layout Standard

The 
\emph on 
PtlMEAttach
\emph default 
 function creates a match list consisting of a single entry and attaches
 this list to the Portal table for 
\family typewriter 
interface
\family default 
.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_INV_NI Indicates that 
\family typewriter 
interface
\family default 
 is not a valid network interface handle.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_PTINDEX Indicates that 
\family typewriter 
index
\family default 
 is not a valid Portal table index.
 
\layout Description

PTL_INV_PROC Indicates that 
\family typewriter 
matchid
\family default 
 is not a valid process identifier.
 
\layout Description

PTL_NOSPACE Indicates that there is insufficient memory to allocate the
 match list entry.
 
\layout Description

PTL_ML_TOOLONG Indicates that the resulting match list is too long.
 The maximum length for a match list is defined by the interface.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="7" columns="3">
<features>
<column alignment="left" valignment="top" width="0.8in">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.75in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
interface
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the interface to use.
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
index
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The Portal table index where the match list  should be attached.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
matchid
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Specifies the match criteria for the process id of the requestor.
  The constants 
\family typewriter 
PTL_PID_ANY
\family default 
 and 
\family typewriter 
PTL_NID_ANY
\family default 
 can be used to  wildcard either of the ids in the 
\family typewriter 
ptl_process_id_t
\family default 
 structure.
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
match_bits, ignorebits
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Specify the match criteria to apply  to the match bits in the incoming request.
  The 
\family typewriter 
ignorebits
\family default 
 are used to mask out insignificant bits in the incoming match bits.
  The resulting bits are then compared to the match list entry's match 
 bits to determine if the incoming request meets the match criteria.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
unlink
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Indicates the match list entry should be unlinked when the last memory descripto
r associated with this match list  entry is unlinked.
  (Note, the check for unlinking a match entry  only occurs when a memory
 descriptor is unlinked.) 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
position
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Indicates whether the new match entry should be prepended or appended to
 the existing match list.
 If there is no existing list, this argument is ignored and the new match
 entry becomes the only entry in the list.
 Allowed constants: 
\family typewriter 
PTL_INS_BEFORE
\family default 
, 
\family typewriter 
PTL_INS_AFTER
\family default 
.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
handle
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold a  handle for the newly created
 match list entry.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlMEAttachAny
\begin_inset LatexCommand \label{sec:attachany}

\end_inset 


\layout LyX-Code

int PtlMEAttachAny( ptl_handle_ni_t  interface,
\newline 
                    ptl_pt_index_t   *index,
\newline 
                    ptl_process_id_t matchid,
\newline 
                    ptl_match_bits_t match_bits,
\newline 
                    ptl_match_bits_t ignorebits,
\newline 
                    ptl_unlink_t     unlink,
\newline 
                    ptl_handle_me_t* handle );
\layout Standard

The 
\emph on 
PtlMEAttachAny
\emph default 
 function creates a match list consisting of a single entry and attaches
 this list to an unused Portal table entry for 
\family typewriter 
interface
\family default 
.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_INV_NI Indicates that 
\family typewriter 
interface
\family default 
 is not a valid network interface handle.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_PROC Indicates that 
\family typewriter 
matchid
\family default 
 is not a valid process identifier.
 
\layout Description

PTL_NOSPACE Indicates that there is insufficient memory to allocate the
 match list entry.
 
\layout Description

PTL_PT_FULL Indicates that there are no free entries in the Portal table.
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="4" columns="3">
<features>
<column alignment="left" valignment="top" width="0.8in">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.75in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
interface
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the interface to use.
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
index
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold the Portal index where the
 match list  has been attached.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
matchid, match_bits, ignorebits, unlink
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
See the discussion for 
\shape italic 
PtlMEAttach
\shape default 
.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
handle
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold a  handle for the newly created
 match list entry.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlMEInsert
\begin_inset LatexCommand \label{sec:meinsert}

\end_inset 


\layout LyX-Code

int PtlMEInsert( ptl_handle_me_t  current,
\newline 
                 ptl_process_id_t matchid,
\newline 
                 ptl_match_bits_t match_bits,
\newline 
                 ptl_match_bits_t ignorebits,
\newline 
                 ptl_ins_pos_t    position,
\newline 
                 ptl_handle_me_t* handle );
\layout Standard

The 
\emph on 
PtlMEInsert
\emph default 
 function creates a new match list entry and inserts this entry into the
 match list containing 
\family typewriter 
current
\family default 
.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_PROC Indicates that 
\family typewriter 
matchid
\family default 
 is not a valid process identifier.
 
\layout Description

PTL_INV_ME Indicates that 
\family typewriter 
current
\family default 
 is not a valid match entry handle.
 
\layout Description

PTL_ML_TOOLONG Indicates that the resulting match list is too long.
 The maximum length for a match list is defined by the interface.
 
\layout Description

PTL_NOSPACE Indicates that there is insufficient memory to allocate the
 match entry.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="4" columns="3">
<features>
<column alignment="left" valignment="top" width="0.8in">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
current
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for a match entry.
  The new match entry will be inserted immediately before or immediately
 after this match entry.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
matchid
\family default 
, 
\family typewriter 
match_bits
\family default 
, 
\family typewriter 
ignorebits
\family default 
,  
\family typewriter 
unlink
\family default 
 
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
See the discussion  for 
\emph on 
PtlMEAttach
\emph default 
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
position
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Indicates whether the new match entry should be inserted before or after
 the 
\family typewriter 
current
\family default 
 entry.
 Allowed constants: 
\family typewriter 
PTL_INS_BEFORE
\family default 
, 
\family typewriter 
PTL_INS_AFTER
\family default 
.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 

\family typewriter 
handle
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
See the discussion for 
\emph on 
PtlMEAttach
\emph default 
.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlMEUnlink
\begin_inset LatexCommand \label{sec:meunlink}

\end_inset 


\layout LyX-Code

int PtlMEUnlink( ptl_handle_me_t entry );
\layout Standard
\noindent 
The 
\emph on 
PtlMEUnlink
\emph default 
 function can be used to unlink a match entry from a match list.
 This operation also releases any resources associated with the match entry
 (including the associated memory descriptor).
 It is an error to use the match entry handle after calling 
\emph on 
PtlMEUnlink
\emph default 
.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_ME Indicates that 
\family typewriter 
entry
\family default 
 is not a valid match entry handle.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="1" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
entry
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard

A handle for the match entry to be unlinked.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Section

Memory Descriptors
\begin_inset LatexCommand \label{sec:md}

\end_inset 


\layout Standard

A memory descriptor contains information about a region of an application
 process' memory and an event queue where information about the operations
 performed on the memory descriptor are recorded.
 The Portals API provides two operations to create memory descriptors: 
\emph on 
PtlMDAttach
\emph default 
, and 
\emph on 
PtlMDBind
\emph default 
; an operation to update a memory descriptor, 
\emph on 
PtlMDUpdate
\emph default 
; and an operation to unlink and release the resources associated with a
 memory descriptor, 
\emph on 
PtlMDUnlink
\emph default 
.
\layout Subsection

The Memory Descriptor Type
\begin_inset LatexCommand \label{sec:md-type}

\end_inset 


\layout LyX-Code

typedef struct {
\newline 
    void*           start;
\newline 
    ptl_size_t      length;
\newline 
    int             threshold;
\newline 
    unsigned int    max_offset;
\newline 
    unsigned int    options;
\newline 
    void*           user_ptr;
\newline 
    ptl_handle_eq_t eventq;
\newline 
} ptl_md_t;
\layout Standard
\noindent 
The 
\family typewriter 
ptl_md_t
\family default 
 type defines the application view of a memory descriptor.
 Values of this type are used to initialize and update the memory descriptors.
\layout Subsubsection

Members
\layout Description

start,\SpecialChar ~
length Specify the memory region associated with the memory descriptor.
 The 
\family typewriter 
start
\family default 
 member specifies the starting address for the memory region and the 
\family typewriter 
length
\family default 
 member specifies the length of the region.
 The 
\family typewriter 
start member
\family default 
 can be NULL provided that the 
\family typewriter 
length
\family default 
 member is zero.
 (Zero length buffers are useful to record events.) There are no alignment
 restrictions on the starting address or the length of the region; although,
 unaligned messages may be slower (i.e., lower bandwidth and/or longer latency)
 on some implementations.
 
\layout Description

threshold Specifies the maximum number of operations that can be performed
 on the memory descriptor.
 An operation is any action that could possibly generate an event (see Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 

 for the different types of events).
 In the usual case, the threshold value is decremented for each operation
 on the memory descriptor.
 When the threshold value is zero, the memory descriptor is 
\emph on 
inactive
\emph default 
, and does not respond to operations.
 A memory descriptor can have an initial threshold value of zero to allow
 for manipulation of an inactive memory descriptor by the local process.
 A threshold value of 
\family typewriter 
PTL_MD_THRESH_INF
\family default 
 indicates that there is no bound on the number of operations that may be
 applied to a memory descriptor.
 Note that local operations (e.g., 
\emph on 
PtlMDUpdate
\emph default 
) are not applied to the threshold count.
 
\layout Description

max_offset Specifies the maximum local offset of a memory descriptor.
 When the local offset of a memory descriptor exceeds this maximum, the
 memory descriptor becomes 
\shape italic 
inactive
\shape default 
 and does not respond to further operations.
\layout Description

options Specifies the behavior of the memory descriptor.
 There are five options that can be selected: enable put operations (yes
 or no), enable get operations (yes or no), offset management (local or
 remote), message truncation (yes or no), and acknowledgement (yes or no).
 Values for this argument can be constructed using a bitwise or of the following
 values: 
\begin_deeper 
\begin_deeper 
\layout Description

PTL_MD_OP_PUT Specifies that the memory descriptor will respond to 
\emph on 
put
\emph default 
 operations.
 By default, memory descriptors reject 
\emph on 
put
\emph default 
 operations.
 
\layout Description

PTL_MD_OP_GET Specifies that the memory descriptor will respond to 
\emph on 
get
\emph default 
 operations.
 By default, memory descriptors reject 
\emph on 
get
\emph default 
 operations.
 
\layout Description

PTL_MD_MANAGE_REMOTE Specifies that the offset used in accessing the memory
 region is provided by the incoming request.
 By default, the offset is maintained locally.
 When the offset is maintained locally, the offset is incremented by the
 length of the request so that the next operation (put and/or get) will
 access the next part of the memory region.
\layout Description

PTL_MD_TRUNCATE Specifies that the length provided in the incoming request
 can be reduced to match the memory available in the region.
 (The memory available in a memory region is determined by subtracting the
 offset from the length of the memory region.) By default, if the length
 in the incoming operation is greater than the amount of memory available,
 the operation is rejected.
 
\layout Description

PTL_MD_ACK_DISABLE Specifies that an acknowledgement should 
\emph on 
not
\emph default 
 be sent for incoming 
\emph on 
put
\emph default 
 operations, even if requested.
 By default, acknowledgements are sent for 
\emph on 
put
\emph default 
 operations that request an acknowledgement.
 Acknowledgements are never sent for 
\emph on 
get
\emph default 
 operations.
 The value sent in the reply serves as an implicit acknowledgement.
 
\end_deeper 
\layout Standard


\series bold 
Note
\series default 
: It is not considered an error to have a memory descriptor that does not
 respond to either 
\emph on 
put
\emph default 
 or 
\emph on 
get
\emph default 
 operations: Every memory descriptor responds to 
\emph on 
reply
\emph default 
 operations.
 Nor is it considered an error to have a memory descriptor that responds
 to both 
\emph on 
put
\emph default 
 and 
\emph on 
get
\emph default 
 operations.
 
\end_deeper 
\layout Description

user_ptr A user-specified value that is associated with the memory descriptor.
 The value does not need to be a pointer, but must fit in the space used
 by a pointer.
 This value (along with other values) is recorded in events associated with
 operations on this memory descriptor.
\begin_inset Foot
collapsed true

\layout Standard

Tying the memory descriptor to a user-defined value can be useful when multiple
 memory descriptor share the same event queue or when the memory descriptor
 needs to be associated with a data structure maintained by the application.
 For example, an MPI implementation can set the 
\family typewriter 
user_ptr
\family default 
 argument to the value of an MPI Request.
 This direct association allows for processing of memory descriptor's by
 the MPI implementation without a table lookup or a search for the appropriate
 MPI Request.
\end_inset 


\layout Description

eventq A handle for the event queue used to log the operations performed
 on the memory region.
 If this argument is 
\family typewriter 
PTl_EQ_NONE
\family default 
, operations performed on this memory descriptor are not logged.
 
\layout Subsection

PtlMDAttach
\begin_inset LatexCommand \label{sec:mdattach}

\end_inset 


\layout LyX-Code

int PtlMDAttach( ptl_handle_me_t  match,
\newline 
                 ptl_md_t         mem_desc,
\newline 
                 ptl_unlink_t     unlink_op,
\newline 
                 ptl_unlink_t     unlink_nofit,
\newline 
                 ptl_handle_md_t* handle );
\layout Standard
\noindent 
Values of the type 
\family typewriter 
ptl_unlink_t
\family default 
 are used to control whether an item is unlinked from a list.
 The value 
\family typewriter 
PTL_UNLINK
\family default 
 enables unlinking.
 The value 
\family typewriter 
PTL_RETAIN
\family default 
 disables unlinking.
\layout Standard

The 
\emph on 
PtlMDAttach
\emph default 
 operation is used to create a memory descriptor and attach it to a match
 list entry.
 An error code is returned if this match list entry already has an associated
 memory descriptor.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INUSE Indicates that 
\family typewriter 
match
\family default 
 already has a memory descriptor attached.
 
\layout Description

PTL_INV_ME Indicates that 
\family typewriter 
match
\family default 
 is not a valid match entry handle.
 
\layout Description

PTL_ILL_MD Indicates that 
\family typewriter 
mem_desc
\family default 
 is not a legal memory descriptor.
 This may happen because the memory region defined in 
\family typewriter 
mem_desc
\family default 
 is invalid or because the network interface associated with the 
\family typewriter 
eventq
\family default 
 in 
\family typewriter 
mem_desc
\family default 
 is not the same as the network interface associated with 
\family typewriter 
match
\family default 
.
 
\layout Description

PTL_NOSPACE Indicates that there is insufficient memory to allocate the
 memory descriptor.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="5" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
match
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the match entry that the memory descriptor will be associated
 with.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Provides initial values for the application visible parts of a memory descriptor.
  Other than its use for initialization, there is no linkage between this
 structure and the memory descriptor maintained by the API.
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
unlink_op
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A flag to indicate whether the memory descriptor is  unlinked when it becomes
 inactive, either because the operation threshold drops to zero or because
 the maximum offset has been exceeded.
  (Note, the check for unlinking a memory descriptor only occurs after a
 the completion of a successful operation.
  If the threshold is set to zero during initialization or  using 
\emph on 
PtlMDUpdate
\emph default 
, the memory descriptor is 
\series bold 
not
\series default 
  unlinked.) 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
unlink_nofit
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A flag to indicate whether the memory descriptor is  unlinked when the space
 remaining in the memory descriptor is not sufficient for a matching operation.
 If an incoming message arrives arrives at a memory descriptor that does
 not have sufficient space and the 
\series bold 
PTL_MD_TRUNCATE
\series default 
 operation is not specified, the memory descriptor will be unlinked.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
handle
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold a handle for the newly created
 memory descriptor.
  The 
\family typewriter 
handle
\family default 
 argument can be NULL, in which case the handle will not be returned.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlMDBind
\begin_inset LatexCommand \label{sec:mdbind}

\end_inset 


\layout LyX-Code

int PtlMDBind( ptl_handle_ni_t  interface,
\newline 
               ptl_md_t         mem_desc,
\newline 
               ptl_handle_md_t* handle );
\layout Standard
\noindent 
The 
\emph on 
PtlMDBind
\emph default 
 operation is used to create a 
\begin_inset Quotes eld
\end_inset 

free floating
\begin_inset Quotes erd
\end_inset 

 memory descriptor, i.e., a memory descriptor that is not associated with
 a match list entry.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_NI Indicates that 
\family typewriter 
interface
\family default 
 is not a valid match entry handle.
 
\layout Description

PTL_ILL_MD Indicates that 
\family typewriter 
mem_desc
\family default 
 is not a legal memory descriptor.
 This may happen because the memory region defined in 
\family typewriter 
mem_desc
\family default 
 is invalid or because the network interface associated with the 
\family typewriter 
eventq
\family default 
 in 
\family typewriter 
mem_desc
\family default 
 is not the same as the network interface, 
\family typewriter 
interface
\family default 
.
 
\layout Description

PTL_INV_EQ Indicates that the event queue associated with 
\family typewriter 
mem_desc
\family default 
 is not valid.
 
\layout Description

PTL_NOSPACE Indicates that there is insufficient memory to allocate the
 memory descriptor.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
handle
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="3" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
interface
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the network interface with which the memory descriptor will
 be associated.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Provides initial values for the application visible parts of a memory descriptor.
  Other than its use for initialization, there is no linkage between this
 structure and the  memory descriptor maintained by the API.
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
handle
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold a  handle for the newly created
 memory descriptor.
  The 
\family typewriter 
handle
\family default 
 argument must be a valid address and cannot be NULL.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlMDUnlink
\begin_inset LatexCommand \label{sec:mdfree}

\end_inset 


\layout LyX-Code

int PtlMDUnlink( ptl_handle_md_t mem_desc );
\layout Standard
\noindent 
The 
\emph on 
PtlMDUnlink
\emph default 
 function unlinks the memory descriptor from any match list entry it may
 be linked to and releases the resources associated with a memory descriptor.
 (This function does not free the memory region associated with the memory
 descriptor.) This function also releases the resources associated with a
 floating memory descriptor.
 Only memory descriptors with no pending operations may be unlinked.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_MD Indicates that 
\family typewriter 
mem_desc
\family default 
 is not a valid memory descriptor handle.
\layout Description

PTL_MD_INUSE Indicates that 
\family typewriter 
mem_desc
\family default 
 has pending operations and cannot be unlinked.
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="1" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the memory descriptor to be released.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlMDUpdate
\begin_inset LatexCommand \label{sec:mdupdate}

\end_inset 


\layout LyX-Code

int PtlMDUpdate( ptl_handle_md_t mem_desc,
\newline 
                 ptl_md_t*       old_md,
\newline 
                 ptl_md_t*       new_md,
\newline 
                 ptl_handle_eq_t testq );
\layout Standard
\noindent 
The 
\emph on 
PtlMDUpdate
\emph default 
 function provides a conditional, atomic update operation for memory descriptors.
 The memory descriptor identified by 
\family typewriter 
mem_desc
\family default 
 is only updated if the event queue identified by 
\family typewriter 
testq
\family default 
 is empty.
 The intent is to only enable updates to the memory descriptor when no new
 messages have arrived since the last time the queue was checked.
 See section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:exmpi}

\end_inset 

 for an example of how this function can be used.
\layout Standard

If 
\family typewriter 
new
\family default 
 is not NULL the memory descriptor identified by handle will be updated
 to reflect the values in the structure pointed to by 
\family typewriter 
new
\family default 
 if 
\family typewriter 
testq
\family default 
 has the value 
\family typewriter 
PTL_EQ_NONE
\family default 
 or if the event queue identified by 
\family typewriter 
testq
\family default 
 is empty.
 If 
\family typewriter 
old
\family default 
 is not NULL, the current value of the memory descriptor identified by 
\family typewriter 
mem_desc
\family default 
 is recorded in the location identified by 
\family typewriter 
old
\family default 
.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_NOUPDATE Indicates that the update was not performed because 
\family typewriter 
testq
\family default 
 was not empty.
 
\layout Description

PTL_INV_MD Indicates that 
\family typewriter 
mem_desc
\family default 
 is not a valid memory descriptor handle.
 
\layout Description

PTL_ILL_MD Indicates that the value pointed to by 
\family typewriter 
new
\family default 
 is not a legal memory descriptor (e.g., the memory region specified by the
 memory descriptor may be invalid).
 
\layout Description

PTL_INV_EQ Indicates that 
\family typewriter 
testq
\family default 
 is not a valid event queue handle.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
new
\family default 
 or 
\family typewriter 
old
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="4" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the memory descriptor to update.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
old_md
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
If 
\family typewriter 
old_md
\family default 
 is not the value 
\family typewriter 
NULL
\family default 
, the current value of the memory descriptor will be stored in the location
 identified by 
\family typewriter 
old
\family default 
_md.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
new_md
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
If 
\family typewriter 
new_md
\family default 
 is not the value 
\family typewriter 
NULL
\family default 
, this argument provides the new values for the memory descriptor, if the
 update is performed.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
testq
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for an event queue used to predicate the update.
 If 
\family typewriter 
testq
\family default 
 is equal to 
\family typewriter 
PTL_EQ_NONE
\family default 
, the update is performed unconditionally.
  Otherwise, the update is performed if and only if 
\family typewriter 
testq
\family default 
 is empty.
  If the update is  not performed, the function returns the value 
\family typewriter 
PTL_NOUPDATE
\family default 
.
  (Note, the 
\family typewriter 
testq
\family default 
 argument does not need to be the same as  the event queue associated with
 the memory descriptor.)
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Standard

The conditional update can be used to ensure that the memory descriptor
 has not changed between the time it was examined and the time it is updated.
 In particular, it is needed to support an MPI implementation where the
 activity of searching an unexpected message queue and posting a receive
 must be atomic.
\layout Section

Events and Event Queues
\begin_inset LatexCommand \label{sec:eq}

\end_inset 


\layout Standard

Event queues are used to log operations performed on memory descriptors.
 They can also be used to hold acknowledgements for completed 
\emph on 
put
\emph default 
 operations and to note when the data specified in a 
\emph on 
put
\emph default 
 operation has been sent (i.e., when it is safe to reuse the buffer that holds
 this data).
 Multiple memory descriptors can share a single event queue.
\layout Standard

In addition to the 
\family typewriter 
ptl_handle_eq_t
\family default 
 type, the Portals API defines two types associated with events: The 
\family typewriter 

\newline 
ptl_event_kind_t
\family default 
 type defines the kinds of events that can be stored in an event queue.
 The 
\family typewriter 
ptl_event_t
\family default 
 type defines a structure that holds the information associated with an
 event.
\layout Standard

The Portals API also provides four functions for dealing with event queues:
 The 
\emph on 
PtlEQAlloc
\emph default 
 function is used to allocate the API resources needed for an event queue,
 the 
\emph on 
PtlEQFree
\emph default 
 function is used to release these resources, the 
\emph on 
PtlEQGet
\emph default 
 function can be used to get the next event from an event queue, and the
 
\emph on 
PtlEQWait
\emph default 
 function can be used to block a process (or thread) until an event queue
 has at least one event.
\layout Subsection

Kinds of Events
\begin_inset LatexCommand \label{sec:ek-type}

\end_inset 


\layout LyX-Code

typedef enum { 
\newline 
    PTL_EVENT_GET_START, PTL_EVENT_GET_END, PTL_EVENT_GET_FAIL,
\newline 
    PTL_EVENT_PUT_START, PTL_EVENT_PUT_END, PTL_EVENT_PUT_FAIL,
\newline 
    PTL_EVENT_REPLY_START, PTL_EVENT_REPLY_END, PTL_EVENT_REPLY_FAIL,
\newline 
    PTL_EVENT_SEND_START, PTL_EVENT_SEND_END, PTL_EVENT_SEND_FAIL,
\newline 
    PTL_EVENT_ACK,
\newline 
    PTL_EVENT_UNLINK
\newline 
} ptl_event_kind_t;
\layout Standard
\noindent 
The Portals API defines fourteen types of events that can be logged in an
 event queue: 
\layout Description

PTL_EVENT_GET_START A remote 
\emph on 
get
\emph default 
 operation has been started on the memory descriptor.
 The memory region associated with this descriptor should not be altered
 until the corresponding END or FAIL event is logged.
\layout Description

PTL_EVENT_GET_END A previously initiated 
\emph on 
get
\emph default 
 operation completed successfully.
 This event is logged after the reply has been sent by the local node.
 As such, the process could free the memory descriptor once it sees this
 event.
 
\layout Description

PTL_EVENT_GET_FAIL A previously initiated 
\emph on 
get
\emph default 
 operation completed unsuccessfully.
 This event is logged after the reply has been sent by the local node.
 As such, the process could free the memory descriptor once it sees this
 event.
 
\layout Description

PTL_EVENT_PUT_START A remote 
\emph on 
put
\emph default 
 operation has been started on the memory descriptor.
 The memory region associated with this descriptor should should be considered
 volatile until the corresponding END or FAIL event is logged.
\layout Description

PTL_EVENT_PUT_END A previously initiated 
\emph on 
put
\emph default 
 operation completed successfully.
 The underlying layers will not alter the memory (on behalf of this operation)
 once this event has been logged.
 
\layout Description

PTL_EVENT_PUT_FAIL A previously initiated 
\emph on 
put
\emph default 
 operation completed unsuccessfully.
 The underlying layers will not alter the memory (on behalf of this operation)
 once this event has been logged.
 
\layout Description

PTL_EVENT_REPLY_START A 
\emph on 
reply
\emph default 
 operation has been started on the memory descriptor.
 
\layout Description

PTL_EVENT_REPLY_END A previously initiated 
\emph on 
reply
\emph default 
 operation has completed successfully .
 This event is logged after the data (if any) from the reply has been written
 into the memory descriptor.
 
\layout Description

PTL_EVENT_REPLY_FAIL A previously initiated 
\emph on 
reply
\emph default 
 operation has completed unsuccessfully.
 This event is logged after the data (if any) from the reply has been written
 into the memory descriptor.
 
\layout Description

PTL_EVENT_ACK An 
\emph on 
acknowledgement
\emph default 
 was received.
 This event is logged when the acknowledgement is received 
\layout Description

PTL_EVENT_SEND_START An outgoing 
\emph on 
send
\emph default 
 operation has been started.
 The memory region associated with this descriptor should not be altered
 until the corresponding END or FAIL event is logged.
\layout Description

PTL_EVENT_SEND_END A previously initiated 
\emph on 
send
\emph default 
 operation has completed successfully.
 This event is logged after the entire buffer has been sent and it is safe
 for the application to reuse the buffer.
 
\layout Description

PTL_EVENT_SEND_FAIL A previously initiated 
\emph on 
send
\emph default 
 operation has completed unsuccessfully.
 The process can safely manipulate the memory or free the memory descriptor
 once it sees this event.
\layout Description

PTL_EVENT_UNLINK A memory descriptor associated with this event queue has
 been automatically unlinked.
 This event is not generated when a memory descriptor is explicitly unlinked
 by calling 
\shape italic 
PtlMDUnlink
\shape default 
.
 This event does not decrement the threshold count.
\layout Subsection

Event Ordering
\layout Standard

The Portals API guarantees that a when a process initiates two operations
 on a remote process, the operations will be initiated on the remote process
 in the same order that they were initiated on the original process.
 As an example, if process A intitates two 
\emph on 
put
\emph default 
 operations, 
\emph on 
x
\emph default 
 and 
\emph on 
y
\emph default 
, on process B, the Portals API guarantees that process A will receive the
 
\family typewriter 
PTL_EVENT_SEND_START
\family default 
 events for 
\emph on 
x
\emph default 
 and 
\emph on 
y
\emph default 
 in the same order that process B receives the 
\family typewriter 
PTL_EVENT_PUT_START
\family default 
 events for 
\emph on 
x
\emph default 
 and 
\emph on 
y
\emph default 
.
 Notice that the API does not guarantee that the start events will be delivered
 in the same order that process A initiated the 
\emph on 
x
\emph default 
 and 
\emph on 
y
\emph default 
 operations.
 If process A needs to ensure the ordering of these operations, it should
 include code to wait for the initiation of 
\emph on 
x
\emph default 
 before it initiates 
\emph on 
y
\emph default 
.
\layout Subsection

Failure Notification
\layout Standard

Operations may fail to complete successfully; however, unless the node itself
 fails, every operation that is started will eventually complete.
 While an operation is in progress, the memory associated with the operation
 should not be viewed (in the case of a put or a reply) or altered (in the
 case of a send or get).
 Operation completion, whether successful or unsuccessful, is final.
 That is, when an operation completes, the memory associated with the operation
 will no longer be read or altered by the operation.
 A network interface can use the 
\family typewriter 
ptl_ni_fail_t
\family default 
 to define more specific information regarding the failure of the operation
 and record this information in the 
\family typewriter 
ni_fail_type
\family default 
 field of the event.
\layout Subsection

The Event Type
\begin_inset LatexCommand \label{sec:event-type}

\end_inset 


\layout LyX-Code

typedef struct {
\newline 
    ptl_event_kind_t      type;
\newline 
    ptl_process_id_t      initiator;
\newline 
    ptl_uid_t             uid;
\layout LyX-Code

    ptl_pt_index_t        portal;
\newline 
    ptl_match_bits_t      match_bits;
\newline 
    ptl_size_t            rlength;
\newline 
    ptl_size_t            mlength;
\newline 
    ptl_size_t            offset; 
\newline 
    ptl_handle_md_t       md_handle;
\newline 
    ptl_md_t              mem_desc;
\newline 
    ptl_hdr_data_t        hdr_data;
\newline 
    ptl_seq_t             link;
\newline 
    ptl_ni_fail_t         ni_fail_type;
\newline 
    volatile ptl_seq_t    sequence;
\newline 
} ptl_event_t;
\layout Standard
\noindent 
An event structure includes the following members: 
\layout Description

type Indicates the type of the event.
 
\layout Description

initiator The id of the initiator.
 
\layout Description

portal The Portal table index specified in the request.
 
\layout Description

match_bits A copy of the match bits specified in the request.
 See section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:me}

\end_inset 

 for more information on match bits.
 
\layout Description

rlength The length (in bytes) specified in the request.
 
\layout Description

mlength The length (in bytes) of the data that was manipulated by the operation.
 For truncated operations, the manipulated length will be the number of
 bytes specified by the memory descriptor (possibly with an offset) operation.
 For all other operations, the manipulated length will be the length of
 the requested operation.
 
\layout Description

offset Is the displacement (in bytes) into the memory region that the operation
 used.
 The offset can be determined by the operation (see Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:datamovement}

\end_inset 

) for a remote managed memory descriptor, or by the local memory descriptor
 (see Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:md}

\end_inset 

).
 
\layout Description

md_handle Is the handle to the memory descriptor associated with the event.
\layout Description

mem_desc Is the state of the memory descriptor immediately after the event
 has been processed.
 
\layout Description

hdr_data 64 bits of out-of-band user data (see Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:put}

\end_inset 

).
 
\layout Description

link The 
\emph on 
link
\emph default 
 member is used to link 
\family typewriter 
START
\family default 
 events with the 
\family typewriter 
END
\family default 
 or 
\family typewriter 
FAIL
\family default 
 event that signifies completion of the operation.
 The 
\emph on 
link
\emph default 
 member will be the same for the two events associated with an operation.
 The link member is also used to link an 
\family typewriter 
UNLINK
\family default 
 event with the event that caused the memory descriptor to be unlinked.
\layout Description

sequence The sequence number for this event.
 Sequence numbers are unique to each event.
\layout Comment

The 
\emph on 
sequence
\emph default 
 member is the last member and is volatile to support SMP implementations.
 When an event structure is filled in, the 
\emph on 
sequence
\emph default 
 member should be written after all other members have been updated.
 Moreover, a memory barrier should be inserted between the updating of other
 members and the updating of the 
\emph on 
sequence
\emph default 
 member.
\layout Subsection

PtlEQAlloc
\begin_inset LatexCommand \label{sec:eqalloc}

\end_inset 


\layout LyX-Code

int PtlEQAlloc( ptl_handle_ni_t  interface,
\newline 
                ptl_size_t       count,
\newline 
                ptl_handle_eq_t* handle );
\layout Standard
\noindent 
The 
\emph on 
PtlEQAlloc
\emph default 
 function is used to build an event queue.
 
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_NI Indicates that 
\family typewriter 
interface
\family default 
 is not a valid network interface handle.
 
\layout Description

PTL_NOSPACE Indicates that there is insufficient memory to allocate the
 event queue.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
handle
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="3" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
interface
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the interface with which the event queue  will be associated.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
count
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The number of events that can be stored in the event queue.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
handle
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold a handle for the newly created
 event queue.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlEQFree
\begin_inset LatexCommand \label{sec:eqfree}

\end_inset 


\layout LyX-Code

int PtlEQFree( ptl_handle_eq_t eventq );
\layout Standard
\noindent 
The 
\emph on 
PtlEQFree
\emph default 
 function releases the resources associated with an event queue.
 It is up to the user to insure that no memory descriptors are associated
 with the event queue once it is freed.
 
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_EQ Indicates that 
\family typewriter 
eventq
\family default 
 is not a valid event queue handle.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="1" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
eventq
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard

A handle for the event queue to be released.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlEQGet
\begin_inset LatexCommand \label{sec:eqget}

\end_inset 


\layout LyX-Code

int PtlEQGet( ptl_handle_eq_t eventq,
\newline 
              ptl_event_t*    event );
\layout Standard
\noindent 
The 
\emph on 
PTLEQGet
\emph default 
 function is a nonblocking function that can be used to get the next event
 in an event queue.
 The event is removed from the queue.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_EQ_DROPPED Indicates success (i.e., an event is returned) and that at
 least one event between this event and the last event obtained (using 
\emph on 
PtlEQGet
\emph default 
 or 
\emph on 
PtlEQWait
\emph default 
) from this event queue has been dropped due to limited space in the event
 queue.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_EQ_EMPTY Indicates that 
\family typewriter 
eventq
\family default 
 is empty or another thread is waiting on 
\emph on 
PtlEQWait
\emph default 
.
 
\layout Description

PTL_INV_EQ Indicates that 
\family typewriter 
eventq
\family default 
 is not a valid event queue handle.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
event
\family default 
 is not a legal address.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="2" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.5in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
eventq
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the event queue.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
event
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold the  values associated with
 the next event in the event queue.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlEQWait
\begin_inset LatexCommand \label{sec:eqwait}

\end_inset 


\layout LyX-Code

int PtlEQWait( ptl_handle_eq_t eventq,
\newline 
               ptl_event_t*    event );
\layout Standard
\noindent 
The 
\emph on 
PTLEQWait
\emph default 
 function can be used to block the calling process (thread) until there
 is an event in an event queue.
 This function also returns the next event in the event queue and removes
 this event from the queue.
 This is the only blocking operation in the Portals 3.2 API.
 In the event that multiple threads are waiting on the same event queue,
 PtlEQWait is guaranteed to wake exactly one thread, but the order in which
 they are awakened is not specified.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_EQ_DROPPED Indicates success (i.e., an event is returned) and that at
 least one event between this event and the last event obtained (using 
\emph on 
PtlEQGet
\emph default 
 or 
\emph on 
PtlEQWait
\emph default 
) from this event queue has been dropped due to limited space in the event
 queue.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_EQ Indicates that 
\family typewriter 
eventq
\family default 
 is not a valid event queue handle.
 
\layout Description

PTL_SEGV Indicates that 
\family typewriter 
event
\family default 
 is not a legal address.
 queue handle.
 
\layout Subsubsection

Arguments
\layout Standard
\noindent 

\begin_inset  Tabular
<lyxtabular version="3" rows="2" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
eventq
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the event queue to wait on.
  The calling process (thread) will be blocked until 
\family typewriter 
eventq
\family default 
 is not empty.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
event
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
output
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
On successful return, this location will hold the values associated with
 the next event in the event queue.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Section

The Access Control Table
\begin_inset LatexCommand \label{sec:ac}

\end_inset 


\layout Standard

Processes can use the access control table to control which processes are
 allowed to perform operations on Portal table entries.
 Each communication interface has a Portal table and an access control table.
 The access control table for the default interface contains an entry at
 index zero that allows all processes with the same user id to communicate.
 Entries in the access control table can be manipulated using the 
\emph on 
PtlACEntry
\emph default 
 function.
\layout Subsection

PtlACEntry
\begin_inset LatexCommand \label{sec:acentry}

\end_inset 


\layout LyX-Code

int PtlACEntry( ptl_handle_ni_t  interface,
\newline 
                ptl_ac_index_t   index,
\newline 
                ptl_process_id_t matchid,
\newline 
                ptl_uid_t        user_id,
\newline 
                ptl_pt_index_t   portal );
\layout Standard
\noindent 
The 
\emph on 
PtlACEntry
\emph default 
 function can be used to update an entry in the access control table for
 an interface.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_NI Indicates that 
\family typewriter 
interface
\family default 
 is not a valid network interface handle.
 
\layout Description

PTL_AC_INV_INDEX Indicates that 
\family typewriter 
index
\family default 
 is not a valid access control table index.
 
\layout Description

PTL_INV_PROC Indicates that 
\family typewriter 
matchid
\family default 
 is not a valid process identifier.
 
\layout Description

PTL_PT_INV_INDEX Indicates that 
\family typewriter 
portal
\family default 
 is not a valid Portal table index.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="5" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
interface
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Identifies the interface to use.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
index
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The index of the entry in the access control table to update.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
matchid
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Identifies the process(es) that are allowed to  perform operations.
 The constants 
\family typewriter 
PTL_PID_ANY
\family default 
 and 
\family typewriter 
PTL_NID_ANY
\family default 
 can be used to wildcard either of the ids in the 
\family typewriter 
ptl_process_id_t
\family default 
 structure.
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
user_id
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Identifies the user that is allowed to  perform operations.
 The value 
\family typewriter 
PTL_UID_ANY
\family default 
 can be used to wildcard the user.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
portal
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Identifies the Portal index(es) that can be used.
  The value 
\family typewriter 
PTL_PT_INDEX_ANY
\family default 
 can be used to wildcard the  Portal index.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Section

Data Movement Operations
\begin_inset LatexCommand \label{sec:datamovement}

\end_inset 


\layout Standard

The Portals API provides two data movement operations: 
\emph on 
PtlPut
\emph default 
 and 
\emph on 
PtlGet
\emph default 
.
\layout Subsection

PtlPut
\begin_inset LatexCommand \label{sec:put}

\end_inset 


\layout LyX-Code

typedef enum { PTL_ACK_REQ, PTL_NOACK_REQ } ptl_ack_req_t;
\newline 

\newline 
int PtlPut( ptl_handle_md_t  mem_desc,
\newline 
            ptl_ack_req_t    ack_req,
\newline 
            ptl_process_id_t target,
\newline 
            ptl_pt_index_t   portal,
\newline 
            ptl_ac_index_t   cookie,
\newline 
            ptl_match_bits_t match_bits,
\newline 
            ptl_size_t       offset,
\newline 
            ptl_hdr_data_t   hdr_data );
\layout Standard
\noindent 
Values of the type 
\family typewriter 
ptl_ack_req_t
\family default 
 are used to control whether an acknowledgement should be sent when the
 operation completes (i.e., when the data has been written to a memory descriptor
 of the 
\family typewriter 
target
\family default 
 process).
 The value 
\family typewriter 
PTL_ACK_REQ
\family default 
 requests an acknowledgement, the value 
\family typewriter 
PTL_NOACK_REQ
\family default 
 requests that no acknowledgement should be generated.
\layout Standard

The 
\emph on 
PtlPut
\emph default 
 function initiates an asynchronous put operation.
 There are several events associated with a put operation: initiation of
 the send on the local node (
\family typewriter 
PTL_EVENT_SEND_START
\family default 
), completion of the send on the local node (
\family typewriter 
PTL_EVENT_SEND_END
\family default 
 or 
\family typewriter 
PTL_EVENT_SEND_FAIL
\family default 
), and, when the send completes successfully, the receipt of an acknowledgement
 (
\family typewriter 
PTL_EVENT_ACK
\family default 
) indicating that the operation was accepted by the target.
 These events will be logged in the event queue associated with the memory
 descriptor (
\family typewriter 
mem_desc
\family default 
) used in the put operation.
 Using a memory descriptor that does not have an associated event queue
 results in these events being discarded.
 In this case, the application must have another mechanism (e.g., a higher
 level protocol) for determining when it is safe to modify the memory region
 associated with the memory descriptor.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_MD Indicates that 
\family typewriter 
mem_desc
\family default 
 is not a valid memory descriptor.
 
\layout Description

PTL_INV_PROC Indicates that 
\family typewriter 
target
\family default 
 is not a valid process id.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="8" columns="3">
<features>
<column alignment="center" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the memory descriptor that describes the memory to be sent.
  If the memory descriptor has an event queue  associated with it, it will
 be used to record events when the  message has been sent (PTL_EVENT_SEND_START,
 PTL_EVENT_SEND_END).
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ack_req
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
Controls whether an acknowledgement event is requested.
  Acknowledgements are only sent when they are requested by the initiating
 process 
\series bold 
and
\series default 
 the memory descriptor has an event queue 
\series bold 
and
\series default 
 the target memory descriptor enables them.
 Allowed constants: 
\family typewriter 
PTL_ACK_REQ
\family default 
, 
\family typewriter 
PTL_NOACK_REQ
\family default 
.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
target
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A process id for the target process.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
portal
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The index in the remote Portal table.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
cookie
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The index into the access control table of the target process.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
match_bits
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The match bits to use for message selection at the target process.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
offset
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The offset into the target memory descriptor (only used when the target
 memory descriptor has the 
\family typewriter 
PTL_MD_MANAGE_REMOTE
\family default 
 option set).
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
hdr_data
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
64 bits of user data that can be included in message header.
  This data is written to an event queue entry at the target if an event
 queue is present on the matching memory descriptor.
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Subsection

PtlGet
\begin_inset LatexCommand \label{sec:get}

\end_inset 


\layout LyX-Code

int PtlGet( ptl_handle_md_t  mem_desc,
\newline 
            ptl_process_id_t target,
\newline 
            ptl_pt_index_t   portal,
\newline 
            ptl_ac_index_t   cookie,
\newline 
            ptl_match_bits_t match_bits,
\newline 
            ptl_size_t       offset );
\layout Standard
\noindent 
The 
\emph on 
PtlGet
\emph default 
 function initiates a remote read operation.
 There are two event pairs associated with a get operation , when the data
 is sent from the remote node, a 
\family typewriter 
PTL_EVENT_GET{START|END}
\family default 
 event pair is registered on the remote node; and when the data is returned
 from the remote node a 
\family typewriter 
PTL_EVENT_REPLY{START|END}
\family default 
 event pair is registered on the local node.
\layout Subsubsection

Return Codes
\layout Description

PTL_OK Indicates success.
 
\layout Description

PTL_NOINIT Indicates that the Portals API has not been successfully initialized.
 
\layout Description

PTL_INV_MD Indicates that 
\family typewriter 
mem_desc
\family default 
 is not a valid memory descriptor.
 
\layout Description

PTL_INV_PROC Indicates that 
\family typewriter 
target
\family default 
 is not a valid process id.
 
\layout Subsubsection

Arguments
\layout Standard


\begin_inset  Tabular
<lyxtabular version="3" rows="6" columns="3">
<features>
<column alignment="right" valignment="top" width="0pt">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="4.7in">
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A handle for the memory descriptor that describes the memory into which
 the requested data will be received.
  The memory descriptor can have an event queue associated with it to record
 events, such as when the message receive has started (
\family typewriter 
PTL_EVENT_REPLY
\family default 
_
\family typewriter 
START
\family default 
).
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
target
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
A process id for the target process.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
portal
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The index in the remote Portal table.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
cookie
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The index into the access control table of the target process.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
match_bits
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The match bits to use for message selection at the target process.
\end_inset 
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
offset
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
input
\end_inset 
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
The offset into the target memory descriptor (only used when the target
 memory descriptor has the 
\family typewriter 
PTL_MD_MANAGE_REMOTE
\family default 
 option set).
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\layout Section

Summary
\layout Standard


\begin_inset LatexCommand \label{sec:summary}

\end_inset 

 We conclude this section by summarizing the names introduced by the Portals
 3.2 API.
 We start by summarizing the names of the types introduced by the API.
 This is followed by a summary of the functions introduced by the API.
 Which is followed by a summary of the function return codes.
 Finally, we conclude with a summary of the other constant values introduced
 by the API.
\layout Standard

Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:types}

\end_inset 

 presents a summary of the types defined by the Portals API.
 The first column in this table gives the type name, the second column gives
 a brief description of the type, the third column identifies the section
 where the type is defined, and the fourth column lists the functions that
 have arguments of this type.
\layout Standard


\begin_inset Float table
placement htbp
wide false
collapsed false

\layout Caption

Types Defined by the Portals 3.2 API
\begin_inset LatexCommand \label{tab:types}

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
medskip  
\end_inset 


\layout Standard
\noindent 

\size small 

\begin_inset  Tabular
<lyxtabular version="3" rows="25" columns="4">
<features firstHeadEmpty="true">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="2in">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="2.2in">
<row bottomline="true">
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
 Name
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
 Meaning 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
 Sect
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
 Functions 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_ac_index_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
indexes for an access control table 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:index-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlACEntry, PtlPut, PtlGet 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_ack_req_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
acknowledgement request types 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlPut
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
kinds of events
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlGet
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
information about events 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:event-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlEQGet
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
plt_seq_t
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

event sequence number
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:event-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

PtlEQGet, PtlEQWait
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_handle_any_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
handles for any object 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:handle-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIHandle 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_handle_eq_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
handles for event queues 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:handle-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlEQAlloc, PtlEQFree, PtlEQGet, PtlEQWait, PtlMDUpdate 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_handle_md_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
handles for memory descriptors 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:handle-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMDAlloc, PtlMDUnlink, PtlMDUpdate, PtlMEAttach, PtlMEAttachAny, PtlMEInsert,
 PtlPut, PtlGet 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_handle_me_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
handles for match entries 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:handle-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMEAttach, PtlMEAttachAny, PtlMEInsert, PtlMEUnlink 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_handle_ni_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
handles for network interfaces 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:handle-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIInit, PtlNIFini, PtlNIStatus, PtlNIDist, PtlEQAlloc, PtlACEntry, PtlPut,
 PtlGet 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_nid_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
node identifiers
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:id-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 PtlGetId,PtlACEntry
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_pid_t
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

process identifier
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:id-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

PtlGetId, PtlACEntry
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_uid_t
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

user indentifier
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:id-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

PtlGetUid, PtlACEntry
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_ins_pos_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
insertion position (before or after) 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:meattach}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMEAttach, PtlMEAttachAny, PtlMEInsert 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_interface_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
identifiers for network interfaces 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ni-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIInit 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_match_bits_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
match (and ignore) bits 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:mb-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMEAttach, PtlMEAttachAny, PtlMEInsert, PtlPut, PtlGet 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\family typewriter 
ptl_md_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
memory descriptors 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMDAttach, PtlMDUpdate 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_ni_fail_t
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

network interface-specific failures
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:eq}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

PtlEQGet, PtlEQWait
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_process_id_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
process identifiers 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:pid-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlGetId, PtlNIDist, PtlMEAttach, PtlMEAttachAny, PtlACEntry, PtlPut, PtlGet
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_pt_index_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
indexes for Portal tables 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:index-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMEAttach, PtlMEAttachAny, PtlACEntry 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_size_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
sizes 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:size-t}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlEQAlloc, PtlPut, PtlGet 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_sr_index_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
indexes for status registers 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:stat-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIStatus 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_sr_value_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
values in status registers 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:stat-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIStatus 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\family typewriter 
ptl_unlink_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
unlink options 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:meattach}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMEAttach, PtlMEAttachAny, PtlMEInsert, PtlMDAttach 
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\end_inset 


\layout Standard

Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:func}

\end_inset 

 presents a summary of the functions defined by the Portals API.
 The first column in this table gives the name for the function, the second
 column gives a brief description of the operation implemented by the function,
 and the third column identifies the section where the function is defined.
\layout Standard


\begin_inset Float table
placement htbp
wide false
collapsed false

\layout Caption

Functions Defined by the Portals 3.2 API
\begin_inset LatexCommand \label{tab:func}

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
medskip  
\end_inset 


\layout Standard
\align center 

\size small 

\begin_inset  Tabular
<lyxtabular version="3" rows="24" columns="3">
<features firstHeadEmpty="true">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<row bottomline="true">
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

Name 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 Operation 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 Section 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

PtlACEntry 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 update an entry in an access control table 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ac}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlEQAlloc 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 create an event queue 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:eq}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlEQGet 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 get the next event from an event queue 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:eq}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlEQFree 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 release the resources for an event queue 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:eq}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlEQWait 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 wait for a new event in an event queue 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:eq}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlFini 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 shutdown the Portals API 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:init}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlGet 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 perform a get operation 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:datamovement}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlGetId 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 get the id for the current process 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:pid}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlInit 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 initialize the Portals API 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:init}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlMDAttach 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 create a memory descriptor and attach it to a match entry 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlMDBind 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 create a free-floating memory descriptor 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:mdbind}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlMDUnlink 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 remove a memory descriptor from a list and release its resources 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlMDUpdate 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 update a memory descriptor 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlMEAttach 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

create a match entry and attach it to a Portal table 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:me}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

PtlMEAttachAny
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

create a match entry and attach it to a free Portal table entry
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:attachany}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlMEInsert 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 create a match entry and insert it in a list 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:me}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlMEUnlink 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 remove a match entry from a list and release its resources 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:me}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlNIDist 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 get the distance to another process 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ni}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlNIFini 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 shutdown a network interface 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ni}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlNIHandle 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 get the network interface handle for an object 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ni}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlNIInit 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 initialize a network interface 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ni}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlNIStatus 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 read a network interface status register 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ni}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 PtlPut 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 perform a put operation 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:datamovement}

\end_inset 


\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\end_inset 


\layout Standard

Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:retcodes}

\end_inset 

 summarizes the return codes used by functions defined by the Portals API.
 All of these constants are integer values.
 The first column of this table gives the symbolic name for the constant,
 the second column gives a brief description of the value, and the third
 column identifies the functions that can return this value.
\layout Standard


\begin_inset Float table
placement htbp
wide false
collapsed false

\layout Caption

Function Return Codes for the Portals 3.2 API
\begin_inset LatexCommand \label{tab:retcodes}

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
medskip  
\end_inset 


\layout Standard
\align center 

\size small 

\begin_inset  Tabular
<lyxtabular version="3" rows="27" columns="3">
<features>
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="2.6in">
<row bottomline="true">
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Name
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Meaning 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\series bold 
Functions
\series default 
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_AC_INV_INDEX
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

invalid access control table index 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 PtlACEntry 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EQ_DROPPED
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

at least one event has been dropped 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 PtlEQGet, PtlWait 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EQ_EMPTY
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

no events available in an event queue 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 PtlEQGet 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_FAIL 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

error during initialization or cleanup 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 PtlInit, PtlFini 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_ILL_MD
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

illegal memory descriptor values 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMDAttach, PtlMDBind, PtlMDUpdate 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INIT_DUP 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

duplicate initialization of an interface 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIInit 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INIT_INV
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

initialization of an invalid interface 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIInit 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INUSE
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

the ME already has an MD
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMDAttach
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INV_ASIZE
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

invalid access control table size 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIInit 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INV_EQ
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

invalid event queue handle 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMDUpdate, PtlEQFree, PtlEQGet 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INV_HANDLE 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

invalid handle 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIHandle 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INV_MD 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

invalid memory descriptor handle 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMDUnlink, PtlMDUpdate 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INV_ME
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

invalid match entry handle 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlMDAttach 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INV_NI 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

invalid network interface handle 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIDist, PtlNIFini, PtlMDBind, PtlEQAlloc 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INV_PROC 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

invalid process identifier 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIInit, PtlNIDist, PtlMEAttach, PtlMEInsert, PtlACEntry, PtlPut, PtlGet
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INV_PTINDEX
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

invalid Portal table index 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 PtlMEAttach 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INV_REG 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

invalid status register 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 PtlNIStatus 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INV_SR_INDX 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

invalid status register index 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 PtlNIStatus 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_ML_TOOLONG 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

match list too long 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 PtlMEAttach, PtlMEInsert 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_MD_INUSE
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

MD has pending operations
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

PtlMDUnlink
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_NOINIT 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

uninitialized API 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 
\emph on 
all
\emph default 
, except PtlInit 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_NOSPACE
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

insufficient memory 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIInit, PtlMDAttach, PtlMDBind, PtlEQAlloc, PtlMEAttach, PtlMEInsert
 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\family typewriter 
PTL_NOUPDATE
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 no update was performed 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 PtlMDUpdate 
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_PT_FULL
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

Portal table is full
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

PtlMEAttachAny
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_OK 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 success 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
 
\emph on 
all
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_SEGV 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

addressing violation 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard
\noindent 
PtlNIInit, PtlNIStatus, PtlNIDist, PtlNIHandle, PtlMDBind, PtlMDUpdate,
 PtlEQAlloc, PtlEQGet, PtlEQWait 
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\end_inset 


\layout Standard

Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:oconsts}

\end_inset 

 summarizes the remaining constant values introduced by the Portals API.
 The first column in this table presents the symbolic name for the constant,
 the second column gives a brief description of the value, the third column
 identifies the type for the value, and the fourth column identifies the
 sections in which the value is mentioned.
\layout Standard


\begin_inset Float table
placement htbp
wide false
collapsed false

\layout Caption

Other Constants Defined by the Portals 3.2 API
\begin_inset LatexCommand \label{tab:oconsts}

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
medskip  
\end_inset 


\layout Standard
\align center 

\size small 

\begin_inset  Tabular
<lyxtabular version="3" rows="36" columns="5">
<features>
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<row bottomline="true">
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Name
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Meaning 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Base type
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Intr.
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Ref.
\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_ACK_REQ
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

request an acknowledgement 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_ack_req_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EQ_NONE 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

a NULL event queue handle 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_handle_eq_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:handle-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md}

\end_inset 

, 
\begin_inset LatexCommand \ref{sec:mdupdate}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_GET_START
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

get event start
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:get}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_GET_END
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

get event end
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:get}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_GET_FAIL
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

get event fail
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:get}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_PUT_START
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

put event start
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_PUT_END
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

put event end
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_PUT_FAIL
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

put event fail
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_REPLY_START
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

reply event start
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:get}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_REPLY_END
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

reply event end
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:get}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_REPLY_FAIL
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

reply event fail
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:get}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_ACK_START
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

acknowledgement event start
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_ACK_END
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

acknowledgement event end
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_ACK_FAIL
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

acknowledgement event fail
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_SEND_START
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

send event start
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_SEND_END
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

send event end
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_SEND_FAIL
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

send event fail
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_EVENT_UNLINK
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

unlink event
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_event_kind_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:ek-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:md-type}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_PID_ANY 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

wildcard for process id fields 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_pid_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:id-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:meattach}

\end_inset 

, 
\begin_inset LatexCommand \ref{sec:acentry}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_NID_ANY
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

wildcard for node id fields
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_nid_t
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:id-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:meattach}

\end_inset 

, 
\begin_inset LatexCommand \ref{sec:acentry}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_UID_ANY
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

wildcard for user id
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_uid_t
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:id-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\begin_inset LatexCommand \ref{sec:meattach}

\end_inset 

, 
\begin_inset LatexCommand \ref{sec:acentry}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_IFACE_DEFAULT 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

default interface 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_interface_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:ni-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INS_AFTER 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

insert after 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_ins_pos_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:meinsert}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_INS_BEFORE 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

insert before 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_ins_pos_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:meinsert}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_MD_ACK_DISABLE 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

a flag to disable acknowledgements 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
int
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_MD_MANAGE_REMOTE 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

a flag to enable the use of remote offsets 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
int 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:put}

\end_inset 

, 
\begin_inset LatexCommand \ref{sec:get}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_MD_OP_GET 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

a flag to enable get operations 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
int 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_MD_OP_PUT 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

a flag to enable put operations 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
int
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_MD_THRESH_INF 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

infinite threshold for a memory descriptor 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
int 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_MD_TRUNCATE 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

a flag to enable truncation of a request 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
int 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:md-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_NOACK_REQ 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

request no acknowledgement 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_ack_req_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:put}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_PT_INDEX_ANY 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

wildcard for Portal indexes 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_pt_index_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:acentry}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_RETAIN 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

disable unlinking 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_unlink_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:mdattach}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_SR_DROP_COUNT 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

index for the dropped count register 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_sr_index_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:stat-type}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:nistatus}

\end_inset 


\end_inset 
</cell>
</row>
<row>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
PTL_UNLINK 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

enable unlinking 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_unlink_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\begin_inset LatexCommand \ref{sec:mdattach}

\end_inset 


\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\end_inset 


\layout Chapter

The Semantics of Message Transmission
\begin_inset LatexCommand \label{sec:semantics}

\end_inset 


\layout Standard

The portals API uses four types of messages: put requests, acknowledgements,
 get requests, and replies.
 In this section, we describe the information passed on the wire for each
 type of message.
 We also describe how this information is used to process incoming messages.
\layout Section

Sending Messages
\layout Standard

Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:put-wire}

\end_inset 

 summarizes the information that is transmitted for a put request.
 The first column provides a descriptive name for the information, the second
 column provides the type for this information, the third column identifies
 the source of the information, and the fourth column provides additional
 notes.
 Most information that is transmitted is obtained directly from the 
\emph on 
PtlPut
\emph default 
 operation.
 Notice that the handle for the memory descriptor used in the 
\emph on 
PtlPut
\emph default 
 operation is transmitted even though this value cannot be interpreted by
 the target.
 A value of anything other than 
\family typewriter 
PTL_MD_NONE
\family default 
, is interpreted as a request for an acknowledgement.
\layout Standard


\begin_inset Float table
placement htbp
wide false
collapsed false

\layout Caption

Information Passed in a Put Request
\begin_inset LatexCommand \label{tab:put-wire}

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
medskip
\end_inset 


\layout Standard
\align center 

\size small 

\begin_inset  Tabular
<lyxtabular version="3" rows="12" columns="4">
<features firstHeadEmpty="true">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<row bottomline="true">
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Information 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Type
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
\emph on 
PtlPut
\emph default 
 arg
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Notes 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

operation 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
int 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

indicates a put request 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

initiator 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_process_id_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

local information 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

user
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_uid_t
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

local information
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

target 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_process_id_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
target
\family default 
 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

portal index 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_pt_index_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
portal 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

cookie 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_ac_index_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
cookie
\family default 
 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

match bits 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_match_bits_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
match_bits 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

offset 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_size_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
offset
\family default 
 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

memory desc 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_handle_md_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc
\family default 
 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

no ack if 
\family typewriter 
PTL_MD_NONE
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

length 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_size_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
length
\family default 
 member 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

data 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family roman 
\emph on 
bytes
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
start
\family default 
 and 
\family typewriter 
length
\family default 
 members 
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\end_inset 


\layout Standard

Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:ack-wire}

\end_inset 

 summarizes the information transmitted in an acknowledgement.
 Most of the information is simply echoed from the put request.
 Notice that the initiator and target are obtained directly from the put
 request, but are swapped in generating the acknowledgement.
 The only new piece of information in the acknowledgement is the manipulated
 length which is determined as the put request is satisfied.
\layout Standard


\begin_inset Float table
placement htbp
wide false
collapsed false

\layout Caption

Information Passed in an Acknowledgement
\begin_inset LatexCommand \label{tab:ack-wire}

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
medskip  
\end_inset 


\layout Standard
\align center 

\size small 

\begin_inset  Tabular
<lyxtabular version="3" rows="10" columns="4">
<features firstHeadEmpty="true">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<row bottomline="true">
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\series bold 
Information
\series default 
 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\series bold 
Type
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\series bold 
Put Information 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\series bold 
Notes 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

operation 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\family typewriter 
int 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 indicates an acknowledgement 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 initiator 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\family typewriter 
ptl_process_id_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 target 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 target 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 
\family typewriter 
ptl_process_id_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

 initiator 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 portal index 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\family typewriter 
ptl_pt_index_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 portal index 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 echo 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 match bits 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\family typewriter 
ptl_match_bits_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 match bits 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 echo 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 offset 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 
\family typewriter 
ptl_size_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 offset 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 echo 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 memory desc 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
 ptl_handle_md_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 memory desc 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 echo 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 requested length 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
 ptl_size_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 length 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 echo 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 manipulated length 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
 ptl_size_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

 obtained from the operation 
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\end_inset 


\layout Standard

Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:get-wire}

\end_inset 

 summarizes the information that is transmitted for a get request.
 Like the information transmitted in a put request, most of the information
 transmitted in a get request is obtained directly from the 
\emph on 
PtlGet
\emph default 
 operation.
 Unlike put requests, get requests do not include the event queue handle.
 In this case, the reply is generated whenever the operation succeeds and
 the memory descriptor must not be unlinked until the reply is received.
 As such, there is no advantage to explicitly sending the event queue handle.
\layout Standard


\begin_inset Float table
placement htbp
wide false
collapsed false

\layout Caption

Information Passed in a Get Request
\begin_inset LatexCommand \label{tab:get-wire}

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
medskip  
\end_inset 


\layout Standard
\align center 

\size small 

\begin_inset  Tabular
<lyxtabular version="3" rows="11" columns="4">
<features firstHeadEmpty="true">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<row bottomline="true">
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Information
\series default 
 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Type
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
\emph on 
PtlGet
\emph default 
 argument
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Notes 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

operation 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
int 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

indicates a get operation 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

initiator 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_process_id_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

local information 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

user
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_uid_t
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

local information
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

target 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_process_id_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
target 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

portal index 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_pt_index_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
portal
\family default 
 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

cookie 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_ac_index_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
cookie 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

match bits 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_match_bits_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
match_bits
\family default 
 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

offset 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_size_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
offset 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

memory desc 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_handle_md_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc
\family default 
 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

length 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_size_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
mem_desc 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
length
\family default 
 member 
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\end_inset 


\layout Standard

Table\SpecialChar ~

\begin_inset LatexCommand \ref{tab:reply-wire}

\end_inset 

 summarizes the information transmitted in a reply.
 Like an acknowledgement, most of the information is simply echoed from
 the get request.
 The initiator and target are obtained directly from the get request, but
 are swapped in generating the acknowledgement.
 The only new information in the acknowledgement are the manipulated length
 and the data, which are determined as the get request is satisfied.
\layout Standard


\begin_inset Float table
placement htbp
wide false
collapsed false

\layout Caption

Information Passed in a Reply
\begin_inset LatexCommand \label{tab:reply-wire}

\end_inset 


\layout Standard


\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
medskip  
\end_inset 


\layout Standard
\align center 

\size small 

\begin_inset  Tabular
<lyxtabular version="3" rows="11" columns="4">
<features firstHeadEmpty="true">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="0pt">
<row bottomline="true">
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Information
\series default 
 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Type
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Put Information 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\series bold 
Notes 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

operation 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
int
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

indicates an acknowledgement 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

initiator 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_process_id_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

target 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

target 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_process_id_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

initiator 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

portal index 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_pt_index_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

portal index 
\end_inset 
</cell>
<cell alignment="right" valignment="top" bottomline="true" usebox="none">
\begin_inset Text

\layout Standard

echo 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

match bits 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_match_bits_t 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

match bits 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

echo 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

offset 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_size_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

offset 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

echo 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

memory desc 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_handle_md_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

memory desc 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

echo 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

requested length 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_size_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

length 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

echo 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

manipulated length 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\family typewriter 
ptl_size_t
\family default 
 
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

obtained from the operation 
\end_inset 
</cell>
</row>
<row>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

data 
\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard


\emph on 
bytes
\end_inset 
</cell>
<cell alignment="left" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

\end_inset 
</cell>
<cell alignment="right" valignment="top" usebox="none">
\begin_inset Text

\layout Standard

obtained from the operation 
\end_inset 
</cell>
</row>
</lyxtabular>

\end_inset 


\end_inset 


\layout Section

Receiving Messages
\begin_inset LatexCommand \label{sec:receiving}

\end_inset 


\layout Standard

When an incoming message arrives on a network interface, the communication
 system first checks that the target process identified in the request is
 a valid process that has initialized the network interface (i.e., that the
 target process has a valid Portal table).
 If this test fails, the communication system discards the message and increment
s the dropped message count for the interface.
 The remainder of the processing depends on the type of the incoming message.
 Put and get messages are subject to access control checks and translation
 (searching a match list), while acknowledgement and reply messages bypass
 the access control checks and the translation step.
\layout Standard

Acknowledgement messages include a handle for the memory descriptor used
 in the original 
\emph on 
PtlPut
\emph default 
 operation.
 This memory descriptor will identify the event queue where the event should
 be recorded.
 Upon receipt of an acknowledgement, the runtime system only needs to confirm
 that the memory descriptor and event queue still exist and that there is
 space for another event.
 Should the any of these conditions fail,  the message is simply discarded
 and the dropped message count for the interface is incremented.
 Otherwise, the system builds an acknowledgement event from the information
 in the acknowledgement message and adds it to the event queue.
\layout Standard

Reception of reply messages is also relatively straightforward.
 Each reply message includes a handle for a memory descriptor.
 If this descriptor exists, it is used to receive the message.
 A reply message will be dropped if the memory descriptor identified in
 the request doesn't exist.
 In either of this case, the dropped message count for the interface is
 incremented.
 These are the only reasons for dropping reply messages.
 Every memory descriptor accepts and truncates incoming reply messages,
 eliminating the other potential reasons for rejecting a reply message.
\layout Standard

The critical step in processing an incoming put or get request involves
 mapping the request to a memory descriptor.
 This step starts by using the Portal index in the incoming request to identify
 a list of match entries.
 This list of match entries is searched in order until a match entry is
 found whose match criteria matches the match bits in the incoming request
 and whose memory descriptor accepts the request.
\layout Standard

Because acknowledge and reply messages are generated in response to requests
 made by the process receiving these messages, the checks performed by the
 runtime system for acknowledgements and replies are minimal.
 In contrast, put and get messages are generated by remote processes and
 the checks performed for these messages are more extensive.
 Incoming put or get messages may be rejected because: 
\layout Itemize

the Portal index supplied in the request is not valid; 
\layout Itemize

the cookie supplied in the request is not a valid access control entry;
 
\layout Itemize

the access control entry identified by the cookie does not match the identifier
 of the requesting process; 
\layout Itemize

the access control entry identified by the access control entry does not
 match the Portal index supplied in the request; or 
\layout Itemize

the match bits supplied in the request do not match any of the match entries
 with a memory descriptor that accepts the request.
 
\layout Standard

In all cases, if the message is rejected, the incoming message is discarded
 and the dropped message count for the interface is incremented.
\layout Standard

A memory descriptor may reject an incoming request for any of the following
 reasons: 
\layout Itemize

the 
\family typewriter 
PTL_MD_PUT
\family default 
 or 
\family typewriter 
PTL_MD_GET
\family default 
 option has not been enabled and the operation is put or get, respectively;
 
\layout Itemize

the length specified in the request is too long for the memory descriptor
 and the 
\family typewriter 
PTL_MD_TRUNCATE
\family default 
 option has not been enabled.
\layout Chapter

Examples
\begin_inset LatexCommand \label{sec:examples}

\end_inset 


\layout Comment

The examples presented in this chapter have not been updated to reflect
 the current API.
\layout Standard

In this section we present several example to illustrate expected usage
 patterns for the Portals 3.2 API.
 The first example describes how to implement parallel servers using the
 features of the Portals 3.2 API.
 This example covers the access control list and the use of remote managed
 offsets.
 The second example presents an approach to dealing with dropped requests.
 This example covers aspects of match lists and memory descriptors.
 The final example covers message reception in MPI.
 This example illustrates more sophisticated uses of matching and a procedure
 to update a memory descriptor.
\layout Section

Parallel File Servers
\begin_inset LatexCommand \label{sec:expfs}

\end_inset 


\layout Standard

Figure\SpecialChar ~

\begin_inset LatexCommand \ref{fig:file}

\end_inset 

 illustrates the logical structure of a parallel file server.
 In this case, the parallel server consists of four servers that stripe
 application data across four disks.
 We would like to present applications with the illusion that the file server
 is a single entity.
 We will assume that all of the processes that constitute the parallel server
 have the same user id.
\layout Standard


\begin_inset Float figure
placement htbp
wide false
collapsed false

\layout Standard
\align center 

\begin_inset Graphics FormatVersion 1
        filename file.eps
        display color
        size_type 0
        rotateOrigin center
        lyxsize_type 1
        lyxwidth 196pt
        lyxheight 147pt
\end_inset 


\layout Caption

Parallel File Server
\begin_inset LatexCommand \label{fig:file}

\end_inset 


\end_inset 


\layout Standard

When an application establishes a connection to the parallel file server,
 it will allocate a Portal and access control list entry for communicating
 with the server.
 The access control list entry will include the Portal and match any process
 in the parallel file server's, so all of the file server processes will
 have access to the portal.
 The Portal information and access control entry will be sent to the file
 server at this time.
 If the application and server need to have multiple, concurrent I/O operations,
 they can use additional portals or match entries to keep the operations
 from interfering with one another.
\layout Standard

When an application initiates an I/O operation, it first builds a memory
 descriptor that describes the memory region involved in the operation.
 This memory descriptor will enable the appropriate operation (put for read
 operations and get for write operations) and enable the use of remote offsets
 (this lets the servers decide where their data should be placed in the
 memory region).
 After creating the memory descriptor and linking it into the appropriate
 Portal entry, the application sends a read or write request (using 
\emph on 
PtlPut
\emph default 
) to one of the file server processes.
 The file server processes can then use put or get operations with the appropria
te offsets to fill or retrieve the contents of the application's buffer.
 To know when the operation has completed, the application can add an event
 queue to the memory descriptor and add up the lengths of the remote operations
 until the sum is the size of the requested I/O operation.
\layout Section

Dealing with Dropped Requests
\begin_inset LatexCommand \label{sec:exdrop}

\end_inset 


\layout Standard

If a process does not anticipate unexpected requests, they will be discarded.
 Applications using the Portals API can query the dropped count for the
 interface to determine the number of requests that have been dropped (see
 Section\SpecialChar ~

\begin_inset LatexCommand \ref{sec:nistatus}

\end_inset 

).
 While this approach minimizes resource consumption, it does not provide
 information that might be critical in debugging the implementation of a
 higher level protocol.
\layout Standard

To keep track of more information about dropped requests, we use a memory
 descriptor that truncates each incoming request to zero bytes and logs
 the 
\begin_inset Quotes eld
\end_inset 

dropped
\begin_inset Quotes erd
\end_inset 

 operations in an event queue.
 Note that the operations are not dropped in the Portals sense, because
 the operation succeeds.
\layout Standard

The following code fragment illustrates an implementation of this approach.
 In this case, we assume that a thread is launched to execute the function
 
\family typewriter 
watch_drop
\family default 
.
 This code starts by building an event queue to log truncated operations
 and a memory descriptor to truncate the incoming requests.
 This example only captures 
\begin_inset Quotes eld
\end_inset 

dropped
\begin_inset Quotes erd
\end_inset 

 requests for a single portal.
 In a more realistic situation, the memory descriptor would be appended
 to the match list for every portal.
 We also assume that the thread is capable of keeping up with the 
\begin_inset Quotes eld
\end_inset 

dropped
\begin_inset Quotes erd
\end_inset 

 requests.
 If this is not the case, we could use a finite threshold on the memory
 descriptor to capture the first few dropped requests.
\layout LyX-Code


\size small 
#include <stdio.h>
\newline 
#include <stdlib.h>
\newline 
#include <portals.h>
\newline 

\newline 
#define DROP_SIZE 32       /* number of dropped requests to track */
\newline 

\newline 
int watch_drop( ptl_handle_ni_t ni, ptl_pt_index_t index ) {
\newline 
    ptl_handle_eq_t drop_events;
\newline 
    ptl_event_t event;
\newline 
    ptl_handle_md_t drop_em;
\newline 
    ptl_md_t drop_desc;
\newline 
    ptl_process_id_t any_proc;
\newline 
    ptl_handle_me_t match_any;
\newline 

\newline 
    /* create the event queue */
\newline 
    if( PtlEQAlloc(ni, DROP_SIZE, &drop_events) != PTL_OK ) {
\newline 
        fprintf( stderr, "Couldn't create the event queue
\backslash 
n" );
\newline 
        exit( 1 );
\newline 
    }
\newline 

\newline 
    /* build a match entry */
\newline 
    any_proc.nid = PTL_ID_ANY;
\newline 
    any_proc.pid = PTL_ID_ANY;
\newline 
    PtlMEAttach( index, any_proc, 0, ~(ptl_match_bits_t)0, PTL_RETAIN,
\newline 
                        &match_any );
\newline 

\newline 
    /* create the memory descriptor */
\newline 
    drop_desc.start = NULL;
\newline 
    drop_desc.length = 0;
\newline 
    drop_desc.threshold = PTL_MD_THRESH_INF;
\newline 
    drop_desc.options = PTL_MD_OP_PUT | PTL_MD_OP_GET | PTL_MD_TRUNCATE;
\newline 
    drop_desc.user_ptr = NULL;
\newline 
    drop_desc.eventq = drop_events;
\newline 
    if( PtlMDAttach(match_any, drop_desc, &drop_em) != PTL_OK ) {
\newline 
        fprintf( stderr, "Couldn't create the memory descriptor
\backslash 
n" );
\newline 
        exit( 1 );
\newline 
    }
\newline 

\newline 
    /* watch for "dropped" requests */
\newline 
    while( 1 ) {
\newline 
        if( PtlEQWait( drop_events, &event ) != PTL_OK ) break;
\newline 
        fprintf( stderr, "Dropped request from gid = event.initiator.gid,
 event.initiator.rid );
\newline 
    }
\newline 
}
\layout Section

Message Transmission in MPI
\begin_inset LatexCommand \label{sec:exmpi}

\end_inset 


\layout Standard

We conclude this section with a fairly extensive example that describes
 an approach to implementing message transmission for MPI.
 Like many MPI implementations, we distinguish two message transmission
 protocols: a short message protocol and a long message protocol.
 We use the constant 
\family typewriter 
MPI_LONG_LENGTH
\family default 
 to determine the size of a long message.
\layout Standard

For small messages, the sender simply sends the message and presumes that
 the message will be received (i.e., the receiver has allocated a memory region
 to receive the message body).
 For large messages, the sender also sends the message, but does not presume
 that the message body will be saved.
 Instead, the sender builds a memory descriptor for the message and enables
 get operations on this descriptor.
 If the target does not save the body of the message, it will record an
 event for the put operation.
 When the process later issues a matching MPI receive, it will perform a
 get operation to retrieve the body of the message.
\layout Standard

To facilitate receive side matching based on the protocol, we use the most
 significant bit in the match bits to indicate the protocol: 1 for long
 messages and 0 for short messages.
\layout Standard

The following code presents a function that implements the send side of
 the protocol.
 The global variable 
\family typewriter 
EndGet
\family default 
 is the last match entry attached to the Portal index used for posting long
 messages.
 This entry does not match any incoming requests (i.e., the memory descriptor
 rejects all get operations) and is built during initialization of the MPI
 library.
 The other global variable, 
\family typewriter 
MPI_NI
\family default 
, is a handle for the network interface used by the MPI implementation.
\layout LyX-Code


\size small 
extern ptl_handle_me_t EndGet;
\newline 
extern ptl_handle_ni_t MPI_NI;
\newline 

\newline 
void MPIsend( void *buf, ptl_size_t len, void *data, ptl_handle_eq_t eventq,
\newline 
                    ptl_process_id target, ptl_match_bits_t match ) 
\newline 
{
\newline 
    ptl_handle_md_t send_handle;
\newline 
    ptl_md_t mem_desc;
\newline 
    ptl_ack_req_t want_ack;
\newline 

\newline 
    mem_desc.start = buf;
\newline 
    mem_desc.length = len;
\newline 
    mem_desc.threshold = 1;
\newline 
    mem_desc.options = PTL_MD_GET_OP;
\newline 
    mem_desc.user_ptr = data;
\newline 
    mem_desc.eventq = eventq;
\newline 

\newline 
    if( len >= MPI_LONG_LENGTH ) {
\newline 
        ptl_handle_me_t me_handle;
\newline 

\newline 
        /* add a match entry to the end of the get list */
\newline 
        PtlMEInsert( target, match, 0, PTL_UNLINK, PTL_INS_BEFORE, EndGet,
 &me_handle );
\newline 
        PtlMDAttach( me_handle, mem_desc, PTL_UNLINK, NULL );
\newline 

\newline 
        /* we want an ack for long messages */
\newline 
        want_ack = PTL_ACK_REQ;
\newline 

\newline 
        /* set the protocol bit to indicate that this is a long message
 */
\newline 
        match |= 1<<63;
\newline 
    } else {
\newline 
        /* we don't want an ack for short messages */
\newline 
        want_ack = PTL_ACK_REQ;
\newline 

\newline 
        /* set the protocol bit to indicate that this is a short message
 */
\newline 
        match &= ~(1<<63);
\newline 
    }
\newline 

\newline 
   /* create a memory descriptor and send it */
\newline 
   PtlMDBind( MPI_NI, mem_desc, &send_handle );
\newline 
   PtlPut( send_handle, want_ack, target, MPI_SEND_PINDEX, MPI_AINDEX, match,
 0 );
\newline 
}
\layout Standard

The 
\emph on 
MPISend
\emph default 
 function returns as soon as the message has been scheduled for transmission.
 The event queue argument, 
\family typewriter 
eventq
\family default 
, can be used to determine the disposition of the message.
 Assuming that 
\family typewriter 
eventq
\family default 
 is not 
\family typewriter 
PTL_EQ_NONE
\family default 
, a 
\family typewriter 
PTL_EVENT_SENT
\family default 
 event will be recorded for each message as the message is transmitted.
 For small messages, this is the only event that will be recorded in 
\family typewriter 
eventq
\family default 
.
 In contrast, long messages include an explicit request for an acknowledgement.
 If the 
\family typewriter 
target
\family default 
 process has posted a matching receive, the acknowledgement will be sent
 as the message is received.
 If a matching receive has not been posted, the message will be discarded
 and no acknowledgement will be sent.
 When the 
\family typewriter 
target
\family default 
 process later issues a matching receive, the receive will be translated
 into a get operation and a 
\family typewriter 
PTL_EVENT_GET
\family default 
 event will be recorded in 
\family typewriter 
eventq
\family default 
.
\layout Standard

Figure\SpecialChar ~

\begin_inset LatexCommand \ref{fig:mpi}

\end_inset 

 illustrates the organization of the match list used for receiving MPI messages.
 The initial entries (not shown in this figure) would be used to match the
 MPI receives that have been preposted by the application.
 The preposted receives are followed by a match entry, 
\emph on 
RcvMark
\emph default 
, that marks the boundary between preposted receives and the memory descriptors
 used for 
\begin_inset Quotes eld
\end_inset 

unexpected
\begin_inset Quotes erd
\end_inset 

 messages.
 The 
\emph on 
RcvMark
\emph default 
 entry is followed by a small collection of match entries that match unexpected
 
\begin_inset Quotes eld
\end_inset 

short
\begin_inset Quotes erd
\end_inset 

 messages, i.e., messages that have a 0 in the most significant bit of their
 match bits.
 The memory descriptors associated with these match entries will append
 the incoming message to the associated memory descriptor and record an
 event in an event queue for unexpected messages.
 The unexpected short message matching entries are followed by a match entry
 that will match messages that were not matched by the preceding match entries,
 i.e., the unexpected long messages.
 The memory descriptor associated with this match entry truncates the message
 body and records an event in the event queue for unexpected messages.
 Note that of the memory descriptors used for unexpected messages share
 a common event queue.
 This makes it possible to process the unexpected messages in the order
 in which they arrived, regardless of.
\layout Standard


\begin_inset Float figure
placement htbp
wide false
collapsed false

\layout Standard
\align center 

\begin_inset Graphics FormatVersion 1
        filename mpi.eps
        display color
        size_type 0
        rotateOrigin center
        lyxsize_type 1
        lyxwidth 389pt
        lyxheight 284pt
\end_inset 


\layout Caption

Message Reception in MPI
\begin_inset LatexCommand \label{fig:mpi}

\end_inset 


\end_inset 


\layout Standard

When the local MPI process posts an MPI receive, we must first search the
 events unexpected message queue to see if a matching message has already
 arrived.
 If no matching message is found, a match entry for the receive is inserted
 before the 
\emph on 
RcvMark
\emph default 
 entry--after the match entries for all of the previously posted receives
 and before the match entries for the unexpected messages.
 This ensures that preposted receives are matched in the order that they
 were posted (a requirement of MPI).
 
\layout Standard

While this strategy respects the temporal semantics of MPI, it introduces
 a race condition: a matching message might arrive after the events in the
 unexpected message queue have been searched, but before the match entry
 for the receive has been inserted in the match list.
 
\layout Standard

To avoid this race condition we start by setting the 
\family typewriter 
threshold
\family default 
 of the memory descriptor to 0, making the descriptor inactive.
 We then insert the match entry into the match list and proceed to search
 the events in the unexpected message queue.
 A matching message that arrives as we are searching the unexpected message
 queue will not be accepted by the memory descriptor and, if not matched
 by an earlier match list element, will add an event to the unexpected message
 queue.
 After searching the events in the unexpected message queue, we update the
 memory descriptor, setting the threshold to 1 to activate the memory descriptor.
 This update is predicated by the condition that the unexpected message
 queue is empty.
 We repeat the process of searching the unexpected message queue until the
 update succeeds.
\layout Standard

The following code fragment illustrates this approach.
 Because events must be removed from the unexpected message queue to be
 examined, this code fragment assumes the existence of a user managed event
 list, 
\family typewriter 
Rcvd
\family default 
, for the events that have already been removed from the unexpected message
 queue.
 In an effort to keep the example focused on the basic protocol, we have
 omitted the code that would be needed to manage the memory descriptors
 used for unexpected short messages.
 In particular, we simply leave messages in these descriptors until they
 are received by the application.
 In a robust implementation, we would introduce code to ensure that short
 unexpected messages are removed from these memory descriptors so that they
 can be re-used.
\layout LyX-Code


\size small 
extern ptl_handle_eq_t UnexpQueue;
\newline 
extern ptl_handle_me_t RcvMark;
\newline 
extern ptl_handle_me_t ShortMatch;
\newline 

\newline 
typedef struct event_list_tag {
\newline 
    ptl_event_t            event;
\newline 
    struct event_list_tag* next;
\newline 
} event_list;
\newline 

\newline 
extern event_list Rcvd;
\newline 

\newline 
void AppendRcvd( ptl_event_t event )
\newline 
{
\newline 
    /* append an event onto the Rcvd list */
\newline 
}
\newline 

\newline 
int SearchRcvd( void *buf, ptl_size_t len, ptl_process_id_t sender, ptl_match_bi
ts_t match,
\newline 
                       ptl_match_bits_t ignore, ptl_event_t *event )
\newline 
{
\newline 
    /* Search the Rcvd event queue, looking for a message that matches the
 requested message.
\newline 
     * If one is found, remove the event from the Rcvd list and return it.
 */
\newline 
}
\newline 

\newline 
typedef enum { RECEIVED, POSTED } receive_state;
\newline 

\newline 
receive_state CopyMsg( void *buf, ptl_size_t &length, ptl_event_t event,
 ptl_md_t md_buf )
\newline 
{
\newline 
    ptl_md_t md_buf;
\newline 
    ptl_handle_me_t me_handle;
\newline 

\newline 
    if( event.rlength >= MPI_LONG_LENGTH ) {
\newline 
        PtlMDBind( MPI_NI, md_buf, &md_handle );
\newline 
        PtlGet( event.initiator, MPI_GET_PINDEX, 0, event.match_bits, MPI_AINDEX,
 md_handle );
\newline 
        return POSTED;
\newline 
    } else {
\newline 
        /* copy the message */
\newline 
        if( event.mlength < *length ) *length = event.mlength;
\newline 
        memcpy( buf, (char*)event.md_desc.start+event.offset, *length );
\newline 
        return RECEIVED;
\newline 
    }
\newline 
}
\newline 

\newline 
receive_state MPIreceive( void *buf, ptl_size_t &len, void *MPI_data, ptl_handle
_eq_t eventq, 
\newline 
                           ptl_process_id_t sender, ptl_match_bits_t match,
 ptl_match_bits_t ignore )
\newline 
{
\newline 
    ptl_md_t md_buf;
\newline 
    ptl_handle_md_t md_handle;
\newline 
    ptl_handle_me_t me_handle;
\newline 
    ptl_event_t event;
\newline 

\newline 
    /* build a memory descriptor for the receive */
\newline 
    md_buf.start = buf;
\newline 
    md_buf.length = *len;
\newline 
    md_buf.threshold = 0;     /* temporarily disabled */
\newline 
    md_buf.options = PTL_MD_PUT_OP;
\newline 
    md_buf.user_ptr = MPI_data;
\newline 
    md_buf.eventq = eventq;
\newline 

\newline 
    /* see if we have already received the message */
\newline 
    if( SearchRcvd(buf, len, sender, match, ignore, &event) )
\newline 
         return CopyMsg( buf, len, event, md_buf );
\newline 

\newline 
    /* create the match entry and attach the  memory descriptor */
\newline 
    PtlMEInsert(sender, match, ignore, PTL_UNLINK, PTL_INS_BEFORE, RcvMark,
 &me_handle);
\newline 
    PtlMDAttach( me_handle, md_buf, PTL_UNLINK, &md_handle );
\newline 

\newline 
    md_buf.threshold = 1;
\newline 
    do
\newline 
        if( PtlEQGet( UnexpQueue, &event ) != PTL_EQ_EMPTY ) {
\newline 
            if( MPIMatch(event, match, ignore, sender) ) {
\newline 
                return CopyMsg( buf, len, (char*)event.md_desc.start+event.offset,
 md_buf );
\newline 
            } else {
\newline 
                AppendRcvd( event );
\newline 
            }
\newline 
        }
\newline 
    while( PtlMDUpdate(md_handle, NULL, &md_buf, unexp_queue) == PTL_NOUPDATE
 );
\newline 
    return POSTED;
\newline 
}
\layout Chapter*

Acknowledgments
\layout Standard

Several people have contributed to the philosophy, design, and implementation
 of the Portals message passing architecture as it has evolved.
 We acknowledge the following people for their contributions: Al Audette,
 Lee Ann Fisk, David Greenberg, Tramm Hudson, Gabi Istrail, Chu Jong, Mike
 Levenhagen, Jim Otto, Mark Sears, Lance Shuler, Mack Stallcup, Jeff VanDyke,
 Dave van Dresser, Lee Ward, and Stephen Wheat.
 
\layout Standard


\begin_inset LatexCommand \BibTeX[ieee]{portals3}

\end_inset 


\the_end