%
% Instructions on writing an FAQ entry:
% All entries begin with a set of structured comments that begin
% with two percent signs (%%) and have the form
%    %% Heading: title
%    %% Filename: shortname
%    %% End
% The '%%' must be at the beginning of the line.
% If the heading requires more than one line of text, use the backslash
% character to continue it (and no %% on the continuation lines).
% However, it is best if headings are short, as their purpose in an
% FAQ is to provide a quick index of the topics.
% In addition, the item
%    %% Section: title
% may be used to group the items into major categories.  This statement 
% should immediately preceed the "Heading" statement for the first item
% in that category.
%
% After that comes text.  It may use a subset of LaTeX commands, including
% \texttt, \emph, \textbf, \url, \_, and the environments verbatim, itemize,
% enumerate, and small (there are some others, but these should normally
% be sufficient).  One difference from basic LaTeX is that paragraph
% breaks must be made explicitly with \par.
%
% To rebuild the faq files in all three (HTML, PDF, Text) formats, do
%    make
% To install the HTML and PDF pages on the web site, do
%    make install-web
% ----------------------------------------------------------------------------
%% Section: General Information
%% Heading: Q: What is MPICH2?
%% Filename: intro
%% End
MPICH2 is a freely available, portable implementation of
\href{http://www.mpi-forum.org}{MPI}, the Standard for 
message-passing libraries.  It implements both MPI-1 and MPI-2.

%% Heading: Q: What does MPICH stand for?
%% Filename: mpichname
%% End
A: MPI stands for Message Passing Interface.
The CH comes from Chameleon, the portability layer used in the original 
MPICH to provide portability to the existing message-passing systems.

%% Section: Building MPICH2
%% Heading: Q: What is the difference between the MPD and SMPD process managers?
%% Filename: mpd-smpd
%% End
MPD is the default process manager for MPICH2 on Unix platforms. It is written in
Python. SMPD is the primary process manager for MPICH2 on Windows. It
is also used for running on a combination of Windows and Linux
machines. It is written in C.

%% Heading: Q: When I use the g95 Fortran compiler on a 64-bit platform, some\
  of the tests fail
%% Filename: g95
%% End
A: The g95 compiler incorrectly defines the default Fortran integer as a
64-bit integer while defining Fortran reals as 32-bit values (the Fortran
standard requires that INTEGER and REAL be the same size).  This was 
apparently done to allow a Fortran INTEGER to hold the value of a pointer, 
rather than requiring the programmer to select an INTEGER of a suitable KIND. 
To force the g95 compiler to correctly implement the Fortran standard, use the
\texttt{-i4} flag.  For example, set the environment variable
\texttt{F90FLAGS} before configuring MPICH2:
\begin{verbatim}
   setenv F90FLAGS "-i4"
\end{verbatim}
G95 users should note that there (at this writing) are two distributions of
g95 for 64-bit Linux platforms.  One uses 32-bit integers and reals (and
conforms to the Fortran standard) and one uses 32-bit integers and 64-bit
reals.  We recommend using the one that conforms to the standard (note that
the standard specifies the \emph{ratio} of sizes, not the absolute sizes, so a
Fortran 95 compiler that used 64 bits for \emph{both} INTEGER and REAL would
also conform to the Fortran standard.  However, such a compiler would need to
use 128 bits for DOUBLE PRECISION quantities).

%% Section: Compiling MPI Programs
%% Heading: C++ and \texttt{SEEK\_SET}
%% Filename: cxxseek
%% End
Some users may get error messages such as
\begin{verbatim}
    SEEK_SET is #defined but must not be for the C++ binding of MPI
\end{verbatim}
The problem is that both \texttt{stdio.h} and the MPI C++ interface use
\texttt{SEEK\_SET}, \texttt{SEEK\_CUR}, and \texttt{SEEK\_END}.  This is really a
bug in the MPI-2 standard.  You can try adding 
\begin{verbatim}
    #undef SEEK_SET
    #undef SEEK_END
    #undef SEEK_CUR
\end{verbatim}
before \texttt{mpi.h} is included, or add the definition
\begin{verbatim}
    -DMPICH_IGNORE_CXX_SEEK
\end{verbatim}
to the command line (this will cause the MPI versions of \texttt{SEEK\_SET}
etc. to be skipped).


%% Heading: When building the ssm or sshm channel, I get the \
error "mpidu\_process\_locks.h:234:2: error: \#error *** No atomic memory \
operation specified to implement busy locks ***"
%% Filename: asmlocks
%% End
The ssm and sshm channels do not work on all platforms because they
use special interprocess locks (often assembly) that may not work
with some compilers or machine architectures. They work on Linux with
gcc, Intel, and Pathscale compilers on various Intel
architectures. They also work in Windows and Solaris environments.



%% Section: Running MPI Programs
%% Heading: Q: How do I pass environment variables to the processes of my parallel program
%% Filename: envvars
%% End
A: The specific method depends on the process manager and version of 
\texttt{mpiexec} that you are using. 

%% Heading: Q:  How do I pass environment variables to the processes of \
my parallel program when using the mpd process manager?
%% Filename: envvarsmpd
%% End
A:  By default, all the environment variables in the shell where
\texttt{mpiexec} is run are passed to all processes of the application
program.  (The one exception is \texttt{LD\_LIBRARY\_PATH} when the
mpd's are being run as root.)  This default can be overridden in many
ways, and individual environment variables can be passed to specific
processes using arguments to \texttt{mpiexec}.  A synopsis of the
possible arguments can be listed by typing
\begin{verbatim}
    mpiexec -help
\end{verbatim}
and further details are available in the 
\href{http://www-unix.mcs.anl.gov/mpi/mpich2/downloads/mpich2-doc-user.pdf}{Users Guide}.

%% Heading: Q:  What determines the hosts on which my MPI processes run?
%% Filename: whererun
%% End
A:  Where processes run, whether by default or by specifying them
yourself, depends on the process manager being used.

If you are using the \texttt{gforker} process manager, then all MPI
processes run on the same host where you are running \texttt{mpiexec}.

If you are using the \texttt{mpd} process manager, which is the default,
then many options are available.  If you are using \texttt{mpd}, then
before you run \texttt{mpiexec}, you will have started, or will have had
started for you, a ring of processes called \texttt{mpd}'s
(multi-purpose daemons), each running on its own host.  It is likely,
but not necessary, that each \texttt{mpd} will be running on a separate
host.  You can find out what this ring of hosts consists of by running
the program \texttt{mpdtrace}.  One of the \texttt{mpd}'s will be
running on the ``local'' machine, the one where you will run
\texttt{mpiexec}.  The default placement of MPI processes, if one runs
\begin{verbatim}
    mpiexec -n 10 a.out
\end{verbatim}
is to start the first MPI process (rank 0) on the local machine and then
to distribute the rest around the \texttt{mpd} ring one at a time.  If
there are more processes than \texttt{mpd}'s, then wraparound occurs.
If there are more \texttt{mpd}'s than MPI processes, then some
\texttt{mpd}'s will not run MPI processes.  Thus any number of processes
can be run on a ring of any size.  While one is doing development, it is
handy to run only one \texttt{mpd}, on the local machine.  Then all the
MPI processes will run locally as well.

The first modification to this default behavior is the \texttt{-1}
option to \texttt{mpiexec} (not a great argument name).  If \texttt{-1}
is specified, as in 
\begin{verbatim}
    mpiexec -1 -n 10 a.out
\end{verbatim}
then the first application process will be started by the first
\texttt{mpd} in the ring \emph{after} the local host.  (If there is only
one \texttt{mpd} in the ring, then this will be on the local host.)
This option is for use when a cluster of compute nodes has a ``head
node'' where commands like \texttt{mpiexec} are run but not application
processes. 

If an \texttt{mpd} is started with the \texttt{--ncpus} option, then
when it is its turn to start a process, it will start several
application processes rather than just one before handing off the task
of starting more processes to the next \texttt{mpd} in the ring.  For
example, if the \texttt{mpd} is started with
\begin{verbatim}
    mpd --ncpus=4
\end{verbatim}
then it will start as many as four application processes, with
consecutive ranks, when it is its turn to start processes.  This option
is for use in clusters of SMP's, when the user would like consecutive
ranks to appear on the same machine.  (In the default case, the same
number of processes might well run on the machine, but their ranks would
be different.)

(A feature of the --ncpus=<n> argument is that it has the above effect only
until all of the mpd's have started n processes at a time once;
afterwards each mpd starts one process at a time.  This is in order to 
balance the number of processes per machine to the extent possible.)

Other ways to control the placement of processes are by direct use of
arguments to \texttt{mpiexec}.  See the 
\href{http://www-unix.mcs.anl.gov/mpi/mpich2/downloads/mpich2-doc-user.pdf}{Users Guide}.

%% Heading: Q: On Windows, I get an error when I attempt to call \texttt{MPI\_Comm\_spawn}.
%% Filename: windows-spawn
%% End
A: On Windows, you need to start the program with \texttt{mpiexec} for
any of the MPI-2 dynamic process functions to work.

