21.1.4. prterun

prterun — start a PRRTE DVM, run one job in it, and shut it down

21.1.4.1. SYNOPSIS

Single Process Multiple Data (SPMD) model:

prterun [options] <program> [<args>]

Multiple Instruction Multiple Data (MIMD) model:

prterun [global options] [local options1] <program1> [<args1>] : \
        [local options2] <program2> [<args2>] : ... : \
        [local optionsN] <programN> [<argsN>]

21.1.4.2. DESCRIPTION

prterun combines prte(1) and prun(1) in one command: it instantiates a PMIx Reference Runtime Environment (PRRTE) distributed virtual machine (DVM) across the allocation, launches the given job in it, waits for the job to complete, and then tears the DVM down again. Use prte and prun instead to start a persistent DVM and run many jobs in it, or give prterun the --dvm option to have it submit the job to an existing DVM rather than start one of its own.

Each application context on the command line is separated by a :. The options that precede a context’s program apply to that context. Options are recognized only before the program name: the first token that is not an option is taken as the executable, and everything after it is passed to the program as its arguments. See “Per-app-context mapping” under --mapby for how placement options given on different contexts combine.

All of the text below is also available from the command itself: prterun --help lists the options, and prterun --help <option> (for example, prterun --help mapby) prints the full description of one. prterun --help placement describes the mapping, ranking and binding procedure in detail.

21.1.4.3. DIRECTIVES AND QUALIFIERS

Several options take a value that is a small language of its own: --map-by, --rank-by, --bind-to, --output, --display and --rtos. The words in that value come from a fixed vocabulary, listed with each option below, and are put together the same way for all of them.

Separators. A value is built from directives, each of which may be followed by qualifiers, and any word may carry a value of its own:

: separates a directive from its qualifiers, and one qualifier from the next: --map-by package:span:pe=2.
, separates one directive from the next, in the options that accept several: --output tag,timestamp or --display map,bind. A directive’s qualifiers follow it with : as usual, so --output tag,file=out:nocopy is the tag directive, then the file directive qualified by nocopy.
= separates a word from its value: pe=2, device=gpu, file=out.

A qualifier written after a , instead of a : is therefore not a qualifier of the directive before it. --map-by device=gpu,ndev=2 names a device called gpu,ndev=2, and since no device is named with a comma, it is refused with the spelling that was almost certainly meant:

$ prun --map-by device=gpu,ndev=2 ./a.out
The device named in a mapping request contains a comma:
  Given:  gpu,ndev=2
...
  device=gpu:ndev=2

--rtos takes no qualifiers, and its values are not split at : at all: a time is written with colons (--rtos timeout=1:30:00, one hour thirty minutes).

Abbreviations. Words are case-insensitive, and any of them may be shortened to a prefix that names only that word:

--map-by pack is --map-by package; --bind-to hwt is --bind-to hwthread; --map-by core:ov is --map-by core:oversubscribe.
A prefix that fits more than one word is refused, and the words it fits are listed, rather than one of them being picked:
```
$ prun --bind-to n ./a.out
The --bind-to option was given a word that abbreviates more than one of the
words it accepts:
  Given:    n
  Matches:  none,numa
```
So --bind-to n must be written no or nu; --map-by :s must be :sp (span) or :sh (shared); --map-by :i must be :inh (inherit) or :int (interleave); --rank-by s must be sl or sp; and --output ta must be tag, tag-d or tag-f.
A word given in full is always that word, even when it is also the start of a longer one: pe=2 is the pe qualifier, not pe-list, and --output tag is tag, not tag-detailed.
A word that merely begins with one in the vocabulary is refused, not read as the word it begins with. --map-by nodes, --bind-to cores and --map-by package:spanish are all errors.

Values. Each word takes no value, may take one, or requires one:

A word that takes no value refuses one. --map-by core:span=false is an error, not a request for span - to not ask for something, leave it out.
A word that requires a value refuses to go without: --map-by core:pe and --rtos timeout are errors, as is pe= with nothing after the =.
The directives of --output, --display and --rtos that are simply on or off may be given a truth value, and are on when given bare: --output tag and --output tag=true ask for tagging, and --output tag=false does not.

Examples.

# two GPUs per process; bind each to a core beside them
$ prun -n 2 --map-by device=gpu:ndev=2 --bind-to core ./a.out

# map by package across the nodes, two cpus per process
$ prun -n 8 --map-by package:span:pe=2 ./a.out

# the same, abbreviated
$ prun -n 8 --map-by pack:sp:pe=2 ./a.out

# tag the output, and also write it to files without copying it to
# the terminal
$ prun -n 4 --output tag,dir=/tmp/out:nocopy ./a.out

# show the map in a form a script can parse, and the bindings
$ prun -n 4 --display map:parseable,bind ./a.out

# stop the job if it runs longer than an hour and a half
$ prun -n 4 --rtos timeout=1:30:00 ./a.out

21.1.4.4. OPTIONS

A value may follow its option either as the next argument or after an = (--np 4 or --np=4).

General options

21.1.4.4.1. `-h` | `--help [<option>]`

Print the list of options, or the full help for the named option.

21.1.4.4.2. `-v` | `--verbose`

Enable typical debug options.

21.1.4.4.3. `-V` | `--version`

Print version and exit.

MCA parameters

21.1.4.4.4. `--prtemca <key> <value>`

Pass a PRRTE MCA parameter.

Syntax: --prtemca <key> <value>, where key is the parameter name and value is the parameter value.

21.1.4.4.5. `--pmixmca <key> <value>`

Pass a PMIx MCA parameter

Syntax: --pmixmca <key> <value>, where key is the parameter name and value is the parameter value.

21.1.4.4.6. `--tune <files>`

Comma-delimited list of one or more files containing MCA parameters for tuning DVM and/or application operations. The option may be given more than once. A file named by a relative path is looked for first in the current directory and then among the parameter sets installed with PRRTE.

Syntax in the file is:

param = value

with one parameter per line. Empty lines and lines beginning with the # character are ignored, as is any whitespace around the = character. Quotes around the value are removed.

Each parameter is a generic MCA parameter, so it is treated exactly like --mca param value: it is applied to PRRTE if it belongs to a PRRTE framework, and otherwise to PMIx. A parameter that belongs to neither is an error, as is a parameter given twice with different values. A parameter given explicitly on the command line (--prtemca, --pmixmca) overrides the same parameter in a tune file.

DVM options

21.1.4.4.7. `--dvm <arg>`

A required argument is passed to the --dvm directive to specify the location of the DVM controller (e.g., --dvm pid:12345) or by passing the string search to instead search for an existing controller.

Supported options include:

search: directs the tool to search for available DVM controllers it is authorized to use, connecting to the first such candidate it finds.
pid:<arg>: provides the PID of the target DVM controller. This can be given as either the PID itself (arg = int) or the path to a file that contains the PID (arg = file:<path>)
file:<path>: provides the path to a PMIx rendezvous file that is output by PMIx servers — the file contains all the required information for completing the connection
uri:<arg>: specifies the URI of the DVM controller, or the name of the file (specified as file:filename) that contains that info
ns:<arg>: specifies the namespace of the DVM controller
system: exclusively find and use the system-level DVM controller
system-first: look for a system-level DVM controller, fall back to searching for an available DVM controller the command is authorized to use if a system-level controller is not found

Examples:

prterun --dvm file:dvm_uri.txt --np 4 ./a.out

prterun --dvm pid:12345 --np 4 ./a.out

prterun --dvm uri:file:dvm_uri.txt --np 4 ./a.out

prterun --dvm ns:prte-node1-2095 --np 4 ./a.out

prterun --dvm pid:file:prte_pid.txt --np 4 ./a.out

prterun --dvm search --np 4 ./a.out

21.1.4.4.8. `--default-hostfile <filename>`

Specify a default hostfile.

Also see --hostfile.

21.1.4.4.9. `--uniform-nodes`

The uniform-nodes command line directive is used to indicate that the allocated nodes should be treated as having only one topology, so optimize the launch for that scenario. This includes ensuring that all CPU allocations are the same on each node, that each node contains the same number of devices and topological layers, etc.

Note

The runtime does not currently support mixes of chips with different endianness.

21.1.4.4.10. `--prefix <dir>`

Prefix to be used to look for PRRTE executables. PRRTE automatically sets the prefix for remote daemons if it was either configured with the --enable-prte-prefix-by-default option OR prte itself was executed with an absolute path to the prte command. This option overrides those settings, if present, and forces use of the provided path.

21.1.4.4.11. `--noprefix`

Disable automatic --prefix behavior. PRRTE automatically sets the prefix for remote daemons if it was either configured with the --enable-prte-prefix-by-default option OR prte itself was executed with an absolute path to the prte command. This option disables that behavior.

21.1.4.4.12. `--pmix-prefix <dir>`

Prefix to be used by a PRRTE executable to look for its PMIx installation on remote nodes. This is the location of the top-level directory for the installation. If the installation has not been moved, it would be the value given to “–prefix” when the installation was configured.

Note that PRRTE cannot determine the exact name of the library subdirectory under this location. For example, some systems will call it “lib” while others call it “lib64”. Accordingly, PRRTE will use the library subdirectory name of the PMIx installation used to build PRRTE.

21.1.4.4.13. `--launch-agent <executable>`

Name of daemon executable used to start processes on remote nodes (default: prted).

This is the executable prterun shall start on each remote node when establishing the DVM.

21.1.4.4.14. `--max-vm-size <num>`

Maximum number of daemons to start.

21.1.4.4.15. `--keepalive <fd>`

Pipe for prterun to monitor — job will terminate upon closure.

21.1.4.4.16. `--system-server`

Start prterun and its daemons as the system server on their nodes.

21.1.4.4.17. `--set-sid`

Direct the DVM (controller and daemons) to separate from the current session.

21.1.4.4.18. `--report-pid <arg>`

Print the PID of this process: on stdout if the argument is -, on stderr if it is +, into the already-open file descriptor it names if it is a non-negative integer, and otherwise into the file it names.

21.1.4.4.19. `--report-uri <arg>`

Print the PMIx contact URI of this process: on stdout if the argument is -, on stderr if it is +, and otherwise into the file it names.

21.1.4.4.20. `--allow-run-as-root`

Allow execution as root (STRONGLY DISCOURAGED).

Running as root exposes the user to potentially catastrophic file system corruption and damage — e.g., if the user accidentally points the root of the session directory to a system required point, this directory and all underlying elements will be deleted upon job completion, thereby rendering the system inoperable.

It is recognized that some environments (e.g., containers) may require operation as root, and that the user accepts the risks in those scenarios. Accordingly, one can override PRRTE’s run-as-root protection by providing one of the following:

The --allow-run-as-root command line directive
Adding BOTH of the following environmental parameters:
- PRTE_ALLOW_RUN_AS_ROOT=1
- PRTE_ALLOW_RUN_AS_ROOT_CONFIRM=1

Again, we recommend this only be done if absolutely necessary.

21.1.4.4.21. `--forward-signals <signals>`

Comma-delimited list of the signals (names or integers) to be forwarded to application processes (none = forward nothing).

The list replaces the default set rather than adding to it, so it names every signal that is to be forwarded. The default set, used when this option is not given, is SIGTSTP, SIGUSR1, SIGUSR2, SIGABRT, SIGALRM, and SIGCONT.

21.1.4.4.22. `--debug-daemons`

Debug daemon output enabled. This is a somewhat limited stream of information normally used to simply confirm that the daemons started. Includes leaving the output streams open.

21.1.4.4.23. `--debug-daemons-file`

Debug daemon output is enabled and all output from the daemons is redirected into files with names of the form:

output-prted-<daemon-nspace>-<nodename>.log

These names avoid conflict on shared file systems. The files are located in the top-level session directory assigned to the DVM.

See the “Session directory” HTML documentation for additional details about the PRRTE session directory.

21.1.4.4.24. `--leave-session-attached`

Do not discard stdout/stderr of remote PRRTE daemons. The primary use for this option is to ensure that the daemon output streams (i.e., stdout and stderr) remain open after launch, thus allowing the user to see any daemon-generated error messages. Otherwise, the daemon will “daemonize” itself upon launch, thereby closing its output streams.

Launch options

21.1.4.4.25. `--personality <name>`

Specify the personality to be used. This governs selection of the plugin responsible for defining and parsing the command line, harvesting and forwarding environmental variables, and providing library-dependent support to the launched processes. Examples include ompi for an application compiled with Open MPI, mpich for one built against the MPICH library, or oshmem for an OpenSHMEM application compiled against SUNY’s reference library.

21.1.4.4.26. `-n` | `-c` | `--np <num>`

Specify the number of application processes to be started for this application context. -n, -c and --n are synonyms for --np.

21.1.4.4.27. `-N <num>`

Specify the number of application processes to be started on each node. -N <num> is shorthand for the placement directive --mapby ppr:<num>:node, and is converted to it.

21.1.4.4.28. `--app <filename>`

Provide an appfile describing the application contexts to be launched. Each line of the file that is neither blank nor a comment (a line whose first non-blank character is #) describes one application context, written as it would be on the command line: the options that apply to that context, followed by the executable and its arguments. Lines are split at spaces; quotes are not interpreted.

Options given on the command line alongside --app are combined with the first application context in the file, so job-level options (such as --mapby) apply to the whole job as usual, while an option that is also given on the file’s first line is refused as a duplicate. The command line may not also name an application — an executable, or a :-separated application context — as well.

21.1.4.4.29. `-H` | `--host <hosts>`

Host syntax consists of a comma-delimited list of node names, each entry optionally containing a :N extension indicating the number of slots to assign to that entry:

--host node01:5,node02

In the absence of the slot extension, one slot will be assigned to the node. Duplicate entries are aggregated and the number of slots assigned to that node are summed together.

Note

A “slot” is the PRRTE term for an allocatable unit where we can launch a process. Thus, the number of slots equates to the maximum number of processes PRRTE may start on that node without oversubscribing it.

Given to a job, --host selects from the hosts already available to the DVM — those a resource manager allocated, or those the DVM was started with. It does not add any: naming a host that is not among them is an error. Use --add-host or --add-hostfile to bring a new host into a running DVM, or --activate to start a daemon on a host the allocation already contains.

The :N count applies to placement, and not merely to the size of the job: it is the number of processes that may be placed on that host, whatever mapping policy is in effect. Asking for more slots on a host than it has is an error under a resource manager, which decided how big the host is. Without one, the larger count is taken as the size of the host for that job only; the allocation itself is unchanged.

See the “Host specification” documentation for details about the format and content of hostfiles.

21.1.4.4.30. `--hostfile <filename>`

PRRTE supports several levels of user-specified hostfiles based on an established precedence order. Users can specify a hostfile that contains a list of nodes to be used for the job, or can provide a comma-delimited list of nodes to be used for that job via the --host command line option.

The precedence order applied to these various options depends to some extent on the local environment. The following table illustrates how host and hostfile directives work together to define the set of hosts upon which a DVM will execute the job in the absence of a resource manager (RM):

host	hostfile	Result
unset	unset	The DVM will utilize all its available resources when mapping the job.
set	unset	Host option defines resource list for the job
unset	set	Hostfile defines resource list for the job
set	set	Hostfile defines resource list for the job, then host filters the list to define the final set of nodes to be used for the job

Hostfiles (sometimes called “machine files”) are a combination of two things:

A listing of hosts on which to launch processes.
Optionally, limit the number of processes which can be launched on each host.

Hostfile syntax consists of one node name on each line, optionally including a designated number of “slots”:

# This is a comment line, and will be ignored
node01  slots=10
node13  slots=5

node15
node16
node17  slots=3
...

Blank lines and lines beginning with a # are ignored.

A node name may carry the account PRRTE is to use when reaching that node, written in front of it and separated by a single @:

user01@node01  slots=4

An entry may contain at most one @, and both the account and the node name must be given: a second @, or an @ with nothing on one side of it, is reported as a parse error naming the hostfile and the line it is on.

A node name written with a leading ^ is excluded rather than used. The ^ goes in front of the whole entry, account included:

node01  slots=4
node02  slots=4
node03  slots=4
^user01@node02

Where the hostfile also names nodes, an exclusion takes a node back out of what it named — the file above names node01 and node03. A hostfile given to a job in a running DVM may consist of nothing but exclusions, and then it selects every node of the allocation except the ones it excludes.

A “slot” is the PRRTE term for an allocatable unit where we can launch a process. See the section on definition of the term slot for a longer description of slots.

In the absence of the slot parameter, PRRTE will assign either the number of slots to be the number of CPUs detected on the node or the resource manager-assigned value if operating in the presence of an RM.

Important

If using a resource manager, the user-specified number of slots is capped by the RM-assigned value.

A hostfile given to a job that is being submitted to an already-running DVM selects within the DVM’s allocation: it names the subset of nodes that job may use, and a slots count smaller than the node’s own is the number of slots that job may take there. It says nothing about how big the node is, so it applies to that job alone — the node is back to its allocated size for the next job, which may be someone else’s. Changing the allocation is what --add-hostfile is for.

Relative host indexing

Hostfile and --host specifications can also be made using relative indexing. This allows a user to stipulate which hosts are to be used for a given app context without specifying the particular host name, but rather its relative position in the allocation.

This can probably best be understood through consideration of a few examples. Consider the case where a DVM is comprised of a set of nodes named foo1, foo2, foo3, foo4. The user wants the first app context to have exclusive use of the first two nodes, and a second app context to use the last two nodes. Of course, the user could printout the allocation to find the names of the nodes allocated to them and then use --host to specify this layout, but this is cumbersome and would require hand-manipulation for every invocation.

A simpler method is to utilize PRRTE’s relative indexing capability to specify the desired layout. In this case, a command line containing:

--host +n0,+n1 ./app1 : --host +n2,+n3 ./app2

would provide the desired pattern. The + syntax indicates that the information is being provided as a relative index into the existing allocation. Two methods of relative indexing are supported:

+n#: A relative index into the allocation, counted from zero — +n0 is the first node of the allocation, +n1 the second, and so on. PRRTE will substitute that node of the allocation. Note that the index counts nodes the job was actually given: if the node you launched from is not part of the allocation, it does not occupy an index. An index past the end of the allocation is an error.
+e[:#]: A request for # empty nodes — i.e., PRRTE is to substitute this reference with nodes that have not yet been used by any other app_context. If the :# is not provided, PRRTE will substitute the reference with all empty nodes. Note that PRRTE does track the empty nodes that have been assigned in this manner, so multiple uses of this option will result in assignment of unique nodes up to the limit of the available empty nodes. Requests for more empty nodes than are available will generate an error.

Relative indexing can be combined with absolute naming of hosts in any arbitrary manner, and can be used in hostfiles as well as with the --host command line option. In addition, any slot specification provided in hostfiles will be respected — thus, a user can specify that only a certain number of slots from a relative indexed host are to be used for a given app context.

Another example may help illustrate this point. Consider the case where the user has a hostfile containing:

dummy1 slots=4
dummy2 slots=4
dummy3 slots=4
dummy4 slots=4
dummy5 slots=4

This may, for example, be a hostfile that describes a set of commonly-used resources that the user wishes to execute applications against. For this particular application, the user plans to map byslot, and wants the first two ranks to be on the third node of any allocation, the next ranks to land on an empty node, have one rank specifically on dummy4, the next rank to be on the third node of the allocation again, and finally any remaining ranks to be on whatever empty nodes are left. To accomplish this, the user provides a hostfile of:

+n2 slots=2
+e:1
dummy4 slots=1
+n2
+e

The user can now use this information in combination with PRRTE’s sequential mapper to obtain their specific layout:

<launcher> --hostfile dummyhosts --hostfile mylayout --mapby seq ./my_app

which will result in:

rank0 being mapped to dummy3
rank1 to dummy1 as the first empty node
rank2 to dummy4
rank3 to dummy3
rank4 to dummy2 and rank5 to dummy5 as the last remaining unused nodes

Note that the sequential mapper ignores the number of slots arguments as it only maps one rank at a time to each node in the list.

If the default round-robin mapper had been used, then the mapping would have resulted in:

ranks 0 and 1 being mapped to dummy3 since two slots were specified
ranks 2-5 on dummy1 as the first empty node, which has four slots
rank6 on dummy4 since the hostfile specifies only a single slot from that node is to be used
ranks 7 and 8 on dummy3 since only two slots remain available
ranks 9-12 on dummy2 since it is the next available empty node and has four slots
ranks 13-16 on dummy5 since it is the last remaining unused node and has four slots

Thus, the use of relative indexing can allow for complex mappings to be ported across allocations, including those obtained from automated resource managers, without the need for manual manipulation of scripts and/or command lines.

See the “Host specification” documentation for details about the format and content of hostfiles.

21.1.4.4.31. `--machinefile <filename>`

Synonym for --hostfile.

21.1.4.4.32. `--add-host <hosts>`

PRRTE allows a user to expand an existing DVM prior to launching an application. Users can specify a a comma-delimited list of node names, each entry optionally containing a :N extension indicating the number of slots to assign to that entry:

--add-host node01:5,node02

Only a DVM started in elastic mode (--prtemca prte_elastic_mode 1) can change its membership. Any other DVM is fixed for its lifetime, and this option is refused.

In the absence of the slot extension, one slot will be assigned to the node. Duplicate entries are aggregated and the number of slots assigned to that node are summed together.

Note

A “slot” is the PRRTE term for an allocatable unit where we can launch a process. Thus, the number of slots equates to the maximum number of processes PRRTE may start on that node without oversubscribing it.

The list can include nodes that are already part of the DVM — in this case, the number of slots available on those nodes will be set to the new specification, or adjusted as directed:

--add-host node01:5,node02

would direct that node01 be set to 5 slots and node02 will have 1 slot, while

--add-host node01:+5,node02

would add 5 slots to the current value for node01, and

--add-host node01:-5,node02

would subtract 5 slots from the current value.

Slot adjustments for existing nodes will have no impact on currently executing jobs, but will be applied to any new spawn requests. Nodes contained in the add-host specification are available for immediate use by the accompanying application.

Users desiring to constrain the accompanying application to the newly added nodes should also include the --host command line directive, giving the same hosts in its argument:

--add-host node01:+5,node02 --host node01:5,node02

Note that the --host argument indicates the number of slots to assign node01 for this spawn request, and not the number of slots being added to the node01 allocation.

21.1.4.4.33. `--add-hostfile <filename>`

PRRTE allows a user to expand an existing DVM prior to launching an application. Users can specify a hostfile that contains a list of nodes to be added to the DVM using normal hostfile syntax.

Only a DVM started in elastic mode (--prtemca prte_elastic_mode 1) can change its membership. Any other DVM is fixed for its lifetime, and this option is refused.

The list can include nodes that are already part of the DVM — in this case, the number of slots available on those nodes will be set to the new specification, or adjusted as directed:

node01  slots=5

would direct that node01 be set to 5 slots, while

node01 slots+=5

would add 5 slots to the current value for node01, and

node01  slots-=5

would subtract 5 slots from the current value.

Slot adjustments for existing nodes will have no impact on currently executing jobs, but will be applied to any new spawn requests. Nodes contained in the add-hostfile specification are available for immediate use by the accompanying application.

Users desiring to constrain the accompanying application to the newly added nodes should also include the --hostfile command line directive, giving the same hostfile as its argument:

--add-hostfile <filename> --hostfile <filename>

21.1.4.4.34. `--activate <hosts>`

A DVM does not necessarily span every node of its allocation. A --host or --hostfile given when the DVM was started narrows which allocated nodes get a daemon, and a released reservation hands its nodes back without one. Such a node is allocated and up, but no part of the DVM: nothing can be launched on it.

--activate starts a daemon on those nodes, bringing them into the DVM before the accompanying application is launched.

Only a DVM started in elastic mode (--prtemca prte_elastic_mode 1) can change its membership. Any other DVM is fixed for its lifetime, and this option is refused.

It takes the same argument syntax as --host: a comma-delimited list of node names,

--activate node01,node02

or, to bring in everything the allocation holds that is not already in the DVM,

--activate +all

or the relative form that names a node by its position in the allocation,

--activate +n3

which selects the fourth node (relative node indices count from zero). Entries may also name a file, in the same format --hostfile reads, and the forms may be mixed in one list:

--activate file=/path/to/hostfile
--activate node01,file=/path/to/hostfile

Only the node names are taken from the file. A slots= it carries is not applied — a hostfile given to a launcher selects nodes, it does not resize them — but everything else the hostfile format offers, including ^host exclusions, works as it does anywhere else.

Note

+e is not accepted here. For --host it means “nodes with no application process running on them”, which says nothing about whether a node is in the DVM: most of the nodes it picks are already in it, so the request would start no daemon and still report success.

Unlike --add-host, this adds nothing to the allocation. It can only name nodes the allocation already contains, it changes no slot counts, and it asks no resource manager for anything — which is why it is permitted even where the allocation is owned by a scheduler and --add-host is refused. For the same reason a :N slot extension is not accepted here: activate has no authority to set slot counts, so a count given to it is refused rather than silently ignored.

Naming a node that is already in the DVM is not an error — the request is a statement about the DVM’s membership, and for that node it is already satisfied. For the same reason +all is satisfied, not refused, when every allocated node is already in the DVM.

Users desiring to constrain the accompanying application to the newly activated nodes should also include the --host command line directive, giving the same hosts in its argument:

--activate node01,node02 --host node01,node02

21.1.4.4.35. `--alloc-id <id>`

Direct that the application be executed using the resources of the allocation identified by the given ID, where that ID is the one the host environment (e.g., the scheduler) assigned to the allocation when it was created. The DVM reports this value back to the requester when an allocation request completes, and it is the value returned by a query of the allocation.

The named allocation must be one the requester is entitled to use — a job can only be spawned onto resources its requester has been granted. Entitlement is by namespace and by user: the namespace that requested the allocation holds it, as does every job spawned into it, and so does any tool run by the user the allocation was granted to. That last part is what lets a later command reach an allocation an earlier one created, since a tool’s namespace lasts only as long as the command that made it.

Naming an allocation the DVM does not know about is an unrecoverable error and reports PMIX_ERR_NOT_FOUND; naming one the requester may not use is likewise unrecoverable and reports PMIX_ERR_NO_PERMISSIONS. In either case the job is not launched.

If no allocation is named, the job is mapped onto the allocation of the session that requested it (the DVM’s default session, in the case of a tool such as prun).

Note

The same allocation can be named in three different ways — by the host-assigned ID given here, by the reference ID the user attached to the allocation request (--alloc-refid), or by the numerical ID of the session that holds it (--session-id). These are alternative spellings of one directive and therefore cannot be combined on a single command line.

21.1.4.4.36. `--alloc-refid <id>`

Direct that the application be executed using the resources of the allocation identified by the given reference ID, where that ID is the one the user attached to the allocation request at the time the allocation was made. This is the convenient counterpart to --alloc-id: it lets a script name an allocation using a label it chose itself, without having to capture the identifier the host environment later assigned to it.

The named allocation must be one the requester is entitled to use — a job can only be spawned onto resources its requester has been granted. Entitlement is by namespace and by user: the namespace that requested the allocation holds it, as does every job spawned into it, and so does any tool run by the user the allocation was granted to. That last part is what lets a later command reach an allocation an earlier one created, since a tool’s namespace lasts only as long as the command that made it.

Naming an allocation the DVM does not know about is an unrecoverable error and reports PMIX_ERR_NOT_FOUND; naming one the requester may not use is likewise unrecoverable and reports PMIX_ERR_NO_PERMISSIONS. In either case the job is not launched.

If no allocation is named, the job is mapped onto the allocation of the session that requested it (the DVM’s default session, in the case of a tool such as prun).

Note

The same allocation can be named in three different ways — by the reference ID given here, by the ID the host environment assigned to the allocation (--alloc-id), or by the numerical ID of the session that holds it (--session-id). These are alternative spellings of one directive and therefore cannot be combined on a single command line.

21.1.4.4.37. `--session-id <id>`

Direct that the application be executed using the resources of the session identified by the given numerical ID. A session is the container the DVM uses to hold an allocation together with the jobs running against it, and every allocation the DVM grants is assigned one. The session ID is reported back to the requester when an allocation request completes.

The value must be an unsigned 32-bit integer; anything else is rejected before the job is submitted.

The named session must be one the requester is entitled to use — a job can only be spawned onto resources its requester has been granted. Entitlement is by namespace and by user: the namespace that requested the session’s allocation holds it, as does every job spawned into it, and so does any tool run by the user it was granted to. That last part is what lets a later command reach a session an earlier one created, since a tool’s namespace lasts only as long as the command that made it.

Naming a session the DVM does not know about is an unrecoverable error and reports PMIX_ERR_NOT_FOUND; naming one the requester may not use is likewise unrecoverable and reports PMIX_ERR_NO_PERMISSIONS. In either case the job is not launched.

If no session is named, the job is mapped onto the session that requested it (the DVM’s default session, in the case of a tool such as prun).

Note

The same allocation can be named in three different ways — by the session ID given here, by the ID the host environment assigned to the allocation (--alloc-id), or by the reference ID the user attached to the allocation request (--alloc-refid). These are alternative spellings of one directive and therefore cannot be combined on a single command line.

21.1.4.4.38. `--path <dir>`

Directory in which to find the executable of this application context: the executable named on the command line is taken relative to it. A relative directory is taken relative to the current working directory of the command, and converted to an absolute path.

21.1.4.4.39. `--wdir <dir>`

Set the working directory of the started processes. A relative path is taken relative to the current working directory of the command, and converted to an absolute path. Without this option (or --set-cwd-to-session-dir), the processes start in the command’s current working directory. --wd is a synonym for --wdir.

21.1.4.4.40. `--wd <dir>`

Synonym for --wdir.

21.1.4.4.41. `--set-cwd-to-session-dir`

Set the working directory of the started processes to their session directory. This is ignored if --wdir is also given.

21.1.4.4.42. `--pset <name>`

User-specified name assigned to the processes in their given application context. The processes are told it as the name of the PMIx process set (PMIX_PSET_NAME) they belong to.

21.1.4.4.43. `--preload-files <files>`

Syntax: --preload-files <files>

Preload the comma-separated list of files to the remote machines’ current working directory before starting the remote process.

21.1.4.4.44. `-s` | `--preload-binary`

Syntax: -s or --preload-binary

Preload the binary on the remote machine before starting the remote process.

21.1.4.4.45. `--app-pmix-prefix <dir>`

Prefix to be used by an app to look for its PMIx installation on remote nodes. This is the location of the top-level directory for the installation. If the installation has not been moved, it would be the value given to “–prefix” when the installation was configured.

Note that PRRTE cannot determine the exact name of the library subdirectory under this location. For example, some systems will call it “lib” while others call it “lib64”. Accordingly, PRRTE will use the library subdirectory name of the PMIx installation used to build PRRTE.

In the absence of providing an application-specific prefix, the PMIx prefix (if given) used by PRRTE’s own executables will be applied unless the “–no-app-prefix” directive is given.

21.1.4.4.46. `--no-app-prefix`

Do not apply any prefix to this application. This is needed when a default PMIx prefix has been given to PRRTE, but the application has been built against a PMIx library that (a) is different from the one used by PRRTE, and (b) was not moved. Otherwise, PRRTE will apply its default prefix to the application.

21.1.4.4.47. `--gpu-support <bool>`

Direct the application to either enable (true) or disable (false) its internal library’s GPU support. The value is passed to the application’s library via PMIx; a value that is neither true nor false is refused.

21.1.4.4.48. `--memory-alloc-kinds <kinds>`

Comma-delimited list of the memory allocation kinds the application requires. The list is passed through to the application’s library (for example, an MPI implementation) via PMIx; PRRTE itself does not interpret the individual kinds.

21.1.4.4.49. `--rtos <directives>` | `--runtime-options <directives>`

The --rtos command line directive must be accompanied by a comma-delimited list of case-insensitive options that control the runtime behavior of the job. The full directive need not be provided — only enough characters are required to uniquely identify the directive.

Runtime options are typically true or false, though this is not a requirement on developers. Since the value of each option may need to be set (e.g., to override a default set by MCA parameter), the syntax of the command line directive includes the use of an = character to allow inclusion of a value for the option. For example, one can set the ERROR-NONZERO-STATUS option to true by specifying it as ERROR-NONZERO-STATUS=1. A boolean option can be set to true using a non-zero integer, the single letter T or Y, or the whole word TRUE, YES or ENABLE; and to false using zero, the single letter F or N, or the whole word FALSE, NO or DISABLE. All of these are case-insensitive. Note that these are the whole words — TR is not an abbreviation of TRUE — and that a value which is neither true nor false is refused rather than guessed at.

Note that a boolean option will default to true if provided without a value. Thus, --rtos error-nonzero is sufficient to set the ERROR-NONZERO-STATUS option to true.

The --runtime-options command line directive is accepted as a synonym for --rtos.

Supported values include:

ERROR-NONZERO-STATUS[=(bool)]: if set to false, this directs the runtime to treat a process that exits with non-zero status as a normal termination. If set to true, the runtime will consider such an occurrence as an error termination and take appropriate action — i.e., the job will be terminated unless a runtime option directs otherwise. This option defaults to a true value if the option is given without a value.
DONOTLAUNCH: directs the runtime to map but not launch the specified job. This is provided to help explore possible process placement patterns before actually starting execution. No value need be passed as this is not an option that can be set by default in PRRTE.
DONOTSPAWN: directs the runtime to carry out the entire launch procedure — including starting any daemons it needs and delivering the job to them — but to not actually start the application processes, which are instead marked as having terminated. This is provided to help exercise the launch procedure itself.
SHOW-PROGRESS[=(bool)]: requests that the runtime provide progress reports on its startup procedure — i.e., the launch of its daemons in support of a job. This is typically used to debug DVM startup on large systems. This option defaults to a true value if the option is given without a value.
NOTIFYERRORS[=(bool)]: if set to true, requests that the runtime provide a PMIx event whenever a job encounters an error — e.g., a process fails. The event is to be delivered to each remaining process in the job. This option defaults to a true value if the option is given without a value. See --help notifications for more detail as to the PMIx event codes available for capturing failure events.
RECOVERABLE[=(bool)]: if set to true, this indicates that the application wishes to consider the job as recoverable — i.e., the application is assuming responsibility for recovering from any process failure. This could include application-driven spawn of a substitute process or internal compensation for the missing process. This option defaults to a true value if the option is given without a value.
AUTORESTART[=(bool)]: if set to true, this requests that the runtime automatically restart failed processes up to “max restarts” number of times. This option defaults to a true value if the option is given without a value.
CONTINUOUS[=(bool)]: if set to true, this informs the runtime that the processes in this job are to run until explicitly terminated. Processes that fail are to be automatically restarted up to “max restarts” number of times. Notification of process failure is to be delivered to all processes in the application. This is the equivalent of specifying RECOVERABLE, NOTIFYERRORS, and AUTORESTART options except that the runtime, not the application, assumes responsibility for process recovery. This option defaults to a true value if the option is given without a value.
MAX-RESTARTS=<int>: indicates the maximum number of times a given process is to be restarted. This can be set at the application or job level (which will then apply to all applications in that job).
EXEC-AGENT=<path> indicates the executable that shall be used to start an application process. The resulting command for starting an application process will be <path> app <app-argv>. The path may contain its own command line arguments.
DEFAULT-EXEC-AGENT: directs the runtime to use the system default exec agent to start an application process. No value need be passed as this is not an option that can be set by default in PRRTE.
OUTPUT-PROCTABLE[(=channel)]: directs the runtime to report the conventional debugger process table (includes PID and host location of each process in the application). Output is directed to stdout if the channel is -, stderr if +, or into the specified file otherwise. If no channel is specified, output will be directed to stdout.
STOP-ON-EXEC: directs the runtime to stop the application process(es) immediately upon exec’ing them. The directive will apply to all processes in the job.
STOP-IN-INIT: indicates that the runtime should direct the application process(es) to stop in PMIx_Init(). The directive will apply to all processes in the job.
STOP-IN-APP[=<breakpoint>]: indicates that the runtime should direct application processes to stop at some application-defined place and notify they are ready-to-debug. The directive will apply to all processes in the job. Given without a value, the processes stop at whichever such place they reach first. Given a value, that string is the identifier of the one breakpoint at which they are to stop: the runtime passes it to the application in the PMIX_BREAKPOINT environment variable and then waits for the application to report itself ready for debug, so it is up to the application to recognize the name and stop in the corresponding place. This is the one directive whose value may be something other than a truth value, and it is read as a truth value whenever it spells one — so a breakpoint cannot be named true, false, or any other spelling of a boolean.
TIMEOUT=<string>: directs the runtime to terminate the job after it has executed for the specified time. Time is specified in colon-delimited format — e.g., 01:20:13:05 to indicate 1 day, 20 hours, 13 minutes and 5 seconds. Time specified without colons will be assumed to have been given in seconds.
SPAWN-TIMEOUT=<string>: directs the runtime to terminate the job if job launch is not completed within the specified time. Time is specified in colon-delimited format — e.g., 01:20:13:05 to indicate 1 day, 20 hours, 13 minutes and 5 seconds. Time specified without colons will be assumed to have been given in seconds.
REPORT-STATE-ON-TIMEOUT[(=bool)]: directs the runtime to provide a detailed report on job and application process state upon job timeout. This option defaults to a true value if the option is given without a value.
GET-STACK-TRACES[(=bool)]: requests that the runtime provide stack traces on all application processes still executing upon timeout. This option defaults to a true value if the option is given without a value.
REPORT-CHILD-JOBS-SEPARATELY[(=bool)]: directs the runtime to report the exit status of any child jobs spawned by the primary job separately. If false, then the final exit status reported will be zero if the primary job and all spawned jobs exit normally, or the first non-zero status returned by either primary or child jobs. This option defaults to a true value if the option is given without a value.
AGGREGATE-HELP-MESSAGES[(=bool)]: directs the runtime to aggregate help messages, reporting each unique help message once accompanied by the number of processes that reported it. This option defaults to a true value if the option is given without a value.
FWD-ENVIRONMENT[(=bool)]: directs the runtime to forward the entire local environment in support of the application. This option defaults to a true value if the option is given without a value.

The --rtos command line option has no qualifiers.

Note

Directives are case-insensitive. FWD-ENVIRONMENT is the same as fwd-environment.

A value that is neither true nor false is refused rather than guessed at: the truth test underneath reads anything it does not recognize as false, so donotlaunch=maybe would otherwise quietly launch.

--rtos describes the job as a whole — there is no such thing as one app context of an MPMD command line not launching — so it may be written in any app context and applies to all of them. Two app contexts that ask for opposite things are refused, since there is no way to honor both.

Environment options

21.1.4.4.50. `-x <name>[=<value>]`

Export an environment variable, optionally specifying a value. For example:

-x foo exports the environment variable foo and takes its value from the current environment.
-x foo=bar exports the environment variable name foo and sets its value to bar in the started processes.
-x foo* exports all current environmental variables starting with foo.

21.1.4.4.51. `--set-env <name>=<value>`

Set the named environmental variable to the specified value. This will overwrite the existing value, if it exists. Equivalent to the -x foo=val option.

These directives are applied in the order they are given on the command line. --set-env replaces a value outright while --prepend-env and --append-env edit the value already there, so the order is the result: --set-env FOO=1 --prepend-env FOO[:] x leaves FOO=x:1, while --prepend-env FOO[:] x --set-env FOO=1 leaves FOO=1.

21.1.4.4.52. `--unset-env <name>`

Unset the named environmental variable. Note --unset-env foo* unsets all current environmental variables starting with foo.

These directives are applied in the order they are given on the command line. --set-env replaces a value outright while --prepend-env and --append-env edit the value already there, so the order is the result: --set-env FOO=1 --prepend-env FOO[:] x leaves FOO=x:1, while --prepend-env FOO[:] x --set-env FOO=1 leaves FOO=1.

21.1.4.4.53. `--append-env <name>[<c>] <value>`

Append the given value to the named environment variable. The [c] must be appended to the name to specify the separator to be used when appending the value. For example:

--append-env LD_LIBRARY_PATH[:] foo/lib

will result in:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:foo/lib

These directives are applied in the order they are given on the command line. --set-env replaces a value outright while --prepend-env and --append-env edit the value already there, so the order is the result: --set-env FOO=1 --prepend-env FOO[:] x leaves FOO=x:1, while --prepend-env FOO[:] x --set-env FOO=1 leaves FOO=1.

21.1.4.4.54. `--prepend-env <name>[<c>] <value>`

Prepend the given value to the named environment variable. The [c] must be appended to the name to specify the separator to be used when prepending the value. For example:

--prepend-env LD_LIBRARY_PATH[:] foo/lib

will result in:

LD_LIBRARY_PATH=foo/lib:$LD_LIBRARY_PATH

These directives are applied in the order they are given on the command line. --set-env replaces a value outright while --prepend-env and --append-env edit the value already there, so the order is the result: --set-env FOO=1 --prepend-env FOO[:] x leaves FOO=x:1, while --prepend-env FOO[:] x --set-env FOO=1 leaves FOO=1.

Placement options

21.1.4.4.55. `--mapby <directive>`

Note

PRRTE accepts both the new --mapby and the older deprecated --map-by cmd line options. For simplicity, the following description will refer to the new --mapby form.

Processes are mapped based on one of the following directives as applied at the job level:

SLOT assigns procs to each node up to the number of available slots on that node before moving to the next node in the allocation
HWTHREAD assigns a proc to each hardware thread on a node in a round-robin manner up to the number of available slots on that node before moving to the next node in the allocation
CORE (default) assigns a proc to each core on a node in a round-robin manner up to the number of available slots on that node before moving to the next node in the allocation
L1CACHE assigns a proc to each L1 cache on a node in a round-robin manner up to the number of available slots on that node before moving to the next node in the allocation
L2CACHE assigns a proc to each L2 cache on a node in a round-robin manner up to the number of available slots on that node before moving to the next node in the allocation
L3CACHE assigns a proc to each L3 cache on a node in a round-robin manner up to the number of available slots on that node before moving to the next node in the allocation
NUMA assigns a proc to each NUMA region on a node in a round-robin manner up to the number of available slots on that node before moving to the next node in the allocation
PACKAGE assigns a proc to each package on a node in a round-robin manner up to the number of available slots on that node before moving to the next node in the allocation
NODE assigns processes in a round-robin fashion to all nodes in the allocation, with the number assigned to each node capped by the number of available slots on that node
SEQ (often accompanied by the file=<path> qualifier) assigns one process to each node specified in the file. The sequential file is to contain an entry for each desired process, one per line of the file.
PPR:N:resource maps N procs to each instance of the specified resource type in the allocation. The resource may be an hwloc object (ppr:2:package) or a class of device (ppr:2:device=gpu), which places N procs on each such device
RANKFILE (often accompanied by the file=<path> qualifier) assigns one process to the node/resource specified in each entry of the file, one per line of the file.
PE-LIST=a,b assigns procs to each node in the allocation based on the ORDERED qualifier. The list is comprised of comma-delimited ranges of CPUs to use for this job. If the ORDERED qualifier is not provided, then each node will be assigned procs up to the number of available slots, capped by the total number of specified processes and the availability of the specified CPUs (i.e., the number of procs cannot exceed the number of specified CPUs), with each proc bound to all the specified CPUs. If ORDERED is given, then one proc will be assigned to each of the specified CPUs, if available, capped by the number of slots on each node and the total number of specified processes. Providing the OVERLOAD qualifier to the --bindto option removes the check on availability of the CPU in both cases.
DEVICE=<class|name> assigns one proc to each device in the node’s topology, in PCI bus order, placing it on the CPUs local to that device. The value is either a class of device – gpu, network, or block – or the name or UUID of one particular device such as mlx5_0, in which case every proc is placed near that one device. nic, fabric and openfabrics are accepted as spellings of network and mean exactly the same set: one entry per card, whether the node presents it as an OpenFabrics device (mlx5_0), a network interface (ib0), or both.

Note there is no bare --mapby gpu: the class is the value of the device directive, which is what allows other classes of device to be supported later without adding a directive for each.

A device is assigned to a process rather than subdivided between them, so each device takes one process and asking for more processes than there are devices is an error unless SHARED is given.

Binding descends from the device’s locality: the nearest object in the topology that both contains the device and has CPUs. Asking to bind to an object larger than that locality is an error rather than a silent widening, since such a binding is not “near the device” at all. Whether a given --bindto is legal therefore depends on the machine, not on the command line alone. Where every device on a node is equally close to every CPU, the job runs and each proc is still assigned its own device, but a warning reports that the binding could not be made any more specific.

Mapping by a GPU additionally requires that the GPUs can be named to the vendor’s runtime. hwloc records a GPU’s vendor identity – an NVIDIA GPU-<uuid> and its AMD and Intel equivalents – only when it is built with that vendor’s backend (NVML, RSMI, or Level Zero). Built without them, hwloc still reports the GPUs and PRRTE can still place processes next to them, but no process can be told which GPU it got in terms the library it links will accept. PRRTE refuses the request in that case rather than mapping and saying nothing, because the two are indistinguishable while the job runs: the placement looks correct and the only symptom is that every process on the node ends up using the same GPU.

The hwloc that decides this is the one PRRTE was built against, since each daemon discovers its own node – not whatever hwloc happens to be installed alongside it. The other device classes are unaffected: a network or fabric device is named by its own GUIDs or MAC address, which hwloc always has.

Given that identity, each process is also handed its GPUs in the variable its vendor’s runtime reads – CUDA_VISIBLE_DEVICES for NVIDIA, ROCR_VISIBLE_DEVICES for AMD – naming them by the vendor’s own identifier. Only processes actually mapped against a device get this, and a variable already set in the environment is replaced, since --mapby device= is the more specific request. PRRTE never sets the vendor’s device ordering variable (CUDA_DEVICE_ORDER and its equivalents): the identifiers do not depend on the ordering, which is the reason for using them, and changing it would renumber devices for the rest of the process’s life.

Intel is the exception to “named by identity rather than by index”. ZE_AFFINITY_MASK has no identifier form – it takes Level Zero device ordinals – but the ordinals are not guessed: hwloc’s Level Zero backend records the driver and device index zeDeviceGet returned for each device, so the value is read from that enumeration rather than predicted, on the node that will run the process. Because a Level Zero driver reads ZE_FLAT_DEVICE_HIERARCHY first and interprets the mask against the devices that model exposes – the same ordinals name a card under COMPOSITE and a tile under FLAT – the model is stated alongside the mask whenever the process’s environment does not already name one. If it names one and it disagrees, nothing is set and a message says so: overriding a deliberate choice would change how many devices the program sees, and writing a mask that will be read under a different model would silently hand it half the hardware it was assigned.

A process mapped against a network device is handed it the same way, in the variables the fabric libraries read: NCCL_IB_HCA and UCX_NET_DEVICES for a Mellanox or NVIDIA InfiniBand adapter, and PSM3_NIC for an Intel Omni-Path adapter. The device is named by the name those libraries accept – mlx5_0 – which is the name hwloc gave it, so unlike the GPU case there is no identity that can be missing. Nothing is set for an adapter whose fabric has no component behind it, and no variable that takes a device unit number is set at all (HFI_UNIT, FI_OPX_HFI_SELECT): a unit number is meaningful only against the enumeration it came from, and a wrong one there does not fail, it quietly puts the process on another adapter.

The assignment is also readable directly, whether or not a device variable was set, as the PMIX_DEVICE_ID key of the process’s own job data.

Any directive can include qualifiers by adding a colon (:) and any combination of one or more of the following (delimited by colons) to the --mapby option (except where noted). A comma does not separate qualifiers: --mapby device=gpu:ndev=2 gives each process two GPUs, while --mapby device=gpu,ndev=2 names a device called gpu,ndev=2 and is refused. For example:

--mapby package:span                  load balance across the nodes
--mapby core:pe=2:hwtcpus             two hwthreads per process
--mapby device=gpu:ndev=2:interleave  two GPUs each, one per package
--mapby ppr:2:numa:pe=4               two processes per NUMA domain

PE=n bind n CPUs to each process (can not be used in combination with rankfile or pe-list directives)
SPAN load balance the processes across the allocation by treating the allocation as a single “super-node” (can not be used in combination with slot, node, seq, ppr, rankfile, or pe-list directives). One process is placed on each object in turn, cycling across the nodes, so a job that does not fill the allocation spreads over all of it instead of filling the first nodes. On three 4-slot nodes, -n 8 --mapby core:SPAN places 3, 3 and 2 processes, where --mapby core alone places 4 and 4 and leaves the third node empty.
OVERSUBSCRIBE allow more processes on a node than processing elements. Note that this describes the whole job. It may be written in a per-app --mapby string, where it is taken to describe the job (see “Per-app-context mapping” below).
NOOVERSUBSCRIBE means !OVERSUBSCRIBE. Note that this describes the whole job (see above).
NOLOCAL do not launch processes on the same node as prun. This qualifier may be applied per app context in an MPMD job — see “Per-app-context mapping” below.
HWTCPUS use hardware threads as CPUs. The processes are bound to hardware threads by default, and a node whose slot count PRRTE determined by counting its cores offers this job one slot per hardware thread instead — for this job only (see the definition of “slot”)
CORECPUS use cores as CPU slots (default)
INHERIT indicates that a child job (i.e., one spawned from within an application) shall inherit the placement policies of the parent job that spawned it. Note that this describes the whole job; written in a per-app --mapby string it is taken to describe the job.
NOINHERIT means !INHERIT. Note that this describes the whole job (see above).
FILE=<path> (path to file containing sequential or rankfile entries).
INTERLEAVE[=<level>] only applies to the DEVICE directive. It reorders the device list so that consecutive processes land on different objects of the given level — package (the default), numa, l3cache, l2cache or l1cache. On a node with two GPUs per socket, device=gpu:interleave places the first two processes on different sockets rather than filling the first. node is not an accepted level: interleaving across nodes is what SPAN already expresses. A level that does not divide the devices into groups leaves the order unchanged, so the qualifier is safe to leave in a default mapping policy.
SHARED[=true|false] only applies to the DEVICE directive. It permits several processes to be assigned the same device. The default is false: a device is assigned to a process rather than subdivided between them, so a job asking for more processes than there are devices is an error unless this is given. This is a separate question from the overload-allowed qualifier to --bindto, which concerns running more processes than there are CPUs.
NDEV=<n> only applies to the DEVICE directive. It assigns n devices to each process rather than one.

This changes what a process is placed against, and that is worth understanding. A process holding two GPUs attached to different NUMA domains is local to neither of them alone — it is local to whatever contains them both. So the locality of a process becomes the common ancestor of its devices’ localities, and that decides how coarse a binding may be. On a node with two GPUs per socket, ndev=2 therefore makes each process package-local, which means --bindto package is legitimate in that case and remains an error without ndev.

A finer binding is chosen from the devices’ own localities, not from anywhere in that ancestor: --bindto numa binds to the NUMA domain of one of the process’s GPUs, and --bindto core to a core in one of them — never to a part of the package that is local to neither.

Devices are handed out in groups taken in order from the device list, so a group is a contiguous run of that order and the INTERLEAVE qualifier composes with this one. On a node with two GPUs per socket, interleaving across packages and then taking ndev=2 gives each process one GPU from each socket.
ORDERED only applies to the PE-LIST option to indicate that procs are to be bound to each of the specified CPUs in the order in which they are assigned (i.e., the first proc on a node shall be bound to the first CPU in the list, the second proc shall be bound to the second CPU, etc.)

Note

Directives and qualifiers are case-insensitive and can be shortened to the minimum number of characters to uniquely identify them. Thus, L1CACHE can be given as l1cache or simply as L1. A shortening that fits more than one of them is refused rather than guessed at — N could be NUMA or NODE, and :S could be SPAN or SHARED — as is a value given to one that takes none: SPAN=false is an error, not SPAN.

The type of CPU (core vs hwthread) used in the mapping algorithm is determined as follows:

by user directive on the command line via the HWTCPUS qualifier to the --mapby directive
by setting the mapby MCA parameter to include the HWTCPUS qualifier (e.g., --prtemca mapby :HWTCPUS when starting the DVM). This parameter sets the default value for a PRRTE DVM — qualifiers are carried across to DVM jobs started via prun unless overridden by the user’s command line
defaults to CORE in topologies where core CPUs are defined, and to hwthreads otherwise.

If your application uses threads, then you probably want to ensure that you are either not bound at all (by specifying --bindto none), or bound to multiple cores using an appropriate binding level or specific number of processing elements per application process via the PE=# qualifier to the --mapby command line directive.

Per-app-context mapping (MPMD jobs)

In a multi-program multiple-data (MPMD) job, each application context separated by : on the prun command line may carry its own --mapby, --rankby, and --bindto directives.

Which app such a directive describes follows one rule: the first app segment is where the command line speaks for the job. A directive written there and nowhere else applies to the whole job, however many apps follow. A directive written on any later app describes that app alone; apps that were given none take the ordinary defaults, since an app that says nothing is not agreeing with one that did.

Examples:

prun --mapby core -n 4 app1 : -n 2 app2

Both apps are mapped by core: the only directive is on the first app, so it describes the job.

prun --mapby core -n 4 app1 : --mapby node --rankby fill -n 2 app2

Here app1 is mapped by core and app2 is mapped by node with fill ranking. No binding directive was given, so both apps take the default binding policy.

prun -n 4 app1 : --mapby node -n 2 app2

Only app2 is mapped by node; app1 takes the default mapping.

prun --mapby slot:nolocal -n 8 app1 : --mapby slot -n 1 app2

Here app1 is excluded from the head node while app2 may run anywhere, including the head node.

Every mapping policy may be given per app — seq, rankfile, ppr:N:obj and pe-list=... included, each with the file or pattern it needs — so two apps of one job may be placed by two different mapping components. --display map-devel names the component that placed each app.

Qualifiers that describe the whole job:

OVERSUBSCRIBE / NOOVERSUBSCRIBE and INHERIT / NOINHERIT describe the job, not an app, so wherever they are written they are applied to the job. Apps that say nothing about them are silent, not dissenting; only apps that answer the same question in opposite ways are refused, and that aborts the job with an error.
NOLOCAL may be applied per app. It prevents that specific app’s processes from running on the head node without affecting other apps.

Per-app directives can also be supplied via the PMIx_Spawn API by placing PMIX_MAPBY, PMIX_RANKBY, PMIX_BINDTO and PMIX_PPR keys in the per-app info[] array on the corresponding pmix_app_t. On that path there is no “first app” rule: the array a key was written in says what it describes.

Note that PMIX_MAPPER is not supported, per job or per app, and a spawn request carrying it is refused. Naming a mapping component says nothing that PMIX_MAPBY has not already said — the mapping policy is what selects the component — and the two can contradict each other. Describe the placement you want with PMIX_MAPBY and let PRRTE choose the component that performs it.

A more detailed description of the mapping, ranking, and binding procedure can be obtained via the --help placement option.

21.1.4.4.56. `--rankby <directive>`

Note

PRRTE accepts both the new --rankby and the older deprecated --rank-by cmd line options. For simplicity, the following description will refer to the new --rankby form.

PRRTE automatically ranks processes for each job starting from zero. Regardless of the algorithm used, rank assignments span applications in the same job — i.e., a command line of

-n 3 app1 : -n 2 app2

will result in app1 having three processes ranked 0-2 and app2 having two processes ranked 3-4.

By default, process ranks are assigned in accordance with the mapping directive: jobs mapped by node are ranked by NODE, jobs mapped with the SPAN qualifier are ranked by SPAN, jobs mapped to a hardware object (hwthread, core, cache, NUMA region, or package) are ranked by FILL, and all other jobs are ranked by SLOT. However, users can override the default by specifying any of the following directives using the --rankby command line option:

SLOT assigns ranks to each process on a node in the order in which the mapper assigned them. This is the default for jobs mapped by slot, and is provided as an explicit option to allow users to override any alternative default. When mapping to a specific resource type, procs assigned to a given instance of that resource on a node will be ranked on a per-resource basis on that node before moving to the next node.
NODE assigns ranks round-robin on a per-node basis
FILL assigns ranks to procs mapped to a particular resource type on each node, filling all ranks on that resource before moving to the next resource on that node. For example, procs mapped by L1cache would have all procs on the first L1cache ranked sequentially before moving to the second L1cache on the node. Once all procs on the node have been ranked, ranking would continue on the next node.
SPAN assigns ranks round-robin to procs mapped to a particular resource type, treating the collection of resource instances spanning the entire allocation as a single “super node” before looping around for the next pass. Thus, ranking would begin with the first proc on the first L1cache on the first node, then the next rank would be assigned to the first proc on the second L1cache on that node, proceeding across until the first proc had been ranked on all L1cache used by the job before circling around to rank the second proc on each object.

The rankby command line option has no qualifiers.

Note

Directives are case-insensitive. SPAN is the same as span.

Per-app-context ranking (MPMD jobs)

In a multi-program multiple-data (MPMD) job, each application context may carry its own --rankby directive, placed ahead of that app’s executable. The rule is the one described for --mapby: a directive written on the first app segment and nowhere else describes the whole job; otherwise each app that carries a directive is ranked by its own, and apps that carry none take the default ranking policy.

Example:

prun --rankby fill -n 4 app1 : --rankby node -n 2 app2

Rank assignments always span all application contexts in the job and remain globally contiguous: the first app in the command line receives ranks starting from 0, and each subsequent app starts from the next unassigned rank, regardless of the per-app ranking directive. The per-app directive controls only the order in which processes within that app are assigned their ranks relative to one another.

For example, a command line of

--rankby fill -n 3 app1 : --rankby node -n 2 app2

will result in app1 having three processes ranked 0-2 (assigned fill-style) and app2 having two processes ranked 3-4 (assigned node-style).

A more detailed description of the mapping, ranking, and binding procedure can be obtained via the --help placement option.

21.1.4.4.57. `--bindto <directive>`

Note

PRRTE accepts both the new --bindto and the older deprecated --bind-to cmd line options. For simplicity, the following description will refer to the new --bindto form.

By default, processes are bound to individual CPUs (either COREs or HWTHREADs, as defined by default or by user specification for the job — a job or app mapped with the HWTCPUS qualifier binds each process to a hardware thread). On nodes that are OVERSUBSCRIBEd (i.e., where the number of procs exceeds the number of assigned slots), the default is to not bind the processes.

Note

Processes from prior jobs that are already executing on a node are not “unbound” when a new job mapping results in the node becoming oversubscribed.

Binding is performed to the first available specified object type within the object where the process was mapped. In other words, binding can only be done to the mapped object or to a resource located beneath that object.

An object is considered completely consumed when the number of processes bound to it equals the number of CPUs within it. Unbound processes are not considered in this computation. Additional processes cannot be mapped to consumed objects unless the OVERLOAD qualifier is provided via the --bindto command line option.

Note that directives and qualifiers are case-insensitive and can be shortened to the minimum number of characters to uniquely identify them. Thus, L1CACHE can be given as l1cache or simply as L1. A shortening that fits more than one of them is refused rather than guessed at — N could be NONE or NUMA.

Supported binding directives include:

NONE does not bind the processes
HWTHREAD binds each process to a single hardware thread. This requires that hwthreads be treated as independent CPUs (i.e., that either the HWTCPUS qualifier be provided to the --mapby option or that hwthreads be designated as CPUs by default).
CORE binds each process to a single core. This can be done whether hwthreads or cores are being treated as independent CPUs provided that mapping is performed at the core or higher level.
L1CACHE binds each process to all the CPUs in an L1 cache.
L2CACHE binds each process to all the CPUs in an L2 cache
L3CACHE binds each process to all the CPUs in an L3 cache
NUMA binds each process to all the CPUs in a NUMA region
PACKAGE binds each process to all the CPUs in a PACKAGE

Any directive can include qualifiers by adding a colon (:) and any combination of one or more of the following to the --bindto option, each separated from the next by a colon as well - never a comma. For example, --bindto core:overload-allowed:report or --bindto numa:limit=2:

OVERLOAD indicates that objects can have more processes bound to them than CPUs within them
NO-OVERLOAD indicates that they cannot. This is the default, so the qualifier is only useful for overriding an OVERLOAD that a DVM-wide default (the bindto MCA parameter) has already applied.
IF-SUPPORTED indicates that the job should continue to be launched and executed even if binding cannot be performed as requested.
LIMIT=n limits the number of processes bound to each eligible representative of the specified type to the given number. For example, specifying --bindto l3:limit=2 would direct PRRTE to bind ranks to the L3caches, limiting the number of processes bound to each l3cache to two - i.e., bind 2 processes to a given l3cache, and then move on to the next.
REPORT reports each process’ binding as it is applied, which is the same thing --report-bindings does. It describes the job as a whole: unlike the other qualifiers here it cannot be given per application context, and it cannot be given as a DVM-wide default through the bindto MCA parameter.

Note

Directives and qualifiers are case-insensitive. OVERLOAD is the same as overload.

Per-app-context binding (MPMD jobs)

In a multi-program multiple-data (MPMD) job, each application context may carry its own --bindto directive, placed ahead of that app’s executable. The rule is the one described for --mapby: a directive written on the first app segment and nowhere else describes the whole job; otherwise each app that carries a directive is bound by its own, and apps that carry none take the default binding policy — they do not inherit another app’s directive.

Example:

prun --bindto core -n 4 app1 : --bindto none -n 2 app2

Here app1 processes are bound to individual cores while app2 processes are left unbound, all within the same job.

Per-app binding is derived from the mapping options resolved for each app context. It therefore follows the same constraints as job-level binding: binding can only be done to the mapped object or to a resource located beneath it.

A more detailed description of the mapping, ranking, and binding procedure can be obtained via the --help placement option.

Input and output options

21.1.4.4.58. `--output <directives>`

The output command line directive must be accompanied by a comma-delimited list of case-insensitive options that control how output is generated. The full directive need not be provided — only enough characters are required to uniquely identify the directive. For example, MERGE is sufficient to represent the MERGE-STDERR-TO-STDOUT directive — while TAG can not be used to represent TAG-DETAILED (though TAG-D would suffice).

Supported values include:

TAG marks each output line with the [job,rank]<stream>: of the process that generated it
TAG-DETAILED marks each output line with a detailed annotation containing [namespace,rank][hostname:pid]<stream>: of the process that generated it
TAG-FULLNAME marks each output line with the [namespace,rank]<stream>: of the process that generated it
RANK marks each output line with the [rank]<stream>: of the process that generated it
TIMESTAMP prefixes each output line with a [datetime]<stream>: stamp. Note that the timestamp will be the time when the line is output by the DVM and not the time when the source output it
XML provides all output in a pseudo-XML format
MERGE-STDERR-TO-STDOUT merges stderr into stdout
DIR=DIRNAME redirects output from application processes into DIRNAME/job/rank/std[out,err,diag]. The provided name will be converted to an absolute path
FILE=FILENAME redirects output from application processes into filename.rank. The provided name will be converted to an absolute path

Only one of DIR and FILE may be given.

Supported qualifiers include:

COPY — when combined with the DIR or FILE option, directs that a copy of the output also be sent to the stdout/err streams
NOCOPY — when combined with the DIR or FILE option, do not copy the output to the stdout/err streams (default). COPY and NOCOPY cannot both be given.
RAW — do not buffer the output into complete lines, but instead output it as it is received.

PATTERN — when combined with the FILE option, the given name is a pattern you compose yourself rather than a stem to be annotated with the namespace and rank. These conversions are expanded in it:

Conversion	Expands to
`%n`	the job’s namespace
`%r`	the process’s rank
`%R`	the rank, zero-padded to the width of the job’s largest rank
`%h`	the hostname of the node the process ran on
`%%`	a literal `%` character

The stream suffix (.out or .err) is still appended, so stdout and stderr can never land on the same file. A pattern containing no % at all is therefore simply a fixed name shared by every process in the job.

For example, --output file=run/%h/rank-%R:pattern writes each process’s stdout to run/<hostname>/rank-<rank>.out.

Any conversion other than those listed above is an error, reported when the command line is parsed. PATTERN given without FILE is also an error.

Note

Directives and qualifiers are case-insensitive.

Every directive and qualifier above that asks a yes-or-no question — everything except DIR and FILE, which name a place to write — may also be given an explicit truth value:

--output tag        the directive is requested
--output tag=1      the same, said explicitly
--output tag=0      the directive is NOT requested

True may also be written T, Y, TRUE, YES or ENABLE, and false F, N, FALSE, NO or DISABLE — case-insensitively, and as whole words (TR is not an abbreviation of TRUE).

A value that is neither true nor false is refused rather than guessed at: the truth test underneath reads anything it does not recognize as false, so tag=maybe would otherwise quietly turn tagging off.

--output describes the job as a whole: there is no way to give one app context of an MPMD command line different output handling from another. It may therefore be written in any app context and applies to all of them. Two app contexts that ask for opposite things are refused, since there is no way to honor both.

21.1.4.4.59. `--stdin <rank>`

Specify which application process is to receive the command’s stdin: all, none, or the rank of a single process (default: 0, indicating rank 0). Any other value is refused.

21.1.4.4.60. `--xterm <ranks>`

Display the output of the specified application processes in their own xterm window. Ranks are given as a comma-delimited list of ranks and inclusive ranges (for example, 1,3-6,9), or as all. A trailing ! (for example, 1,3!) keeps each window open after its process exits. The xterm is started by the daemon on the node where the process runs, so it needs a usable DISPLAY there: forward it with -x DISPLAY, or start the DVM with an ssh -X launch agent.

Debug options

21.1.4.4.61. `--display <directives>`

The display command line directive must be accompanied by a comma-delimited list of case-insensitive options indicating what information about the job and/or allocation is to be displayed. The full directive need not be provided — only enough characters are required to uniquely identify the directive. For example, ALL is sufficient to represent the ALLOCATION directive — while MAP can not be used to represent MAP-DEVEL (though MAP-D would suffice).

Supported values include:

ALLOCATION displays the detected hosts and slot assignments for this job
BINDINGS displays the resulting bindings applied to processes in this job
MAP displays the resulting locations assigned to processes in this job
MAP-DEVEL displays a more detailed report on the locations assigned to processes in this job that includes local and node ranks, assigned bindings, and other data
TOPO[=LIST] displays the topology of each node in the provided semicolon-delimited list of nodes allocated to the job (defaults to all nodes). An empty list (TOPO=) is refused.
CPUS[=LIST] displays the available CPUs on the provided semicolon-delimited list of nodes (defaults to all nodes)

The display command line directive can include qualifiers by adding a colon (:) and any combination of one or more of the following (delimited by colons):

PARSEABLE directs that the output be provided in a format that is easily parsed by machines. Note that PARSABLE is also accepted as a typical spelling for the qualifier.
PHYSICAL directs that the output of the BINDINGS option be displayed using physical (instead of logical) CPU IDs.

Provided qualifiers will apply to all of the display directives unless noted. Note that directives and qualifiers are case-insensitive.

Every directive and qualifier above that asks a yes-or-no question — everything except TOPO and CPUS, which name a list of nodes — may also be given an explicit truth value:

--display map        the directive is requested
--display map=1      the same, said explicitly
--display map=0      the directive is NOT requested

True may also be written T, Y, TRUE, YES or ENABLE, and false F, N, FALSE, NO or DISABLE — case-insensitively, and as whole words (TR is not an abbreviation of TRUE).

A value that is neither true nor false is refused rather than guessed at: the truth test underneath reads anything it does not recognize as false, so map=maybe would otherwise quietly turn the display off.

--display describes the job as a whole: there is no such thing as one app context of an MPMD command line being displayed. It may therefore be written in any app context and applies to all of them. Two app contexts that ask for opposite things are refused, since there is no way to honor both.

21.1.4.4.62. `--timeout <seconds>`

Timeout the job if execution is not complete after the specified number of seconds. The value must be a non-negative integer; zero means no timeout. If this option is not given, the value of the MPIEXEC_TIMEOUT environment variable, if set, is used instead. See also --report-state-on-timeout and --get-stack-traces.

21.1.4.4.63. `--spawn-timeout <seconds>`

Timeout the job if spawn takes more than the specified number of seconds.

21.1.4.4.64. `--report-state-on-timeout`

Report all job and process states upon timeout.

21.1.4.4.65. `--get-stack-traces`

Get stack traces of all application processes still executing upon timeout.

21.1.4.4.66. `--stop-on-exec`

If supported by the platform, stop each application process immediately upon exec’ing it, pending release by a debugger. The directive applies to all processes in the job. This is the same as --rtos stop-on-exec.

21.1.4.4.67. `--stop-in-init`

Include the PMIX_DEBUG_STOP_IN_INIT attribute in the application’s job info directing that the processes stop in PMIx_Init pending release. The directive applies to all processes in the job. This is the same as --rtos stop-in-init.

21.1.4.4.68. `--stop-in-app[=<breakpoint>]`

Include the PMIX_DEBUG_STOP_IN_APP attribute in the application’s job info directing that the processes stop at an application-determined point pending release. The directive applies to all processes in the job. This is the same as --rtos stop-in-app.

An optional argument names the one breakpoint at which they are to stop — e.g., --stop-in-app=mpi-init. PRRTE cannot know where any given breakpoint lives; all it can do is pass the name to the application in the PMIX_BREAKPOINT environment variable and then wait for the “ready for debug” event the application generates when it gets there. It is therefore up to the application to recognize the name and stop in the corresponding place. Given without an argument, the processes stop at whichever such place they reach first.

Since the argument is read as a boolean when it spells one, a breakpoint cannot be named true, false, or any other spelling of a truth value.

21.1.4.4.69. `--no-aggregate-help`

Do not aggregate help messages issued by the application processes.

By default, identical help output generated by many processes is combined into a single report naming the number of processes that hit it. This option turns that off, so every process’s message is reported as it is received.

21.1.4.5. DEPRECATED COMMAND LINE OPTIONS

The following options are still accepted, but are converted to their modern equivalent and a warning is printed. They will be removed in a future release; use the replacement shown.

--map-by, --rank-by, --bind-to, --runtime-options

Hyphenated spellings of --mapby, --rankby, --bindto and --rtos. These are converted silently.

--mca <key> <value>

Syntax: --mca <key> <value>, where key is the parameter name and value is the parameter value.

Pass generic MCA parameters — i.e., parameters whose project affiliation must be determined by PRRTE based on matching the name of the parameter with defined values from various projects that PRRTE knows about.

Deprecated

This translation can be incomplete (e.g., if a project adds or changes parameters) — thus, it is strongly recommended that users use project-specific parameters such as --prtemca or --pmixmca.

--rankfile <filename>

Replaced by --mapby rankfile:file=<filename>.

--nolocal

Replaced by the nolocal qualifier to --mapby.

--oversubscribe, --nooversubscribe

Replaced by the oversubscribe and nooversubscribe qualifiers to --mapby.

--use-hwthread-cpus

Replaced by the hwtcpus qualifier to --mapby.

--cpu-set <list>, --cpu-list <list>

Replaced by --mapby pe-list=<list>.

--cpus-per-proc <n>, --cpus-per-rank <n>

Replaced by the pe=<n> qualifier to --mapby.

--bind-to-core

Replaced by --bindto core.

--bynode, --bycore, --byslot

Replaced by --mapby node, --mapby core and --mapby slot respectively.

--npernode <n>, --pernode

Replaced by --mapby ppr:<n>:node and --mapby ppr:1:node respectively.

--npersocket <n>

Replaced by --mapby ppr:<n>:package.

--ppr <n>:<resource>

Replaced by --mapby ppr:<n>:<resource>.

--tag-output, --timestamp-output, --xml, --merge-stderr-to-stdout

Replaced by --output tag, --output timestamp, --output xml and --output merge-stderr-to-stdout respectively.

--output-directory <dir>, --output-filename <file>

Replaced by --output dir=<dir> and --output file=<file> respectively.

--display-map, --display-devel-map, --display-allocation, --display-devel-allocation, --report-bindings

Replaced by --display map, --display map-devel, --display allocation (for both allocation options) and --display bind respectively.

--display-topo

Replaced by the topo directive to --display.

--do-not-launch

Replaced by --rtos donotlaunch.

--show-progress, --report-child-jobs-separately, --fwd-environment

Replaced by the show-progress, report-child-jobs-separately and fwd-environment runtime options (--rtos).

--output-proctable[=<channel>]

Replaced by --rtos output-proctable[=<channel>].

--debug

Has no effect; it is accepted, with a warning, so that old command lines still parse.

--hetero-nodes: Has no effect beyond a warning: heterogeneous nodes are detected automatically. Give --uniform-nodes if the nodes are known to be identical.

21.1.4.6. ENVIRONMENT

MPIEXEC_TIMEOUT: The job timeout in seconds, used when --timeout is not given.
PRTEPROXY_USE_DVM: When set, prterun behaves as if --dvm had been given without an argument, submitting the job to an existing DVM.
PRTE_ALLOW_RUN_AS_ROOT, PRTE_ALLOW_RUN_AS_ROOT_CONFIRM: When both are set to 1, permit execution as root — see --allow-run-as-root.
PRTE_MCA_<name>, PMIX_MCA_<name>: Set the PRRTE or PMIx MCA parameter <name>, as --prtemca and --pmixmca do on the command line.

21.1.4.7. EXIT STATUS

prterun reports the outcome of the job it launched, exactly as prun(1) does:

0 if the job completed successfully.
the application’s exit status, if a process of the job exited with a non-zero status or was killed. Only the low eight bits survive, and it is the status recorded for the first process whose failure caused the job to be terminated, which is also the process named in the error message.
a non-zero status derived from the reason the job ended, when the job failed without any process having produced an exit status of its own — a job that could not be mapped or launched, for instance. Do not attach meaning to the particular value beyond “not zero”.

A failure to start the DVM, or an invalid command line, also exits non-zero.

21.1.4.8. EXAMPLES

Run four copies of a program across the hosts named in a hostfile:

shell$ prterun --hostfile myhosts -n 4 ./a.out

Show where the processes of a job would be placed and bound, without launching it:

shell$ prterun --rtos donotlaunch --display map,bind -n 16 ./a.out

Run the same command against a persistent DVM started earlier with prte:

shell$ prterun --dvm search -n 4 ./a.out

21.1.4. prterun

21.1.4.1. SYNOPSIS

21.1.4.2. DESCRIPTION

21.1.4.3. DIRECTIVES AND QUALIFIERS

21.1.4.4. OPTIONS

21.1.4.4.1. -h | --help [<option>]

21.1.4.4.2. -v | --verbose

21.1.4.4.3. -V | --version

21.1.4.4.4. --prtemca <key> <value>

21.1.4.4.5. --pmixmca <key> <value>

21.1.4.4.6. --tune <files>

21.1.4.4.7. --dvm <arg>

21.1.4.4.8. --default-hostfile <filename>

21.1.4.4.9. --uniform-nodes

21.1.4.4.10. --prefix <dir>

21.1.4.4.11. --noprefix

21.1.4.4.12. --pmix-prefix <dir>

21.1.4.4.13. --launch-agent <executable>

21.1.4.4.14. --max-vm-size <num>

21.1.4.4.15. --keepalive <fd>

21.1.4.4.16. --system-server

21.1.4.4.17. --set-sid

21.1.4.4.18. --report-pid <arg>

21.1.4.4.19. --report-uri <arg>

21.1.4.4.20. --allow-run-as-root

21.1.4.4.21. --forward-signals <signals>

21.1.4.4.22. --debug-daemons

21.1.4.4.23. --debug-daemons-file

21.1.4.4.24. --leave-session-attached

21.1.4.4.25. --personality <name>

21.1.4.4.26. -n | -c | --np <num>

21.1.4.4.27. -N <num>

21.1.4.4.28. --app <filename>

21.1.4.4.29. -H | --host <hosts>

21.1.4.4.30. --hostfile <filename>

21.1.4.4.31. --machinefile <filename>

21.1.4.4.32. --add-host <hosts>

21.1.4.4.33. --add-hostfile <filename>

21.1.4.4.34. --activate <hosts>

21.1.4.4.35. --alloc-id <id>

21.1.4.4.36. --alloc-refid <id>

21.1.4.4.37. --session-id <id>

21.1.4.4.38. --path <dir>

21.1.4.4.39. --wdir <dir>

21.1.4.4.40. --wd <dir>

21.1.4.4.41. --set-cwd-to-session-dir

21.1.4.4.42. --pset <name>

21.1.4.4.43. --preload-files <files>

21.1.4.4.44. -s | --preload-binary

21.1.4.4.45. --app-pmix-prefix <dir>

21.1.4.4.46. --no-app-prefix

21.1.4.4.47. --gpu-support <bool>

21.1.4.4.48. --memory-alloc-kinds <kinds>

21.1.4.4.49. --rtos <directives> | --runtime-options <directives>

21.1.4.4.50. -x <name>[=<value>]

21.1.4.4.51. --set-env <name>=<value>

21.1.4.4.52. --unset-env <name>

21.1.4.4.53. --append-env <name>[<c>] <value>

21.1.4.4.54. --prepend-env <name>[<c>] <value>

21.1.4.4.55. --mapby <directive>

21.1.4.4.56. --rankby <directive>

21.1.4.4.57. --bindto <directive>

21.1.4.4.58. --output <directives>

21.1.4.4.59. --stdin <rank>

21.1.4.4.60. --xterm <ranks>

21.1.4.4.61. --display <directives>

21.1.4.4.62. --timeout <seconds>

21.1.4.4.63. --spawn-timeout <seconds>

21.1.4.4.64. --report-state-on-timeout

21.1.4.4.65. --get-stack-traces

21.1.4.4.66. --stop-on-exec

21.1.4.4.67. --stop-in-init

21.1.4.4.68. --stop-in-app[=<breakpoint>]

21.1.4.4.69. --no-aggregate-help

21.1.4.5. DEPRECATED COMMAND LINE OPTIONS

21.1.4.6. ENVIRONMENT

21.1.4.7. EXIT STATUS

21.1.4.8. EXAMPLES

21.1.4.4.1. `-h` | `--help [<option>]`

21.1.4.4.2. `-v` | `--verbose`

21.1.4.4.3. `-V` | `--version`

21.1.4.4.4. `--prtemca <key> <value>`

21.1.4.4.5. `--pmixmca <key> <value>`

21.1.4.4.6. `--tune <files>`

21.1.4.4.7. `--dvm <arg>`

21.1.4.4.8. `--default-hostfile <filename>`

21.1.4.4.9. `--uniform-nodes`

21.1.4.4.10. `--prefix <dir>`

21.1.4.4.11. `--noprefix`

21.1.4.4.12. `--pmix-prefix <dir>`

21.1.4.4.13. `--launch-agent <executable>`

21.1.4.4.14. `--max-vm-size <num>`

21.1.4.4.15. `--keepalive <fd>`

21.1.4.4.16. `--system-server`

21.1.4.4.17. `--set-sid`

21.1.4.4.18. `--report-pid <arg>`

21.1.4.4.19. `--report-uri <arg>`

21.1.4.4.20. `--allow-run-as-root`

21.1.4.4.21. `--forward-signals <signals>`

21.1.4.4.22. `--debug-daemons`

21.1.4.4.23. `--debug-daemons-file`

21.1.4.4.24. `--leave-session-attached`

21.1.4.4.25. `--personality <name>`

21.1.4.4.26. `-n` | `-c` | `--np <num>`

21.1.4.4.27. `-N <num>`

21.1.4.4.28. `--app <filename>`

21.1.4.4.29. `-H` | `--host <hosts>`

21.1.4.4.30. `--hostfile <filename>`

21.1.4.4.31. `--machinefile <filename>`

21.1.4.4.32. `--add-host <hosts>`

21.1.4.4.33. `--add-hostfile <filename>`

21.1.4.4.34. `--activate <hosts>`

21.1.4.4.35. `--alloc-id <id>`

21.1.4.4.36. `--alloc-refid <id>`

21.1.4.4.37. `--session-id <id>`

21.1.4.4.38. `--path <dir>`

21.1.4.4.39. `--wdir <dir>`

21.1.4.4.40. `--wd <dir>`

21.1.4.4.41. `--set-cwd-to-session-dir`

21.1.4.4.42. `--pset <name>`

21.1.4.4.43. `--preload-files <files>`

21.1.4.4.44. `-s` | `--preload-binary`

21.1.4.4.45. `--app-pmix-prefix <dir>`

21.1.4.4.46. `--no-app-prefix`

21.1.4.4.47. `--gpu-support <bool>`

21.1.4.4.48. `--memory-alloc-kinds <kinds>`

21.1.4.4.49. `--rtos <directives>` | `--runtime-options <directives>`

21.1.4.4.50. `-x <name>[=<value>]`

21.1.4.4.51. `--set-env <name>=<value>`

21.1.4.4.52. `--unset-env <name>`

21.1.4.4.53. `--append-env <name>[<c>] <value>`

21.1.4.4.54. `--prepend-env <name>[<c>] <value>`

21.1.4.4.55. `--mapby <directive>`

21.1.4.4.56. `--rankby <directive>`

21.1.4.4.57. `--bindto <directive>`

21.1.4.4.58. `--output <directives>`

21.1.4.4.59. `--stdin <rank>`

21.1.4.4.60. `--xterm <ranks>`

21.1.4.4.61. `--display <directives>`

21.1.4.4.62. `--timeout <seconds>`

21.1.4.4.63. `--spawn-timeout <seconds>`

21.1.4.4.64. `--report-state-on-timeout`

21.1.4.4.65. `--get-stack-traces`

21.1.4.4.66. `--stop-on-exec`

21.1.4.4.67. `--stop-in-init`

21.1.4.4.68. `--stop-in-app[=<breakpoint>]`

21.1.4.4.69. `--no-aggregate-help`