15.1.6. psched

psched — a modest scheduler for PRRTE

15.1.6.1. SYNOPSIS

shell$ psched ...options...

15.1.6.2. DESCRIPTION

psched is a standalone daemon that acts as a dynamic scheduler for PRRTE.

Extensive help documentation for this command is provided through psched --help [topic].

15.1.6.3. COMMAND LINE OPTIONS

The following command line options are recognized by psched.

15.1.6.3.1. General command line options

The following general command line options are available.

  • -h | --help: This help message

  • -v | --verbose: Enable extra verbose output

  • -V | --version: Print version and exit

15.1.6.3.2. MCA parameters

  • --pmixmca <key> <value>: Pass context-specific PMIx MCA parameters (<key>> is the parameter name; <value> is the parameter value). See below for details.

  • --prtemca <key> <value>: Pass context-specific PRRTE MCA parameters to the scheduler. See below for details.

  • --tune <filename>: File(s) containing MCA params for tuning scheduler operations. See below for details.

15.1.6.3.3. Output options

15.1.6.3.4. Resource options

  • --default-hostfile <filename>: Provide a default hostfile.

  • -H | --host <list>: Comma-delimited list of hosts to be included in scheduler queues See below for details.

  • --hostfile <filename>: Provide a hostfile. See below for details.

  • --machinefile <filename>: Synonym for --hostfile.

15.1.6.3.5. Specific options

  • --allow-run-as-root: Allow execution as root (STRONGLY DISCOURAGED). See below for details.

  • --daemonize: Daemonize the scheduler into the background.

  • --no-ready-msg: Do not output a “ready” message when the scheduler has completed initializing.

  • --set-sid: Direct the scheduler to separate from the current session.

  • --tmpdir <dir>: Set the root for the session directory tree.

  • --report-pid <value>: Print out PID on stdout (-), stderr (+), or a filename (anything else)

  • --report-uri <value>: Print out URI on stdout (-), stderr (+), or a filename (anything else)

  • --keepalive <filename>: Named pipe filename to monitor — psched will terminate upon closure

15.1.6.3.6. Debug options

  • --debug: Synonym for --leave-session-attached

  • --leave-session-attached: Do not discard stdout/stderr of remote PRTE daemons. See below for details.

15.1.6.3.7. Details of individual command line options

The sections below offer more detail than the abbreviated lists, above.

15.1.6.3.7.1. The --pmixmca option

Pass a PMIx MCA parameter

Syntax: --pmixmca <key> <value>, where key is the parameter name and value is the parameter value.

15.1.6.3.7.2. The --prtemca option

Pass a PRRTE MCA parameter.

Syntax: --prtemca <key> <value>, where key is the parameter name and value is the parameter value.

15.1.6.3.7.3. The --tune option

Comma-delimited list of one or more files containing PRRTE and PMIx MCA params for tuning DVM and/or application operations. Parameters in the file will be treated as generic parameters and subject to the translation rules/uncertainties. See --help mca for more information.

Syntax in the file is:

param = value

with one parameter and its associated value per line. Empty lines and lines beginning with the # character are ignored, as is any whitespace around the = character.

15.1.6.3.7.4. The --allow-run-as-root option

Allow execution as root (STRONGLY DISCOURAGED).

Running as root exposes the user to potentially catastrophic file system corruption and damage — e.g., if the user accidentally points the root of the session directory to a system required point, this directory and all underlying elements will be deleted upon job completion, thereby rendering the system inoperable.

It is recognized that some environments (e.g., containers) may require operation as root, and that the user accepts the risks in those scenarios. Accordingly, one can override PRRTE’s run-as-root protection by providing one of the following:

  • The --allow-run-as-root command line directive

  • Adding BOTH of the following environmental parameters:

    • PRTE_ALLOW_RUN_AS_ROOT=1

    • PRTE_ALLOW_RUN_AS_ROOT_CONFIRM=1

Again, we recommend this only be done if absolutely necessary.

15.1.6.3.7.5. The --output option

The output command line directive must be accompanied by a comma-delimited list of case-insensitive options that control how output is generated. The full directive need not be provided — only enough characters are required to uniquely identify the directive. For example, MERGE is sufficient to represent the MERGE-STDERR-TO-STDOUT directive — while TAG can not be used to represent TAG-DETAILED (though TAG-D would suffice).

Supported values include:

  • TAG marks each output line with the [job,rank]<stream>: of the process that generated it

  • TAG-DETAILED marks each output line with a detailed annotation containing [namespace,rank][hostname:pid]<stream>: of the process that generated it

  • TAG-FULLNAME marks each output line with the [namespace,rank]<stream>: of the process that generated it

  • TAG-FULLNAME marks each output line with the [namespace,rank]<stream>: of the process that generated it

  • TIMESTAMP prefixes each output line with a [datetime]<stream>: stamp. Note that the timestamp will be the time when the line is output by the DVM and not the time when the source output it

  • XML provides all output in a pseudo-XML format MERGE-STDERR-TO-STDOUT merges stderr into stdout

  • DIR=DIRNAME redirects output from application processes into DIRNAME/job/rank/std[out,err,diag]. The provided name will be converted to an absolute path

  • FILE=FILENAME redirects output from application processes into filename.rank. The provided name will be converted to an absolute path

Supported qualifiers include NOCOPY (do not copy the output to the stdout/err streams), and RAW (do not buffer the output into complete lines, but instead output it as it is received).

15.1.6.3.7.6. The --stream-buffering option

Adjust buffering for stdout/stderr. Allowable values:

  • 0: unbuffered

  • 1: line buffered

  • 2: fully buffered

15.1.6.3.7.7. The --host option

Host syntax consists of a comma-delimited list of node names, each entry optionally containing a :N extension indicating the number of slots to assign to that entry:

--host node01:5,node02

In the absence of the slot extension, one slot will be assigned to the node. Duplicate entries are aggregated and the number of slots assigned to that node are summed together.

Note

A “slot” is the PRRTE term for an allocatable unit where we can launch a process. Thus, the number of slots equates to the maximum number of processes PRRTE may start on that node without oversubscribing it.

15.1.6.3.7.8. The --hostfile option

PRRTE supports several levels of user-specified host lists based on an established precedence order. Users can specify a default hostfile that contains a list of nodes to be used by the DVM. Only one default hostfile can be provided for a given DVM. In addition, users can specify a hostfile that contains a list of nodes to be used for a DVM, or can provide a comma-delimited list of nodes to be used for that DVM via the --host command line option.

The precedence order applied to these various options depends to some extent on the local environment. The following table illustrates how host and hostfile directives work together to define the set of hosts upon which a DVM will execute in the absence of a resource manager (RM):

Default hostfile

host

hostfile

Result

unset

unset

unset
The DVN will consist solely of the
local host where the DVM
was started.

unset

set

unset
Host option defines resource list for the DVM.

unset

unset

set
Hostfile option defines resource list for the DVM.

unset

set

set
Hostfile option defines resource list for the DVM,
then host filters the list to define the final
set of nodes to be used by the DVM

set

unset

unset
Default hostfile defines resource list for the DVM

set

set

unset
Default hostfile defines resource list for the DVM,
then host filters the list to define the final
set of nodes to be used by the DVM

set

set

set
Default hostfile defines resource list for the DVM,
then hostfile filters the list, and then host filters
the list to define the final set of nodes to be
used by the DVM

This changes somewhat in the presence of an RM as that entity specifies the initial allocation of nodes. In this case, the default hostfile, hostfile and host directives are all used to filter the RM’s specification so that a user can utilize different portions of the allocation for different DVMs. This is done according to the same precedence order as in the prior table, with the RM providing the initial pool of nodes.

Hostfiles (sometimes called “machine files”) are a combination of two things:

  1. A listing of hosts on which to launch processes.

  2. Optionally, limit the number of processes which can be launched on each host.

Hostfile syntax consists of one node name on each line, optionally including a designated number of “slots”:

# This is a comment line, and will be ignored
node01  slots=10
node13  slots=5

node15
node16
node17  slots=3
...

Blank lines and lines beginning with a # are ignored.

A “slot” is the PRRTE term for an allocatable unit where we can launch a process. See the section on definition of the term slot for a longer description of slots.

In the absence of the slot parameter, PRRTE will assign either the number of slots to be the number of CPUs detected on the node or the resource manager-assigned value if operating in the presence of an RM.

Important

If using a resource manager, the user-specified number of slots is capped by the RM-assigned value.

15.1.6.3.7.9. The --leave-session-attached option

Do not discard stdout/stderr of remote PRRTE daemons. The primary use for this option is to ensure that the daemon output streams (i.e., stdout and stderr) remain open after launch, thus allowing the user to see any daemon-generated error messages. Otherwise, the daemon will “daemonize” itself upon launch, thereby closing its output streams.

15.1.6.3.7.10. The --display option

The display command line directive must be accompanied by a comma-delimited list of case-insensitive options indicating what information about the job and/or allocation is to be displayed. The full directive need not be provided — only enough characters are required to uniquely identify the directive. For example, ALL is sufficient to represent the ALLOCATION directive — while MAP can not be used to represent MAP-DEVEL (though MAP-D would suffice).

Supported values include:

  • ALLOCATION displays the detected hosts and slot assignments for this job

  • BINDINGS displays the resulting bindings applied to processes in this job

  • MAP displays the resulting locations assigned to processes in this job

  • MAP-DEVEL displays a more detailed report on the locations assigned to processes in this job that includes local and node ranks, assigned bindings, and other data

  • TOPO=LIST displays the topology of each node in the semicolon-delimited list that is allocated to the job

  • CPUS[=LIST] displays the available CPUs on the provided semicolon-delimited list of nodes (defaults to all nodes)

The display command line directive can include qualifiers by adding a colon (:) and any combination of one or more of the following (delimited by colons):

  • PARSEABLE directs that the output be provided in a format that is easily parsed by machines. Note that PARSABLE is also accepted as a typical spelling for the qualifier.

Provided qualifiers will apply to all of the display directives.

See also

prte(1)