USER: job environment

srun and sbatch

export SLURMEXPORTENV=NONE # Same as --export implies environment is created using the --get-user-env mechanism.

just sbatch (probably preferred)

export SBATCH_EXPORT=NONE # Same as --export implies environment is created using the --get-user-env mechanism.

WIP (Work In Progress)

Slurm processes are not run under a shell, but directly exec'ed by the slurmd daemon (assuming srun is used to launch the processes). The environment variables in effect at the time the srun command is executed are propagated to the spawned processes. The ~/.profile and ~/.bashrc scripts are not executed as part of the process launch. You can also look at the --export option of srun and sbatch. See man pages for details.

This is rarely what is desired for batch jobs; especially in environments using the module(1) command. The job should explicitly set its limits with ulimit(1) or limit(1) and possibly source $HOME/.bashrc or other setup scripts and simulate a login in many cases. Batch jobs should be self-contained and independent of who submitted them and what their environment was in general. Inheriting an environment is generally only useful for interactive sessions.

test which features create the environment you desire

#!/bin/bash -l 
# -l switch makes shell act as if a login shell
#  environment-related
#SBATCH --export=NONE
#SBATCH --propagate=NONE
#SBATCH --mem=10G
$SBATCH --get-user-env=100L
#  regular options
#SBATCH -N 1 -n 1
#SBATCH --time=14-0:0:0
#SBATCH --time-min=1-0:0:0
#SBATCH --tmp=20G
env
####################################################################
# display before:
ulimit -a

# optionally set
# if submitting thousands of jobs and they all start dumping core
# can be a big problem in disk space, time, network bandwidth ...
# only allow coredumps when debugging or running small numbers of
# jobs
ulimit -c 0         # core file size (blocks)

# setting a limit can help stop a run-away job from inadvertently
# filling a filesystem
ulimit -f unlimited # file size(blocks)

# usually want unlimited
ulimit -d unlimited # data seg size (kbytes)
ulimit -m unlimited # max memory size(kbytes) 
ulimit -s unlimited # stack size              (kbytes) unlimited
ulimit -t unlimited # cpu time per process(seconds)
ulimit -v unlimited # virtual memory(kbytes) 
ulimit -x unlimited # file locks            

# defaults are usually fine or you should ask admin to change defaults
ulimit -i 14546     # pending signals
ulimit -l 65536     # max locked memory(kbytes) 
ulimit -n 1024      # open files 
ulimit -p 512       # pipe size(bytes) 
ulimit -q 819200    # POSIX message queues(bytes)
ulimit -u 14546     # max user processes 
# rarely changed by users
# ulimit -r 0         # real-time priority 
# ulimit -e 0         # scheduling priority 
# display after:
ulimit -a
####################################################################
printenv RAN_BASHRC

   --export=<environment variables [ALL] | NONE>

          Identify  which  environment variables from the submission envi‐
          ronment are propagated to the launched application. By  default,
          all  are propagated.  Multiple environment variable names should
          be comma separated.  Environment variable names may be specified
          to  propagate the current value (e.g. "--export=EDITOR") or spe‐
          cific   values   may   be    exported    (e.g.    "--export=EDI‐
          TOR=/bin/emacs").   In  these two examples, the propagated envi‐
          ronment will  only  contain  the  variable  EDITOR,  along  with
          SLURM_* environment variables.  However, Slurm will then implic‐
          itly attempt to load the user's environment on  the  node  where
          the  script  is  being executed, as if --get-user-env was speci‐
          fied. This will happen whenever NONE  or  environment  variables
          are specified.  If one desires to add to the submission environ‐
          ment instead of replacing it,  have  the  argument  include  ALL
          (e.g. "--export=ALL,EDITOR=/bin/emacs"). Make sure ALL is speci‐
          fied first, since sbatch applies the environment  from  left  to
          right,  overwriting  as necessary.  Environment variables propa‐
          gated from the submission environment will always overwrite  en‐
          vironment  variables  found in the user environment on the node.
          If one desires no environment variables be propagated  from  the
          submitting  machine,  use the argument NONE.  Regardless of this
          setting, the appropriate SLURM_* task environment variables  are
          always exported to the environment.  This option is particularly
          important for jobs that are submitted on one cluster and execute
          on  a  different  cluster (e.g. with different paths).  To avoid
          steps inheriting environment export settings  (e.g.  NONE)  from
          sbatch command, the environment variable SLURM_EXPORT_ENV should
          be set to ALL in the job script.

   --export-file=<filename | fd>
          If a number between 3 and OPEN_MAX is specified as the  argument
          to  this  option,  a  readable  file  descriptor will be assumed
          (STDIN and STDOUT are not supported as valid arguments).  Other‐
          wise  a  filename  is assumed.  Export environment variables de‐
          fined in <filename> or read from <fd> to the job's execution en‐
          vironment. The content is one or more environment variable defi‐
          nitions of the form NAME=value, each separated by a null charac‐
          ter.   This  allows the use of special characters in environment
          definitions.

   --get-user-env[=timeout][mode]
          This option will tell sbatch to retrieve the  login  environment
          variables for the user specified in the --uid option.  The envi‐
          ronment variables are retrieved by  running  something  of  this
          sort  "su  - <username> -c /usr/bin/env" and parsing the output.
          Be aware that any environment variables already set in  sbatch's
          environment  will take precedence over any environment variables
          in the user's login environment. Clear any environment variables
          before  calling  sbatch  that  you do not want propagated to the
          spawned program.  The optional timeout value is in seconds.  De‐
          fault  value  is 8 seconds.  The optional mode value control the
          "su" options.  With a mode value of "S", "su" is executed  with‐
          out  the "-" option.  With a mode value of "L", "su" is executed
          with the "-" option, replicating the login environment.  If mode
          not specified, the mode established at Slurm build time is used.
          Example of  use  include  "--get-user-env",  "--get-user-env=10"
          "--get-user-env=10L",  and  "--get-user-env=S".  This option was
          originally created for use by Moab.

   --propagate[=rlimit[,rlimit...]]
          Allows users to specify which of the modifiable (soft)  resource
          limits  to  propagate  to  the  compute nodes and apply to their
          jobs. If no rlimit is specified, then all resource  limits  will
          be  propagated.   The  following  rlimit  names are supported by
          Slurm (although some options may not be supported on  some  sys‐
          tems):

          ALL       All limits listed below (default)

          NONE      No limits listed below

          AS        The maximum address space for a process

          CORE      The maximum size of core file

          CPU       The maximum amount of CPU time

          DATA      The maximum size of a process's data segment

          FSIZE     The  maximum  size  of files created. Note that if the
                    user sets FSIZE to less than the current size  of  the
                    slurmd.log,  job  launches will fail with a 'File size
                    limit exceeded' error.

          MEMLOCK   The maximum size that may be locked into memory

          NOFILE    The maximum number of open files

          NPROC     The maximum number of processes available

          RSS       The maximum resident set size

          STACK     The maximum stack size
#

using env -i to understand how the environment is inherited

env -i bash -c env env -i bash -l -c env so to get a true login without getting any variables from current environment cat FILE|env -i sbatch --export=NONE --propagate=NONE

#

The "%#" specifiers control the field width, and the letter suffixes (e.g. "%10i") reference the format field (JobID width=10). Although the format specifiers for squeue are a bit obscure, if you find a format that is particularly useful, you can define the format to use whenever you login to Matilda by setting the value of "SQUEUE_FORMAT" in your ".bashrc" file. For example:

export SQUEUE_FORMAT="%10i%15u%15j%5t%15M%15l%8C%30N%10D%" UP

category: index