Help for package BatchJobs

Title:

Batch Computing with R

Description:

Provides Map, Reduce and Filter variants to generate jobs on batch computing systems like PBS/Torque, LSF, SLURM and Sun Grid Engine. Multicore and SSH systems are also supported. For further details see the project web page.

Author:

Bernd Bischl <bernd_bischl@gmx.net>, Michel Lang <michellang@gmail.com>, Henrik Bengtsson <henrikb@braju.com>

Maintainer:

Bernd Bischl <bernd_bischl@gmx.net>

URL:

https://github.com/tudo-r/BatchJobs

BugReports:

https://github.com/tudo-r/BatchJobs/issues

MailingList:

batchjobs@googlegroups.com

License:

BSD_2_clause + file LICENSE

Depends:

R (≥ 3.0.0), BBmisc (≥ 1.9), methods

Imports:

backports (≥ 1.1.1), brew, checkmate (≥ 1.8.0), data.table (≥ 1.9.6), DBI, digest, parallel, RSQLite (≥ 1.0.9011), sendmailR, stats, stringi (≥ 0.4-1), utils

Suggests:

MASS, testthat

Version:

1.9

RoxygenNote:

7.1.2

Encoding:

UTF-8

NeedsCompilation:

Packaged:

2022-03-21 10:39:57 UTC; michel

Repository:

CRAN

Date/Publication:

2022-03-21 11:30:02 UTC

The BatchJobs package

Description

Additional information

Homepage:: https://github.com/tudo-r/BatchJobs
Wiki:: https://github.com/tudo-r/BatchJobs/wiki
FAQ:: https://github.com/tudo-r/BatchJobs/wiki/FAQ
Configuration:: https://github.com/tudo-r/BatchJobs/wiki/Configuration

The package currently support the following further R options, which you can set either in your R profile file or a script via options:

BatchJobs.verbose: This boolean flag can be set to FALSE to reduce the console output of the package operations. Usually you want to see this output in interactive work, but when you use the package in e.g. knitr documents, it clutters the resulting document too much.
BatchJobs.check.posix: If this boolean flag is enabled, the package checks your registry file dir (and related user-defined directories) quite strictly to be POSIX compliant. Usually this is a good idea, you do not want to have strange chars in your file paths, as this might results in problems when these paths get passed to the scheduler or other command-line tools that the package interoperates with. But on some OS this check might be too strict and cause problems. Setting the flag to FALSE allows to disable the check entirely. The default is FALSE on Windows systems and TRUE else.

Add packages to registry.

Description

Mutator function for packages in makeRegistry.

Usage

addRegistryPackages(reg, packages)

Arguments

reg

[Registry]
Registry.

packages

[character]
Packages to add to registry.

Value

[Registry]. Changed registry.

Add source dirs to registry.

Description

Mutator function for src.dirs in makeRegistry.

Usage

addRegistrySourceDirs(reg, src.dirs, src.now = TRUE)

Arguments

reg

[Registry]
Registry.

src.dirs

[character]
Paths to add to registry. See makeRegistry.

src.now

[logical(1)] Source files now on master? Default is TRUE.

Value

[Registry]. Changed registry.

Add source files to registry.

Description

Mutator function for src.files in makeRegistry.

Usage

addRegistrySourceFiles(reg, src.files, src.now = TRUE)

Arguments

reg

[Registry]
Registry.

src.files

[character]
Paths to add to registry. See makeRegistry.

src.now

[logical(1)] Source files now on master? Default is TRUE.

Value

[Registry]. Changed registry.

applyJobFunction ONLY FOR INTERNAL USAGE.

Description

applyJobFunction ONLY FOR INTERNAL USAGE.

Usage

applyJobFunction(reg, job, cache)

Arguments

reg

[Registry]
Registry.

job

[Job]
Job.

cache

[FileCache]
Instance of makeFileCache.

Value

[any]. Result of job.

Map function over all combinations.

Description

Maps an n-ary-function over a list of all combinations which are given by some vectors. Internally expand.grid is used to compute the combinations, then batchMap is called.

Usage

batchExpandGrid(reg, fun, ..., more.args = list())

Arguments

reg

[Registry]
Empty Registry that will store jobs for the mapping.

fun

[function]
Function to map over the combinations.

...

[any]
Vectors that are used to compute all combinations. If the arguments are named, these names are used to bind to arguments of fun.

more.args

[list]
A list of other arguments passed to fun. Default is empty list.

Value

[data.frame]. Expanded grid of combinations produced by expand.grid.

Examples

reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
f = function(x, y, z) x * y  + z
# lets store the param grid
grid = batchExpandGrid(reg, f, x = 1:2, y = 1:3, more.args = list(z = 10))
submitJobs(reg)
waitForJobs(reg)
y = reduceResultsVector(reg)
# later, we can always access the param grid like this
grid = getJobParamDf(reg)
cbind(grid, y = y)

Export R object to be available on the slaves.

Description

Saves objects as RData files in the “exports” subdirectory of your file.dir to be later loaded on the slaves.

Usage

batchExport(reg, ..., li = list(), overwrite = FALSE)

Arguments

reg

[Registry]
Registry.

...

[any]
Objects to export. You must provide a valid name.

li

[list]
More objects to export provided as a named list.

overwrite

[logical(1)]
If set to FALSE (default), exported objects are protected from being overwritten by multiple calls of this function. Setting this to TRUE disables this check.

Value

[character]. Invisibly returns a character vector of exported objects.

Maps a function over lists or vectors, adding jobs to a registry.

Description

You can then submit these jobs to the batch system.

Usage

batchMap(reg, fun, ..., more.args = list(), use.names = FALSE)

Arguments

reg

[Registry]
Empty Registry that will store jobs for the mapping.

fun

[function]
Function to map over ....

...

[any]
Arguments to vectorize over (list or vector).

more.args

[list]
A list of other arguments passed to fun. Default is empty list.

use.names

[logical(1)]
Store parameter names to enable named results in loadResults and some other functions. Default is FALSE.

Value

Vector of type integer with job ids.

Examples

reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
f = function(x) x^2
batchMap(reg, f, 1:10)
print(reg)

Combination of makeRegistry, batchMap and submitJobs.

Description

Combination of makeRegistry, batchMap and submitJobs for quick computations on the cluster. Should only be used by skilled users who know what they are doing. Creates the file.dir, maps function, potentially chunks jobs and submits them.

Usage

batchMapQuick(
  fun,
  ...,
  more.args = list(),
  file.dir = NULL,
  packages = character(0L),
  chunk.size,
  n.chunks,
  chunks.as.arrayjobs = FALSE,
  inds,
  resources = list()
)

Arguments

fun

[function]
Function to map over ....

...

[any]
Arguments to vectorize over (list or vector).

more.args

[list]
A list of other arguments passed to fun. Default is empty list.

file.dir

[character]
See makeRegistry. Default is NULL, which means that it is created in the current directory under the name “bmq_[random alphanumerics]”.

packages

[character]
See makeRegistry.

chunk.size

[integer(1)]
Preferred number of jobs in each chunk. Can not be used in combination with n.chunks. Note that the ids will get shuffled to balance out possible run time differences. Default is not to use chunking.

n.chunks

[integer(1)]
Preferred number chunks. Can not be used in combination with chunk.size. Note that the ids will get shuffled to balance out possible run time differences. Default is not to use chunking.

chunks.as.arrayjobs

[logical(1)]
Submit chunks as array jobs? Default is FALSE.

inds

[integer]
Indices of ids / chunks to submit. Default is all. If ids get chunked, this subsets the list of shuffled ids.

resources

[list]
Required resources for all batch jobs. Default is empty list.

Value

[Registry]

Maps a function over the results of a registry by using batchMap.

Description

Maps a function over the results of a registry by using batchMap.

Usage

batchMapResults(
  reg,
  reg2,
  fun,
  ...,
  ids,
  part = NA_character_,
  more.args = list()
)

Arguments

reg

[Registry]
Registry whose results should be mapped by fun.

reg2

[Registry]
Empty registry that should store the job for the mapping.

fun

[function(job, res, ...)]
Function to map over results of reg. Further arguments come from ... of batchMapResults and more.args.

...

[any]
Furher arguments to vectorize over (list or vector). Must all be the same length as number of results in reg.

ids

[integer]
Ids of jobs whose results should be mapped with fun. Default is all jobs.

part

[character] Only useful for multiple result files, then defines which result file part(s) should be loaded. NA means all parts are loaded, which is the default.

more.args

[list]
A list of other arguments passed to fun. Default is empty list.

Value

Vector of type integer with job ids.

Examples

reg1 = makeRegistry(id = "BatchJobsExample1", file.dir = tempfile(), seed = 123)
# square some numbers
f = function(x) x^2
batchMap(reg1, f, 1:10)

# submit jobs and wait for the jobs to finish
submitJobs(reg1)
waitForJobs(reg1)

# look at results
reduceResults(reg1, fun = function(aggr,job,res) c(aggr, res))

reg2 = makeRegistry(id = "BatchJobsExample2", file.dir = tempfile(), seed = 123)

# define function to tranform results, we simply do the inverse of the squaring
g = function(job, res) sqrt(res)
batchMapResults(reg1, reg2, fun = g)

# submit jobs and wait for the jobs to finish
submitJobs(reg2)
waitForJobs(reg2)

# check results
reduceResults(reg2, fun = function(aggr,job,res) c(aggr, res))

Manually query the BatchJobs database

Description

Manually query the BatchJobs database

Usage

batchQuery(reg, query, flags = "ro")

Arguments

reg

[Registry]
Registry.

query

[character(1)]
SQL query to send to the database.

flags

[character(1)]
One of “ro”, “rw” or “rwc” which is translated to SQLITE_RO, SQLITE_RW or SQLITE_RWC, respectively. See SQLITE_RO for more info.

Value

[data.frame] Result of the query.

Examples

reg = makeRegistry("test", file.dir = tempfile())
batchMap(reg, identity, i = 1:10)
batchQuery(reg, "SELECT * FROM test_job_status")

Reduces via a binary function over a list adding jobs to a registry.

Description

Each jobs reduces a certain number of elements on one slave. You can then submit these jobs to the batch system.

Usage

batchReduce(reg, fun, xs, init, block.size, more.args = list())

Arguments

reg

[Registry]
Empty Registry.

fun

[function(aggr, x, ...)]
Function to reduce xs with.

xs

[vector]
Vector to reduce.

init

[any]
Initial object for reducing.

block.size

[integer(1)]
Number of elements of xs reduced in one job.

more.args

[list]
A list of other arguments passed to fun. Default is empty list.

Value

Vector of type integer with job ids.

Examples

# define function to reduce on slave, we want to sum a vector
f = function(aggr, x) aggr + x
reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)

# sum 20 numbers on each slave process, i.e. 5 jobs
batchReduce(reg, fun = f, 1:100, init = 0, block.size = 5)
submitJobs(reg)
waitForJobs(reg)

# now reduce one final time on master
reduceResults(reg, fun = function(aggr,job,res) f(aggr, res))

Reduces results via a binary function and adds jobs for this to a registry.

Description

Each jobs reduces a certain number of results on one slave. You can then submit these jobs to the batch system. Later, you can do a final reduction with reduceResults on the master.

Usage

batchReduceResults(
  reg,
  reg2,
  fun,
  ids,
  part = NA_character_,
  init,
  block.size,
  more.args = list()
)

Arguments

reg

[Registry]
Registry whose results should be reduced by fun.

reg2

[Registry]
Empty registry that should store the job for the mapping.

fun

[function(aggr, job, res, ...)]
Function to reduce results with.

ids

[integer]
Ids of jobs whose results should be reduced with fun. Default is all jobs.

part

[character] Only useful for multiple result files, then defines which result file part(s) should be loaded. NA means all parts are loaded, which is the default.

init

[any]
Initial object for reducing.

block.size

[integer(1)]
Number of results reduced in one job.

more.args

[list]
A list of other arguments passed to fun. Default is empty list.

Value

Vector of type integer with job ids.

Examples

# generating example results:
reg1 = makeRegistry(id = "BatchJobsExample1", file.dir = tempfile(), seed = 123)
f = function(x) x^2
batchMap(reg1, f, 1:20)
submitJobs(reg1)
waitForJobs(reg1)

# define function to reduce on slave, we want to sum the squares
myreduce = function(aggr, job, res) aggr + res

# sum 5 results on each slave process, i.e. 4 jobs
reg2 = makeRegistry(id = "BatchJobsExample2", file.dir = tempfile(), seed = 123)
batchReduceResults(reg1, reg2, fun = myreduce, init = 0, block.size = 5)
submitJobs(reg2)
waitForJobs(reg2)

# now reduce one final time on master
reduceResults(reg2, fun = myreduce)

Unload exported R objects.

Description

Removes RData files from the “exports” subdirectory of your file.dir and thereby prevents loading on the slave.

Usage

batchUnexport(reg, what)

Arguments

reg

[Registry]
Registry.

what

[character]
Names of objects to remove.

Value

[character]. Invisibly returns a character vector of unexported objects.

Call an arbitrary function on specified SSH workers.

Description

Calls can be made in parallel or consecutively, the function waits until all calls have finished and returns call results. In consecutive mode the output on the workers can also be shown on the master during computation.

Please read and understand the comments for argument dir.

Note that this function should only be used for short administrative tasks or information gathering on the workers, the true work horse for real computation is submitJobs.

In makeSSHWorker various options for load management are possible. Note that these will be ignored for the current call to execute it immediatly.

Usage

callFunctionOnSSHWorkers(
  nodenames,
  fun,
  ...,
  consecutive = FALSE,
  show.output = consecutive,
  simplify = TRUE,
  use.names = TRUE,
  dir = getwd()
)

Arguments

nodenames

[character]
Nodenames of workers to call function on. Only workers which were specified in your makeClusterFunctionsSSH configuration can be used.

fun

[function]
Function to call on workers.

...

[any]
Arguments for fun.

consecutive

[logical(1)]
Do calls consecutively and always wait until each worker is done. Default is FALSE.

show.output

[logical(1)]
Show output of workers on master during call. Can be useful to see what is happening. Can only be used in consecutive mode. Default is consecutive.

simplify

[logical(1)]
Should the result be simplified? See sapply. Default is TRUE.

use.names

[logical(1)]
Name results by nodenames. Default is TRUE.

dir

[character(1)]
Directory under which a temporary registry will be created in a subdirectory for communication. This has to be somewhere on the shared filesystem. The created subdirectory will be cleaned up on exit. Default is current working directory.

Value

Results of function calls, either a list or simplified.

Cluster functions helper: Brew your template into a job description file.

Description

This function is only intended for use in your own cluster functions implementation.

Calls brew silently on your template, any error will lead to an exception. If debug mode is turned on in the configuration, the file is stored at the same place as the corresponding R script in the “jobs”-subdir of your files directory, otherwise in the temp dir via tempfile.

Usage

cfBrewTemplate(conf, template, rscript, extension)

Arguments

conf

[environment]
BatchJobs configuration.

template

[character(1)]
Job desfription template as a char vecrtor, possibly read in via cfReadBrewTemplate.

rscript

[character(1)]
File path to you your corresponding R script for the job.

extension

[character(1)]
Extension for the job description file, e.g. “pbs”.

Value

[character(1)]. File path of result.

Cluster functions helper: Handle an unknown error during job submission.

Description

This function is only intended for use in your own cluster functions implementation.

Simply constructs a SubmitJobResult object with status code 101, NA as batch job id and an informative error message containing the output of the OS command in output.

Usage

cfHandleUnknownSubmitError(cmd, exit.code, output)

Arguments

cmd

[character(1)]
OS command used to submit the job, e.g. qsub.

exit.code

[integer(1)]
Exit code of the OS command, should not be 0.

output

[character]
Output of the OS command, hopefully an informative error message. If these are mutiple lines in a vector, they are automatically pasted together.

Value

[SubmitJobResult].

Cluster functions helper: Kill a batch job via OS command

Description

This function is only intended for use in your own cluster functions implementation.

Calls the OS command to kill a job via system like this: “cmd batch.job.id”. If the command returns an exit code > 0, the command is repeated after a 1 second sleep max.tries-1 times. If the command failed in all tries, an exception is generated.

Usage

cfKillBatchJob(cmd, batch.job.id, max.tries = 3L)

Arguments

cmd

[character(1)]
OS command, e.g. “qdel”.

batch.job.id

[character(1)]
Id of the batch job on the batch system.

max.tries

[integer(1)]
Number of total times to try execute the OS command in cases of failures. Default is 3.

Value

Nothing.

Cluster functions helper: Read in your brew template file.

Description

This function is only intended for use in your own cluster functions implementation.

Simply reads your template and returns it as a character vector. If you do this in the constructor of your cluster functions once, you can avoid this repeated file access later on.

Usage

cfReadBrewTemplate(template.file)

Arguments

template.file

[character(1)]
File path.

Value

[character].

Check job ids.

Description

Simply checks if probided vector of job ids is valid and throws an error if something is odd.

Usage

checkIds(reg, ids, check.present = TRUE, len = NULL)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Vector of job ids.

check.present

[logical(1)]
Check if the ids are present in the database? Default is TRUE.

len

[integer(1)]
Expected length of id vector. Passed to asInteger.

Value

Invisibly returns the vector of ids, converted to integer.

BatchJobs configuration.

Description

In order to understand how the package should be configured please read https://github.com/tudo-r/BatchJobs/wiki/Configuration.

ONLY FOR INTERNAL USAGE.

Description

ONLY FOR INTERNAL USAGE.

Usage

copyRequiredJobFiles(reg1, reg2, id)

Arguments

reg1

[Registry]
Source registry.

reg2

[Registry]
Detination registry.

id

[character(1)]
Job id.

Value

Nothing.

ONLY FOR INTERNAL USAGE.

Description

ONLY FOR INTERNAL USAGE.

Usage

dbCreateJobDefTable(reg)

Arguments

reg

[Registry]
Registry.

Value

Nothing.

ONLY FOR INTERNAL USAGE.

Description

ONLY FOR INTERNAL USAGE.

Usage

dbGetJobs(reg, ids)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of selected jobs.

Value

[list of Job]. Retrieved jobs from DB.

Helper function to debug multicore mode.

Description

Useful in case of severe errors. Tries different operations of increasing difficulty and provides debug output on the console

Usage

debugMulticore(r.options)

Arguments

r.options

[list] Options for R and Rscript, one option per element of the vector, a la “–vanilla”. Default is c("--no-save", "--no-restore", "--no-init-file", "--no-site-file").

Value

Nothing.

Helper function to debug SSH mode.

Description

Useful in case of configuration problems. Tries different operations of increasing difficulty and provides debug output on the console.

Note that this function does not access nor use information specified for your cluster functions in your configuration.

Usage

debugSSH(
  nodename,
  ssh.cmd = "ssh",
  ssh.args = character(0L),
  rhome = "",
  r.options = c("--no-save", "--no-restore", "--no-init-file", "--no-site-file"),
  dir = getwd()
)

Arguments

nodename

[character(1)]
Node on which worker should be constructed for the test.

ssh.cmd

[character(1))]
CLI command to ssh into remote node. Default is “ssh”.

ssh.args

[character]
CLI args for ssh.cmd. Default is none.

rhome

[character(1)]
Path to R installation on the worker. “” means R installation on the PATH is used, of course this implies that it must be on the PATH (also for non-interactive shells)! Default is “”.

r.options

[list] Options for R and Rscript, one option per element of the vector, a la “–vanilla”. Default is c("--no-save", "--no-restore", "--no-init-file", "--no-site-file").

dir

[character(1)]
Path where internally used test registries can be created. Note that this must be shared for the worker. Default is current working directory.

Value

Nothing.

Find all results where a specific condition is true.

Description

Find all results where a specific condition is true.

Usage

filterResults(reg, ids, fun, ...)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs whose results you want to test for the condition. Default is all jobs for which results are available.

fun

[fun(job, res)]
Predicate function that returns TRUE or FALSE.

...

[any]
Additional arguments to fun.

Value

[integer]. Ids of jobs where fun(job, result) returns TRUE.

Examples

reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
f = function(x) x^2
batchMap(reg, f, 1:10)
submitJobs(reg)
waitForJobs(reg)

# which square numbers are even:
filterResults(reg, fun = function(job, res) res %% 2 == 0)

Find jobs depending on computional state.

Description

findDone: Find jobs which succesfully terminated.

Usage

findDone(reg, ids, limit = NULL)

findNotDone(reg, ids, limit = NULL)

findMissingResults(reg, ids, limit = NULL)

findErrors(reg, ids, limit = NULL)

findNotErrors(reg, ids, limit = NULL)

findTerminated(reg, ids, limit = NULL)

findNotTerminated(reg, ids, limit = NULL)

findSubmitted(reg, ids, limit = NULL)

findNotSubmitted(reg, ids, limit = NULL)

findOnSystem(reg, ids, limit = NULL)

findNotOnSystem(reg, ids, limit = NULL)

findRunning(reg, ids, limit = NULL)

findNotRunning(reg, ids, limit = NULL)

findStarted(reg, ids, limit = NULL)

findNotStarted(reg, ids, limit = NULL)

findExpired(reg, ids, limit = NULL)

findDisappeared(reg, ids, limit = NULL)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Subset of job ids to restrict the result to. Default is all jobs.

limit

[integer(1)]
Limit the number of returned ids. Default is all ids.

Value

[integer]. Ids of jobs.

Finds ids of jobs that match a query.

Description

Finds ids of jobs that match a query.

Usage

findJobs(reg, ids, pars, jobnames)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Subset of job ids to restrict the result to. Default is all jobs.

pars

[R expression]
All jobs whose parameters match the given expression are selected. This implies that you have named the parameters when you passed the vectors. If you forgot to do this you can use .arg1, .arg2, etc., to refer to the the unnamed ones.

jobnames

[character]
Restrict to jobs with stored names. Exact matching is used.

Value

[integer]. Ids for jobs which match the query.

Examples

reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
f = function(x, y) x * y
batchExpandGrid(reg, f, x = 1:2, y = 1:3)
findJobs(reg, pars = (y > 2))

Returns a list of BatchJobs configuration settings

Description

Returns a list of BatchJobs configuration settings

Usage

getConfig()

Value

list of current configuration variables with classs “Config”.

Get error messages of jobs.

Description

Get error messages of jobs.

Usage

getErrorMessages(reg, ids)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs. Default is all jobs with errors.

Value

[character]. Error messages for jobs as character vector
NA if job has terminated successfully.

Get job from registry by id.

Description

Get job from registry by id.

Usage

getJob(reg, id, check.id = TRUE)

Arguments

reg

[Registry]
Registry.

id

[integer(1)]
Id of job.

check.id

[logical(1)]
Check the job id? Default is TRUE.

Value

[Job].

Get ids of jobs in registry.

Description

Get ids of jobs in registry.

Usage

getJobIds(reg)

Arguments

reg

[Registry]
Registry.

Value

[character].

Get computational information of jobs.

Description

Returns time stamps (submitted, started, done, error), time running, approximate memory usage (in Mb), error messages (shortened, see showLog for detailed error messages), time in queue, hostname of the host the job was executed, assigned batch ID, the R PID and the seed of the job.

To estimate memory usage the sum of the last column of gc is used.

Column “time.running” displays the time until either the job was done, or an error occured; it will by NA in case of time outs or hard R crashes.

Usage

getJobInfo(
  reg,
  ids,
  pars = FALSE,
  prefix.pars = FALSE,
  select,
  unit = "seconds"
)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs. Default is all jobs.

pars

[logical(1)]
Include job parameters in the output? Default is FALSE.

prefix.pars

[logical(1)]
Should a prefix be added to job parameter names (column names) to avoid name clashes? Default is FALSE.

select

[character]
Select only a subset of columns. Usually this is not required and you can subset yourself, but in some rare cases it may be advantageous to not query all information. Note that the column “id” (job id) is always selected. If not provided, all columns are queried and returned.

unit

[character(1)]
Unit to convert execution and queing times to. Possible values: “seconds”, “minutes”, “hours”, “days” and “weeks”. Default is “seconds”.

Value

[data.frame].

Get the physical location of job files on the hard disk.

Description

Get the physical location of job files on the hard disk.

Usage

getJobLocation(reg, ids)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs. Default is all jobs.

Value

[character] Vector of directories.

Get number of jobs in registry.

Description

Get number of jobs in registry.

Usage

getJobNr(reg)

Arguments

reg

[Registry]
Registry.

Value

[integer(1)].

Retrieve Job Parameters.

Description

Returns parameters for all jobs as the rows of a data.frame.

Usage

getJobParamDf(reg, ids)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs. Default is all jobs.

Value

[data.frame]. Rows are named with job ids.

Examples

# see batchExpandGrid

Function to get the resources that were submitted for some jobs.

Description

Throws an error if call it for unsubmitted jobs.

Usage

getJobResources(reg, ids, as.list = TRUE)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs. Default is all submitted jobs.

as.list

[integer(1)]
If FALSE a data.frame will be returned. Default is TRUE.

Value

[list | data.frame]. List (or data.frame) of resource lists as passed to submitJobs.

Get jobs from registry by id.

Description

Get jobs from registry by id.

Usage

getJobs(reg, ids, check.ids = TRUE)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs. Default is all jobs.

check.ids

[logical(1)]
Check the job ids? Default is TRUE.

Value

[list of Job].

Get log file paths for jobs.

Description

Get log file paths for jobs.

Usage

getLogFiles(reg, ids)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs. Default is all jobs.

Value

[character]. Vector of file paths to log files.

Function to get job resources in job function.

Description

Return the list passed to submitJobs, e.g. nodes, walltime, etc.

Usage

getResources()

Details

Can only be called in job function during job execution on slave.

Value

[list].

Print and return R installation and other information for SSH workers.

Description

Workers are queried in parallel via callFunctionOnSSHWorkers.

The function will display a warning if the first lib path on the worker is not writable as this indicates potential problems in the configuration and installPackagesOnSSHWorkers will not work.

Usage

getSSHWorkersInfo(nodenames)

Arguments

nodenames

[character]
Nodenames of workers.

Value

[list]. Displayed information as a list named by nodenames.

Grep log files for a pattern.

Description

Searches for occurence of pattern in log files.

Usage

grepLogs(
  reg,
  ids,
  pattern = "warn",
  ignore.case = TRUE,
  verbose = FALSE,
  range = 2L
)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs to grep. Default is all terminated jobs (done + errors).

pattern

[character(1)]
Pattern to search for. See grep. Default is "warn".

ignore.case

[logical(1)]
Ignore case. See grep. Default is TRUE.

verbose

[logical(1)]
Print matches. Default is FALSE.

range

[integer(1)]
If verbose is set to TRUE, print range leading and trailing lines for contextual information about the warning. Default is 2.

Value

[integer]. Ids of jobs where pattern was found in the log file.

Install packages on SSH workers.

Description

Installation is done via callFunctionOnSSHWorkers and install.packages.

Note that as usual the function tries to install the packages into the first path of .libPaths() of each each worker.

Usage

installPackagesOnSSHWorkers(
  nodenames,
  pkgs,
  repos = getOption("repos"),
  consecutive = TRUE,
  show.output = consecutive,
  ...
)

Arguments

nodenames

[character]
Nodenames of workers.

pkgs

[character]
See install.packages.

repos

[character]
See install.packages. If the user must be queried this is of course done on the master.

consecutive

[logical(1)]
See callFunctionOnSSHWorkers. Default is TRUE.

show.output

[logical(1)]
See callFunctionOnSSHWorkers. Default is consecutive.

...

[any]
Passed to install.packages.

Value

Nothing.

Kill some jobs on the batch system.

Description

Kill jobs which have already been submitted to the batch system. If a job is killed its internal state is reset as if it had not been submitted at all.

The function informs if (a) the job you want to kill has not been submitted, (b) the job has already terminated, (c) for some reason no batch job id is available. In all 3 cases above, nothing is changed for the state of this job and no call to the internal kill cluster function is generated.

In case of an error when killing, the function tries - after a short sleep - to kill the remaining batch jobs again. If this fails again for some jobs, the function gives up. Only jobs that could be killed are reset in the DB.

Usage

killJobs(reg, ids, progressbar = TRUE)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs to kill. Default is none.

progressbar

[logical(1)]
Set to FALSE to disable the progress bar. To disable all progress bars, see makeProgressBar.

Value

[integer]. Ids of killed jobs.

Examples

## Not run: 
reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
f = function(x) Sys.sleep(x)
batchMap(reg, f, 1:10 + 5)
submitJobs(reg)
waitForJobs(reg)

# kill all jobs currently _running_
killJobs(reg, findRunning(reg))
# kill all jobs queued or running
killJobs(reg, findNotTerminated(reg))

## End(Not run)

Load a specific configuration file.

Description

Load a specific configuration file.

Usage

loadConfig(conffile = ".BatchJobs.R")

Arguments

conffile

[character(1)]
Location of the configuration file to load. Default is “.BatchJobs.conf” in the current working directory.

Value

Invisibly returns a list of configuration settings.

Load exported R data objects.

Description

Loads exported RData object files in the “exports” subdirectory of your file.dir and assigns the objects to the global environment.

Usage

loadExports(reg, what = NULL)

Arguments

reg

[Registry]
Registry.

what

[character]
Names of objects to load. Defaults to all objects exported.

Value

[character]. Invisibly returns a character vector of loaded objects.

Load a previously saved registry.

Description

Loads a previously created registry from the file system. The file.dir is automatically updated upon load if adjust.paths is set to TRUE, so be careful if you use the registry on multiple machines simultaneously, e.g. via sshfs or a samba share.

There is a heuristic included which tries to detect if the location of the registry has changed and returns a read-only registry if necessary.

Usage

loadRegistry(file.dir, work.dir, adjust.paths = FALSE)

Arguments

file.dir

[character(1)]
Location of the file.dir to load the registry from.

work.dir

[character(1)]
Location of the work. Unchanged if missing.

adjust.paths

[logical(1)]
If set to FALSE (default), the paths for the file.dir and work.dir will not be updated. Set to TRUE if you moved the directoy to another system after all computations finished.

Value

[Registry].

Loads a specific result file.

Description

Loads a specific result file.

Usage

loadResult(
  reg,
  id,
  part = NA_character_,
  missing.ok = FALSE,
  impute.val = NULL
)

Arguments

reg

[Registry]
Registry.

id

[integer(1)]
Id of job.

part

[character] Only useful for multiple result files, then defines which result file part(s) should be loaded. NA means all parts are loaded, which is the default.

missing.ok

[logical(1)]
If FALSE an error is thrown if no result file is found. Otherwise NULL is returned. Default is FALSE.

impute.val

[any]
The value to return when no result is available. Defaults to NULL (the previous behavior)

Value

[any]. Result of job.

Loads result files for id vector.

Description

Loads result files for id vector.

Usage

loadResults(
  reg,
  ids,
  part = NA_character_,
  simplify = FALSE,
  use.names = "ids",
  missing.ok = FALSE,
  impute.val = NULL
)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs. Default is all done jobs.

part

[character] Only useful for multiple result files, then defines which result file part(s) should be loaded. NA means all parts are loaded, which is the default.

simplify

[logical(1)]
Should the result be simplified to a vector, matrix or higher dimensional array if possible? Default is TRUE.

use.names

[character(1)]
Name the results with job ids (“ids”), stored job names (“names”) or return a unnamed result (“none”). Default is ids.

missing.ok

[logical(1)]
If FALSE an error is thrown if the results are not found. Otherwise missing results are imputed to NULL. Default is FALSE.

impute.val

[any]
The value to return when no result is available. Defaults to NULL (the previous behavior)

Value

[list]. Results of jobs as list, possibly named by ids.

Create a ClusterFuntions object.

Description

Use this funtion when you implement a backend for a batch system. You must define the functions specified in the arguments.

Usage

makeClusterFunctions(
  name,
  submitJob,
  killJob,
  listJobs,
  getArrayEnvirName,
  class = NULL,
  ...
)

Arguments

name

[character(1)]
Name of cluster functions.

submitJob

[function(conf, reg, job.name, rscript, log.file, job.dir, resources, ...)]
Function to submit a new job. Must return a SubmitJobResult object.
The arguments are:
conf [environment]: The user configuration.
reg [Registry]: The registry.
job.name [character(1)]: Name of job, used if the job is displayed on the batch system. This is just for display and not an id!
rscript [character(1)]: File path to R script that is used to execute the job.
log.file [character(1)]: File path where log file (.Rout) has to be placed.
job.dir [character(1)]: Directory where all files relating to this job are placed.
resources [list]: Freely definable list of required resources for this job, e.g. walltime or memory.

killJob

[function(conf, reg, batch.job.id)]
Function to kill a job on the batch system. Make sure that you definately kill the job! Return value is currently ignored.
The arguments are:
conf [environment]: The user configuration.
reg [Registry]: The registry.
batch.job.id [character(1)]: Batch job id, as produced by submitJob.
Set killJob to NULL if killing jobs cannot be supported.

listJobs

[function(conf, reg)]
List all jobs on the batch system for the current user / registry. This includes queued, running, held, idle, etc. jobs. Must return an integer vector of batch job ids, same format as they are produced by submitJob. It does not matter if you return a few job ids too many (e.g. all for the current user instead of all for the current registry), but you have to include all relevant ones. The arguments are:
conf [environment]: The user configuration.
reg [Registry]: The registry.
Set listJobs to NULL if listing jobs cannot be supported.

getArrayEnvirName

[function()]
Returns the name of the environment variable specifying the array ID. Should return NA if not supported.

class

[character(1)]
Optional class name for cluster functions object. Useful to provide a nice print method which might show additional information about the workers. Default is NULL.

...

[any]
Currently ignored.

Create cluster functions for sequential execution in same session.

Description

All jobs executed under these cluster functions are executed sequentially, in the same interactive R process that you currently are. That is, submitJob does not return until the job has finished. The main use of this ClusterFunctions implementation is to test and debug programs on a local computer.

Listing jobs returns an empty vector (as no jobs can be running when you call this) and killJob returns at once (for the same reason).

Usage

makeClusterFunctionsInteractive(write.logs = TRUE)

Arguments

write.logs

[logical(1)]
Sink the output to log files. Turning logging off can increase the speed of calculations but makes it next to impossible to debug. Default is TRUE.

Value

[ClusterFunctions].

Create cluster functions for LSF systems.

Description

Job files are created based on the brew template template.file. This file is processed with brew and then submitted to the queue using the bsub command. Jobs are killed using the bkill command and the list of running jobs is retrieved using bjobs -u $USER -w. The user must have the appropriate privileges to submit, delete and list jobs on the cluster (this is usually the case).

The template file can access all arguments passed to the submitJob function, see here ClusterFunctions. It is the template file's job to choose a queue for the job and handle the desired resource allocations. Examples can be found on https://github.com/tudo-r/BatchJobs/tree/master/examples/cfLSF.

Usage

makeClusterFunctionsLSF(
  template.file,
  list.jobs.cmd = c("bjobs", "-u $USER", "-w")
)

Arguments

template.file

[character(1)]
Path to a brew template file that is used for the PBS job file.

list.jobs.cmd

[character]
Change default system command / options to list jobs. The first entry is the command, the following the options. See system3.

Value

[ClusterFunctions].

Create cluster functions for sequential execution on local host.

Description

All jobs executed under these cluster functions are executed sequentially, but in an independent, new R session. That is, submitJob does not return until the job has finished. The main use of this ClusterFunctions implementation is to test and debug programs on a local computer.

Listing jobs returns an empty vector (as no jobs can be running when you call this) and killJob returns at once (for the same reason).

Usage

makeClusterFunctionsLocal()

Value

[ClusterFunctions].

Use multiple cores on local Linux machine to spawn parallel jobs.

Description

Jobs are spawned by starting multiple R sessions on the commandline (similar like on true batch systems). Packages parallel or multicore are not used in any way.

Usage

makeClusterFunctionsMulticore(
  ncpus = max(getOption("mc.cores", parallel::detectCores()) - 1L, 1L),
  max.jobs,
  max.load,
  nice,
  r.options = c("--no-save", "--no-restore", "--no-init-file", "--no-site-file"),
  script
)

Arguments

ncpus

[integer(1)]
Number of VPUs of worker. Default is to use all cores but one, where total number of cores "available" is given by option mc.cores and if that is not set it is inferred by detectCores.

max.jobs

[integer(1)]
Maximal number of jobs that can run concurrently for the current registry. Default is ncpus.

max.load

[numeric(1)]
Load average (of the last 5 min) at which the worker is considered occupied, so that no job can be submitted. Default is inferred by detectCores, cf. argument ncpus.

nice

[integer(1)]
Process priority to run R with set via nice. Integers between -20 and 19 are allowed. If missing, processes are not nice'd and the system default applies (usually 0).

r.options

[character] Options for R and Rscript, one option per element of the vector, a la “–vanilla”. Default is c("--no-save", "--no-restore", "--no-init-file", "--no-site-file").

script

[character(1)]
Path to helper bash script which interacts with the worker. You really should not have to touch this, as this would imply that we have screwed up and published an incompatible version for your system. This option is only provided as a last resort for very experienced hackers. Note that the path has to be absolute. This is what is done in the package: https://github.com/tudo-r/BatchJobs/blob/master/inst/bin/linux-helper Default means to take it from package directory.

Value

[ClusterFunctions].

Create cluster functions for OpenLava systems.

Description

Usage

makeClusterFunctionsOpenLava(
  template.file,
  list.jobs.cmd = c("bjobs", "-u $USER", "-w")
)

Arguments

template.file

[character(1)]
Path to a brew template file that is used for the PBS job file.

list.jobs.cmd

[character]
Change default system command / options to list jobs. The first entry is the command, the following the options. See system3.

Value

[ClusterFunctions].

Create cluster functions for Sun Grid Engine systems.

Description

Job files are created based on the brew template template.file. This file is processed with brew and then submitted to the queue using the qsub command. Jobs are killed using the qdel command and the list of running jobs is retrieved using qselect. The user must have the appropriate privileges to submit, delete and list jobs on the cluster (this is usually the case).

Usage

makeClusterFunctionsSGE(template.file, list.jobs.cmd = c("qstat", "-u $USER"))

Arguments

template.file

[character(1)]
Path to a brew template file that is used for the PBS job file.

list.jobs.cmd

[character]
Change default system command / options to list jobs. The first entry is the command, the following the options. See system3.

Value

[ClusterFunctions].

Create cluster functions for SLURM-based systems.

Description

Job files are created based on the brew template template.file. This file is processed with brew and then submitted to the queue using the sbatch command. Jobs are killed using the scancel command and the list of running jobs is retrieved using squeue. The user must have the appropriate privileges to submit, delete and list jobs on the cluster (this is usually the case).

Usage

makeClusterFunctionsSLURM(
  template.file,
  list.jobs.cmd = c("squeue", "-h", "-o %i", "-u $USER")
)

Arguments

template.file

[character(1)]
Path to a brew template file that is used for the PBS job file.

list.jobs.cmd

[character]
Change default system command / options to list jobs. The first entry is the command, the following the options. See system3.

Value

[ClusterFunctions].

Create an SSH cluster to execute jobs.

Description

Worker nodes must share the same file system and be accessible by ssh without manually entering passwords (e.g. by ssh-agent or passwordless pubkey). Note that you can also use this function to parallelize on multiple cores on your local machine. But you still have to run an ssh server and provide passwordless access to localhost.

Usage

makeClusterFunctionsSSH(..., workers)

Arguments

...

[SSHWorker]
Worker objects, all created with makeSSHWorker.

workers

[list of SSHWorker]
Alternative way to pass workers.

Value

[ClusterFunctions].

Examples

## Not run: 

# Assume you have three nodes larry, curley and moe. All have 6
# cpu cores. On curley and moe R is installed under
# "/opt/R/R-current" and on larry R is installed under
# "/usr/local/R/". larry should not be used extensively because
# somebody else wants to compute there as well.
# Then a call to 'makeClusterFunctionsSSH'
# might look like this:

cluster.functions = makeClusterFunctionsSSH(
  makeSSHWorker(nodename = "larry", rhome = "/usr/local/R", max.jobs = 2),
  makeSSHWorker(nodename = "curley", rhome = "/opt/R/R-current"),
  makeSSHWorker(nodename = "moe", rhome = "/opt/R/R-current"))

## End(Not run)

Create cluster functions for torque-based systems.

Description

Usage

makeClusterFunctionsTorque(
  template.file,
  list.jobs.cmd = c("qselect", "-u $USER", "-s EHQRTW")
)

Arguments

template.file

[character(1)]
Path to a brew template file that is used for the PBS job file.

list.jobs.cmd

[character]
Change default system command / options to list jobs. The first entry is the command, the following the options. See system3.

Value

[ClusterFunctions].

Creates a job description.

Description

Usually you will not do this manually. Every object is a list that contains the passed arguments of the constructor.

Usage

makeJob(id = NA_integer_, fun, fun.id = digest(fun), pars, name, seed)

Arguments

id

[integer(1)]
Job id, determined by DB autoincrement. Default is NA.

fun

[function]
Job function to apply on parameters.

fun.id

[character(1)]
Id used to store function on disk. Default is digest(fun).

pars

[list]
Parameter list for job function.

name

[character(1)]
Alias name for this job.

seed

[integer(1)]
Random seed for job.

Construct a registry object.

Description

Note that if you don't want links in your paths (file.dir, work.dir) to get resolved and have complete control over the way the path is used internally, pass an absolute path which begins with “/”.

Usage

makeRegistry(
  id,
  file.dir,
  sharding = TRUE,
  work.dir,
  multiple.result.files = FALSE,
  seed,
  packages = character(0L),
  src.dirs = character(0L),
  src.files = character(0L),
  skip = TRUE
)

Arguments

id

[character(1)]
Name of registry. Displayed e.g. in mails or in cluster queue.

file.dir

[character(1)]
Path where files regarding the registry / jobs should be saved. Default is “<id>-files” in current working directory if id is set.

sharding

[logical(1)]
Enable sharding to distribute result files into different subdirectories? Important if you have many experiments. Default is TRUE.

work.dir

[character(1)]
Working directory for R process when experiment is executed. Default is the current working directory when registry is created.

multiple.result.files

[logical(1)]
Should a result file be generated for every list element of the returned list of the job function? Note that the function provided to batchMap or batchReduce must return a named list if this is set to TRUE. The result file will be named “<id>-result-<element name>.RData” instead of “<id>-result.RData”. Default is FALSE.

seed

[integer(1)]
Start seed for experiments. The first experiment in the registry will use this seed, for the subsequent ones the seed is incremented by 1. Default is a random number from 1 to .Machine$integer.max/2.

packages

[character]
Packages that will always be loaded on each node. Default is character(0).

src.dirs

[character]
Directories containing R scripts to be sourced on registry load (both on slave and master). Files not matching the pattern “\.[Rr]$” are ignored. Useful if you have many helper functions that are needed during the execution of your jobs. These files should only contain function definitions and no executable code. Default is character(0).

src.files

[character]
R scripts files to be sourced on registry load (both on slave and master). Useful if you have many helper functions that are needed during the execution of your jobs. These files should only contain function and constant definitions and no long running, executable code. These paths are considered to be relative to your work.dir. As a last remedy in problematic cases you can use absolute paths, by passing paths that start with “/”, see the comment about file.dir and work.dir above, where we allow the same thing. Note that this is a less portable approach and therefore usually a less good idea. Default is character(0).

skip

[logical(1)]
Skip creation of a new registry if a registry is found in file.dir. Defaults to TRUE.

Details

Every object is a list that contains the passed arguments of the constructor.

Value

[Registry]

Examples

reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
print(reg)

Create SSH worker for SSH cluster functions.

Description

Create SSH worker for SSH cluster functions.

Usage

makeSSHWorker(
  nodename,
  ssh.cmd = "ssh",
  ssh.args = character(0L),
  rhome = "",
  ncpus,
  max.jobs,
  max.load,
  nice,
  r.options = c("--no-save", "--no-restore", "--no-init-file", "--no-site-file"),
  script
)

Arguments

nodename

[character(1)]
Host name of node.

ssh.cmd

[character(1))]
CLI command to ssh into remote node. Default is “ssh”.

ssh.args

[character]
CLI args for ssh.cmd. Default is none.

rhome

[character(1)]
Path to R installation on worker. “” means R installation on the PATH is used, of course this implies that it must be on the PATH (also for non-interactive shells)! Default is “”.

ncpus

[integers(1)]
Number of VPUs of worker. Default means to query the worker via “/proc/cpuinfo”.

max.jobs

[integer(1)]
Maximal number of jobs that can run concurrently for the current registry. Default is ncpus.

max.load

[numeric(1)]
Load average (of the last 5 min) at which the worker is considered occupied, so that no job can be submitted. Default is ncpus-1.

nice

[integer(1)]
Process priority to run R with set via nice. Integers between -20 and 19 are allowed. If missing, processes are not nice'd and the system default applies (usually 0). Default is no niceing.

r.options

[character]
Options for R and Rscript, one option per element of the vector, a la “–vanilla”. Default is c("--no-save", "--no-restore", "--no-init-file", "--no-site-file").

script

Value

[SSHWorker].

Create a SubmitJobResult object.

Description

Use this function in your implementation of makeClusterFunctions to create a return value for the submitJob function.

Usage

makeSubmitJobResult(status, batch.job.id, msg, ...)

Arguments

status

[integer(1)]
Launch status of job. 0 means success, codes bewteen 1 and 100 are temporary errors and any error greater than 100 is a permanent failure.

batch.job.id

[character(1)]
Unique id of this job on batch system. Note that this is not the usual job id used in BatchJobs! Must be globally unique so that the job can be terminated using just this information.

msg

[character(1)]
Optional error message in case status is not equal to 0. Default is “OK”, “TEMPERR”, “ERROR”, depending on status.

...

[any]
Currently unused.

Value

[SubmitJobResult]. A list, containing status, batch.job.id and msg.

Reduce results from result directory.

Description

The following functions provide ways to reduce result files into either specific R objects (like vectors, lists, matrices or data.frames) or to arbitrarily aggregate them, which is a more general operation.

Usage

reduceResults(
  reg,
  ids,
  part = NA_character_,
  fun,
  init,
  impute.val,
  progressbar = TRUE,
  ...
)

reduceResultsList(
  reg,
  ids,
  part = NA_character_,
  fun,
  ...,
  use.names = "ids",
  impute.val,
  progressbar = TRUE
)

reduceResultsVector(
  reg,
  ids,
  part = NA_character_,
  fun,
  ...,
  use.names = "ids",
  impute.val
)

reduceResultsMatrix(
  reg,
  ids,
  part = NA_character_,
  fun,
  ...,
  rows = TRUE,
  use.names = "ids",
  impute.val
)

reduceResultsDataFrame(
  reg,
  ids,
  part = NA_character_,
  fun,
  ...,
  use.names = "ids",
  impute.val,
  strings.as.factors = FALSE
)

reduceResultsDataTable(
  reg,
  ids,
  part = NA_character_,
  fun,
  ...,
  use.names = "ids",
  impute.val
)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of selected jobs. Default is all jobs for which results are available.

part

[character] Only useful for multiple result files, then defines which result file part(s) should be loaded. NA means all parts are loaded, which is the default.

fun

[function]
For reduceResults, a function function(aggr, job, res, ...) to reduce things, for all others, a function function(job, res, ...) to select stuff. Here, job is the current job descriptor (see Job), result is the current result object and aggr are the so far aggregated results. When using reduceResults, your function should add the stuff you want to have from job and result to aggr and return that. When using the other reductions, you should select the stuff you want to have from job and result and return something that can be coerced to an element of the selected return data structure (reasonable conversion is tried internally). Default behavior for this argument is to return res, except for reduceResults where no default is available.

init

[ANY]
Initial element, as used in Reduce. Default is first result.

impute.val

[any]
For reduceResults: If not missing, the value of impute.val is passed to function fun as argument res for jobs with missing results.
For the specialized reduction functions reduceResults[Type]: If not missing, impute.val is used as a replacement for the return value of fun on missing results.

progressbar

[logical(1)]
Set to FALSE to disable the progress bar. To disable all progress bars, see makeProgressBar.

...

[any]
Additional arguments to fun.

use.names

[character(1)]
Name the results with job ids (“ids”), stored job names (“names”) or return a unnamed result (“none”). Default is ids.

rows

[logical(1)]
Should the selected vectors be used as rows (or columns) in the result matrix? Default is TRUE.

strings.as.factors

[logical(1)] Should all character columns in result be converted to factors? Default is FALSE.

Value

Aggregated results, return type depends on function. If ids is empty: reduceResults returns init (if available) or NULL, reduceResultsVector returns c(), reduceResultsList returns list(), reduceResultsMatrix returns matrix(0,0,0), reduceResultsDataFrame returns data.frame().

Examples

# generate results:
reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
f = function(x) x^2
batchMap(reg, f, 1:5)
submitJobs(reg)
waitForJobs(reg)

# reduce results to a vector
reduceResultsVector(reg)
# reduce results to sum
reduceResults(reg, fun = function(aggr, job, res) aggr+res)

reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
f = function(x) list(a = x, b = as.character(2*x), c = x^2)
batchMap(reg, f, 1:5)
submitJobs(reg)
waitForJobs(reg)

# reduce results to a vector
reduceResultsVector(reg, fun = function(job, res) res$a)
reduceResultsVector(reg, fun = function(job, res) res$b)
# reduce results to a list
reduceResultsList(reg)
# reduce results to a matrix
reduceResultsMatrix(reg, fun = function(job, res) res[c(1,3)])
reduceResultsMatrix(reg, fun = function(job, res) c(foo = res$a, bar = res$c), rows = TRUE)
reduceResultsMatrix(reg, fun = function(job, res) c(foo = res$a, bar = res$c), rows = FALSE)
# reduce results to a data.frame
print(str(reduceResultsDataFrame(reg)))
# reduce results to a sum
reduceResults(reg, fun = function(aggr, job, res) aggr+res$a, init = 0)

Remove a registry object.

Description

If there are no live/running jobs, the registry will be closed and all of its files will be removed from the file system. If there are live/running jobs, an informative error is generated. The default is to prompt the user for confirmation.

Usage

removeRegistry(reg, ask = c("yes", "no"))

Arguments

reg

[Registry]
Registry.

ask

[character(1)]
If "yes" the user is prompted to confirm the action. If trying to prompt the user this way in a non-interactive session, then an informative error is generated. If "no", the registry will be removed without further confirmation.

Value

[logical[1]]

Remove packages from registry.

Description

Mutator function for packages in makeRegistry.

Usage

removeRegistryPackages(reg, packages)

Arguments

reg

[Registry]
Registry.

packages

[character]
Packages to remove from registry.

Value

[Registry]. Changed registry.

Remove packages from registry.

Description

Mutator function for src.dirs in makeRegistry.

Usage

removeRegistrySourceDirs(reg, src.dirs)

Arguments

reg

[Registry]
Registry.

src.dirs

[character]
Paths to remove from registry.

Value

[Registry]. Changed registry.

Remove source files from registry.

Description

Mutator function for src.files in makeRegistry.

Usage

removeRegistrySourceFiles(reg, src.files)

Arguments

reg

[Registry]
Registry.

src.files

[character]
Paths to remove from registry.

Value

[Registry]. Changed registry.

Reset computational state of jobs.

Description

Reset state of jobs in the database. Useful under two circumstances: Either to re-submit them because of changes in e.g. external data or to resolve rare issues when jobs are killed in an unfortunate state and therefore blocking your registry.

The function internally lists all jobs on the batch system and if those include some of the jobs you want to reset, it informs you to kill them first by raising an exception. If you really know what you are doing, you may set force to TRUE to omit this sanity check. Note that this is a dangerous operation to perform which may harm the database integrity. In this case you HAVE to make externally sure that none of the jobs you want to reset are still running.

Usage

resetJobs(reg, ids, force = FALSE)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs to kill. Default is none.

force

[logical(1)]
Reset jobs without checking whether they are currently running. READ THE DETAILS SECTION! Default is FALSE.

Value

Vector of reseted job ids.

Sanitize a path

Description

Replaces backward slashes with forward slashes and optionally normalizes the path.

Usage

sanitizePath(path, make.absolute = TRUE, normalize.absolute = FALSE)

Arguments

path

[character]
Vector of paths to sanitize.

make.absolute

[logical]
If TRUE convert to an absolute path.

normalize.absolute

[logical]
Also call normalizePath on absolute paths? This will immediately resolve symlinks.

Value

character with sanitized paths.

Set and overwrite configuration settings

Description

Set and overwrite configuration settings

Usage

setConfig(conf = list(), ...)

Arguments

conf

[Config or list]
List of configuration parameters as returned by loadConfig or getConfig.

...

[ANY]
Named configuration parameters. Overwrites parameters in conf, if provided.

Value

Invisibly returns a list of configuration settings.

Sets the job function for already existing jobs.

Description

Use this function only as last measure when there is a bug in a part of your job function and you have already computed a large number of (unaffected) results. This function allows you to fix the error and to associate the jobs with the corrected function.

Note that by default the computational state of the affected jobs is also reset.

Usage

setJobFunction(reg, ids, fun, more.args = list(), reset = TRUE, force = FALSE)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs. Default is all jobs.

fun

[function]
Replacement function.

more.args

[list]
A list of other arguments passed to fun. Default is empty list.

reset

[logical(1)]
Reset job status via resetJobs. Default is TRUE.

force

[logical(1)]
See resetJobs. Default is FALSE.

Value

Nothing.

Set job names.

Description

Set job names.

Usage

setJobNames(reg, ids, jobnames)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of jobs. Default is all jobs.

jobnames

[character]
Character vector with length equal to length(ids). NA removes the names stored in the registry. A single NA is replicated to match the length of ids provided.

Value

Named vector of job ids.

Set packages for a registry.

Description

Mutator function for packages in makeRegistry.

Usage

setRegistryPackages(reg, packages)

Arguments

reg

[Registry]
Registry.

packages

[character]
Character vector of package names to load.

Value

[Registry]. Changed registry.

Show information about available computational resources on cluster.

Description

Currently only supported for multicore and SSH mode. Displays: Name of node, current load, number of running R processes, number of R processes with more than 50 The latter counts either jobs belonging to reg or all BatchJobs jobs if reg was not passed.

Usage

showClusterStatus(reg)

Arguments

reg

[Registry]
Registry. Must not be passed and this is the default.

Value

[data.frame].

Display the contents of a log file.

Description

Display the contents of a log file, useful in case of errors.

Note this rare special case: When you use chunking, submit some jobs, some jobs fail, then you resubmit these jobs again in different chunks, the log files will contain the log of the old, failed job as well. showLog tries to jump to the correct part of the new log file with a supported pager.

Usage

showLog(reg, id, pager = getOption("pager"))

Arguments

reg

[Registry]
Registry.

id

[integer(1)]
Id of selected job. Default is first id in registry.

pager

[any]
Pager to use to display the log. Defaults to getOption("pager"). This option is passed to file.show and is highly OS dependant and GUI dependant. If either R's pager or the environment variable “PAGER” is set to “less” or “vim”, the correct part of the log file will be shown. Otherwise you find information about the correct part in the beginning of the displayed file.

Value

[character(1)]. Invisibly returns path to log file.

Retrieve or show status information about jobs.

Description

E.g.: How many there are, how many are done, any errors, etc. showStatus displays on the console, getStatus returns an informative result without console output.

Usage

showStatus(reg, ids, run.and.exp = TRUE, errors = 10L)

getStatus(reg, ids, run.and.exp = TRUE)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Ids of selected jobs. Default is all jobs.

run.and.exp

[logical(1)]
Show running and expired jobs? Requires to list the job on the batch system. If not possible, because that cluster function is not avilable, this option is ignored anyway. Default is TRUE.

errors

[integer(1)]
How many of the error messages should be displayed if any errors occured in the jobs? Default is 10.

Value

[list]. List of absolute job numbers. showStatus returns them invisibly.

Examples

reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
f = function(x) x^2
batchMap(reg, f, 1:10)
submitJobs(reg)
waitForJobs(reg)

# should show 10 submitted jobs, which are all done.
showStatus(reg)

Source registry files

Description

Sources all files found in src.dirs and specified via src.files.

Usage

sourceRegistryFiles(reg, envir = .GlobalEnv)

Arguments

reg

[Registry]
Registry.

envir

[environment]
Environment to source the files into. Default is the global environment.

Value

Nothing.

Submit jobs or chunks of jobs to batch system via cluster function.

Description

If the internal submit cluster function completes successfully, the retries counter is set back to 0 and the next job or chunk is submitted. If the internal submit cluster function returns a fatal error, the submit process is completely stopped and an exception is thrown. If the internal submit cluster function returns a temporary error, the submit process waits for a certain time, which is determined by calling the user-defined wait-function with the current retries counter, the counter is increased by 1 and the same job is submitted again. If max.retries is reached the function simply terminates.

Potential temporary submit warnings and errors are logged inside your file directory in the file “submit.log”. To keep track you can use tail -f [file.dir]/submit.log in another terminal.

Usage

submitJobs(
  reg,
  ids,
  resources = list(),
  wait,
  max.retries = 10L,
  chunks.as.arrayjobs = FALSE,
  job.delay = FALSE,
  progressbar = TRUE
)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Vector for job id or list of vectors of chunked job ids. Only corresponding jobs are submitted. Chunked jobs will get executed sequentially as a single job for the scheduler. Default is all jobs which were not yet submitted to the batch system.

resources

[list]
Required resources for all batch jobs. The elements of this list (e.g. something like “walltime” or “nodes” are defined by your template job file. Defaults can be specified in your config file. Default is empty list.

wait

[function(retries)]
Function that defines how many seconds should be waited in case of a temporary error. Default is exponential back-off with 10*2^retries.

max.retries

[integer(1)]
Number of times to submit one job again in case of a temporary error (like filled queues). Each time wait is called to wait a certain number of seconds. Default is 10 times.

chunks.as.arrayjobs

[logical(1)]
If ids are passed as a list of chunked job ids, execute jobs in a chunk as array jobs. Note that your scheduler and your template must be adjusted to use this option. Default is FALSE.

job.delay

[function(n, i) or logical(1)]
Function that defines how many seconds a job should be delayed before it starts. This is an expert option and only necessary to change when you want submit extremely many jobs. We then delay the jobs a bit to write the submit messages as early as possible to avoid writer starvation. n is the number of jobs and i the number of the ith job. The default function used with job.delay set to TRUE is no delay for 100 jobs or less and otherwise runif(1, 0.1*n, 0.2*n). If set to FALSE (the default) delaying jobs is disabled.

progressbar

[logical(1)]
Set to FALSE to disable the progress bar. To disable all progress bars, see makeProgressBar.

Value

[integer]. Vector of submitted job ids.

Examples

reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
f = function(x) x^2
batchMap(reg, f, 1:10)
submitJobs(reg)
waitForJobs(reg)

# Submit the 10 jobs again, now randomized into 2 chunks:
chunked = chunk(getJobIds(reg), n.chunks = 2, shuffle = TRUE)
submitJobs(reg, chunked)

Sweep obsolete files from the file system.

Description

Removes R scripts, log files, resource informations and temporarily stored configuration files from the registry's file directory. Assuming all your jobs completed successfully, none of these are needed for further work. This operation potentially releases quite a lot of disk space, depending on the number of your jobs. BUT A HUGE WORD OF WARNING: IF you later notice something strange and need to determine the reason for it, you are at a huge disadvantage. Only do this at your own risk and when you are sure that you have successfully completed a project and only want to archive your produced experiments and results.

Usage

sweepRegistry(reg, sweep = c("scripts", "conf"))

Arguments

reg

[Registry]
Registry.

sweep

[character]
Possible choices: Temporary R scripts of jobs, really not needed for anything else then execution (“scripts”), log file of jobs, think about whether you later want to inspect them (“logs”), BatchJobs configuration files which are temporarily stored on submit, really not needed for anything else then execution (“conf”), resource lists of submitJobs which are temporarily stored on submit, think about whether you later want to inspect them (“resources”), Default is c("scripts", "conf").

Value

[logical]. Invisibly returns TRUE on success and FALSE if some files could not be removed.

Syncronize staged queries into the registry.

Description

If the option “staged.queries” is enabled, all communication from the nodes to the master is done via files in the subdirectory “pending” of the file.dir. This function checks for such files and merges the information into the database. Usually you do not have to call this function yourself.

Usage

syncRegistry(reg)

Arguments

reg

[Registry]
Registry.

Value

Invisibly returns TRUE on success.

Tests a job by running it with Rscript in a new process.

Description

Useful for debugging. Note that neither the registry, database or file directory are changed.

Usage

testJob(reg, id, resources = list(), external = TRUE)

Arguments

reg

[Registry]
Registry.

id

[integer(1)]
Id of job to test. Default is first job id of registry.

resources

[list]
Usually not needed, unless you call the function getResources in your job. See submitJobs. Default is empty list.

external

[logical(1)]
Run test in an independent external R session instead of current. The former allows to uncover missing variable definitions (which may accidentially be defined in the current global environment) and the latter is useful to get traceable execeptions. Default is TRUE.

Value

[any]. Result of job. If the job did not complete because of an error, NULL is returned.

Examples

reg = makeRegistry(id = "BatchJobsExample", file.dir = tempfile(), seed = 123)
f = function(x) if (x==1) stop("oops") else x
batchMap(reg, f, 1:2)
testJob(reg, 2)

ONLY FOR INTERNAL USAGE.

Description

ONLY FOR INTERNAL USAGE.

Usage

updateRegistry(reg)

Arguments

reg

[Registry]
Registry.

Value

[any]. Updated Registry or FALSE if no updates were performed.

Wait for termination of jobs on the batch system.

Description

Waits for termination of jobs while displaying a progress bar containing summarizing informations of the jobs. The following abbreviations are used in the progress bar: “S” for number of jobs on system, “D” for number of jobs successfully terminated, “E” for number ofjobs terminated with an R exception and “R” for number of jobs currently running on the system.

Usage

waitForJobs(
  reg,
  ids,
  sleep = 10,
  timeout = 604800,
  stop.on.error = FALSE,
  progressbar = TRUE
)

Arguments

reg

[Registry]
Registry.

ids

[integer]
Vector of job ids. Default is all submitted jobs not yet terminated.

sleep

[numeric(1)]
Seconds to sleep between status updates. Default is 10.

timeout

[numeric(1)]
After waiting timeout seconds, show a message and return FALSE. This argument may be required on some systems where, e.g., expired jobs or jobs on hold are problematic to detect. If you don't want a timeout, set this to Inf. Default is 604800 (one week).

stop.on.error

[logical(1)]
Immediately return if a job terminates with an error? Default is FALSE.

progressbar

[logical(1)]
Set to FALSE to disable the progress bar. To disable all progress bars, see makeProgressBar.

Value

[logical(1)]. Returns TRUE if all jobs terminated successfully and FALSE if either an error occurred or the timeout is reached.

The BatchJobs package

Description

Additional information

Add packages to registry.

Description

Usage

Arguments

Value

See Also

Add source dirs to registry.

Description

Usage

Arguments

Value

See Also

Add source files to registry.

Description

Usage

Arguments

Value

See Also

applyJobFunction ONLY FOR INTERNAL USAGE.

Description

Usage

Arguments

Value

Map function over all combinations.

Description

Usage

Arguments

Value

Examples

Export R object to be available on the slaves.

Description

Usage

Arguments

Value

See Also

Maps a function over lists or vectors, adding jobs to a registry.

Description

Usage

Arguments

Value

Examples

Combination of makeRegistry, batchMap and submitJobs.

Description

Usage

Arguments

Value

Maps a function over the results of a registry by using batchMap.

Description

Usage

Arguments

Value

Examples

Manually query the BatchJobs database

Description

Usage

Arguments

Value

Examples

Reduces via a binary function over a list adding jobs to a registry.

Description

Usage

Arguments

Value

Examples

Reduces results via a binary function and adds jobs for this to a registry.

Description

Usage

Arguments

Value

Examples

Unload exported R objects.

Description

Usage

Arguments

Value

See Also

Call an arbitrary function on specified SSH workers.