mirai - Reference Manual

This is a reference vignette of the package’s core functionality. Other package vignettes cover additional features.

1. Introduction

mirai (Japanese for ‘future’) implements the concept of futures in R.

Futures represent results from code that will complete later. Code executes in a separate R process (daemon) and returns results to the main process (host).

mirai

mirai() creates a mirai object from an expression.

It returns immediately without blocking. While the expression evaluates on a daemon, the host process continues working.

Expressions must be self-contained:

This explicit design perfectly matches message-passing parallelism - attempting to infer global variables introduces unreliability, which we do not compromise on.

This example mimics an expensive calculation:

library(mirai)

m <- mirai(
  {
    Sys.sleep(time)
    rnorm(5L, mean)
  },
  time = 2L,
  mean = 4.5
)

m
#> < mirai [] >
m$data
#> 'unresolved' logi NA
unresolved(m)
#> [1] TRUE

# Do work whilst unresolved

m[]
#> [1] 3.371159 4.680047 4.968465 4.693166 3.895580
m$data
#> [1] 3.371159 4.680047 4.968465 4.693166 3.895580

A mirai is unresolved until its result is received, then resolved. Use unresolved() to check its state.

Access results via m$data once resolved. This will be the return value, or an ‘errorValue’ if the expression errored, crashed, or timed out (see Error Handling).

Use m[] to efficiently wait for and collect the value instead of repeatedly checking unresolved(m).

You may also wait efficiently for mirai (or lists of mirai) to resolve using:

mirai (advanced)

For programmatic use, ‘.expr’ accepts a pre-constructed language object and ‘.args’ accepts a named list of arguments. The following is equivalent:

expr <- quote({Sys.sleep(time); rnorm(5L, mean)})
args <- list(time = 2L, mean = 4)

m1 <- mirai(.expr = expr, .args = args)
m1[]
#> [1] 2.150206 3.954099 5.314667 3.547804 4.939130

This example performs an asynchronous write operation. Passing environment() to ‘.args’ conveniently provides all objects from the calling environment (like x and file):

write.csv.async <- function(x, file) {
  mirai(write.csv(x, file), .args = environment())
}

m <- write.csv.async(x = rnorm(1e6), file = tempfile())

while (unresolved(m)) {
  cat("Writing file...\n")
  Sys.sleep(0.5) # or do other work
}
#> Writing file...
#> Writing file...
cat("Write complete:", is.null(m$data))
#> Write complete: TRUE

daemons

When writing a mirai() call, don’t worry about where or how it executes. End-users declare available resources using daemons().

Without daemons configured, each mirai() call creates a new local background process (ephemeral daemon).

daemons() sets up persistent daemons to evaluate mirai expressions:

See local daemons for setup instructions.

2. Error Handling

Errors return as a character string with classes ‘miraiError’ and ‘errorValue’.

Use is_mirai_error() to test for errors:

m1 <- mirai(stop("occurred with a custom message", call. = FALSE))
m1[]
#> 'miraiError' chr Error: occurred with a custom message

m2 <- mirai(mirai::mirai())
m2[]
#> 'miraiError' chr Error in mirai::mirai(): missing expression, perhaps wrap in {}?

is_mirai_error(m2$data)
#> [1] TRUE
is_error_value(m2$data)
#> [1] TRUE

Error objects include $stack.trace for full stack traces and $condition.class for original condition classes:

f <- function(x) if (x > 0) stop("positive")

m3 <- mirai({f(-1); f(1)}, f = f)
m3[]
#> 'miraiError' chr Error in f(1): positive

m3$data$stack.trace
#> [[1]]
#> stop("positive")
#> 
#> [[2]]
#> f(1)
m3$data$condition.class
#> [1] "simpleError" "error"       "condition"

Original error condition elements and rlang::abort() metadata are preserved:

f <- function(x) if (x > 0) stop("positive")

m4 <- mirai(rlang::abort("aborted", meta_uid = "UID001"))
m4[]
#> 'miraiError' chr Error: aborted

m4$data$meta_uid
#> [1] "UID001"

User interrupts resolve to class ‘miraiInterrupt’ and ‘errorValue’. Use is_mirai_interrupt() to test for interrupts:

m4 <- mirai(rlang::interrupt()) # simulates a user interrupt
is_mirai_interrupt(m4[])
#> [1] TRUE

Timeouts (via ‘.timeout’) resolve to ‘errorValue’ of 5L, guarding against hanging processes:

m5 <- mirai(nanonext::msleep(1000), .timeout = 500)
m5[]
#> 'errorValue' int 5 | Timed out

is_mirai_error(m5$data)
#> [1] FALSE
is_mirai_interrupt(m5$data)
#> [1] FALSE
is_error_value(m5$data)
#> [1] TRUE

is_error_value() tests for all mirai execution errors, user interrupts and timeouts.

3. Local Daemons

Daemons are persistent background processes that receive mirai() requests.

Daemons inherit system configuration (’.Renviron’, ‘.Rprofile’) and load default packages. To load only the base package (cutting startup time in half), set R_SCRIPT_DEFAULT_PACKAGES=NULL before launching.

Specify the number of daemons to launch:

daemons(6)

Set n to one less than available cores for optimal performance. Consider cores reserved for other purposes.

With Dispatcher (default)

The default dispatcher = TRUE creates a background dispatcher process that manages daemon connections. Tasks dispatch efficiently in FIFO order, queueing at the dispatcher and sending to daemons as they become available. The event-driven approach consumes no resources while waiting and stays synchronized with events.

info() provides current statistics as an integer vector:

info()
#> connections  cumulative    awaiting   executing   completed 
#>           6           6           0           0           0

Set daemons to zero to reset. This reverts to creating a new background process per request.

daemons(0)

Without Dispatcher

With dispatcher = FALSE, daemons connect directly to the host process:

daemons(6, dispatcher = FALSE)

Tasks send immediately in round-robin fashion, ensuring even distribution. However, scheduling isn’t optimal since task duration is unknown beforehand. Tasks may queue behind long-running tasks while other daemons sit idle.

This resource-light approach suits similar-length tasks or when concurrent tasks don’t exceed available daemons.

Info now shows 6 connections:

info()
#> connections  cumulative    awaiting   executing   completed 
#>           6          NA          NA          NA          NA

everywhere()

everywhere() evaluates expressions on all daemons and persists state regardless of cleanup settings:

everywhere(library(DBI))

This keeps the DBI package loaded. You can also set up common resources like database connections:

everywhere(con <<- dbConnect(RSQLite::SQLite(), file), file = tempfile())

Super-assignment makes ‘con’ available globally in all daemons:

mirai(exists("con"))[]
#> [1] TRUE

Disconnect everywhere:

everywhere(dbDisconnect(con))

To evaluate in the global environment of each daemon (since mirai evaluations occur in an environment inheriting from global), use evalq(envir = globalenv()). Example with box::use():

everywhere(
  evalq(
    box::use(dplyr[select], mirai[...]),
    envir = globalenv()
  )
)

daemons(0)

4. mirai_map

mirai_map() performs asynchronous parallel mapping over lists or vectors.

Requires daemons() to be set (avoids launching too many ephemeral daemons).

Basic Usage

Returns immediately. Collect results with x[]:

with(daemons(3, seed = 1234L), mirai_map(1:3, rnorm, .args = list(mean = 20, sd = 2))[])
#> [[1]]
#> [1] 19.86409
#> 
#> [[2]]
#> [1] 19.55834 22.30159
#> 
#> [[3]]
#> [1] 20.62193 23.06144 19.61896

Use .args for constant arguments to .f, and ... for objects referenced in .f:

daemons(4, seed = 2345L)
fn <- function(x, range) runif(x, x, x + range)
ml <- mirai_map(c(a = 1, b = 2, c = 3), \(x) fn(x, x * 2), fn = fn)
ml
#> < mirai map [0/3] >
ml[]
#> $a
#> [1] 2.637793
#> 
#> $b
#> [1] 2.328183 5.649959
#> 
#> $c
#> [1] 5.302906 3.531788 6.389231

Collecting Options

mirai_map(list(a = 1, b = "a", c = 3), function(x) exp(x))[.stop]
#> Error in `mirai_map()`:
#> ℹ In index: 2.
#> ℹ With name: b.
#> Caused by error in `exp()`:
#> ! non-numeric argument to mathematical function

mirai_map(c(0.1, 0.2, 0.3), Sys.sleep)[.progress, .flat]
#> NULL

Multiple Map

Dataframes and matrices map over rows. .f must accept as many arguments as there are columns:

fruit <- c("melon", "grapes", "coconut")
df <- data.frame(i = seq_along(fruit), fruit = fruit)

mirai_map(df, sprintf, .args = list(fmt = "%d. %s"))[.flat]
#> [1] "1. melon"   "2. grapes"  "3. coconut"

Matrices also map over rows:

mat <- matrix(1:4, nrow = 2L, dimnames = list(c("a", "b"), c("y", "z")))
mirai_map(mat, function(x = 10, y = 0, z = 0) x + y + z)[.flat]
#>  a  b 
#> 14 16

daemons(0)

To map over columns instead, use as.list() for dataframes or t() for matrices.

Nested Maps

For nested mapping, don’t launch local daemons from within mirai_map(). Instead:

daemons(url = local_url())
launch_local(n)

5. Remote Infrastructure

This section covers setting up remote daemons, launching them on remote machines, and securing connections with TLS.

Remote Daemons Overview

Remote daemons run on network machines to process tasks remotely.

Call daemons() with a ‘url’ (e.g., ‘tcp://10.75.32.70:5555’) or use host_url() to construct one automatically. The host listens on a single port for daemons to connect.

IPv6 addresses are also supported and must be enclosed in square brackets [] to avoid confusion with the final colon separating the port. For example, port 5555 on the IPv6 address ::ffff:a6f:50d would be specified as tcp://[::ffff:a6f:50d]:5555.

Calling host_url() without a port uses ‘0’, which automatically assigns a free ephemeral port:

daemons(url = host_url())

Query launch_remote() for the assigned port:

launch_remote()
#> [1]
#> Rscript -e 'mirai::daemon("tcp://10.154.3.14:49440")'

Dynamically scale the number of daemons up or down as needed.

Reset all connections:

daemons(0)

Closing connections exits all daemons. With dispatcher, this exits the dispatcher first, then all connected daemons.

Launching Remote Daemons

Launchers deploy daemons on remote machines. Once deployed, daemons connect back to the host via TCP or TLS.

Local launchers run Rscript via a local shell. Remote launchers run Rscript on remote machines.

Supply a remote launch configuration to the ‘remote’ argument of daemons() or launch_remote().

Four configuration options:

  1. ssh_config() for SSH access
  2. cluster_config() for HPC resource managers (Slurm, SGE, Torque/PBS, LSF)
  3. http_config() for HTTP API launch (e.g., Posit Workbench)
  4. remote_config() for generic/custom launchers

All return simple lists that can be pre-constructed, saved, and reused.

SSH Direct Connection

Use for internal networks where the host can accept incoming connections. Remote daemons connect back directly to the host port.

TLS is recommended for additional security.

Launch 4 daemons on 10.75.32.90 (SSH port 22 is default):

daemons(
  n = 4,
  url = host_url(tls = TRUE, port = 5555),
  remote = ssh_config("ssh://10.75.32.90")
)

Launch one daemon on each machine using custom SSH port 222:

daemons(
  n = 1,
  url = host_url(tls = TRUE, port = 5555),
  remote = ssh_config(c("ssh://10.75.32.90:222", "ssh://10.75.32.91:222"))
)

SSH Tunnelling

Use SSH tunnelling when firewall policies prevent direct connections. Requires SSH key-based authentication to be setup.

SSH tunnelling creates a tunnel after the initial SSH connection, using the same port on both host and daemon.

Supply a ‘127.0.0.1’ URL to daemons():

With local_url(tcp = TRUE, port = 5555), the host listens at 127.0.0.1:5555 and daemons dial into 127.0.0.1:5555 on their own machines.

Launch 2 daemons on 10.75.32.90 with tunnelling:

daemons(
  n = 2,
  url = local_url(tcp = TRUE),
  remote = ssh_config("ssh://10.75.32.90", tunnel = TRUE)
)

HPC Cluster Resource Managers

cluster_config() deploys daemons via cluster resource managers.

Specify command:

The options argument accepts scheduler options (lines typically preceded by #):

  Slurm: "#SBATCH --job-name=mirai
          #SBATCH --mem=10G
          #SBATCH --output=job.out"
  SGE: "#$ -N mirai
        #$ -l mem_free=10G
        #$ -o job.out"
  Torque/PBS: "#PBS -N mirai
               #PBS -l mem=10gb
               #PBS -o job.out"
  LSF: "#BSUB -J mirai
        #BSUB -M 10000
        #BSUB -o job.out"

module load R

or for a specific R version:

module load R/4.5.0

The rscript argument defaults to "Rscript" (assumes R is on PATH). Specify full path if needed: file.path(R.home("bin"), "Rscript").

Job Arrays

For many daemons, use job arrays instead of individual jobs.

Instead of:

daemons(n = 100, url = host_url(), remote = cluster_config())

rather use:

daemons(
  n = 1,
  url = host_url(),
  remote = cluster_config(options = "#SBATCH --array=1-100")
)

HTTP Launcher

http_config() launches daemons via HTTP API.

It takes the following arguments:

Each argument accepts either a character value or a function returning a value. When a function is supplied, it is called at launch time (when launch_remote() runs), not when the configuration is created. This lazy evaluation ensures that dynamic values such as session cookies, API tokens, or endpoint URLs are always fresh at the moment of use.

Default: Posit Workbench

Requires Posit Workbench 2026.01 or later, which supports authenticating the launcher using the session cookie.

By default, http_config() auto-configures for Posit Workbench. The defaults for url, cookie, and data are functions (not function calls) that read Workbench environment information:

http_config(
  url = posit_workbench_url,     # reads server address at launch time
  method = "POST",
  cookie = posit_workbench_cookie, # reads session cookie at launch time
  token = NULL,
  data = posit_workbench_data    # queries the compute environment at launch time
)

Because these are stored as functions, calling http_config() does no work — it simply saves the functions into the configuration list. Only when daemons are actually launched are the functions evaluated, at which point the environment variables are read and the API is queried. This means the configuration can be created early (e.g., at session start) while credentials that may change or expire are always obtained fresh.

Launch daemons in Posit Workbench:

daemons(n = 2, url = host_url(), remote = http_config())
Custom HTTP APIs

For custom HTTP APIs, provide URL, authentication, and request body. The data argument should include "%s" as a placeholder where the daemon launch command is inserted at launch time:

daemons(
  n = 2,
  url = host_url(),
  remote = http_config(
    url = "https://api.example.com/launch",
    method = "POST",
    token = function() Sys.getenv("MY_API_KEY"),
    data = '{"command": "%s"}'
  )
)

Here, token is a function so the API key environment variable is read each time daemons are launched. The remaining arguments are plain character values used as-is.

Troubleshooting

launch_remote() with an http_config() configuration returns a list of server response data (invisibly). Capture and inspect these to diagnose launch failures:

daemons(url = host_url())
res <- launch_remote(remote = http_config())

Each element of res is the response for a single daemon launch request.

Generic Remote Configuration

remote_config() provides a generic framework for custom deployment commands.

The args argument must contain ".", which is replaced with the daemon launch command.

cluster_config() is easier for HPC, but remote_config() offers flexibility. Slurm example:

daemons(
  n = 2,
  url = host_url(),
  remote = remote_config(
    command = "sbatch",
    args = c("--mem 512", "-n 1", "--wrap", "."),
    rscript = file.path(R.home("bin"), "Rscript"),
    quote = TRUE
  )
)

Manual Deployment

Call launch_remote() without ‘remote’ to get shell commands for manual deployment:

daemons(url = host_url())
launch_remote()
#> [1]
#> Rscript -e 'mirai::daemon("tcp://10.154.3.14:49441")'
daemons(0)

TLS Secure Connections

TLS secures communications between host and remote daemons.

Automatic Zero-configuration Default

Use tls+tcp:// scheme or host_url(tls = TRUE):

daemons(url = host_url(tls = TRUE))

Keys and certificates generate automatically. Private keys remain on the host.

Self-signed certificates are included in launch_remote() commands:

launch_remote(1)
#> [1]
#> Rscript -e 'mirai::daemon("tls+tcp://10.154.3.14:49442",tlscert=c("-----BEGIN CERTIFICATE-----
#> MIIFPTCCAyWgAwIBAgIBATANBgkqhkiG9w0BAQsFADA2MRQwEgYDVQQDDAsxMC4x
#> NTQuMy4xNDERMA8GA1UECgwITmFub25leHQxCzAJBgNVBAYTAkpQMB4XDTAxMDEw
#> MTAwMDAwMFoXDTMwMTIzMTIzNTk1OVowNjEUMBIGA1UEAwwLMTAuMTU0LjMuMTQx
#> ETAPBgNVBAoMCE5hbm9uZXh0MQswCQYDVQQGEwJKUDCCAiIwDQYJKoZIhvcNAQEB
#> BQADggIPADCCAgoCggIBAItezzjpIfLF1RT3QbuKUXS/YKDF6bd4TlEAfAH58BGh
#> Ds8sR9gmiguockMONgYCCKZMY7ccFqycQGa0phE69udpGFvpucx1R7GblzlY9jHa
#> rHhivqSdSAXXwMILA7OcOj/j+tteTCdnCTDVUQYYghN+BLTxqxRwsVuw140aFLRF
#> 2xi986TeHK5noUd+T5IUiijlRBF+nb2I+CtkA/EoEr6/+tJyfMPB5MO5tjq/S8/G
#> mIfhOZB7OBVYeuWiA83ZSQuOVCj+8HRbTMa8IinfmO/+9ukv1veykj7MJeEl0MBk
#> OvtOuCKhqc+/XqdNO03BQIKznQ8zsuWuEyjVcH3K/7R4/DdkRRDs9OMF25ccZsal
#> EHcoI4YPdgY4G6GIgo5CDXy/bHddffh0fuO8GwowfpfDX1jVcLH6zceUW4gaKto5
#> 8592yH/TzWEOxf31ie0P1G/xxkJzgunL9/32dUstAXyNM+LMm4DoJDrP484l40/i
#> +OkthXT2VZIngD8XiTF+3pdEZ4L3p/TJhCi33jeMMHk2auoB/DDKZDXzHcqHedR5
#> epB97DNDl+mzK6o5g1PC/EW5/DmDXbs7XeYC8LdCWYRbarBgAO31EhvuWOYn6pnm
#> L8fFRAUbGQwpK/0G7XSXLPS7WbMIxEyju+7g6fyT43AXxk/OAKdpQrSuo9mcn8TN
#> AgMBAAGjVjBUMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFBfzosBcf9Le
#> NliCnLkwPfh+l0y4MB8GA1UdIwQYMBaAFBfzosBcf9LeNliCnLkwPfh+l0y4MA0G
#> CSqGSIb3DQEBCwUAA4ICAQASYDmKlYK9rNoGaEp8Ol4KI4Pu1rMEj8pLGfmuRDYQ
#> 7pDTU29dGDym3Cy5ViHfHBTgPq6opj5P7ByykIIt55efr9zzkw1MS2NF2LRWYiHF
#> jxT4DCoukDSLFELv8Jkoc8mWpST6Pr2hE3QKx63Kh6GjHUzX0BV87/Gy1jM9sP63
#> 03/rVI5VG1BQFKmqZeQfJBxstLzoEB/xVZ44ytHN+Q2QuwhzMKEPcA1jqWNlq94u
#> hg6FySyYSZxmR1otdky2q63vLyxNsS/1h1FE2YLV6VBhLH4UdQr9OfuQghSFXcov
#> 9U11zhBqx5ANyjYDxyMqI6V4+XOHXL4nxNtTTCZX0YIoSRgieYh3WTmqtM0j6EDS
#> 2QPn2W4m1oYEBPKfgeCtICFf6PkrpJ4vM+iUI9jA+AY71VvdvlZx+bXZvrNZBIs4
#> mj6XHiAcN6nlKwTcdfhhvgGJ4VTadZwzeJpLZS5ZhxnwUIlgBMfL8rFdWTYZRxG/
#> 6MApxl/eeHXbFafN2n5wWMhlNXSBFQMlxoCppONzrVgaq8/O2nsXhBpxC3/RklWm
#> yU3T2QMRQ+dQgZm30TsEnEHsndN6OpAJtaTwgD7pQb2zFAE+4JNFjWN389UwmQ7N
#> Xa5eYIyPVUoYuZWvQ95RQ9xquU9LAOqU4owcoINiEwAjhSQ4tUUUKQLhajW91urG
#> xA==
#> -----END CERTIFICATE-----
#> ",""))'

daemons(0)
CA Signed Certificates

Alternatively, generate certificates via a Certificate Signing Request (CSR) to a Certificate Authority (public or internal).

  1. Generate a private key and CSR:
  1. Provide the generated CSR to the CA for it to sign a new TLS certificate.
  1. When setting daemons, the TLS certificate and private key should be provided to the ‘tls’ argument of daemons().
  1. The certificate chain to the CA should be supplied to the ‘tlscert’ argument of daemons().

6. Compute Profiles

The .compute argument to daemons() creates separate, independent daemon pools (compute profiles) for heterogeneous compute requirements:

Pass a character string to .compute as the profile name (NULL defaults to ‘default’). Settings save under this name.

Specify .compute in mirai() to use a profile (NULL uses ‘default’).

Other functions (info(), launch_local(), launch_remote()) also accept .compute.

with_daemons() and local_daemons()

with_daemons() or local_daemons() with a profile name sets the default for all functions within that scope:

daemons(1, .compute = "cpu")
daemons(1, .compute = "gpu")

with_daemons("cpu", {
  m1 <- mirai(Sys.getpid())
})

with_daemons("gpu", {
  m2 <- mirai(Sys.getpid())
  m3 <- mirai(Sys.getpid(), .compute = "cpu")
  local_daemons("cpu")
  m4 <- mirai(Sys.getpid())
})

m1[]
#> [1] 11456
m2[] # different to m1
#> [1] 11482

m3[] # same as m1
#> [1] 11456
m4[] # same as m1
#> [1] 11456

with_daemons("cpu", daemons(0))
with_daemons("gpu", daemons(0))

With Method

The with() method creates daemons for an expression’s duration, then automatically resets them. Functions within the scope use the daemons’ compute profile.

Designed for running Shiny apps with specific daemon counts:

with(daemons(4), shiny::runApp(app))
# Or:
with(daemons(4, .compute = "shiny"), shiny::runApp(app))

Note: The app must already be created. Don’t wrap shiny::shinyApp() since runApp() is called when printed, after with() returns.

Shiny apps execute all mirai calls before returning (blocking). For other expressions, collect all mirai values to ensure completion before daemon reset.

7. Advanced Topics

Random Number Generation

mirai uses L’Ecuyer-CMRG streams (like base R’s parallel package) for statistically-sound parallel RNG.

Streams divide the RNG sequence at far-apart intervals that don’t overlap, ensuring valid parallel results.

Default (seed = NULL): New stream per daemon (like base R):

Reproducible (seed = integer): New stream per mirai() call (not per daemon):

Synchronous Mode

daemons(sync = TRUE) enables synchronous mode. Mirai evaluate immediately without async operation, useful for testing and debugging with browser().

Restrict to a specific profile by specifying .compute. Only seed affects behavior with sync = TRUE.

Example usage:

# run everything in sync:
daemons(sync = TRUE)
mp <- mirai_map(1:2, \(x) Sys.getpid())
daemons(0)
mp[]
#> [[1]]
#> [1] 7224
#> 
#> [[2]]
#> [1] 7224


# Use sync with the 'sync' compute profile:
daemons(sync = TRUE, .compute = "sync")
with_daemons("sync", {
  mp <- mirai_map(1:2, \(x) Sys.getpid())
})
daemons(0, .compute = "sync")
mp[]
#> [[1]]
#> [1] 7224
#> 
#> [[2]]
#> [1] 7224