zcurve: An Implementation of Z-Curves

Description

An implementation of z-curves - a method for estimating expected discovery and replicability rates on the bases of test-statistics of published studies. The package provides functions for fitting the density, EM, and censored EM version (Bartoš & Schimmack, 2022, doi:10.15626/MP.2021.2720; Schimmack & Bartoš, 2023, doi: 10.1371/journal.pone.0290084), as well as the original density z-curve (Brunner & Schimmack, 2020, doi:10.15626/MP.2018.874). Furthermore, the package provides summarizing and plotting functions for the fitted z-curve objects. See the aforementioned articles for more information about the z-curves, expected discovery and replicability rates, validation studies, and limitations.

Author(s)

Maintainer: František Bartoš f.bartos96@gmail.com

Authors:

• Ulrich Schimmack ulrich.schimmack@utoronto.ca

See Also

Useful links:

• https://fbartos.github.io/zcurve/

• Report bugs at https://github.com/FBartos/zcurve/issues

Z-scores from subset of original studies featured in OSC 2015 reproducibility project

Description

The dataset contains z-scores from subset of original studies featured in psychology reproducibility project (Collaboration and others 2015). Only z-scores from studies with unambiguous original outcomes are supplied (eliminating 7 studies with marginally significant results). The real replication rate for those studies is 35/90 (the whole project reports 36/97).

Usage

OSC.z

Format

A vector with 90 observations

References

Collaboration OS, others (2015). “Estimating the reproducibility of psychological science.” Science, 349(6251). doi:10.1126/science.aac4716.

Control settings for the zcurve EM algorithm

Description

All these settings are passed to the Expectation Maximization fitting algorithm. All unspecified settings are set to the default value. Setting model = "EM" sets all settings to the default value irrespective of any other setting and fits z-curve as described in Bartoš and Schimmack (2022)

Arguments

References

Bartoš F, Schimmack U (2022). “Z-curve 2.0: Estimating replication rates and discovery rates.” Meta-Psychology, 6. doi:10.15626/MP.2021.2720.

See Also

zcurve(), control_density

Examples

# to increase the number of starting fits
# and change the means of the mixture components

ctrl <- list(
   fit_reps  = 50,
   mu = c(0, 1.5, 3, 4.5, 6)
)
## Not run: zcurve(OSC.z, method = "EM", control = ctrl)

Control settings for the z-curve 2.0 density algorithm

Description

All settings are passed to the density fitting algorithm. All unspecified settings are set to the default value. Setting model = "KD2" sets all settings to the default value irrespective of any other setting and fits z-curve as describe in Bartoš and Schimmack (2022). In order to fit the z-curve 1.0 density algorithm, set model = "KD1" and go to control_density_v1

Arguments

References

Bartoš F, Schimmack U (2022). “Z-curve 2.0: Estimating replication rates and discovery rates.” Meta-Psychology, 6. doi:10.15626/MP.2021.2720.

See Also

zcurve(), control_density_v1, control_EM

Examples

# to decrease the criterion and increase the number of iterations
ctrl <- list(
   max_iter  = 300,
   criterion = 1e-4
)
## Not run: zcurve(OSC.z, method = "density", control = ctrl)

Control settings for the original z-curve density algorithm

Description

All settings are passed to the density fitting algorithm. All unspecified settings are set to the default value. Setting model = "KD1" sets all settings to the default value irrespective of any other setting and fits z-curve as described in Brunner and Schimmack (2020).

Arguments

References

Brunner J, Schimmack U (2020). “Estimating population mean power under conditions of heterogeneity and selection for significance.” Meta-Psychology, 4. doi:10.15626/MP.2018.874.

See Also

zcurve(), control_density, control_EM

Examples

# to increase the number of iterations
ctrl <- list(
   version   = 1,
   max_iter  = 300
)
## Not run: zcurve(OSC.z, method = "density", control = ctrl)

Prints first few rows of a z-curve data object

Description

Prints first few rows of a z-curve data object

Usage

## S3 method for class 'zcurve_data'
head(x, ...)

Arguments

See Also

zcurve_data()

Reports whether x is a zcurve object

Description

Reports whether x is a zcurve object

Usage

is.zcurve(x)

Arguments

Plot fitted z-curve object

Description

Plot fitted z-curve object

Usage

## S3 method for class 'zcurve'
plot(
  x,
  annotation = FALSE,
  CI = FALSE,
  extrapolate = FALSE,
  plot_type = "base",
  y.anno = c(0.95, 0.88, 0.78, 0.71, 0.61, 0.53, 0.43, 0.35),
  x.anno = 0.6,
  cex.anno = 1,
  ...
)

Arguments

See Also

zcurve()

Examples

## Not run: 
# simulate some z-statistics and fit a z-curve
z <- abs(rnorm(300,3))
m.EM <- zcurve(z, method = "EM", bootstrap = 100)

# plot the z-curve
plot(m.EM)

# add annotation text and model fit CI
plot(m.EM, annotation = TRUE, CI = TRUE)

# change the location of the annotation to the left
plot(m.EM, annotation = TRUE, CI = TRUE, x_text = 0)

## End(Not run)

Compute z-score corresponding to a power

Description

A function for computing z-scores of two-sided tests corresponding to power power for a given significance level alpha alpha (or corresponding cut-off z-statistic a).

Usage

power_to_z(
  power,
  alpha = 0.05,
  a = stats::qnorm(alpha/2, lower.tail = FALSE),
  two.sided = TRUE,
  nleqslv_control = list(xtol = 1e-15, maxit = 300, stepmax = 0.5)
)

Arguments

Examples

# z-scores corresponding to the (aproximate) power of components of EM2
power_to_z(c(0.05, 0.20, 0.40, 0.60, 0.80, 0.974, 0.999), alpha = .05)

Prints estimates from z-curve object

Description

Prints estimates from z-curve object

Usage

## S3 method for class 'zcurve'
print.estimates(x, ...)

Arguments

See Also

zcurve()

Prints summary object for z-curve method

Description

Prints summary object for z-curve method

Usage

## S3 method for class 'zcurve'
print.summary(x, ...)

Arguments

See Also

zcurve()

Prints a fitted z-curve object

Description

Prints a fitted z-curve object

Usage

## S3 method for class 'zcurve'
print(x, ...)

Arguments

See Also

zcurve()

Prints a z-curve data object

Description

Prints a z-curve data object

Usage

## S3 method for class 'zcurve_data'
print(x, ...)

Arguments

See Also

zcurve_data()

Summarize fitted z-curve object

Description

Summarize fitted z-curve object

Usage

## S3 method for class 'zcurve'
summary(
  object,
  type = "results",
  all = FALSE,
  ERR.adj = 0.03,
  EDR.adj = 0.05,
  round.coef = 3,
  conf.level = 0.95,
  ...
)

Arguments

Value

Summary of a z-curve object

See Also

zcurve()

Compute power corresponding to z-scores

Description

A function for computing power of two-sided tests corresponding to z-scores for a given significance level. alpha (or corresponding cut-off z-score a)

Usage

z_to_power(
  z,
  alpha = 0.05,
  a = stats::qnorm(alpha/2, lower.tail = FALSE),
  two.sided = TRUE
)

Arguments

Examples

# mean powers corresponding to the mean components of KD2
z_to_power(0:6, alpha = .05)

Fit a z-curve

Description

zcurve is used to fit z-curve models. The function takes input of z-statistics or two-sided p-values and returns object of class "zcurve" that can be further interrogated by summary and plot function. It default to EM model, but different version of z-curves can be specified using the method and control arguments. See 'Examples' and 'Details' for more information.

Usage

zcurve(
  z,
  z.lb,
  z.ub,
  p,
  p.lb,
  p.ub,
  data,
  method = "EM",
  bootstrap = 1000,
  parallel = FALSE,
  control = NULL
)

Arguments

Details

The function returns the EM method by default and changing method = "density" gives the KD2 version of z-curve as outlined in Bartoš and Schimmack (2022). For the original z-curve (Brunner and Schimmack 2020), referred to as KD1, specify 'control = "density", control = list(model = "KD1")'. Specifying the lower and upper bounds of z-scores or p-values will fit the censored version of z-curve described in (Schimmack and Bartoš 2023).

Value

The fitted z-curve object

References

Bartoš F, Schimmack U (2022). “Z-curve 2.0: Estimating replication rates and discovery rates.” Meta-Psychology, 6. doi:10.15626/MP.2021.2720.

Brunner J, Schimmack U (2020). “Estimating population mean power under conditions of heterogeneity and selection for significance.” Meta-Psychology, 4. doi:10.15626/MP.2018.874.

Schimmack U, Bartoš F (2023). “Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values.” PLoS ONE, 18(8), e0290084. doi:10.1371/journal.pone.0290084.

See Also

summary.zcurve(), plot.zcurve(), control_EM, control_density

Examples

# load data from OSC 2015 reproducibility project
OSC.z

# fit an EM z-curve (with disabled bootstrap due to examples times limits)
m.EM <- zcurve(OSC.z, method = "EM", bootstrap = FALSE)
# a version with 1000 boostraped samples would looked like:
m.EM <- zcurve(OSC.z, method = "EM", bootstrap = 1000)

# or KD2 z-curve (use larger bootstrap for real inference)
m.D <- zcurve(OSC.z, method = "density", bootstrap = FALSE)

# inspect the results
summary(m.EM)
summary(m.D)
# see '?summary.zcurve' for more output options

# plot the results
plot(m.EM)
plot(m.D)
# see '?plot.zcurve' for more plotting options

# to specify more options, set the control arguments
# ei. increase the maximum number of iterations and change alpha level
ctr1 <- list(
  "max_iter" = 9999,
  "alpha"    = .10
  )
## Not run: m1.EM <- zcurve(OSC.z, method = "EM", bootstrap = FALSE, control = ctr1)
# see '?control_EM' and '?control_density' for more information about different
# z-curves specifications

z-curve estimates

Description

The following functions extract estimates from the z-curve object.

Usage

ERR(object, round.coef = 3)

EDR(object, round.coef = 3)

ODR(object, round.coef = 3)

Soric(object, round.coef = 3)

file_drawer_ration(object, round.coef = 3)

expected_n(object, round.coef = 0)

missing_n(object, round.coef = 0)

significant_n(object)

included_n(object)

Arguments

Details

Technically, ODR, significant n, and included n are not z-curve estimates but they are grouped in this category for convenience.

See Also

zcurve()

Fit a z-curve to clustered data

Description

zcurve_clustered is used to fit z-curve models to clustered data. The function requires a data object created with the zcurve_data() function as the input (where id denotes clusters). Two different methods that account for clustering ar implemented via the EM model: "w" for down weighting the likelihood of the test statistics proportionately to the number of repetitions in the clusters, and "b" for a nested bootstrap where only a single study from each bootstrap is selected for model fitting.

Usage

zcurve_clustered(
  data,
  method = "b",
  bootstrap = 1000,
  parallel = FALSE,
  control = NULL
)

Arguments

Value

The fitted z-curve object

References

There are no references for Rd macro ⁠\insertAllCites⁠ on this help page.

See Also

zcurve(), summary.zcurve(), plot.zcurve(), control_EM, control_density

Prepare data for z-curve

Description

zcurve_data is used to prepare data for the zcurve() function. The function transform strings containing reported test statistics "z", "t", "f", "chi", "p" into two-sided p-values. Test statistics reported as inequalities are as considered to be censored as well as test statistics reported with low accuracy (i.e., rounded to too few decimals). See details for more information.

Usage

zcurve_data(data, id = NULL, rounded = TRUE, stat_precise = 2, p_precise = 3)

Arguments

Details

By default, the function extract the type of test statistic:

"F(df1, df2)=x"

F-statistic with df1 and df2 degrees of freedom,

"chi(df)=x"

Chi-square statistic with df degrees of freedom,

"t(df)=x"

for t-statistic with df degrees of freedom,

"z=x"

for z-statistic,

"p=x"

for p-value.

The input is not case sensitive and automatically removes empty spaces. Furthermore, inequalities ("<" and ">") can be used to denote censoring. I.e., that the p-value is lower than "x" or that the test statistic is larger than "x" respectively. The automatic de-rounding procedure (if rounded = TRUE) treats p-values with less decimal places than specified in p_precise or test statistics with less decimal places than specified in stat_precise as censored on an interval that could result in a given rounded value. I.e., a "p = 0.03" input would be de-rounded as a p-value lower than 0.035 but larger than 0.025.

Value

An object of type "zcurve_data".

See Also

zcurve(), print.zcurve_data(), head.zcurve_data()

Examples

# Specify a character vector containing the test statistics
data <- c("z = 2.1", "t(34) = 2.21", "p < 0.03", "F(2,23) > 10", "p = 0.003")

# Obtain the z-curve data object
data <- zcurve_data(data)

# inspect the resulting object
data

Options for the zcurve package

Description

A placeholder object and functions for the zcurve package. (adapted from the runjags R package).

Usage

zcurve.options(...)

zcurve.get_option(name)

Arguments

Value

The current value of all available zcurve options (after applying any changes specified) is returned invisibly as a named list.