Testing Workbench for Precision-Recall Curves


License
GPL-3.0

Documentation

prcbench

R-CMD-check AppVeyor Build Status codecov.io CodeFactor CRAN_Status_Badge CRAN_Logs_Badge

The aim of the prcbench package is to provide a testing workbench for evaluating precision-recall curves under various conditions. It contains integrated interfaces for the following five tools. It also contains predefined test data sets.

Tool Language Link
precrec R Tool web site, CRAN
ROCR R Tool web site, CRAN
PRROC R CRAN
AUCCalculator Java Tool web site
PerfMeas R CRAN

Disclaimer: prcbench was originally develop to help our precrec library in order to provide fast and accurate calculations of precision-recall curves with extra functionality.

Accuracy evaluation of precision-recall curves

prcbench uses pre-defined test sets to help evaluate the accuracy of precision-recall curves.

  1. create_toolset: creates objects of different tools for testing (5 different tools)
  2. create_testset: selects pre-defined data sets (c1, c2, and c3)
  3. run_evalcurve: evaluates the selected tools on the simulation data
  4. autoplot: shows the results with ggplot2 and patchwork
## Load library
library(prcbench)

## Plot base points and the result of 5 tools on pre-defined test sets (c1, c2, and c3)
toolset <- create_toolset(c("precrec", "ROCR", "AUCCalculator", "PerfMeas", "PRROC"))
testset <- create_testset("curve", c("c1", "c2", "c3"))
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1, ncol = 3, nrow = 2)

Running-time evaluation of precision-recall curves

prcbench helps create simulation data to measure computational times of creating precision-recall curves.

  1. create_toolset: creates objects of different tools for testing
  2. create_testset: creates simulation data
  3. run_benchmark: evaluates the selected tools on the simulation data
## Load library
library(prcbench)

## Run benchmark for auc5 (5 tools) on b10 (balanced 5 positives and 5 negatives)
toolset <- create_toolset(set_names = "auc5")
testset <- create_testset("bench", "b10")
res <- run_benchmark(testset, toolset)

print(res)
testset toolset toolname min lq mean median uq max neval
b10 auc5 AUCCalculator 0.93 0.96 1.12 1.00 1.00 1.68 5
b10 auc5 PerfMeas 0.06 0.06 0.08 0.06 0.07 0.17 5
b10 auc5 precrec 3.40 3.45 3.73 3.47 3.58 4.74 5
b10 auc5 PRROC 0.14 0.14 0.17 0.14 0.16 0.28 5
b10 auc5 ROCR 1.57 1.59 1.69 1.60 1.63 2.06 5

Documentation

  • Introduction to prcbench – a package vignette that contains the descriptions of the functions with several useful examples. View the vignette with vignette("introduction", package = "prcbench") in R.

  • Help pages – all the functions including the S3 generics have their own help pages with plenty of examples. View the main help page with help(package = "prcbench") in R.

Installation

CRAN

install.packages("prcbench")

Dependencies

AUCCalculator requires a Java runtime environment (>= 6) if AUCCalculator needs to be evaluated.

GitHub

You can install a development version of prcbench from our GitHub repository.

devtools::install_github("evalclass/prcbench")
  1. Make sure you have a working development environment.

    • Windows: Install Rtools (available on the CRAN website).

    • Mac: Install Xcode from the Mac App Store.

    • Linux: Install a compiler and various development libraries (details vary across different flavors of Linux).

  2. Install devtools from CRAN with install.packages("devtools").

  3. Install prcbench from the GitHub repository with devtools::install_github("evalclass/prcbench").

Troubleshooting

microbenchmark

microbenchmark does not work on some OSs. prcbench uses system.time when microbenchmark is not available.

rJava

  • Some OSs require en extra configuration step after rJava installation.
sudo R CMD javareconf
  • JDKs
  1. Oracle JDK
  2. OpenJDK
  • JDKs for macOS
  1. AdoptOpenJDK
  2. AdoptOpenJDK with homebrew
install.packages("rJava", configure.args = "--disable-jri")

Citation

Precrec: fast and accurate precision-recall and ROC curve calculations in R

Takaya Saito; Marc Rehmsmeier

Bioinformatics 2017; 33 (1): 145-147.

doi: 10.1093/bioinformatics/btw570

External links