Usage | Release | Development |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
drake
— or, Data Frames in R for Make — is a general-purpose workflow manager for data-driven tasks. It rebuilds intermediate data objects when their dependencies change, and it skips work when the results are already up to date. Not every runthrough starts from scratch, there is native support for parallel and distributed computing, and completed workflows have tangible evidence of reproducibility.
Too many data science projects follow a Sisyphean loop:
Ordinarily, it is hard to avoid rerunning the code from scratch.
But with drake
, you can automatically
To set up a project, load your packages,
load your custom functions,
create_plot <- function(data) {
ggplot(data, aes(x = Petal.Width, fill = Species)) +
geom_histogram()
}
check any supporting files (optional),
# Get the files with drake_example("main").
file.exists("raw_data.xlsx")
#> [1] TRUE
file.exists("report.Rmd")
#> [1] TRUE
and plan what you are going to do.
plan <- drake_plan(
raw_data = readxl::read_excel(file_in("raw_data.xlsx")),
data = raw_data %>%
mutate(Species = forcats::fct_inorder(Species)),
hist = create_plot(data),
fit = lm(Sepal.Width ~ Petal.Width + Species, data),
report = rmarkdown::render(
knitr_in("report.Rmd"),
output_file = file_out("report.html"),
quiet = TRUE
)
)
plan
#> # A tibble: 5 x 2
#> target command
#> <chr> <expr>
#> 1 raw_data readxl::read_excel(file_in("raw_data.xlsx")) …
#> 2 data raw_data %>% mutate(Species = forcats::fct_inorder(Species)) …
#> 3 hist create_plot(data) …
#> 4 fit lm(Sepal.Width ~ Petal.Width + Species, data) …
#> 5 report rmarkdown::render(knitr_in("report.Rmd"), output_file = file_ou…
So far, we have just been setting the stage. Use make()
to do the real work. Targets are built in the correct order regardless of the row order of plan
.
Except for files like report.html
, your output is stored in a hidden .drake/
folder. Reading it back is easy.
readd(data) # See also loadd().
#> # A tibble: 150 x 5
#> Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#> <dbl> <dbl> <dbl> <dbl> <fct>
#> 1 5.1 3.5 1.4 0.2 setosa
#> 2 4.9 3 1.4 0.2 setosa
#> 3 4.7 3.2 1.3 0.2 setosa
#> 4 4.6 3.1 1.5 0.2 setosa
#> 5 5 3.6 1.4 0.2 setosa
#> # … with 145 more rows
You may look back on your work and see room for improvement, but it’s all good! The whole point of drake
is to help you go back and change things quickly and painlessly. For example, we forgot to give our histogram a bin width.
So let’s fix the plotting function.
create_plot <- function(data) {
ggplot(data, aes(x = Petal.Width, fill = Species)) +
geom_histogram(binwidth = 0.25) +
theme_gray(20)
}
drake
knows which results are affected.
The next make()
just builds hist
and report.html
. No point in wasting time on the data or model.
The R community emphasizes reproducibility. Traditional themes include scientific replicability, literate programming with knitr, and version control with git. But internal consistency is important too. Reproducibility carries the promise that your output matches the code and data you say you used. With the exception of non-default triggers and hasty mode, drake
strives to keep this promise.
Suppose you are reviewing someone else’s data analysis project for reproducibility. You scrutinize it carefully, checking that the datasets are available and the documentation is thorough. But could you re-create the results without the help of the original author? With drake
, it is quick and easy to find out.
make(plan)
#> All targets are already up to date.
config <- drake_config(plan)
outdated(config)
#> character(0)
With everything already up to date, you have tangible evidence of reproducibility. Even though you did not re-create the results, you know the results are re-creatable. They faithfully show what the code is producing. Given the right package environment and system configuration, you have everything you need to reproduce all the output by yourself.
When it comes time to actually rerun the entire project, you have much more confidence. Starting over from scratch is trivially easy.
With even more evidence and confidence, you can invest the time to independently replicate the original code base if necessary. Up until this point, you relied on basic drake
functions such as make()
, so you may not have needed to peek at any substantive author-defined code in advance. In that case, you can stay usefully ignorant as you reimplement the original author’s methodology. In other words, drake
could potentially improve the integrity of independent replication.
Ideally, independent observers should be able to read your code and understand it. drake
helps in several ways.
vis_drake_graph()
visualizes how those steps depend on each other.drake
takes care of the parallel scheduling and high-performance computing (HPC) for you. That means the HPC code is no longer tangled up with the code that actually expresses your ideas.Not every project can complete in a single R session on your laptop. Some projects need more speed or computing power. Some require a few local processor cores, and some need large high-performance computing systems. But parallel computing is hard. Your tables and figures depend on your analysis results, and your analyses depend on your datasets, so some tasks must finish before others even begin. drake
knows what to do. Parallelism is implicit and automatic. See the high-performance computing guide for all the details.
# Use the spare cores on your local machine.
make(plan, jobs = 4)
# Or scale up to a supercomputer.
drake_batchtools_tmpl_file("slurm") # https://slurm.schedmd.com/
library(future.batchtools)
future::plan(batchtools_slurm, template = "batchtools.slurm.tmpl", workers = 100)
make(plan, parallelism = "future_lapply")
You can choose among different versions of drake
. The CRAN release often lags behind the online manual but may have fewer bugs.
# Install the latest stable release from CRAN.
install.packages("drake")
# Alternatively, install the development version from GitHub.
install.packages("devtools")
library(devtools)
install_github("ropensci/drake")
A few technical details:
drake
using install.packages()
, devtools::install_github()
, or similar. It is not enough to use devtools::load_all()
, particularly for the parallel computing functionality, in which multiple R sessions initialize and then try to require(drake)
.make(parallelism = "Makefile")
, Windows users may need to download and install Rtools
.make(parallelism = "future")
or make(parallelism = "future_lapply")
to deploy your work to a computing cluster (see the high-performance computing guide), you will need the future.batchtools
package.The main resources to learn drake
are the user manual and the reference website. Others are below.
Thanks to Kirill for preparing a drake
cheat sheet for the workshop.
The FAQ page is an index of links to appropriately-labeled issues on GitHub. To contribute, please submit a new issue and ask that it be labeled as a frequently asked question.
The reference section lists all the available functions. Here are the most important ones.
drake_plan()
: create a workflow data frame (like my_plan
).make()
: build your project.loadd()
: load one or more built targets into your R session.readd()
: read and return a built target.drake_config()
: create a master configuration list for other user-side functions.vis_drake_graph()
: show an interactive visual network representation of your workflow.outdated()
: see which targets will be built in the next make()
.deps()
: check the dependencies of a command or function.failed()
: list the targets that failed to build in the last make()
.diagnose()
: return the full context of a build, including errors, warnings, and messages.Thanks to Kirill for constructing two interactive learnr
tutorials: one supporting drake
itself, and a prerequisite walkthrough of the cooking
package.
There are multiple drake
-powered example projects available here, ranging from beginner-friendly stubs to demonstrations of high-performance computing. You can generate the files for a project with drake_example()
(e.g. drake_example("gsp")
), and you can list the available projects with drake_examples()
. You can contribute your own example project with a fork and pull request.
Author | Venue | Date | Materials |
---|---|---|---|
Amanda Dobbyn | R-Ladies NYC | 2019-02-12 | slides, source |
Will Landau | Harvard DataFest | 2019-01-22 | slides, source |
Karthik Ram | RStudio Conference | 2019-01-18 | video, slides, resources |
Sina Rüeger | Geneva R User Group | 2018-10-04 | slides, example code |
Will Landau | R in Pharma | 2018-08-16 | video, slides, source |
Christine Stawitz | R-Ladies Seattle | 2018-06-25 | materials |
Kirill Müller | Swiss Institute of Bioinformatics | 2018-03-05 | workshop, slides, source, exercises |
Kirill Müller | RStudio Conference | 2018-02-01 | slides, source |
Here are some real-world applications of drake
in the wild.
If you have a project of your own, we would love to add it. Click here to edit the README.Rmd
file.
For context and history, check out this post on the rOpenSci blog and episode 22 of the R Podcast.
The following resources document many known issues and challenges.
If you are still having trouble, please submit a new issue with a bug report or feature request, along with a minimal reproducible example where appropriate.
The GitHub issue tracker is mainly intended for bug reports and feature requests. While questions about usage etc. are also highly encouraged, you may alternatively wish to post to Stack Overflow and use the drake-r-package
tag.
Development is a community effort, and we encourage participation.
The environment for collaboration should be friendly, inclusive, respectful, and safe for everyone, so all participants must obey this repository’s code of conduct.
drake
thrives on the suggestions, questions, and bug reports you submit to the issue tracker. Before posting, please search both the open and closed issues to help us avoid duplication. Usage questions are welcome, but you may also wish to post to Stack Overflow with the drake-r-package
tag.
If you would like to work on the code or documentation, please fork this repository, make the changes in your fork, and then submit a pull request. We will discuss your work and then hopefully merge it into the project.
The original idea of a time-saving reproducible build system extends back at least as far as GNU Make, which still aids the work of data scientists as well as the original user base of complied language programmers. In fact, the name “drake” stands for “Data Frames in R for Make”. Make is used widely in reproducible research. Below are some examples from Karl Broman’s website.
Makefile
at https://github.com/kbroman/ailProbPaper/blob/master/Makefile.Makefile
at https://github.com/kbroman/phyloQTLpaper/blob/master/Makefile.There are several reasons for R users to prefer drake
instead.
drake
already has a Make-powered parallel backend. Just run make(..., parallelism = "Makefile", jobs = 2)
to enjoy most of the original benefits of Make itself.drake
, you can use wildcard templating to automatically generate massive collections of targets with minimal code.make(..., parallelism = "mclapply, jobs = 4")
, drake
launches 4 persistent workers up front and efficiently processes the targets in R.drake
saves all the results for you automatically in a storr cache so you do not have to micromanage the results.drake overlaps with its direct predecessor, remake. In fact, drake owes its core ideas to remake and Rich FitzJohn. Remake’s development repository lists several real-world applications. drake surpasses remake in several important ways, including but not limited to the following.
drake::example_drake()
.Memoization is the strategic caching of the return values of functions. Every time a memoized function is called with a new set of arguments, the return value is saved for future use. Later, whenever the same function is called with the same arguments, the previous return value is salvaged, and the function call is skipped to save time. The memoise package is an excellent implementation of memoization in R.
However, memoization does not go far enough. In reality, the return value of a function depends not only on the function body and the arguments, but also on any nested functions and global variables, the dependencies of those dependencies, and so on upstream. drake
surpasses memoise because it uses the entire dependency network graph of a project to decide which pieces need to be rebuilt and which ones can be skipped.
Much of the R community uses knitr for reproducible research. The idea is to intersperse code chunks in an R Markdown or *.Rnw
file and then generate a dynamic report that weaves together code, output, and prose. Knitr is not designed to be a serious pipeline toolkit, and it should not be the primary computational engine for medium to large data analysis projects.
cache
and autodep
), this functionality is not the focus of knitr. It is deactivated by default, and remake and drake
are more dependable ways to skip work that is already up to date.drake
was designed to manage the entire workflow with knitr reports as targets. The strategy is analogous for knitr reports within remake projects.
Factual’s Drake is similar in concept, but the development effort is completely unrelated to the drake R package.
There are countless other successful pipeline toolkits. The drake
package distinguishes itself with its R-focused approach, Tidyverse-friendly interface, and a thorough selection of parallel computing technologies and scheduling algorithms.
Special thanks to Jarad Niemi, my advisor from graduate school, for first introducing me to the idea of Makefiles for research. He originally set me down the path that led to drake
.
Many thanks to Julia Lowndes, Ben Marwick, and Peter Slaughter for reviewing drake for rOpenSci, and to Maëlle Salmon for such active involvement as the editor. Thanks also to the following people for contributing early in development.
Credit for images is attributed here.