rmarkdown::render problem when called from a package
I've made a small package to reproduce the problem:
# example package
devtools::install_github("privefl/minipkg")
# example Rmd
rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
writeLines(readLines(rmd)) ## see content
# works fine
rmarkdown::render(
rmd,
"all",
envir = new.env(),
encoding = "UTF-8"
)
# !! does not work !!
minipkg::my_render(rmd)
minipkg::my_render ## see source code
I don't understand why the behaviour is different and how to fix this.
Edit: I know I can use Matrix::t()
. My question is more "why do I need to use it in this particular case and not in all the other cases (such as calling rmarkdown::render()
outside of a package)?".
Error
Quitting from lines 10-13 (Matrix.Rmd)
Error in t.default(mat) : argument is not a matrix
Matrix.Rmd File
---
output: html_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
```{r}
library(Matrix)
mat <- rsparsematrix(10, 10, 0.1)
t(mat)
```
Console Output:
> # example package
> devtools::install_github("privefl/minipkg")
Downloading GitHub repo privefl/minipkg@master
✔ checking for file ‘/private/var/folders/md/03gdc4c14z18kbqwpfh4jdfc0000gr/T/RtmpKefs4h/remotes685793b9df4/privefl-minipkg-c02ae62/DESCRIPTION’ ...
─ preparing ‘minipkg’:
✔ checking DESCRIPTION meta-information ...
─ checking for LF line-endings in source and make files and shell scripts
─ checking for empty or unneeded directories
─ building ‘minipkg_0.1.0.tar.gz’
* installing *source* package ‘minipkg’ ...
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (minipkg)
> # example Rmd
> rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
> writeLines(readLines(rmd)) ## see content
---
output: html_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
```{r}
library(Matrix)
mat <- rsparsematrix(10, 10, 0.1)
t(mat)
```
> # works fine
> rmarkdown::render(
+ rmd,
+ "all",
+ envir = new.env(),
+ encoding = "UTF-8"
+ )
processing file: Matrix.Rmd
|............. | 20%
ordinary text without R code
|.......................... | 40%
label: setup (with options)
List of 1
$ include: logi FALSE
|....................................... | 60%
ordinary text without R code
|.................................................... | 80%
label: unnamed-chunk-1
|.................................................................| 100%
ordinary text without R code
output file: Matrix.knit.md
/usr/local/bin/pandoc +RTS -K512m -RTS Matrix.utf8.md --to html4 --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash+smart --output Matrix.html --email-obfuscation none --self-contained --standalone --section-divs --template /Library/Frameworks/R.framework/Versions/3.5/Resources/library/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable 'theme:bootstrap' --include-in-header /var/folders/md/03gdc4c14z18kbqwpfh4jdfc0000gr/T//RtmpKefs4h/rmarkdown-str68525040df1.html --mathjax --variable 'mathjax-url:https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --metadata pagetitle=Matrix.utf8.md
Output created: Matrix.html
> # !! does not work !!
> minipkg::my_render(rmd)
processing file: Matrix.Rmd
|............. | 20%
ordinary text without R code
|.......................... | 40%
label: setup (with options)
List of 1
$ include: logi FALSE
|....................................... | 60%
ordinary text without R code
|.................................................... | 80%
label: unnamed-chunk-1
Quitting from lines 10-13 (Matrix.Rmd)
Error in t.default(mat) : argument is not a matrix
> minipkg::my_render ## see source code
function (rmd)
{
rmarkdown::render(rmd, "all", envir = new.env(), encoding = "UTF-8")
}
<bytecode: 0x7f89c416c2a8>
<environment: namespace:minipkg>
>
r r-markdown
add a comment |
I've made a small package to reproduce the problem:
# example package
devtools::install_github("privefl/minipkg")
# example Rmd
rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
writeLines(readLines(rmd)) ## see content
# works fine
rmarkdown::render(
rmd,
"all",
envir = new.env(),
encoding = "UTF-8"
)
# !! does not work !!
minipkg::my_render(rmd)
minipkg::my_render ## see source code
I don't understand why the behaviour is different and how to fix this.
Edit: I know I can use Matrix::t()
. My question is more "why do I need to use it in this particular case and not in all the other cases (such as calling rmarkdown::render()
outside of a package)?".
Error
Quitting from lines 10-13 (Matrix.Rmd)
Error in t.default(mat) : argument is not a matrix
Matrix.Rmd File
---
output: html_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
```{r}
library(Matrix)
mat <- rsparsematrix(10, 10, 0.1)
t(mat)
```
Console Output:
> # example package
> devtools::install_github("privefl/minipkg")
Downloading GitHub repo privefl/minipkg@master
✔ checking for file ‘/private/var/folders/md/03gdc4c14z18kbqwpfh4jdfc0000gr/T/RtmpKefs4h/remotes685793b9df4/privefl-minipkg-c02ae62/DESCRIPTION’ ...
─ preparing ‘minipkg’:
✔ checking DESCRIPTION meta-information ...
─ checking for LF line-endings in source and make files and shell scripts
─ checking for empty or unneeded directories
─ building ‘minipkg_0.1.0.tar.gz’
* installing *source* package ‘minipkg’ ...
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (minipkg)
> # example Rmd
> rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
> writeLines(readLines(rmd)) ## see content
---
output: html_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
```{r}
library(Matrix)
mat <- rsparsematrix(10, 10, 0.1)
t(mat)
```
> # works fine
> rmarkdown::render(
+ rmd,
+ "all",
+ envir = new.env(),
+ encoding = "UTF-8"
+ )
processing file: Matrix.Rmd
|............. | 20%
ordinary text without R code
|.......................... | 40%
label: setup (with options)
List of 1
$ include: logi FALSE
|....................................... | 60%
ordinary text without R code
|.................................................... | 80%
label: unnamed-chunk-1
|.................................................................| 100%
ordinary text without R code
output file: Matrix.knit.md
/usr/local/bin/pandoc +RTS -K512m -RTS Matrix.utf8.md --to html4 --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash+smart --output Matrix.html --email-obfuscation none --self-contained --standalone --section-divs --template /Library/Frameworks/R.framework/Versions/3.5/Resources/library/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable 'theme:bootstrap' --include-in-header /var/folders/md/03gdc4c14z18kbqwpfh4jdfc0000gr/T//RtmpKefs4h/rmarkdown-str68525040df1.html --mathjax --variable 'mathjax-url:https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --metadata pagetitle=Matrix.utf8.md
Output created: Matrix.html
> # !! does not work !!
> minipkg::my_render(rmd)
processing file: Matrix.Rmd
|............. | 20%
ordinary text without R code
|.......................... | 40%
label: setup (with options)
List of 1
$ include: logi FALSE
|....................................... | 60%
ordinary text without R code
|.................................................... | 80%
label: unnamed-chunk-1
Quitting from lines 10-13 (Matrix.Rmd)
Error in t.default(mat) : argument is not a matrix
> minipkg::my_render ## see source code
function (rmd)
{
rmarkdown::render(rmd, "all", envir = new.env(), encoding = "UTF-8")
}
<bytecode: 0x7f89c416c2a8>
<environment: namespace:minipkg>
>
r r-markdown
add a comment |
I've made a small package to reproduce the problem:
# example package
devtools::install_github("privefl/minipkg")
# example Rmd
rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
writeLines(readLines(rmd)) ## see content
# works fine
rmarkdown::render(
rmd,
"all",
envir = new.env(),
encoding = "UTF-8"
)
# !! does not work !!
minipkg::my_render(rmd)
minipkg::my_render ## see source code
I don't understand why the behaviour is different and how to fix this.
Edit: I know I can use Matrix::t()
. My question is more "why do I need to use it in this particular case and not in all the other cases (such as calling rmarkdown::render()
outside of a package)?".
Error
Quitting from lines 10-13 (Matrix.Rmd)
Error in t.default(mat) : argument is not a matrix
Matrix.Rmd File
---
output: html_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
```{r}
library(Matrix)
mat <- rsparsematrix(10, 10, 0.1)
t(mat)
```
Console Output:
> # example package
> devtools::install_github("privefl/minipkg")
Downloading GitHub repo privefl/minipkg@master
✔ checking for file ‘/private/var/folders/md/03gdc4c14z18kbqwpfh4jdfc0000gr/T/RtmpKefs4h/remotes685793b9df4/privefl-minipkg-c02ae62/DESCRIPTION’ ...
─ preparing ‘minipkg’:
✔ checking DESCRIPTION meta-information ...
─ checking for LF line-endings in source and make files and shell scripts
─ checking for empty or unneeded directories
─ building ‘minipkg_0.1.0.tar.gz’
* installing *source* package ‘minipkg’ ...
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (minipkg)
> # example Rmd
> rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
> writeLines(readLines(rmd)) ## see content
---
output: html_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
```{r}
library(Matrix)
mat <- rsparsematrix(10, 10, 0.1)
t(mat)
```
> # works fine
> rmarkdown::render(
+ rmd,
+ "all",
+ envir = new.env(),
+ encoding = "UTF-8"
+ )
processing file: Matrix.Rmd
|............. | 20%
ordinary text without R code
|.......................... | 40%
label: setup (with options)
List of 1
$ include: logi FALSE
|....................................... | 60%
ordinary text without R code
|.................................................... | 80%
label: unnamed-chunk-1
|.................................................................| 100%
ordinary text without R code
output file: Matrix.knit.md
/usr/local/bin/pandoc +RTS -K512m -RTS Matrix.utf8.md --to html4 --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash+smart --output Matrix.html --email-obfuscation none --self-contained --standalone --section-divs --template /Library/Frameworks/R.framework/Versions/3.5/Resources/library/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable 'theme:bootstrap' --include-in-header /var/folders/md/03gdc4c14z18kbqwpfh4jdfc0000gr/T//RtmpKefs4h/rmarkdown-str68525040df1.html --mathjax --variable 'mathjax-url:https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --metadata pagetitle=Matrix.utf8.md
Output created: Matrix.html
> # !! does not work !!
> minipkg::my_render(rmd)
processing file: Matrix.Rmd
|............. | 20%
ordinary text without R code
|.......................... | 40%
label: setup (with options)
List of 1
$ include: logi FALSE
|....................................... | 60%
ordinary text without R code
|.................................................... | 80%
label: unnamed-chunk-1
Quitting from lines 10-13 (Matrix.Rmd)
Error in t.default(mat) : argument is not a matrix
> minipkg::my_render ## see source code
function (rmd)
{
rmarkdown::render(rmd, "all", envir = new.env(), encoding = "UTF-8")
}
<bytecode: 0x7f89c416c2a8>
<environment: namespace:minipkg>
>
r r-markdown
I've made a small package to reproduce the problem:
# example package
devtools::install_github("privefl/minipkg")
# example Rmd
rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
writeLines(readLines(rmd)) ## see content
# works fine
rmarkdown::render(
rmd,
"all",
envir = new.env(),
encoding = "UTF-8"
)
# !! does not work !!
minipkg::my_render(rmd)
minipkg::my_render ## see source code
I don't understand why the behaviour is different and how to fix this.
Edit: I know I can use Matrix::t()
. My question is more "why do I need to use it in this particular case and not in all the other cases (such as calling rmarkdown::render()
outside of a package)?".
Error
Quitting from lines 10-13 (Matrix.Rmd)
Error in t.default(mat) : argument is not a matrix
Matrix.Rmd File
---
output: html_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
```{r}
library(Matrix)
mat <- rsparsematrix(10, 10, 0.1)
t(mat)
```
Console Output:
> # example package
> devtools::install_github("privefl/minipkg")
Downloading GitHub repo privefl/minipkg@master
✔ checking for file ‘/private/var/folders/md/03gdc4c14z18kbqwpfh4jdfc0000gr/T/RtmpKefs4h/remotes685793b9df4/privefl-minipkg-c02ae62/DESCRIPTION’ ...
─ preparing ‘minipkg’:
✔ checking DESCRIPTION meta-information ...
─ checking for LF line-endings in source and make files and shell scripts
─ checking for empty or unneeded directories
─ building ‘minipkg_0.1.0.tar.gz’
* installing *source* package ‘minipkg’ ...
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (minipkg)
> # example Rmd
> rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
> writeLines(readLines(rmd)) ## see content
---
output: html_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
```{r}
library(Matrix)
mat <- rsparsematrix(10, 10, 0.1)
t(mat)
```
> # works fine
> rmarkdown::render(
+ rmd,
+ "all",
+ envir = new.env(),
+ encoding = "UTF-8"
+ )
processing file: Matrix.Rmd
|............. | 20%
ordinary text without R code
|.......................... | 40%
label: setup (with options)
List of 1
$ include: logi FALSE
|....................................... | 60%
ordinary text without R code
|.................................................... | 80%
label: unnamed-chunk-1
|.................................................................| 100%
ordinary text without R code
output file: Matrix.knit.md
/usr/local/bin/pandoc +RTS -K512m -RTS Matrix.utf8.md --to html4 --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash+smart --output Matrix.html --email-obfuscation none --self-contained --standalone --section-divs --template /Library/Frameworks/R.framework/Versions/3.5/Resources/library/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable 'theme:bootstrap' --include-in-header /var/folders/md/03gdc4c14z18kbqwpfh4jdfc0000gr/T//RtmpKefs4h/rmarkdown-str68525040df1.html --mathjax --variable 'mathjax-url:https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --metadata pagetitle=Matrix.utf8.md
Output created: Matrix.html
> # !! does not work !!
> minipkg::my_render(rmd)
processing file: Matrix.Rmd
|............. | 20%
ordinary text without R code
|.......................... | 40%
label: setup (with options)
List of 1
$ include: logi FALSE
|....................................... | 60%
ordinary text without R code
|.................................................... | 80%
label: unnamed-chunk-1
Quitting from lines 10-13 (Matrix.Rmd)
Error in t.default(mat) : argument is not a matrix
> minipkg::my_render ## see source code
function (rmd)
{
rmarkdown::render(rmd, "all", envir = new.env(), encoding = "UTF-8")
}
<bytecode: 0x7f89c416c2a8>
<environment: namespace:minipkg>
>
r r-markdown
r r-markdown
edited Nov 23 at 20:02
asked Nov 21 at 10:16
F. Privé
6,6952840
6,6952840
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
How it works
The problem is envir = new.env()
.
What you need is envir = new.env(parent = globalenv())
:
devtools::install_github("privefl/minipkg")
rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
minipkg::my_render(rmd)
# Fails
f <- minipkg::my_render
body(f) <- quote(rmarkdown::render(rmd, "all", envir = new.env(parent = globalenv()), encoding = "UTF-8"))
ns <- getNamespace("minipkg")
unlockBinding("my_render", ns)
assign("my_render", f, envir = ns)
minipkg::my_render(rmd)
# Patched one works :)
Why it works
Look at the default arguments of new.env()
to find that the default parent environment is parent.frame()
. Note that from the console, this will be globalenv()
and from within a package it will be this packages namespace (not the same as package environment!).
You can get a package namespace with getNamespace("pkg")
. It is the environment that contains all (also internal) objects of the package. The problem is that this environment is in a sense "disconnected" from the usual search / method lookup mechanic in R and so you won't find the necessary methods even though they are attached to search()
.
Now chosing new.env(parent = globalenv())
sets the parent environment to be on the top of the search path and thus able to find all attached methods.
Benchmarking different approaches
These three approaches all produce proper html files:
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render <- function(rmd) {
rmarkdown::render(
rmd,
"all",
envir = new.env(parent = globalenv()),
encoding = "UTF-8"
)
}
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render2 <- function(rmd) {
cl <- parallel::makePSOCKcluster(1)
on.exit(parallel::stopCluster(cl), add = TRUE)
parallel::clusterExport(cl, "rmd", envir = environment())
parallel::clusterEvalQ(cl, {
rmarkdown::render(rmd, "all", encoding = "UTF-8")
})[[1]]
}
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render3 <- function(rmd) {
system2(
command = "R",
args = c("-e", shQuote(sprintf("rmarkdown::render('%s', 'all', encoding = 'UTF-8')", gsub("\\", "/", normalizePath(rmd))))),
wait = TRUE
)
}
Now it's interesting to compare their speed:
> microbenchmark::microbenchmark(my_render("inst/extdata/Matrix.Rmd"), my_render2("inst/extdata/Matrix.Rmd"), my_render3("inst/extdata/Matrix.Rmd"), times = 10L)
[...]
Unit: milliseconds
expr min lq mean median uq max neval
my_render("inst/extdata/Matrix.Rmd") 352.7927 410.604 656.5211 460.0608 560.3386 1836.452 10
my_render2("inst/extdata/Matrix.Rmd") 1981.8844 2015.541 2163.1875 2118.0030 2307.2812 2407.027 10
my_render3("inst/extdata/Matrix.Rmd") 2061.7076 2079.574 2152.0351 2138.9546 2181.1284 2377.623 10
Conclusions
envir = new.env(globalenv())
is by far the fastest (almost 4x faster than the alternatives)
I expect the overhead to be constant, so it should be irrelevant for larger Rmd files.- There is no discernible difference between spawning a new proces with
system2
and using a parallel SOCK cluster with 1 node.
Good job at finding the root of the problem. Yet, it feels wrong to add the global environment to the search path for the Rmd. Does this means that the Rmd has access to the current global env? I was thinking about usingnew.env(parent = parent.env(globalenv()))
. But this means that the Rmd has access to packages loaded before, right?
– F. Privé
Nov 23 at 22:49
@F.Privé That might work with already attached packages but new library() calls might modify the search path „above“. Also, source() calls modify globalenv() unless local = TRUE is specified. All in all it‘s probably safer to do system(„RScript -e rmarkdown::render(...)“) and fire up a fresh R session for the knitting. This is essentially what RStudio does when you click Knit.
– AlexR
Nov 24 at 5:40
I'm not a great fan of system calls inside packages. But using the "same idea" and executing it in a PSOCK cluster, it seems to work fine.
– F. Privé
Nov 24 at 6:15
Seeminipkg::my_render2()
.
– F. Privé
Nov 24 at 6:19
@F.Privé That‘s not much different, maybe a bit more overhead for setting up R in slave mode. Anyway, using a parallel package just hides the creation of the R process - still safer than playing with the envir parameter but I think system or system2 are more efficient and clearer as to why. Plus on linux a (default) FORK cluster will have a copy of globalenv(), so care must be taken to specify the cluster type.
– AlexR
Nov 24 at 6:19
|
show 4 more comments
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53409789%2frmarkdownrender-problem-when-called-from-a-package%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
How it works
The problem is envir = new.env()
.
What you need is envir = new.env(parent = globalenv())
:
devtools::install_github("privefl/minipkg")
rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
minipkg::my_render(rmd)
# Fails
f <- minipkg::my_render
body(f) <- quote(rmarkdown::render(rmd, "all", envir = new.env(parent = globalenv()), encoding = "UTF-8"))
ns <- getNamespace("minipkg")
unlockBinding("my_render", ns)
assign("my_render", f, envir = ns)
minipkg::my_render(rmd)
# Patched one works :)
Why it works
Look at the default arguments of new.env()
to find that the default parent environment is parent.frame()
. Note that from the console, this will be globalenv()
and from within a package it will be this packages namespace (not the same as package environment!).
You can get a package namespace with getNamespace("pkg")
. It is the environment that contains all (also internal) objects of the package. The problem is that this environment is in a sense "disconnected" from the usual search / method lookup mechanic in R and so you won't find the necessary methods even though they are attached to search()
.
Now chosing new.env(parent = globalenv())
sets the parent environment to be on the top of the search path and thus able to find all attached methods.
Benchmarking different approaches
These three approaches all produce proper html files:
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render <- function(rmd) {
rmarkdown::render(
rmd,
"all",
envir = new.env(parent = globalenv()),
encoding = "UTF-8"
)
}
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render2 <- function(rmd) {
cl <- parallel::makePSOCKcluster(1)
on.exit(parallel::stopCluster(cl), add = TRUE)
parallel::clusterExport(cl, "rmd", envir = environment())
parallel::clusterEvalQ(cl, {
rmarkdown::render(rmd, "all", encoding = "UTF-8")
})[[1]]
}
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render3 <- function(rmd) {
system2(
command = "R",
args = c("-e", shQuote(sprintf("rmarkdown::render('%s', 'all', encoding = 'UTF-8')", gsub("\\", "/", normalizePath(rmd))))),
wait = TRUE
)
}
Now it's interesting to compare their speed:
> microbenchmark::microbenchmark(my_render("inst/extdata/Matrix.Rmd"), my_render2("inst/extdata/Matrix.Rmd"), my_render3("inst/extdata/Matrix.Rmd"), times = 10L)
[...]
Unit: milliseconds
expr min lq mean median uq max neval
my_render("inst/extdata/Matrix.Rmd") 352.7927 410.604 656.5211 460.0608 560.3386 1836.452 10
my_render2("inst/extdata/Matrix.Rmd") 1981.8844 2015.541 2163.1875 2118.0030 2307.2812 2407.027 10
my_render3("inst/extdata/Matrix.Rmd") 2061.7076 2079.574 2152.0351 2138.9546 2181.1284 2377.623 10
Conclusions
envir = new.env(globalenv())
is by far the fastest (almost 4x faster than the alternatives)
I expect the overhead to be constant, so it should be irrelevant for larger Rmd files.- There is no discernible difference between spawning a new proces with
system2
and using a parallel SOCK cluster with 1 node.
Good job at finding the root of the problem. Yet, it feels wrong to add the global environment to the search path for the Rmd. Does this means that the Rmd has access to the current global env? I was thinking about usingnew.env(parent = parent.env(globalenv()))
. But this means that the Rmd has access to packages loaded before, right?
– F. Privé
Nov 23 at 22:49
@F.Privé That might work with already attached packages but new library() calls might modify the search path „above“. Also, source() calls modify globalenv() unless local = TRUE is specified. All in all it‘s probably safer to do system(„RScript -e rmarkdown::render(...)“) and fire up a fresh R session for the knitting. This is essentially what RStudio does when you click Knit.
– AlexR
Nov 24 at 5:40
I'm not a great fan of system calls inside packages. But using the "same idea" and executing it in a PSOCK cluster, it seems to work fine.
– F. Privé
Nov 24 at 6:15
Seeminipkg::my_render2()
.
– F. Privé
Nov 24 at 6:19
@F.Privé That‘s not much different, maybe a bit more overhead for setting up R in slave mode. Anyway, using a parallel package just hides the creation of the R process - still safer than playing with the envir parameter but I think system or system2 are more efficient and clearer as to why. Plus on linux a (default) FORK cluster will have a copy of globalenv(), so care must be taken to specify the cluster type.
– AlexR
Nov 24 at 6:19
|
show 4 more comments
How it works
The problem is envir = new.env()
.
What you need is envir = new.env(parent = globalenv())
:
devtools::install_github("privefl/minipkg")
rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
minipkg::my_render(rmd)
# Fails
f <- minipkg::my_render
body(f) <- quote(rmarkdown::render(rmd, "all", envir = new.env(parent = globalenv()), encoding = "UTF-8"))
ns <- getNamespace("minipkg")
unlockBinding("my_render", ns)
assign("my_render", f, envir = ns)
minipkg::my_render(rmd)
# Patched one works :)
Why it works
Look at the default arguments of new.env()
to find that the default parent environment is parent.frame()
. Note that from the console, this will be globalenv()
and from within a package it will be this packages namespace (not the same as package environment!).
You can get a package namespace with getNamespace("pkg")
. It is the environment that contains all (also internal) objects of the package. The problem is that this environment is in a sense "disconnected" from the usual search / method lookup mechanic in R and so you won't find the necessary methods even though they are attached to search()
.
Now chosing new.env(parent = globalenv())
sets the parent environment to be on the top of the search path and thus able to find all attached methods.
Benchmarking different approaches
These three approaches all produce proper html files:
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render <- function(rmd) {
rmarkdown::render(
rmd,
"all",
envir = new.env(parent = globalenv()),
encoding = "UTF-8"
)
}
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render2 <- function(rmd) {
cl <- parallel::makePSOCKcluster(1)
on.exit(parallel::stopCluster(cl), add = TRUE)
parallel::clusterExport(cl, "rmd", envir = environment())
parallel::clusterEvalQ(cl, {
rmarkdown::render(rmd, "all", encoding = "UTF-8")
})[[1]]
}
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render3 <- function(rmd) {
system2(
command = "R",
args = c("-e", shQuote(sprintf("rmarkdown::render('%s', 'all', encoding = 'UTF-8')", gsub("\\", "/", normalizePath(rmd))))),
wait = TRUE
)
}
Now it's interesting to compare their speed:
> microbenchmark::microbenchmark(my_render("inst/extdata/Matrix.Rmd"), my_render2("inst/extdata/Matrix.Rmd"), my_render3("inst/extdata/Matrix.Rmd"), times = 10L)
[...]
Unit: milliseconds
expr min lq mean median uq max neval
my_render("inst/extdata/Matrix.Rmd") 352.7927 410.604 656.5211 460.0608 560.3386 1836.452 10
my_render2("inst/extdata/Matrix.Rmd") 1981.8844 2015.541 2163.1875 2118.0030 2307.2812 2407.027 10
my_render3("inst/extdata/Matrix.Rmd") 2061.7076 2079.574 2152.0351 2138.9546 2181.1284 2377.623 10
Conclusions
envir = new.env(globalenv())
is by far the fastest (almost 4x faster than the alternatives)
I expect the overhead to be constant, so it should be irrelevant for larger Rmd files.- There is no discernible difference between spawning a new proces with
system2
and using a parallel SOCK cluster with 1 node.
Good job at finding the root of the problem. Yet, it feels wrong to add the global environment to the search path for the Rmd. Does this means that the Rmd has access to the current global env? I was thinking about usingnew.env(parent = parent.env(globalenv()))
. But this means that the Rmd has access to packages loaded before, right?
– F. Privé
Nov 23 at 22:49
@F.Privé That might work with already attached packages but new library() calls might modify the search path „above“. Also, source() calls modify globalenv() unless local = TRUE is specified. All in all it‘s probably safer to do system(„RScript -e rmarkdown::render(...)“) and fire up a fresh R session for the knitting. This is essentially what RStudio does when you click Knit.
– AlexR
Nov 24 at 5:40
I'm not a great fan of system calls inside packages. But using the "same idea" and executing it in a PSOCK cluster, it seems to work fine.
– F. Privé
Nov 24 at 6:15
Seeminipkg::my_render2()
.
– F. Privé
Nov 24 at 6:19
@F.Privé That‘s not much different, maybe a bit more overhead for setting up R in slave mode. Anyway, using a parallel package just hides the creation of the R process - still safer than playing with the envir parameter but I think system or system2 are more efficient and clearer as to why. Plus on linux a (default) FORK cluster will have a copy of globalenv(), so care must be taken to specify the cluster type.
– AlexR
Nov 24 at 6:19
|
show 4 more comments
How it works
The problem is envir = new.env()
.
What you need is envir = new.env(parent = globalenv())
:
devtools::install_github("privefl/minipkg")
rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
minipkg::my_render(rmd)
# Fails
f <- minipkg::my_render
body(f) <- quote(rmarkdown::render(rmd, "all", envir = new.env(parent = globalenv()), encoding = "UTF-8"))
ns <- getNamespace("minipkg")
unlockBinding("my_render", ns)
assign("my_render", f, envir = ns)
minipkg::my_render(rmd)
# Patched one works :)
Why it works
Look at the default arguments of new.env()
to find that the default parent environment is parent.frame()
. Note that from the console, this will be globalenv()
and from within a package it will be this packages namespace (not the same as package environment!).
You can get a package namespace with getNamespace("pkg")
. It is the environment that contains all (also internal) objects of the package. The problem is that this environment is in a sense "disconnected" from the usual search / method lookup mechanic in R and so you won't find the necessary methods even though they are attached to search()
.
Now chosing new.env(parent = globalenv())
sets the parent environment to be on the top of the search path and thus able to find all attached methods.
Benchmarking different approaches
These three approaches all produce proper html files:
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render <- function(rmd) {
rmarkdown::render(
rmd,
"all",
envir = new.env(parent = globalenv()),
encoding = "UTF-8"
)
}
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render2 <- function(rmd) {
cl <- parallel::makePSOCKcluster(1)
on.exit(parallel::stopCluster(cl), add = TRUE)
parallel::clusterExport(cl, "rmd", envir = environment())
parallel::clusterEvalQ(cl, {
rmarkdown::render(rmd, "all", encoding = "UTF-8")
})[[1]]
}
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render3 <- function(rmd) {
system2(
command = "R",
args = c("-e", shQuote(sprintf("rmarkdown::render('%s', 'all', encoding = 'UTF-8')", gsub("\\", "/", normalizePath(rmd))))),
wait = TRUE
)
}
Now it's interesting to compare their speed:
> microbenchmark::microbenchmark(my_render("inst/extdata/Matrix.Rmd"), my_render2("inst/extdata/Matrix.Rmd"), my_render3("inst/extdata/Matrix.Rmd"), times = 10L)
[...]
Unit: milliseconds
expr min lq mean median uq max neval
my_render("inst/extdata/Matrix.Rmd") 352.7927 410.604 656.5211 460.0608 560.3386 1836.452 10
my_render2("inst/extdata/Matrix.Rmd") 1981.8844 2015.541 2163.1875 2118.0030 2307.2812 2407.027 10
my_render3("inst/extdata/Matrix.Rmd") 2061.7076 2079.574 2152.0351 2138.9546 2181.1284 2377.623 10
Conclusions
envir = new.env(globalenv())
is by far the fastest (almost 4x faster than the alternatives)
I expect the overhead to be constant, so it should be irrelevant for larger Rmd files.- There is no discernible difference between spawning a new proces with
system2
and using a parallel SOCK cluster with 1 node.
How it works
The problem is envir = new.env()
.
What you need is envir = new.env(parent = globalenv())
:
devtools::install_github("privefl/minipkg")
rmd <- system.file("extdata", "Matrix.Rmd", package = "minipkg")
minipkg::my_render(rmd)
# Fails
f <- minipkg::my_render
body(f) <- quote(rmarkdown::render(rmd, "all", envir = new.env(parent = globalenv()), encoding = "UTF-8"))
ns <- getNamespace("minipkg")
unlockBinding("my_render", ns)
assign("my_render", f, envir = ns)
minipkg::my_render(rmd)
# Patched one works :)
Why it works
Look at the default arguments of new.env()
to find that the default parent environment is parent.frame()
. Note that from the console, this will be globalenv()
and from within a package it will be this packages namespace (not the same as package environment!).
You can get a package namespace with getNamespace("pkg")
. It is the environment that contains all (also internal) objects of the package. The problem is that this environment is in a sense "disconnected" from the usual search / method lookup mechanic in R and so you won't find the necessary methods even though they are attached to search()
.
Now chosing new.env(parent = globalenv())
sets the parent environment to be on the top of the search path and thus able to find all attached methods.
Benchmarking different approaches
These three approaches all produce proper html files:
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render <- function(rmd) {
rmarkdown::render(
rmd,
"all",
envir = new.env(parent = globalenv()),
encoding = "UTF-8"
)
}
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render2 <- function(rmd) {
cl <- parallel::makePSOCKcluster(1)
on.exit(parallel::stopCluster(cl), add = TRUE)
parallel::clusterExport(cl, "rmd", envir = environment())
parallel::clusterEvalQ(cl, {
rmarkdown::render(rmd, "all", encoding = "UTF-8")
})[[1]]
}
#' Render an Rmd file
#' @param rmd Path of the R Markdown file to render.
#' @export
my_render3 <- function(rmd) {
system2(
command = "R",
args = c("-e", shQuote(sprintf("rmarkdown::render('%s', 'all', encoding = 'UTF-8')", gsub("\\", "/", normalizePath(rmd))))),
wait = TRUE
)
}
Now it's interesting to compare their speed:
> microbenchmark::microbenchmark(my_render("inst/extdata/Matrix.Rmd"), my_render2("inst/extdata/Matrix.Rmd"), my_render3("inst/extdata/Matrix.Rmd"), times = 10L)
[...]
Unit: milliseconds
expr min lq mean median uq max neval
my_render("inst/extdata/Matrix.Rmd") 352.7927 410.604 656.5211 460.0608 560.3386 1836.452 10
my_render2("inst/extdata/Matrix.Rmd") 1981.8844 2015.541 2163.1875 2118.0030 2307.2812 2407.027 10
my_render3("inst/extdata/Matrix.Rmd") 2061.7076 2079.574 2152.0351 2138.9546 2181.1284 2377.623 10
Conclusions
envir = new.env(globalenv())
is by far the fastest (almost 4x faster than the alternatives)
I expect the overhead to be constant, so it should be irrelevant for larger Rmd files.- There is no discernible difference between spawning a new proces with
system2
and using a parallel SOCK cluster with 1 node.
edited Nov 28 at 11:06
answered Nov 23 at 20:19
AlexR
1,961922
1,961922
Good job at finding the root of the problem. Yet, it feels wrong to add the global environment to the search path for the Rmd. Does this means that the Rmd has access to the current global env? I was thinking about usingnew.env(parent = parent.env(globalenv()))
. But this means that the Rmd has access to packages loaded before, right?
– F. Privé
Nov 23 at 22:49
@F.Privé That might work with already attached packages but new library() calls might modify the search path „above“. Also, source() calls modify globalenv() unless local = TRUE is specified. All in all it‘s probably safer to do system(„RScript -e rmarkdown::render(...)“) and fire up a fresh R session for the knitting. This is essentially what RStudio does when you click Knit.
– AlexR
Nov 24 at 5:40
I'm not a great fan of system calls inside packages. But using the "same idea" and executing it in a PSOCK cluster, it seems to work fine.
– F. Privé
Nov 24 at 6:15
Seeminipkg::my_render2()
.
– F. Privé
Nov 24 at 6:19
@F.Privé That‘s not much different, maybe a bit more overhead for setting up R in slave mode. Anyway, using a parallel package just hides the creation of the R process - still safer than playing with the envir parameter but I think system or system2 are more efficient and clearer as to why. Plus on linux a (default) FORK cluster will have a copy of globalenv(), so care must be taken to specify the cluster type.
– AlexR
Nov 24 at 6:19
|
show 4 more comments
Good job at finding the root of the problem. Yet, it feels wrong to add the global environment to the search path for the Rmd. Does this means that the Rmd has access to the current global env? I was thinking about usingnew.env(parent = parent.env(globalenv()))
. But this means that the Rmd has access to packages loaded before, right?
– F. Privé
Nov 23 at 22:49
@F.Privé That might work with already attached packages but new library() calls might modify the search path „above“. Also, source() calls modify globalenv() unless local = TRUE is specified. All in all it‘s probably safer to do system(„RScript -e rmarkdown::render(...)“) and fire up a fresh R session for the knitting. This is essentially what RStudio does when you click Knit.
– AlexR
Nov 24 at 5:40
I'm not a great fan of system calls inside packages. But using the "same idea" and executing it in a PSOCK cluster, it seems to work fine.
– F. Privé
Nov 24 at 6:15
Seeminipkg::my_render2()
.
– F. Privé
Nov 24 at 6:19
@F.Privé That‘s not much different, maybe a bit more overhead for setting up R in slave mode. Anyway, using a parallel package just hides the creation of the R process - still safer than playing with the envir parameter but I think system or system2 are more efficient and clearer as to why. Plus on linux a (default) FORK cluster will have a copy of globalenv(), so care must be taken to specify the cluster type.
– AlexR
Nov 24 at 6:19
Good job at finding the root of the problem. Yet, it feels wrong to add the global environment to the search path for the Rmd. Does this means that the Rmd has access to the current global env? I was thinking about using
new.env(parent = parent.env(globalenv()))
. But this means that the Rmd has access to packages loaded before, right?– F. Privé
Nov 23 at 22:49
Good job at finding the root of the problem. Yet, it feels wrong to add the global environment to the search path for the Rmd. Does this means that the Rmd has access to the current global env? I was thinking about using
new.env(parent = parent.env(globalenv()))
. But this means that the Rmd has access to packages loaded before, right?– F. Privé
Nov 23 at 22:49
@F.Privé That might work with already attached packages but new library() calls might modify the search path „above“. Also, source() calls modify globalenv() unless local = TRUE is specified. All in all it‘s probably safer to do system(„RScript -e rmarkdown::render(...)“) and fire up a fresh R session for the knitting. This is essentially what RStudio does when you click Knit.
– AlexR
Nov 24 at 5:40
@F.Privé That might work with already attached packages but new library() calls might modify the search path „above“. Also, source() calls modify globalenv() unless local = TRUE is specified. All in all it‘s probably safer to do system(„RScript -e rmarkdown::render(...)“) and fire up a fresh R session for the knitting. This is essentially what RStudio does when you click Knit.
– AlexR
Nov 24 at 5:40
I'm not a great fan of system calls inside packages. But using the "same idea" and executing it in a PSOCK cluster, it seems to work fine.
– F. Privé
Nov 24 at 6:15
I'm not a great fan of system calls inside packages. But using the "same idea" and executing it in a PSOCK cluster, it seems to work fine.
– F. Privé
Nov 24 at 6:15
See
minipkg::my_render2()
.– F. Privé
Nov 24 at 6:19
See
minipkg::my_render2()
.– F. Privé
Nov 24 at 6:19
@F.Privé That‘s not much different, maybe a bit more overhead for setting up R in slave mode. Anyway, using a parallel package just hides the creation of the R process - still safer than playing with the envir parameter but I think system or system2 are more efficient and clearer as to why. Plus on linux a (default) FORK cluster will have a copy of globalenv(), so care must be taken to specify the cluster type.
– AlexR
Nov 24 at 6:19
@F.Privé That‘s not much different, maybe a bit more overhead for setting up R in slave mode. Anyway, using a parallel package just hides the creation of the R process - still safer than playing with the envir parameter but I think system or system2 are more efficient and clearer as to why. Plus on linux a (default) FORK cluster will have a copy of globalenv(), so care must be taken to specify the cluster type.
– AlexR
Nov 24 at 6:19
|
show 4 more comments
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53409789%2frmarkdownrender-problem-when-called-from-a-package%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown