Posts on Credibly Curious

R Version 4.5.0 is Out!

Mon, 14 Apr 2025 00:00:00 +0000

Some windows. Olympus XA, Portra 800. Photo by Nicholas Tierney

The new R version 4.5.0 is out, and you should get it!

I’ve read through the NEWS file, which details every change - there are many! I would recommend having a skim.

If you’d like to have a skim of the news file, you even use the news() R function, which will open the NEWS file to your help file pane in RStudio. This can also tell us how many news bullets were added:

news_4_5_0 <- news(Version == "4.5.0")
# just show the first 5
head(news_4_5_0)
#>                         Changes in version 4.5.0                        
#> 
#> NEW FEATURES
#> 
#>     o   as.integer(rl) and hence as.raw(rl) now work for a list of
#>   raw(1) elements, as proposed by Michael Chirico's PR#18696.
#> 
#>     o   graphics' grid() gains optional argument nintLog.
#> 
#>     o   New functions check_package_urls() and check_package_dois() in
#>   package tools for checking URLs and DOIs in package sources.
#> 
#>     o   New head() and tail() methods for class "ts" time series,
#>   proposed by Spencer Graves on R-devel.
#> 
#>     o   New qr.influence() function, a "bare bones" interface to the
#>   lm.influence() leave-one-out diagnostics computations; wished
#>   for in PR#18739.
#> 
#>     o   Package citation() results auto-generated from the package
#>   metadata now also provide package DOIs for CRAN and
#>   Bioconductor packages.

The work of this release involved many people, and there are 150 changes. I think it’s worthwhile reading over these changes, and I am incredibly grateful to all the amazing volunteers who put their time and energy into continuing to make R stable, and awesome.

I thought I’d share just a couple of the changes that jumped out at me as I was reading.

Installation is now faster

install.packages() and download.packages() download packages simultaneously using libcurl, significantly reducing download times when installing or downloading multiple packages.

This is cool! Installing packages is now much faster! I love this. Generally, I use pak, which I believe already does this. But it’s a nice win for vanilla R to know I can download things faster.

Data

There are two new datasets added to R! The last time (that I could find) a new dataset was added to R was in R version 3.0.2, which, going to the release dates, was in September 2013. So, nearly 12 years ago! That NEWS item read:

Dataset npk has been copied from MASS to allow more tests to be run without recommended packages being installed.

You might be familiar with one of these new datasets, the penguins data (!) from the palmerpenguins R package.

artwork by Allison Horst

The NEWS item reads:

New datasets penguins and penguins_raw thanks to Ella Kaye, Heather Turner, and Kristen Gorman.

This is huge! Well, I think it is. Now we don’t need to write library(palmerpenguins) if you want to use the penguins or penguins_raw data. Looking at the examples, we can see the following base visualisations:

## bill depth vs. length by species (color) and sex (symbol):
## positive correlations for all species, males tend to have bigger bills
head(penguins)
#>   species    island bill_len bill_dep flipper_len body_mass    sex year
#> 1  Adelie Torgersen     39.1     18.7         181      3750   male 2007
#> 2  Adelie Torgersen     39.5     17.4         186      3800 female 2007
#> 3  Adelie Torgersen     40.3     18.0         195      3250 female 2007
#> 4  Adelie Torgersen       NA       NA          NA        NA   <NA> 2007
#> 5  Adelie Torgersen     36.7     19.3         193      3450 female 2007
#> 6  Adelie Torgersen     39.3     20.6         190      3650   male 2007
names(penguins)
#> [1] "species"     "island"      "bill_len"    "bill_dep"    "flipper_len"
#> [6] "body_mass"   "sex"         "year"
sym <- c(1, 16)
pal <- c(
  "darkorange",
  "purple",
  "cyan4"
  )
plot(
  bill_dep ~ bill_len, 
  data = penguins, 
  pch = sym[sex], 
  col = pal[species]
  )

It is worth noting that the variable names are slightly different in the new base R version - bill_len (not bill_length) bill_dep (not bill_depth), and flipper_len (not flipper_length), which they note in the helpfile: “The penguins data has some shorter variable names than the palmerpenguins version, for compact code and data display.”

There has been a bit of work done on the penguins data, and I wanted to share the relevant references:

Gorman, K. B., Williams, T. D. and Fraser, W. R. (2014) Ecological Sexual Dimorphism and Environmental Variability within a Community of Antarctic Penguins (Genus Pygoscelis). PLoS ONE 9, 3, e90081; doi:10.1371/journal.pone.0090081.
Horst, A. M., Hill, A. P. and Gorman, K. B. (2022) Palmer Archipelago Penguins Data in the palmerpenguins R Package - An Alternative to Anderson’s Irises. R Journal 14, 1; doi:10.32614/RJ-2022-020.
Kaye, E., Turner, H., Gorman, K. B., Horst, A. M. and Hill, A. P. (2025) Preparing the Palmer Penguins Data for the datasets Package in R. doi:10.5281/zenodo.14902740.

And there’s ANOTHER dataset!

New dataset gait thanks to Heather Turner and Ella Kaye, used in examples.

I hadn’t come across this before, the helpfile (?gait) reads:

Hip and knee angle (in degrees) through a 20 point movement cycle for 39 boys.

Which is stored as a 3D array, as stated in the helpfile:

A 3-dimensional array with dimensions c(20, 39, 2) giving the “Hip Angle” and “Knee Angle” (in degrees) for 39 repetitions of a 20 point gait cycle (over standardized gait times).

head(gait)
#> , , Variable = Hip Angle
#> 
#>        Subject
#> Time    boy1 boy2 boy3 boy4 boy5 boy6 boy7 boy8 boy9 boy10 boy11 boy12 boy13
#>   0.025   37   47   46   37   20   57   46   46   46    35    38    35    34
#>   0.075   36   46   44   36   18   48   38   46   42    34    37    31    31
#>   0.125   33   42   39   27   11   44   33   43   37    29    33    29    27
#>   0.175   29   34   34   20    8   35   25   40   34    28    29    26    23
#>   0.225   23   27   33   15    7   31   18   36   31    19    26    22    19
#>   0.275   18   21   27   15    5   27   15   30   25    15    20    19    15
#>        Subject
#> Time    boy14 boy15 boy16 boy17 boy18 boy19 boy20 boy21 boy22 boy23 boy24 boy25
#>   0.025    43    43    40    51    52    36    35    46    43    55    39    37
#>   0.075    41    37    41    49    46    33    37    38    41    51    38    34
#>   0.125    36    35    36    45    41    28    33    30    37    47    31    30
#>   0.175    31    28    32    39    35    22    27    23    30    41    27    27
#>   0.225    26    26    27    31    31    18    22    17    24    35    21    26
#>   0.275    20    21    20    23    24    13    14    13    16    30    14    19
#>        Subject
#> Time    boy26 boy27 boy28 boy29 boy30 boy31 boy32 boy33 boy34 boy35 boy36 boy37
#>   0.025    36    36    42    38    46    54    52    32    46    46    48    44
#>   0.075    33    33    40    34    47    48    44    28    41    44    42    41
#>   0.125    28    30    40    30    44    44    44    26    38    40    42    38
#>   0.175    22    28    34    23    37    37    33    22    31    35    35    32
#>   0.225    18    21    23    17    29    30    28    19    25    31    30    24
#>   0.275    13    15    15    12    23    27    27    13    20    25    23    18
#>        Subject
#> Time    boy38 boy39
#>   0.025    55    48
#>   0.075    56    50
#>   0.125    51    47
#>   0.175    46    42
#>   0.225    41    37
#>   0.275    36    29
#> 
#> , , Variable = Knee Angle
#> 
#>        Subject
#> Time    boy1 boy2 boy3 boy4 boy5 boy6 boy7 boy8 boy9 boy10 boy11 boy12 boy13
#>   0.025   10   16   18    5    2   15   13   14   15     9    13     7     9
#>   0.075   15   25   27   14    6   17   16   17   20    22    24     8    14
#>   0.125   18   28   32   16    6   23   22   18   23    25    27    11    16
#>   0.175   18   25   32   17    6   23   17   19   26    21    23    12    15
#>   0.225   15   18   28   10    5   20   12   19   25    10    18     8    15
#>   0.275   14   12   23    8    6   19    9   15   21     9    13     6    12
#>        Subject
#> Time    boy14 boy15 boy16 boy17 boy18 boy19 boy20 boy21 boy22 boy23 boy24 boy25
#>   0.025    15     6    11    24    16    16     7    21    11    12     8    11
#>   0.075    20    11    19    32    20    20    13    24    14    17    12    20
#>   0.125    22    20    30    35    21    22    14    25    14    20    14    22
#>   0.175    22    18    28    33    20    21    17    21    11    20    13    21
#>   0.225    21    13    25    29    18    20    14    16     8    18    12    21
#>   0.275    19     9    17    24    14    20     8     9     5    12     9    17
#>        Subject
#> Time    boy26 boy27 boy28 boy29 boy30 boy31 boy32 boy33 boy34 boy35 boy36 boy37
#>   0.025    16    19    13    11    17    20    18     9     8     9    13    19
#>   0.075    20    26    23    15    25    20    18    12    10    18    18    23
#>   0.125    22    28    30    19    30    22    25    16    17    19    27    26
#>   0.175    21    28    28    20    30    16    23    15    16    19    26    25
#>   0.225    20    24    19    18    27    10    18    14    12    19    25    21
#>   0.275    20    18    10    17    22    10    19    11    10    15    18    18
#>        Subject
#> Time    boy38 boy39
#>   0.025    16    14
#>   0.075    23    25
#>   0.125    28    32
#>   0.175    28    34
#>   0.225    25    30
#>   0.275    21    20

With the examples having a nifty data visualisation:

plot(
  gait[, 1, ], 
  type = "b",
  xlim = range(gait[,,1]), 
  ylim = range(gait[,,2]), 
  xlab = "Hip Angle", 
  ylab = "Knee Angle", 
  main = "'gait' data : Boy 1"
)
mtext(
  "all other boys", 
  col = "thistle"
  )
grid()
matlines(
  gait[, -1, 1], 
  gait[, -1, 2], 
  type = "l", 
  lty = 1, 
  col = adjustcolor("thistle", 1/3)
  )

It is worth noting the references for this data:

Olshen, R. A., Biden, E. N., Wyatt, M. P., and Sutherland, D. (1989) Gait Analysis and the Bootstrap. Annals of Statistics 17, 4, 1419–1440. doi:10.1214/AOS/1176347372
Ramsay, J. O., and Silverman, B. W. (2006) Functional Data Analysis, 2nd ed., New York: Springer.
Ramsay, J. (2023) fda: Functional Data Analysis. R package version 6.1.4, https://CRAN.R-project.org/package=fda.

BLAS and LAPACK update

The bundled BLAS and LAPACK sources have been updated to those shipped as part of January 2025’s LAPACK 3.12.1.

These help power many linear algebra operations in R, and there will likely be speed improvements and other efficiencies. So, worth installing for that alone.

A note on reading NEWS files

I have tried to read the R NEWS files since about 2017 or 2018. I believe I heard Roger Peng say he read the R NEWS files on the Not So Standard Deviations podcast with Hilary Parker. It seemed like a good idea, so I have tried to do the same.

It usually involves me reading over the NEWS file and looking out for things that seem relevant to me and my work. I don’t always understand everything, in fact, I would say I don’t understand a lot of it. But every now and again something comes up that is important to me. For example, the NEWS item for R 4.0.0 about matrices and arrays:

matrix objects now also inherit from class “array”, so e.g., class(diag(1)) is c(“matrix”, “array”). This invalidates code incorrectly assuming that class(matrix_obj)) has length one.

S3 methods for class “array” are now dispatched for matrix objects.

Resulted in a breaking change in greta, that I was able to identify and fix.

Another time that I find it useful to know how to look up the NEWS file is remembering when some key new features were introduced. For example, knowing that the native pipe |> was introduced in 4.1.0, but the underscore _ placeholder was only introduced in 4.2.0 is really useful if I’m developing packages to know which version of R to depend on.

I also often visit the “Previous releases of R for Windows” page to see when a version of R was released. This is to check if a version of R is more than three years old, which to me, is about old enough that people shouldn’t be depending on it on a regular basis, or if they are and it is mission critical, they would also have other things set up to maintain it. The tidyverse maintains backward compatibility with the past 5 (minor) versions of R - which ends up being about 4 years. I figure 3 years should be good enough for my work.

Yes, it is annoying to reinstall R packages again.

When you install a new minor version of R (that is, when you change its minor version, for example going from 4.1.0 to 4.2.0, not 4.1.0 to 4.1.1), you need to install all your R packages again.

Sure, it’s annoying. But I see it as an opportunity to start afresh. I usually do something like the following

install.packages("pak")
library(pak)
pkg_install("tidyverse", dependencies = TRUE)
pkg_install("devtools", dependencies = TRUE)
pkg_install("naniar", dependencies = TRUE)
pkg_install("blogdown", dependencies = TRUE)
pkg_install("greta", dependencies = TRUE)
pkg_install("brolgar", dependencies = TRUE)
pkg_install("targets", dependencies = TRUE)
pkg_install("ropensci/geotargets", dependencies = TRUE)
pkg_install("milesmcbain/datapasta", dependencies = TRUE)

And this gets me 90% of the way there.

It is worth noting that if you’re a windows user, you can use the installr package to update R and other dependencies.

And on that note it’s ALSO worth mentioning the rig command line utility. This allows you to more easily install different, and release versions of R. If you’ve seen Bob Rudis’s RSwitch software, this is the updated version of this (Bob has archived RSwitch).

{geotargets} 0.2.0

Wed, 27 Nov 2024 00:00:00 +0000

I’m super stoked to announce {geotargets} version 0.2.0! The {geotargets} package extends {targets} to work with geospatial data formats.

I’d like to firstly acknowledge the strong work by Eric Scott on getting this release ready. I do want to emphasise that while post is on my website, this project is very much a team effort.

You can download {geotargets} from the R universe like so:

install.packages("geotargets", repos = c("https://njtierney.r-universe.dev", "https://cran.r-project.org"))

Why should I use geotargets and targets?

You could benefit from using targets and geotargets if you do geospatial data analysis involving rasters or shapefiles, specifically with terra or stars R packages. For example, if you are doing large downloads of rasters, then operations like cropping, reprojection, resampling, and masking.

The main benefit to using targets and geotargets is your analysis will only rerun when you change relevant parts of your data analysis. For example, you might do a lot of geospatial data processing that feeds downstream into a machine learning model to make predictions on bushfire risk. Writing with targets and geotargets means if you change the parts of the code that related to the machine learning components, then only the relevant parts with machine learning would change. This means you can save time by avoiding running computational expensive spatial data processing.

For more details on what targets is, and why we need geotargets, I would recommend reading the 0.1.0 release blog post, as well as reading the {targets} manual. The “Get started in 4 minutes” guide to targets is also excellent.

Main changes in 0.2.0?

In addition to smaller changes and improvements, there are three main additions in this release:

Support for dynamic branching.
Preserving spatRaster metadata.
Support of stars and stars_proxy.

And very in very exciting news, we have a new hex sticker!

Thanks to Hubert Hałun for their work on getting this together, we are really happy with the new sticker!

geotargets hex sticker

Dynamic Branching

The main addition in this release is a demonstration of using dynamic branching using a new “target factory” function, tar_terra_tiles(). This allows you to break raster operations into tiles, and then perform these operations on the tiles and combine them together. This means we can break computationally intensive raster operations that work in pixel-wise manner over tiled subsets of the raster. This is useful when, for example, loading an entire raster into memory and doing computations on it results in out of memory errors.

As part of this addition, we created helper functions:

These help us define different extents that we can pass along as different parts of the dynamic branches. You can think of these as tools that we can use to specify how to slice, or tile up, a raster into smaller pieces that we can then do analysis on separately and combine later.

Let’s briefly unpack these, and then show how these would be used in dynamic branching. First let’s read in some example elevation data from terra and plot it:

f <- system.file("ex/elev.tif", package="terra")
r <- rast(f)
plot(r)

`tile_n()`

We can use tile_n(), which is the simplest of the three. It produces about n tiles in a grid.

r_tile_4 <- tile_n(r, 4)
#> creating 2 * 2 = 4 tile extents
r_tile_4
#> [[1]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.141667 49.816667 50.191667 
#> 
#> [[2]]
#>      xmin      xmax      ymin      ymax 
#>  6.141667  6.533333 49.816667 50.191667 
#> 
#> [[3]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.141667 49.441667 49.816667 
#> 
#> [[4]]
#>      xmin      xmax      ymin      ymax 
#>  6.141667  6.533333 49.441667 49.816667

# some plot helpers
rect_extent <- function(x, ...) {
  rect(x[1], x[3], x[2], x[4], ...)
}
plot_extents <- function(x, ...) {
  invisible(lapply(x, rect_extent, border = "hotpink", lwd = 2))
}

plot(r)
plot_extents(r_tile_4)

plot(r)
tile_n(r, 6) |> plot_extents()
#> creating 2 * 3 = 6 tile extents

`tile_grid()`

For more control, use tile_grid(), which allows specification of the number of rows and columns to split the raster into. Here we are specify that we want three columns and 1 row:

r_grid_3x1 <- tile_grid(r, ncol = 3, nrow = 1)
r_grid_3x1
#> [[1]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.008333 49.441667 50.191667 
#> 
#> [[2]]
#>      xmin      xmax      ymin      ymax 
#>  6.008333  6.266667 49.441667 50.191667 
#> 
#> [[3]]
#>      xmin      xmax      ymin      ymax 
#>  6.266667  6.533333 49.441667 50.191667
plot(r)
plot_extents(r_grid_3x1)


plot(r)
tile_grid(r, ncol = 2, nrow = 3) |> plot_extents()

`tile_blocksize()`

The third included helper is tile_blocksize(), which tiles by file block size. The block size is a property of raster files, and is the number of pixels (in the x and y direction) that is read into memory at a time. Tiling by multiples of block size may therefore be more efficient because only one block should need to be loaded to create each tile target. You can find the blocksize with fileBlocksize:

fileBlocksize(r)
#>      rows cols
#> [1,]   43   95

This tells us that it reads in the raster in 43x95 pixel sizes.

The tile_blocksize function is similar to tile_grid, except instead of saying how many rows and columns, we specify in units of blocksize.

If we just run tile_blocksize() on r we get the extents of the specified blocksize:

tile_blocksize(r)
#> [[1]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.533333 49.833333 50.191667 
#> 
#> [[2]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.533333 49.475000 49.833333 
#> 
#> [[3]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.533333 49.441667 49.475000

Which is the same as specifying blocksize for row and column at unit 1:

r_block_size_1x1 <- tile_blocksize(r, n_blocks_row = 1, n_blocks_col = 1)
r_block_size_1x1
#> [[1]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.533333 49.833333 50.191667 
#> 
#> [[2]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.533333 49.475000 49.833333 
#> 
#> [[3]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.533333 49.441667 49.475000
plot(r)
plot_extents(r_block_size_1x1)

Here the block size is the same size for the first two blocks, and then a much more narrow block. This is different to the two other tile methods.

Here the column block size is the full width of the raster.

So we could instead have the blocksize extent be written out to 2 blocks in a row, and 1 block size for the columns:

r_block_size_2x1 <- tile_blocksize(r, n_blocks_row = 2, n_blocks_col = 1)
r_block_size_2x1
#> [[1]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.533333 49.475000 50.191667 
#> 
#> [[2]]
#>      xmin      xmax      ymin      ymax 
#>  5.741667  6.533333 49.441667 49.475000
plot(r)
plot_extents(r_block_size_2x1)

This only works when the SpatRaster points to a file—in-memory rasters have no inherent block size.

sources(r)
#> [1] "/Users/nick/Library/R/arm64/4.4/library/terra/ex/elev.tif"
#force into memory
r2 <- r + 0
sources(r2)
#> [1] ""
#this now errors
tile_blocksize(r2)
#> Error: [aggregate] values in argument 'fact' should be > 0

For more detail on using this in targets, please see the geotargets vignette, “Dynamic branching with raster tiles”

Preserving spatRaster metadata

tar_terra_rast() gains a preserve_metadata option that when set to "zip" reads/writes targets as zip archives that include aux.json “sidecar” files sometimes written by terra (#58).

Support of `stars` and `stars_proxy`

We have created tar_stars() and tar_stars_proxy() that create stars and stars_proxy objects, respectively. These are currently experimental.

Minor changes in 0.2.0

Other changes include:

Created utility function set_window() mostly for internal use within tar_terra_tiles().
Removes the iteration argument from all tar_*() functions. iteration now hard-coded as "list" since it is the only option that works (for now at least).
Added the description argument to all tar_*() functions which is passed to tar_target().
Suppressed the warning “[rast] skipped sub-datasets” from tar_terra_sprc(), which is misleading in this context (#92, #104).
Requires GDAL 3.1 or greater to use “ESRI Shapefile” driver in tar_terra_vect() (#71, #97)
geotargets now requires targets version 1.8.0 or higher
terra (>= 1.7.71), withr (>= 3.0.0), and zip are now required dependencies of geotargets (moved from Suggests to Imports)

What’s next?

We have finished developing the main milestones for geotargets, but will continue actively developing it. Soon we will be submitting the package for review by rOpenSci, and subsequently submit the work to the Journal of Open Source Software (JOSS), and then submit to CRAN.

Currently, the next release will focus on adding support for:

You can see the full list of issues for more detail on what we are working on.

Thanks

We would like to thank the R Consortium for generously supporting this project, “{geotargets}: Enabling geospatial workflow management with {targets}".

We would also like to thank Michael Sumner, Anthony North, and Miles McBain for their helpful discussions throughout, as well as Will Landau for writing targets, and being incredibly responsive and helpful to the issues and questions we have asked as we wrote {geotargets}.

{geotargets} 0.1.0

Mon, 27 May 2024 00:00:00 +0000

I’m very happy to announce {geotargets} version 0.1.0! The {geotargets} package extends {targets} to work with geospatial data formats. Version 0.1.0 supports terra::vect(), terra::rast() and terra::sprc() formats. This R package is only possible due to the great work by Eric Scott and Andrew Brown. While this blog post is on my website, I want to emphasise that this project is very much a team effort.

You can download {geotargets} from the R universe like so:

install.packages("geotargets", repos = c("https://njtierney.r-universe.dev", "https://cran.r-project.org"))

What is targets? Why do I need geotargets?

The targets package is an R package for managing analytic pipelines. It means that you can write out an analysis in a specific manner, and then as you update code, it will only rerun the necessary parts. Essentially it helps you avoid running large pieces of analysis when you don’t need to. To learn more about targets, I’d highly recommend reading the {targets} manual.

Let’s show an example. Let’s say we want to get an example raster file from {terra}, we can do the following:

terra_rast_example <- system.file(
  "ex/elev.tif", 
  package = "terra"
  ) |>
  terra::rast()

terra_rast_example
#> class       : SpatRaster 
#> dimensions  : 90, 95, 1  (nrow, ncol, nlyr)
#> resolution  : 0.008333333, 0.008333333  (x, y)
#> extent      : 5.741667, 6.533333, 49.44167, 50.19167  (xmin, xmax, ymin, ymax)
#> coord. ref. : lon/lat WGS 84 (EPSG:4326) 
#> source      : elev.tif 
#> name        : elevation 
#> min value   :       141 
#> max value   :       547

Here is the equivalent code in a targets pipeline - the reason we want to use {targets} here is we save the results so we don’t need to run them again. In this case the example code doesn’t take long to run. But imagine reading in the raster was hugely time and computer expensive and we didn’t want to do it again. The {targets} package stores the information so we can just read it back in later, and if we try and run the code again it will not update the code unless the data input has changed. Neat, right?

library(targets)
tar_dir({ # tar_dir() runs code from a temporary directory.
  tar_script({
    library(targets)
    list(
      tar_target(
        terra_rast_example,
        system.file("ex/elev.tif", package = "terra") |> terra::rast()
      )
    )
  })
  tar_make()
  x <- tar_read(terra_rast_example)
  x
})
#> ▶ dispatched target terra_rast_example
#> ● completed target terra_rast_example [1.196 seconds]
#> ▶ ended pipeline [1.825 seconds]
#> 
#> class       : SpatRaster
#> Error: external pointer is not valid

We get an error!

Error: external pointer is not valid

This is a relatively common gotcha moment when using libraries like {terra}. This is due to limitations with its underlying C++ implementation. There are specific ways to write and read these objects. See ?terra for details.

But how do we use {geotargets} to help with this? It helps handle these write and read steps, so you don’t have to worry about them and can use targets as you are used to.

So instead of tar_target(), you use tar_terra_rast() to save a {terra} raster:

library(targets)
tar_dir({ # tar_dir() runs code from a temporary directory.
  tar_script({
    library(targets)
    library(geotargets)
    list(
      tar_terra_rast(
        terra_rast_example,
        system.file("ex/elev.tif", package = "terra") |> terra::rast()
      )
    )
  })
  tar_make()
  x <- tar_read(terra_rast_example)
  x
})
#> ▶ dispatched target terra_rast_example
#> ● completed target terra_rast_example [0.006 seconds]
#> ▶ ended pipeline [0.061 seconds]
#> 
#> class       : SpatRaster 
#> dimensions  : 90, 95, 1  (nrow, ncol, nlyr)
#> resolution  : 0.008333333, 0.008333333  (x, y)
#> extent      : 5.741667, 6.533333, 49.44167, 50.19167  (xmin, xmax, ymin, ymax)
#> coord. ref. : lon/lat WGS 84 (EPSG:4326) 
#> source      : terra_rast_example 
#> name        : elevation 
#> min value   :       141 
#> max value   :       547

Similarly, there are tar_terra_vect() and tar_terra_sprc() for dealing with vector (shapefile) and sprc (collections of rasters). See the README example for more information.

If you’d like to see these functions being used in a more practical context, see the demo-geotargets repository.

What’s next?

We are actively developing {geotargets}, and the next release will focus on adding support for splitting rasters into tiles, preserving SpatRaster metadata, and adding support for {stars}. You can see the full list of issues for more detail on what we are working on.

Thanks

We have recently generously received support from the R Consortium for our project, "{geotargets}: Enabling geospatial workflow management with {targets}", and so we would like to thank them for their support.

I’d also like to thank Michael Sumner, Anthony North, and Miles McBain for their helpful discussions, as well as Will Landau for writing targets, and being incredibly responsive and helpful to the issues and questions we have asked as we wrote {geotargets}.

Find Out How many Times Faster your Code is

Fri, 17 May 2024 00:00:00 +0000

My venetian blinds, in black and white

I recently watched Josiah Parry’s wonderful video, “Making R 300x times faster!" It’s a great demonstration of how to rewrite code to be faster, and it’s worth your time. He rewrites some R code to be faster, then improves the speed again by writing some Rust code, which is called from R. He gets a 300 times speedup, which is really awesome.

Then someone writes in with an example of some code that is even faster than that, just using R code. It ends up being about 6 times faster than his Rust code. So a (300x6) 2000 times speed up. The main thing that helped with that was ensuring to vectorise your R code. Essentially, not working on the rows, but instead working on the columns.

Throughout the video Josiah makes good use of the bench R package to evaluate how much faster your code is. This idea is called “microbenchmarking”, and it involves running your code many times to evaluate how much faster it is than some other option. The reason you want to run your code many times is there is often variation around the runtimes in your code, so you don’t just want to base your improvements around a single measurement. It’s a general standard approach to attempt tp truly compare your approach to another.

All this being said, you should be wary of trying to make your code fast first without good reason. You want to make sure your code does the right thing first. Don’t just start trying to write performant code. Or as Donald Knuth says:

“The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.”.

If you want to learn more about how to speed up your code, I think it’s worthwhile reading up on the measuring performance chapter in Advanced R.

An example microbenchmark

Let’s take an example from the naniar package. I’ll give more detail of this story of this optimisation at the end of this section. For the moment, let’s say we want to get the number of missing values in a row of a data frame. We can do something like this:

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
my_n_miss <- function(data){
  data |> 
  rowwise() |> 
  mutate(
    n_miss = sum(is.na(c_across(everything())))
  ) |> 
    ungroup()
}

my_n_miss(airquality)
#> # A tibble: 153 × 7
#>    Ozone Solar.R  Wind  Temp Month   Day n_miss
#>    <int>   <int> <dbl> <int> <int> <int>  <int>
#>  1    41     190   7.4    67     5     1      0
#>  2    36     118   8      72     5     2      0
#>  3    12     149  12.6    74     5     3      0
#>  4    18     313  11.5    62     5     4      0
#>  5    NA      NA  14.3    56     5     5      2
#>  6    28      NA  14.9    66     5     6      1
#>  7    23     299   8.6    65     5     7      0
#>  8    19      99  13.8    59     5     8      0
#>  9     8      19  20.1    61     5     9      0
#> 10    NA     194   8.6    69     5    10      1
#> # ℹ 143 more rows

But we can speed this up using rowSums() instead:

new_n_miss <- function(data){
  n_misses <- rowSums(is.na(data))
  data |> 
  mutate(
    n_miss = n_misses
  ) |> 
    as_tibble()
}

new_n_miss(airquality)
#> # A tibble: 153 × 7
#>    Ozone Solar.R  Wind  Temp Month   Day n_miss
#>    <int>   <int> <dbl> <int> <int> <int>  <dbl>
#>  1    41     190   7.4    67     5     1      0
#>  2    36     118   8      72     5     2      0
#>  3    12     149  12.6    74     5     3      0
#>  4    18     313  11.5    62     5     4      0
#>  5    NA      NA  14.3    56     5     5      2
#>  6    28      NA  14.9    66     5     6      1
#>  7    23     299   8.6    65     5     7      0
#>  8    19      99  13.8    59     5     8      0
#>  9     8      19  20.1    61     5     9      0
#> 10    NA     194   8.6    69     5    10      1
#> # ℹ 143 more rows
my_n_miss(airquality)
#> # A tibble: 153 × 7
#>    Ozone Solar.R  Wind  Temp Month   Day n_miss
#>    <int>   <int> <dbl> <int> <int> <int>  <int>
#>  1    41     190   7.4    67     5     1      0
#>  2    36     118   8      72     5     2      0
#>  3    12     149  12.6    74     5     3      0
#>  4    18     313  11.5    62     5     4      0
#>  5    NA      NA  14.3    56     5     5      2
#>  6    28      NA  14.9    66     5     6      1
#>  7    23     299   8.6    65     5     7      0
#>  8    19      99  13.8    59     5     8      0
#>  9     8      19  20.1    61     5     9      0
#> 10    NA     194   8.6    69     5    10      1
#> # ℹ 143 more rows

We can measure the speed using bench::mark():

library(bench)

bm <- mark(
  old = my_n_miss(airquality),
  new = new_n_miss(airquality)
)

bm
#> # A tibble: 2 × 6
#>   expression      min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 old          23.9ms   23.9ms      41.8   355.5KB    334. 
#> 2 new         347.5µs  384.5µs    2385.     81.8KB     34.8

This runs the code at least twice, and prints out the amount of time it takes to run the code provided on the right hand side of “old” and “new”. But you can name them whatever you want.

Now, it can be kind of hard to see just how much faster this is, if you just look at comparing the times, as the times are given here in…well, actually I’m not sure why our friend from the greek alphabet mu, µ, from the greek alphabet is here, actually? If, like me, you needed to double check the standard measures of order of magnitude wiki page, you might not know that “ms” means milli - or one thousandth, and µ means “micro”, or one millionth. The point is that the new one is many times faster than the old one.

We can do a plot to help see this:

plot(bm)
#> Loading required namespace: tidyr

So we can see that the new one really is a lot faster.

But if I just want to be able to say something like:

It is XX times faster

then we can use the (somewhat unknown?) relative = TRUE option of bench’s S3 method for summary method:

summary(bm, relative = TRUE)
#> # A tibble: 2 × 6
#>   expression   min median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <dbl>  <dbl>     <dbl>     <dbl>    <dbl>
#> 1 old         68.7   62.2       1        4.34     9.62
#> 2 new          1      1        57.0      1        1

And this is great, from this we can see it is about 60 times faster. And that the old approach uses 15 times more memory.

The story behind this speedup in naniar.

Now, I didn’t just come up with a speedup for missing values on the fly. The story here is that there was going to be some (very generous) improvements to the naniar package from Romain François in the form of C++ code. However, Jim Hester suggested some changes, I think twitter (which I can’t find anymore), and he then kindly submitted a pull request showing that rowSums in R ends up being plenty fast.

This is a similar story to Josiah’s, where he used Rust code to get it faster, but then there was a faster way just staying within R.

Sometimes, you don’t need extra C or Fortran or Rust. R is enough!

And if you want to be able to compare the speeds of things, don’t forget the relative = TRUE argument in summary when using bench::mark.

Other packages for microbenchmarking

bench isn’t the only way to measure things! Other ones I’ve enjoyed using in the past are microbenchmark and tictoc. I’ve particularly enjoyed tictoc because you get to do this:

library(tictoc)
tic()
new_n_miss(airquality)
#> # A tibble: 153 × 7
#>    Ozone Solar.R  Wind  Temp Month   Day n_miss
#>    <int>   <int> <dbl> <int> <int> <int>  <dbl>
#>  1    41     190   7.4    67     5     1      0
#>  2    36     118   8      72     5     2      0
#>  3    12     149  12.6    74     5     3      0
#>  4    18     313  11.5    62     5     4      0
#>  5    NA      NA  14.3    56     5     5      2
#>  6    28      NA  14.9    66     5     6      1
#>  7    23     299   8.6    65     5     7      0
#>  8    19      99  13.8    59     5     8      0
#>  9     8      19  20.1    61     5     9      0
#> 10    NA     194   8.6    69     5    10      1
#> # ℹ 143 more rows
toc()
#> 0.01 sec elapsed

Which feels a bit nicer than using system.time():

system.time({
  new_n_miss(airquality)
})
#>    user  system elapsed 
#>   0.001   0.000   0.002

Also, notice that those two times are different? This is why we use benchmarking, to run those checks many times!

End

And that’s it, that’s the blog post. The relative = TRUE option in mark is super neat, and I don’t think many people know about it. Thanks again to Jim Hester for originally creating the bench package.

{naniar} version 1.1.0 "Prince Caspian"

Mon, 04 Mar 2024 00:00:00 +0000

I’m happy to announce that naniar version 1.1.0 “Prince Caspian” is released. It’s been about a year since the last releaase, so I’m happy to include some new features. Let’s explore some of the new changes in this release

options(tidyverse.quiet = TRUE)
library(tidyverse)
library(naniar)

Imputation functions

I’ve always been a hesitant to include imputation helper functions like impute_mean(), and friends in {naniar}, as they aren’t a great tool to use for imputation. I also really believe the {simputation} R package does a great job at doing much imputation. However, sometimes you do actually want to demonstrate how bad imputing the mean is. Or sometimes you do need to impute the mean. And I’d rather make it easier for people to do this.

This release included some new functions: impute_fixed(), impute_zero(), impute_factor(), and impute_mode(). Notably, these do not implement “scoped variants” which were previously implemented - for example, impute_fixed_if() etc. This is in favour of using the new across() workflow within {dplyr}, which is a bit easier to maintain. This resolves issues #261 and #213.

To demonstrate these imputation functions, let’s create a vector of missing values:

library(naniar)
vec_num <- rnorm(10)
vec_int <- rpois(10, 5)
vec_fct <- factor(LETTERS[1:10])

vec_num <- set_prop_miss(vec_num, 0.4)
vec_int <- set_prop_miss(vec_int, 0.4)
vec_fct <- set_prop_miss(vec_fct, 0.4)

vec_num
#>  [1]         NA  0.9232763         NA  0.5265162         NA         NA
#>  [7]  1.4260531  1.3793303 -0.7855565 -0.5928170
vec_int
#>  [1] NA  5  6  5  5 NA NA  6 NA  6
vec_fct
#>  [1] <NA> B    <NA> D    <NA> F    G    <NA> I    J   
#> Levels: A B C D E F G H I J

We can use impute_fixed() to impute fixed values into the numeric of these:

impute_fixed(vec_num, -999)
#>  [1] -999.0000000    0.9232763 -999.0000000    0.5265162 -999.0000000
#>  [6] -999.0000000    1.4260531    1.3793303   -0.7855565   -0.5928170
impute_fixed(vec_int, -999)
#>  [1] -999    5    6    5    5 -999 -999    6 -999    6

And impute_zero is just a special case of impute_fixed, where the fixed value is 0:

impute_zero(vec_num)
#>  [1]  0.0000000  0.9232763  0.0000000  0.5265162  0.0000000  0.0000000
#>  [7]  1.4260531  1.3793303 -0.7855565 -0.5928170
impute_zero(vec_int)
#>  [1] 0 5 6 5 5 0 0 6 0 6

Similar to impute_mean(), impute_mode() imputes the mode:

impute_mode(vec_num)
#>  [1]  1.0704569  0.9232763  1.0704569  0.5265162  1.0704569  1.0704569
#>  [7]  1.4260531  1.3793303 -0.7855565 -0.5928170
impute_mode(vec_int)
#>  [1] 6 5 6 5 5 6 6 6 6 6

You can’t however use impute_fixed() or impute_zero() on factors, and this doesn’t work for factors, even if it’s a new character.

impute_fixed(vec_fct, -999)
#> Warning in `[<-.factor`(`*tmp*`, is.na(x), value = -999): invalid factor level, NA generated
#>  [1] <NA> B    <NA> D    <NA> F    G    <NA> I    J   
#> Levels: A B C D E F G H I J
impute_zero(vec_fct)
#> Warning in `[<-.factor`(`*tmp*`, is.na(x), value = 0): invalid factor level, NA generated
#>  [1] <NA> B    <NA> D    <NA> F    G    <NA> I    J   
#> Levels: A B C D E F G H I J
impute_fixed(vec_fct, "ZZ")
#> Warning in `[<-.factor`(`*tmp*`, is.na(x), value = "ZZ"): invalid factor level, NA generated
#>  [1] <NA> B    <NA> D    <NA> F    G    <NA> I    J   
#> Levels: A B C D E F G H I J

However, you can use impute_mode():

impute_mode(vec_fct)
#>  [1] I B I D I F G I I J
#> Levels: A B C D E F G H I J

For factors, you can impute a specific value:

impute_factor(vec_fct, value = "ZZ")
#>  [1] ZZ B  ZZ D  ZZ F  G  ZZ I  J 
#> Levels: A B C D E F G H I J ZZ

Think of it like impute_fixed(), but a special case for factors.

Now let’s demonstrate how to do this in a data frame. First we create the data

library(dplyr)

dat <- tibble(
  num = vec_num,
  int = vec_int,
  fct = vec_fct
)

dat
#> # A tibble: 10 × 3
#>       num   int fct  
#>     <dbl> <int> <fct>
#>  1 NA        NA NA   
#>  2  0.923     5 B    
#>  3 NA         6 NA   
#>  4  0.527     5 D    
#>  5 NA         5 NA   
#>  6 NA        NA F    
#>  7  1.43     NA G    
#>  8  1.38      6 NA   
#>  9 -0.786    NA I    
#> 10 -0.593     6 J

You can use it inside mutate like so:

dat %>%
  mutate(
    num = impute_fixed(num, -9999),
    int = impute_zero(int),
    fct = impute_factor(fct, "out")
  )
#> # A tibble: 10 × 3
#>          num   int fct  
#>        <dbl> <dbl> <fct>
#>  1 -9999         0 out  
#>  2     0.923     5 B    
#>  3 -9999         6 out  
#>  4     0.527     5 D    
#>  5 -9999         5 out  
#>  6 -9999         0 F    
#>  7     1.43      0 G    
#>  8     1.38      6 out  
#>  9    -0.786     0 I    
#> 10    -0.593     6 J

Or if you want to impute across all applicable variables with a single function, you could use where like so, to focus on numeric data:

dat %>%
  mutate(
    across(
      .cols = where(is.numeric),
      .fn = impute_zero
    )
  )
#> # A tibble: 10 × 3
#>       num   int fct  
#>     <dbl> <dbl> <fct>
#>  1  0         0 NA   
#>  2  0.923     5 B    
#>  3  0         6 NA   
#>  4  0.527     5 D    
#>  5  0         5 NA   
#>  6  0         0 F    
#>  7  1.43      0 G    
#>  8  1.38      6 NA   
#>  9 -0.786     0 I    
#> 10 -0.593     6 J

dat %>%
  mutate(
    across(
      .cols = where(is.numeric),
      .fn = \(x) impute_fixed(x, -99)
    )
  )
#> # A tibble: 10 × 3
#>        num   int fct  
#>      <dbl> <dbl> <fct>
#>  1 -99       -99 NA   
#>  2   0.923     5 B    
#>  3 -99         6 NA   
#>  4   0.527     5 D    
#>  5 -99         5 NA   
#>  6 -99       -99 F    
#>  7   1.43    -99 G    
#>  8   1.38      6 NA   
#>  9  -0.786   -99 I    
#> 10  -0.593     6 J

Improvements

Add digit argument to miss_var_summary to help display percentage missing data correctly when there is a very small fraction of missingness - solving #284.

N <- 30000000

df <- tibble(x = rep(NA_real_, N)) %>%
  add_row(x = 0)

df %>% miss_var_summary()
#> # A tibble: 1 × 3
#>   variable   n_miss pct_miss
#>   <chr>       <int>    <num>
#> 1 x        30000000     100.
df %>% miss_var_summary(digits = 6)
#> # A tibble: 1 × 3
#>   variable   n_miss  pct_miss
#>   <chr>       <int> <num:.6!>
#> 1 x        30000000 99.999997

geom_miss_point() works with shape argument #290

ggplot(
  airquality,
  aes(
    x = Ozone,
    y = Solar.R,
    shape = factor(Month)
  )
) +
  geom_miss_point(size = 4)

Implemented Date, POSIXct and POSIXlt methods for impute_below() - #158

dat_date <- tibble(
  values = 1:7,
  number = c(111, 112, NA, NA, 108, 150, 160),
  posixct = as.POSIXct(number, origin = "1970-01-01"),
  posixlt = as.POSIXlt(number, origin = "1970-01-01"),
  date = as.Date(number)
  )
dat_date
#> # A tibble: 7 × 5
#>   values number posixct             posixlt             date      
#>    <int>  <dbl> <dttm>              <dttm>              <date>    
#> 1      1    111 1970-01-01 10:01:51 1970-01-01 10:01:51 1970-04-22
#> 2      2    112 1970-01-01 10:01:52 1970-01-01 10:01:52 1970-04-23
#> 3      3     NA NA                  NA                  NA        
#> 4      4     NA NA                  NA                  NA        
#> 5      5    108 1970-01-01 10:01:48 1970-01-01 10:01:48 1970-04-19
#> 6      6    150 1970-01-01 10:02:30 1970-01-01 10:02:30 1970-05-31
#> 7      7    160 1970-01-01 10:02:40 1970-01-01 10:02:40 1970-06-10

date_date_imp <- dat_date %>% 
  mutate(
    number_imp = impute_below(number),
    .after = number
  ) %>% 
  mutate(
    posixct_imp = impute_below(posixct),
    .after = posixct
    ) %>% 
  mutate(
    posixlt_imp = impute_below(posixlt),
    .after = posixlt
  ) %>% 
  mutate(
    date_imp = impute_below(date),
    .after = date
  )

date_date_imp
#> # A tibble: 7 × 9
#>   values number number_imp posixct             posixct_imp        
#>    <int>  <dbl>      <dbl> <dttm>              <dttm>             
#> 1      1    111       111  1970-01-01 10:01:51 1970-01-01 10:01:51
#> 2      2    112       112  1970-01-01 10:01:52 1970-01-01 10:01:52
#> 3      3     NA       103. NA                  1970-01-01 10:01:43
#> 4      4     NA       102. NA                  1970-01-01 10:01:41
#> 5      5    108       108  1970-01-01 10:01:48 1970-01-01 10:01:48
#> 6      6    150       150  1970-01-01 10:02:30 1970-01-01 10:02:30
#> 7      7    160       160  1970-01-01 10:02:40 1970-01-01 10:02:40
#> # ℹ 4 more variables: posixlt <dttm>, posixlt_imp <dttm>, date <date>,
#> #   date_imp <date>

Bug fixes

Fix bug with all_complete(), which was implemented as !anyNA(x) but should be all(complete.cases(x)).
Correctly implement any_na() (and any_miss()) and any_complete(). Rework examples to demonstrate workflow for finding complete variables.
Fix bug with shadow_long() not working when gathering variables of mixed type. Fix involved specifying a value transform, which defaults to character. #314
Provide replace_na_with(), a complement to replace_with_na() - #129

x <- c(1:5, NA, NA, NA)
x
#> [1]  1  2  3  4  5 NA NA NA
replace_na_with(x, 0)
#> [1] 1 2 3 4 5 0 0 0

dat <- tibble(
  ones = c(1,1,NA),
  twos = c(NA,NA, 2),
  threes = c(NA, NA, NA)
)

dat %>% 
  mutate(
    ones = replace_na_with(ones, 0),
    twos = replace_na_with(twos, -2),
    threes = replace_na_with(threes, -3)
  )
#> # A tibble: 3 × 3
#>    ones  twos threes
#>   <dbl> <dbl>  <dbl>
#> 1     1    -2     -3
#> 2     1    -2     -3
#> 3     0     2     -3

dat %>% 
  mutate(
    across(
      everything(),
      \(x) replace_na_with(x, -99)
    )
  )
#> # A tibble: 3 × 3
#>    ones  twos threes
#>   <dbl> <dbl>  <dbl>
#> 1     1   -99    -99
#> 2     1   -99    -99
#> 3   -99     2    -99

Fix bug with gg_miss_fct() where it used a deprecated function from forcats - #342

Internal changes

{naniar} now uses cli::cli_abort() and cli::cli_warn() instead of stop() and warning() (#326). Internally in tests we changed to use expect_snapshot() instead of expect_error().

Deprecations

The following functions have been soft deprecated, and will eventually be made defunct in future versions of naniar. shadow_shift() - #193 in favour of impute_below(), and miss_case_cumsum() and miss_var_cumsum() - #257 - in favour of miss_case_summary(data, cumsum = TRUE) and miss_var_summary(data, cumsum = TRUE).

Thanks!

Thanks to everyone for using {naniar}, I’m happy that I’ve got another release out the door and am looking forward to more changes in the future. Especially thanks to everyone who contributed to issues and or pull request for this release, including: @szimmer, @HughParsonage, , @siavash-babaei, @maksymiuks, @jonocarroll, @jzadra, @krlmlr

Posts on Credibly Curious

R Version 4.5.0 is Out!

Installation is now faster

Data

BLAS and LAPACK update

A note on reading NEWS files

Yes, it is annoying to reinstall R packages again.

{geotargets} 0.2.0

Why should I use geotargets and targets?

Main changes in 0.2.0?

Dynamic Branching

tile_n()

tile_grid()

tile_blocksize()

Preserving spatRaster metadata

Support of stars and stars_proxy

Minor changes in 0.2.0

What’s next?

Thanks

{geotargets} 0.1.0

What is targets? Why do I need geotargets?

What’s next?

Thanks

Find Out How many Times Faster your Code is

An example microbenchmark

The story behind this speedup in naniar.

Other packages for microbenchmarking

End

{naniar} version 1.1.0 "Prince Caspian"

Imputation functions

Improvements

Bug fixes

Internal changes

Deprecations

Thanks!

`tile_n()`

`tile_grid()`

`tile_blocksize()`

Support of `stars` and `stars_proxy`