Edits

content/blog/duckplyr-1-0-0/index.Rmd

@@ -6,16 +6,16 @@ title: duckplyr fully joins the tidyverse!
 date: 2025-02-11
 author: Kirill Müller and Maëlle Salmon
 description: >
-    duckplyr 1.0.0 is on CRAN and part of the tidyverse! duckplyr is a drop-in
-    replacement for dplyr, powered by DuckDB for speed. It is the most dplyr-like
-    of dplyr backends.
+    duckplyr 1.0.0 is on CRAN and part of the tidyverse!
+    A drop-in replacement for dplyr, powered by DuckDB for speed.
+    It is the most dplyr-like of dplyr backends.
 
 photo:
   url: https://www.pexels.com/photo/a-mallard-duck-on-water-6918877/
   author: Kiril Gruev
 
 # one of: "deep-dive", "learn", "package", "programming", "roundup", or "other"
-categories: [package] 
+categories: [package]
 tags:
   - duckplyr
   - dplyr
@@ -35,105 +35,112 @@ TODO:
 * [x] `usethis::use_tidy_thanks()`
 -->
 
-We're very chuffed to announce the release of [duckplyr](https://duckplyr.tidyverse.org) 1.0.0. 
+We're very chuffed to announce the release of [duckplyr](https://duckplyr.tidyverse.org) 1.0.0.
 duckplyr is a drop-in, fully compatible replacement for dplyr, powered by [DuckDB](https://duckdb.org/) for speed.
 It joins the rank of dplyr backends together with [dtplyr](https://dtplyr.tidyverse.org) and [dbplyr](https://dbplyr.tidyverse.org).
 You can use it instead of dplyr for data small or large.
 
+<!-- FIXME:
+
+We have many more dplyr backends, the two above are just from the tidyverse.
+GitHub search: https://github.com/search?q=org%3Acran+%2FS3method%5B%28%5D%28mutate%7Csummarise%29+*%2C%2F&type=code
+Do we need an "awesome dplyr" like https://github.com/krlmlr/awesome-vctrs/?
+
+-->
+
 You can install it from CRAN with:
 
 ```{r, eval = FALSE}
 install.packages("duckplyr")
 ```
 
-In this article, we'll introduce you to the basic concepts behind duckplyr, show how it can help you handle normal sized but also large data, and explain how you can help improve the package.
+In this article, we'll introduce you to the basic concepts behind duckplyr, show how it can help you data of different sizes, and explain how you can help improve the package.
 
 ## A drop-in replacement for dplyr
 
 The duckplyr package is a _drop-in replacement for dplyr_ that uses _DuckDB for speed_.
 You can simply _drop_ duckplyr into your pipeline by loading it, then computations will be efficiently carried out by DuckDB.
-DuckDB is a [fast database system](https://www.youtube.com/watch?v=GELhdezYmP0&feature=youtu.be).
+DuckDB is a fast in-memory analytical database system.
+If you haven't heard about it, watch [Hannes Mühleisen's keynote at posit::conf(2024)](https://www.youtube.com/watch?v=GELhdezYmP0&feature=youtu.be).
 
 ```{r}
 library(conflicted)
 library(duckplyr)
 conflict_prefer("filter", "dplyr", quiet = TRUE)
-library("babynames")
-
+library(babynames)
 
 out <- babynames |>
-  filter(n > 1000) |>
+  mutate(prevalence = if_else(prop >= 0.01, "frequent", "rare")) |>
   summarize(
-    .by = c(sex, year),
+    .by = c(sex, year, prevalence),
     babies_n = sum(n)
   ) |>
   filter(sex == "F")
 class(out)
-
+out
 ```
 
 Like with other dplyr backends like dtplyr and dbplyr, duckplyr allows you to get faster results.
 Unlike other dplyr backends, duckplyr does not require you to learn a different syntax.
-
 The duckplyr package is fully compatible with dplyr: if an operation cannot be carried out with DuckDB, it is automatically outsourced to dplyr.
-In that case, the operation is not slower than dplyr but not faster either.
-The duckplyr package is actively developed so that over time, we expect fewer and fewer fallbacks to dplyr to be needed.
+Over time, we expect fewer and fewer fallbacks to dplyr to be needed.
 
 ## How to use duckplyr
 
-To _replace_ dplyr with duckplyr, you can either
+To _replace_ dplyr with duckplyr, you can:
 
-- load duckplyr and then keep your pipeline as is. Calling `library(duckplyr)` overwrites dplyr methods, enabling duckplyr for the entire session no matter how data.frames are created.
+- Load duckplyr and then keep your pipeline as is. Calling `library(duckplyr)` overwrites dplyr methods, enabling duckplyr for the entire session no matter how data.frames are created.
+    This is shown in the example above.
 
-```{r}
-library(conflicted)
-library(duckplyr)
-conflict_prefer("filter", "dplyr", quiet = TRUE)
-```
+- Create individual "duck frames" using _conversion functions_ like `duckdb_tibble()` or `as_duckdb_tibble()`, or _ingestion functions_ like `read_csv_duckdb()`.
+    Then, the data manipulation pipeline uses the exact same syntax as a dplyr pipeline.
+    The duckplyr package performs the computation using DuckDB.
+
+    ```{r}
+    # Undo the effect of library(duckplyr)
+    methods_restore()
 
-- Create individual "duck frames" which allows you to control their automatic materialization parameters to [protect memory](https://duckplyr.tidyverse.org/articles/prudence.html). To do so, you can use _conversion functions_ like `duckdb_tibble()` or `as_duckdb_tibble()`, or _ingestion functions_ like `read_csv_duckdb()`.
+    out <- babynames |>
+      as_duckdb_tibble() |>
+      mutate(prevalence = if_else(prop >= 0.01, "frequent", "rare")) |>
+      summarize(
+        .by = c(sex, year, prevalence),
+        babies_n = sum(n)
+      ) |>
+      filter(sex == "F")
+    class(out)
+    ```
 
-Then, the data manipulation pipeline uses the exact same syntax as a dplyr pipeline.
-The duckplyr package performs the computation using DuckDB.
+In both cases, printing the result only shows the first few rows, as with dbplyr.
 
 ```{r}
-library("babynames")
-out <- babynames |>
-  filter(n > 1000) |>
-  summarize(
-    .by = c(sex, year),
-    babies_n = sum(n)
-  ) |>
-  filter(sex == "F")
+out
 ```
 
 The result can finally be materialized to memory, or computed temporarily, or computed to a file.
 
 ```{r}
 # to memory
-out
+collect(out)
 
 # to a file
 csv_file <- withr::local_tempfile()
-file.size(csv_file)
 compute_csv(out, csv_file)
-file.size(csv_file)
+fs::file_size(csv_file)
 ```
 
 When duckplyr itself does not support specific functionality, it falls back to dplyr.
-For instance, row names are not supported yet:
+For instance, pivoting is not supported yet, still it works thanks to the fallback mechanism.
 
 ```{r}
-mtcars |>
-  summarize(
-    .by = cyl,
-    disp = mean(disp, na.rm = TRUE),
-    sd = sd(disp, na.rm = TRUE)
-  )
+out |>
+  tidyr::pivot_wider(names_from = prevalence, values_from = babies_n, values_fill = 0L) |>
+  mutate(share_frequent = frequent / (frequent + rare))
 ```
 
-Current limitations are documented in a [vignette](https://duckplyr.tidyverse.org/articles/limits.html).
-You can change the verbosity of fallbacks, refer to [`duckplyr::fallback_sitrep()`](https://duckplyr.tidyverse.org/reference/fallback.html).
+For performance reasons, the output order of the result is not guaranteed to be stable.
+If you need a stable order, you can use `arrange()`.
+Other limitations are documented in [`vignette("limits")`](https://duckplyr.tidyverse.org/articles/limits.html).
 
 ### For large data
 
@@ -173,7 +180,7 @@ Our goals for future development of duckplyr include:
 - Enabling users to provide [custom translations](https://github.com/tidyverse/duckplyr/issues/158) of dplyr functionality;
 - Making it easier to contribute code to duckplyr.
 
-You can help! 
+You can help!
 
 - Please report any issue especially regarding unknown incompabilities. See [`vignette("limits")`](https://duckplyr.tidyverse.org/articles/limits.html).
 - Contribute to the codebase after reading duckplyr's [contributing guide](https://duckplyr.tidyverse.org/CONTRIBUTING.html).