03-data_transformation.Rmd

# Data transformation

**Learning objectives:**

- **Pick out rows** of a data frame with the **`dplyr::filter()`** function.
- **Sort rows** of a data frame with **`dplyr::arrange()`**.
- **Pick out columns** of a data frame with **`dplyr::select()`**.
- **Modify columns** of a data frame with **`dplyr::mutate()`**.
- **Group rows** of a data frame with **`dplyr::group()`**.
- **Apply functions to columns** of a (grouped) data frame with **`dplyr::summarize()`**.
- **Streamline data transformations** with the **pipe** operator (`|>`).

## Prerequisites {-}

-   dplyr 📦 = functions to manipulate data frames.
-   data frames consist of columns (variables) and rows (observations).
-   dplyr is part of the tidyverse. 

```{r 04-01, eval=FALSE}
install.packages("tidyverse")
```

```{r 04-02}
library(tidyverse)
```

Or just dplyr

```{r 04-03, eval=FALSE}
install.packages("dplyr")
```

```{r 04-04}
library(dplyr)
```

## nycflights13 {-}

`nycflights13` 📦 = New York City flight data from 2013.

```{r 04-05, eval=FALSE}
install.packages("nycflights13")
```

```{r 04-06}
library(nycflights13)
```

For new data: quickly browse to check if you want to use it.

View in console:

```{r 04-07}
flights
```

View in a spreadsheet-like viewer:

```{r 04-08, eval=FALSE}
View(flights)
```

`?flights` = help viewer to get info about all the variables.

Check size of the flights data:

```{r 04-10, eval=FALSE}
# number of rows
nrow(flights)
#> [1] 336776

# number of columns
ncol(flights)
#> [1] 19

# number of columns
length(flights)
#> [1] 19

# number of rows and columns
dim(flights)
#> [1] 336776     19
```

Column names:

```{r 04-11}
colnames(flights)
```

```{r 04-glimpse-flights}
glimpse(flights)
```

## dplyr basics: General structure {-}

When using `dplyr`:

-   1st arg is always a data frame.
-   Subsequent args typically = columns to operate on, as variable names (without quotes).
-   Output is always a new data frame.

## dplyr basics: Pipe {-}

-   `|>` = pipe symbol
-   Pronounced "and then"
-   Pipes data from one function into 1st arg of next

```{r 04-pipe-basics, eval=FALSE}
flights |>
  filter(dest == "IAH") |> 
  group_by(year, month, day) |> 
  summarize(
    arr_delay = mean(arr_delay, na.rm = TRUE)
  )
```

-   "Take the flights dataset, 
-   *and then* filter the flights that have their destination (dest) as 'IAH', 
-   *and then* group these filtered results by year, month, and day, 
-   *and then* for each group, calculate the average arrival delay (arr_delay), ignoring the missing values."

## Filter rows with filter() {-}

-   `filter()` = pick out certain rows (observations)
-   `filter()` picks rows which evaluate to TRUE for all criteria.
-   1st arg = data frame
-   Subsequent args are logical expressions

```{r 04-filter-flights}
# all flights that departed more than 120 minutes (two hours) late
flights |> 
  filter(dep_delay > 120)

# Flights that departed on January 1
flights |> 
  filter(month == 1 & day == 1)


# Flights that departed in January or February
flights |> 
  filter(month == 1 | month == 2)

# A shorter way to select flights that departed in January or February
flights |> 
  filter(month %in% c(1, 2))

# Save the result of the filtering to a variable
jan1 <- flights |> 
  filter(month == 1 & day == 1)
# "jan1 gets flights and then filter where month is 1 and day is 1."
```

### Comparisons {-}

-   `>` greater than
-   `>=` greater than or equal
-   `<` less than
-   `<=` less than or equal
-   `==` equal
-   `!=` not equal

### Logical operators {-}

- `&` and; all expressions must be true in order to return true
- `|` or; one or more expressions must be true in order to return true; `|` is the key above the return key, not lowercase letter l.
- `!` not; negate the expression

## Common mistakes {-}

-   Mistake: Using `=` instead of `==` 
    -   `filter()` tells you when this happens (read the errors!):

```{r 04-mistake1, eval=FALSE}
flights |> 
  filter(month = 1)
#> Error in `filter()`:
#> ! We detected a named input.
#> ℹ This usually means that you've used `=` instead of `==`.
#> ℹ Did you mean `month == 1`?
```

Mistake: Writing “or” statements like you would in English:

```{r 04-mistake2, eval=FALSE}
flights |> 
  filter(month == 1 | 2)
# "filter where (month is 1), or (2 (is a non-zero number))"
# Equivalent to filter((month == 1) | TRUE)
```

instead of:

```{r 04-mistake2-fixed, eval=FALSE}
flights |> 
  filter(month == 1 | month == 2)
# "filter where month is 1 or month is 2"
```


## Arrange rows with arrange() {-}

-   `arrange()` changes order of rows
-   1st arg = data frame
-   Subsequent args = columns names to sort by
-   Default order = ascending (small to big)
-   `desc()` for descending order (big to small).

```{r 04-arrange1}
# Sort by year (early to late), then within year by month (early to late), then
# within month by day (early to late), then by time
flights |> 
  arrange(year, month, day, dep_time)
```
```{r 04-arrange2}
# Sort flights from most to least delayed
flights |> 
  arrange(desc(dep_delay))
```


## Distinct {-}

-   `distinct()` = find unique rows in dataset
-   Operates on rows
-   Optional args: column names = find unique combos of values

```{r 04-distinct1}
# Remove duplicate rows, if any
flights |> 
  distinct()

# Find unique origin + destination pairs
flights |> 
  distinct(origin, dest)

# keep other columns when filtering for unique rows
flights |> 
  distinct(origin, dest, .keep_all = TRUE)
```

-   `count()` = number of occurrences
-   `sort = TRUE` to arrange in descending order of # occurrences

```{r 04-count}
flights |>
  count(origin, dest, sort = TRUE)
```

## Exercises (rows) {-}

(NOTE: using `relocate()` and `mutate()` from next section for better visualization)

### Question 1 {-}

In a single pipeline for each condition, find all flights that meet the condition:

```{r 04-exercise1}
# Had an arrival delay of two or more hours
flights |>
  filter(arr_delay >= 120) |>
  arrange(desc(arr_delay)) |>
  relocate(arr_delay)
  
# Flew to Houston (IAH or HOU)
flights |>
  filter(dest %in% c("IAH", "HOU")) |>
  relocate(dest)

# Were operated by United, American, or Delta
flights |>
  filter(carrier %in% c("UA", "DL", "AA")) |>
  relocate(carrier)


# Departed in summer (July, August, and September)
flights |>
  filter(month %in% c(7, 8, 9)) |>
  relocate(month)

# Arrived more than two hours late, but didn’t leave late
flights |> 
  filter(arr_delay >= 120 & dep_delay <= 0) |>
  relocate(arr_delay, dep_delay)

# Were delayed by at least an hour, but made up over 30 minutes in flight
flights |> 
  filter(dep_delay >= 60 & dep_delay - arr_delay > 30) |>
  relocate(dep_delay, arr_delay)

```

### Question 2 {-}

Sort flights to find the flights with longest departure delays. Find the flights that left earliest in the morning.

```{r 04-exercise2}
flights |> 
  arrange(desc(dep_delay)) |> 
  arrange(dep_time) |>
  relocate(dep_delay, dep_time)
```

### Question 3 {-}

Sort flights to find the fastest flights. (Hint: Try including a math calculation inside of your function.)
```{r 04-exercise3}
flights |> 
  mutate(speed = distance / (air_time / 60)) |>
  arrange(desc(speed)) |>
  relocate(speed)
```


### Question 4 {-}

Was there a flight on every day of 2013?
```{r 04-exercise4}
# using count
flights |>
  count(year, month, day, sort = TRUE) |>
  arrange(n)

# using distinct
flights |> 
  distinct(year, month, day)

# Yes, there was a flight every day as the resulting tibble has 365 rows
```

### Question 5 {-}

Which flights traveled the farthest distance? Which traveled the least distance?
```{r 04-exercise5}
# Flights that traveled the farthest distance:
flights |> 
  arrange(desc(distance)) |>
  relocate(distance)

# Flights that traveled the shortest distance
flights |> 
  arrange(distance) |>
  relocate(distance)

```

### Question 6 {-}

Does it matter what order you used filter() and arrange() if you’re using both? Why/why not? Think about the results and how much work the functions would have to do.

If `arrange()` is used before `filter()`, the sorting will be applied to all the rows in the dataset, including those that will be filtered out later. This can result in unnecessary work for the computer and slower performance. On the other hand, if `filter()` is used before `arrange()`, the sorting will only be applied to the remaining rows after filtering, which can be more efficient and faster.

## Columns {-}

There are four important verbs that affect the columns without changing the rows:

-   `mutate()` creates new columns that are derived from the existing columns
-   `select()` changes which columns are present
-   `rename()` changes the names of the columns
-   `relocate()` changes the positions of the columns.

## Add new variables with mutate() {-}

`mutate()` adds new columns based on values from existing columns. Data frame includes existing and new columns.

Compute the gain, how much time a delayed flight made up in the air, and the speed in miles per hour:

```{r 04-mutate1}
flights |> 
  mutate(
    gain = dep_delay - arr_delay,
    speed = distance / air_time * 60
  )
```
Use the `.before` argument to instead add the variables to the left hand side

```{r 04-mutate2}
flights |> 
  mutate(
    gain = dep_delay - arr_delay,
    speed = distance / air_time * 60,
    .before = 1
  )
```
Use `.after` to add after a variable, and in both `.before` and `.after` you can use the variable name instead of a position.
Add the new variables after day:

```{r 04-mutate3}
flights |> 
  mutate(
    gain = dep_delay - arr_delay,
    speed = distance / air_time * 60,
    .after = day
  )
```
You can use the `.keep` argument to control which variables are kept after a `mutate()` operation. Setting `.keep = "used"` ensures only columns involved or created in the `mutate()` step are retained, like `dep_delay`, `arr_delay`, `air_time`, `gain`, `hours`, and `gain_per_hour`.

```{r 04-mutate4}
flights |> 
  mutate(
    gain = dep_delay - arr_delay,
    hours = air_time / 60,
    gain_per_hour = gain / hours,
    .keep = "used"
  )
```
If you don't assign the result of the computation back to `flights`, the new variables will only be displayed, not stored. Consider whether to overwrite `flights` with more variables or create a new object for future use.


## Select columns with select() {-}

`select()` lets you pick which columns (variables) to use. The first argument to `select()` is a data frame, the subsequent arguments are columns to use.

The order you list the columns will determine the order of the columns returned by `select()`.

```{r 04-44}
# Select columns by name:
flights |> 
  select(year, month, day)
```

```{r 04-45}
# Select all columns between year and day (inclusive):
flights |> 
  select(year:day)
```

```{r 04-46}
# Select all columns except those from year to day (inclusive):
flights |> 
  select(!year:day)
```

```{r 04-select-characters}
# Select all columns that are characters:
flights |> 
  select(where(is.character))
```
helper functions for `select()`

There are a number of helper functions you can use within `select()`:

-   `starts_with("abc")`: matches names that begin with “abc”.
-   `ends_with("xyz")`: matches names that end with “xyz”.
-   `contains("ijk")`: matches names that contain “ijk”.
-   `num_range("x", 1:3)`: matches x1, x2 and x3.

See `?select` for more details.


You can rename variables as you `select()` them by using `=`. The new name appears on the left hand side of the `=`, and the old variable appears on the right hand side:

```{r 04-select-rename}
flights |> 
  select(tail_num = tailnum)
```

## Rename columns with rename() {-}

If you want to keep all the existing variables and just want to rename a few, you can use `rename()` instead of `select()`:

```{r 04-rename}
flights |> 
  rename(tail_num = tailnum)
```

**Tip**: If you have inconsistently named columns and it's difficult to fix them manually, use `janitor::clean_names()`. It automates the cleaning process.


## Move variables around with relocate() {-}

Use `relocate()` to move variables around. You might want to collect related variables together or move important variables to the front. By default `relocate()` moves variables to the front:

```{r 04-relocate}
flights |> 
  relocate(time_hour, air_time)
```

You can also specify where to put them using the `.before` and `.after` arguments, just like in `mutate()`:

```{r 04-relocate-before-after}
flights |> 
  relocate(year:dep_time, .after = time_hour)
flights |> 
  relocate(starts_with("arr"), .before = dep_time)
```


##  Exercises (columns) {-}

### Question 1 {-}
Compare `dep_time`, `sched_dep_time`, and `dep_delay`. How would you expect those three numbers to be related?

The `dep_time` should be `sched_dep_time + dep_delay`.

```{r 04-exercises-columns-01}
flights |> 
  relocate(dep_time, sched_dep_time, dep_delay)
```

### Question 2 {-}

Brainstorm as many ways as possible to select `dep_time`, `dep_delay`, `arr_time`, and `arr_delay` from `flights`.

You can select them by name, or the columns that starts with dep or with arr.
```{r 04-exercises-columns-02}
flights |> 
  select(dep_time, dep_delay, arr_time, arr_delay)

flights |> 
  select(starts_with("dep"), starts_with("arr"))
```

### Question 3 {-}

What happens if you specify the name of the same variable multiple times in a `select()` call?

It will only show the column at the first position it was specified.

```{r 04-exercises-columns-03}
flights |> 
  select(dep_time, dep_delay, arr_time, arr_delay, dep_time)
```

### Question 4 {-}

What does the `any_of()` function do? Why might it be helpful in conjunction with this vector?

`variables <- c("year", "month", "day", "dep_delay", "arr_delay")`

`any_of()` doesn't check for missing variables. It is especially useful with negative selections, when you would like to make sure a variable is removed.

The order of selected columns is determined by the order in the vector.

```{r 04-exercises-columns-04}
variables <- c("year", "month", "day", "dep_delay", "arr_delay")

flights |> 
  select(any_of(variables))
```

### Question 5 {-}

Does the result of running the following code surprise you? How do the select helpers deal with upper and lower case by default? How can you change that default?

```{r 04-exercises-columns-05}
flights |> select(contains("TIME"))
```
It is suprising because "TIME" is in uppercase, so it seems that `contains` ignores the case. 

To change this default behavior, set ignore.case = FALSE.

```{r 04-exercises-columns-05-1}
flights |> 
  select(contains("TIME", ignore.case = FALSE))
```

### Question 6 {-}

Rename `air_time` to `air_time_min` to indicate units of measurement and move it to the beginning of the data frame.

```{r 04-exercises-columns-06}
flights |>
  rename(air_time_min = air_time) |>
  relocate(air_time_min)
```

### Question 7 {-}

Why doesn’t the following work, and what does the error mean?

```{r 04-exercises-columns-07, eval=FALSE}
flights |> 
  select(tailnum) |> 
  arrange(arr_delay)
#> Error in `arrange()`:
#> ℹ In argument: `..1 = arr_delay`.
#> Caused by error:
#> ! object 'arr_delay' not found
```

The code doesn’t work because the `select()` will give a tibble with only the `tailnum` column, so  the arrange will not find the column `arr_delay`.

## The pipe {-}

The real power of the pipe is to combine multiple verbs. 
To find the fast flights to Houston’s IAH airport: you need to combine filter(), mutate(), select(), and arrange():

```{r 04-pipe1,}
flights |> 
  filter(dest == "IAH") |> 
  mutate(speed = distance / air_time * 60) |> 
  select(year:day, dep_time, carrier, flight, speed) |> 
  arrange(desc(speed))
```

Using the pipe operator makes complex operations easier to read and follow. Each step is clear as the verbs start at the beginning of the line. Without the pipe, functions would be nested inside each other, making the code harder to understand.


```{r 04-pipe2, eval=FALSE}
arrange(
  select(
    mutate(
      filter(
        flights, 
        dest == "IAH"
      ),
      speed = distance / air_time * 60
    ),
    year:day, dep_time, carrier, flight, speed
  ),
  desc(speed)
)
```

Another way but using intermediate objects:

```{r 04-pipe3, eval=FALSE}
flights1 <- filter(flights, dest == "IAH")
flights2 <- mutate(flights1, speed = distance / air_time * 60)
flights3 <- select(flights2, year:day, dep_time, carrier, flight, speed)
arrange(flights3, desc(speed))
```

Both forms have their uses, but the pipe operator usually makes data analysis code easier to write and read.

**Tip**: To add the pipe to your code, use the built-in keyboard shortcut Ctrl/Cmd + Shift + M. 

## Groups {-}

The previous functions work with rows and columns. `dplyr` also allows to work with groups with functions like: `group_by()`, `summarize()`, and the slice family of functions.


### group_by() {-}

Use `group_by()` to divide your dataset into groups meaningful for your analysis:

```{r 04-group1}
flights |> 
  group_by(month)
```

group_by() doesn’t change the data but looking at the output, it indicates that it is “grouped by” month `(Groups: month [12])`

## summarize() {-}

Summarization is the most important grouped operation, because it reduces the data frame to a single row for each group, summarizing the data with a single statistic.
The following example computes the average departure delay by month:


```{r 04-summarize1}
flights |> 
  group_by(month) |> 
  summarize(
    avg_delay = mean(dep_delay)
  )

```

The above example show NAs because some of the observed flights had missing data in the delay column. To ignore the missing data, use:

```{r 04-summarize2}
flights |> 
  group_by(month) |> 
  summarize(
    delay = mean(dep_delay, na.rm = TRUE)
  )
```

In a single call to summarize() you can create any number of summaries. There are different useful summaries available, one of which is n(), which returns the number of rows in each group:

```{r 04-summarize3}
flights |> 
  group_by(month) |> 
  summarize(
    delay = mean(dep_delay, na.rm = TRUE), 
    n = n()
  )
```

## Slice functions {-}

To extract specific rows within each group you can use the following functions:

-   `df |> slice_head(n = 1)` takes the first row from each group.
-   `df |> slice_tail(n = 1)` takes the last row in each group.
-   `df |> slice_min(x, n = 1)` takes the row with the smallest value of column `x`.
-   `df |> slice_max(x, n = 1)` takes the row with the largest value of column `x`.
-   `df |> slice_sample(n = 1)` takes one random row.

You can use n to select multiple rows or use prop = 0.1 to select 10% of the rows in each group. For example, the code finds the most delayed flights upon arrival at each destination.

```{r 04-slice1}
flights |> 
  group_by(dest) |> 
  slice_max(arr_delay, n = 1) |>
  relocate(dest)
```

In the above output we get 108 rows but there are only 105 destinations.
`slice_min()` and `slice_max()` keep tied values so `n = 1` means give us all rows with the highest value, if you want only one row per group you can set `with_ties = FALSE`.

## Grouping by multiple variables {-}

To make a group for each date you can create groups using more than one variable.

```{r 04-grouping-multiple-1}
daily <- flights |>  
  group_by(year, month, day)
daily
```

When you summarize a tibble grouped by more than one variable, each summary peels off the last group.


```{r 04-grouping-multiple-2}
daily_flights <- daily |> 
  summarize(n = n())
```

If this behavior is OK, you can explicitly request it so that it doesn't give the warning message:

```{r 04-grouping-multiple-3}
daily_flights <- daily |> 
  summarize(
    n = n(), 
    .groups = "drop_last"
  )
```

You can change the default behavior by setting a different value, like `"drop"` to remove all grouping or `"keep"` to maintain the existing groups.

## Ungrouping {-}

You might also want to remove grouping from a data frame without using `summarize()`.
You can do this with `ungroup()`.

```{r 04-ungrouping-1}
daily |> 
  ungroup()
```

Now let's see what happens when you summarize an ungrouped data frame.

```{r 04-ungrouping-}
daily |> 
  ungroup() |>
  summarize(
    avg_delay = mean(dep_delay, na.rm = TRUE), 
    flights = n()
  )
```

You get a single row back because dplyr treats all the rows in an ungrouped data frame as belonging to one group.

## Group using .by {-}

You can now also use the `.by` argument to group within a single operation,
`.by` works with all verbs and has the advantage that you don't need to use the `.groups` argument to suppress the grouping message or `ungroup()` when you're done.


```{r 04-groupby-01}
flights |> 
  summarize(
    delay = mean(dep_delay, na.rm = TRUE), 
    n = n(),
    .by = month
  )
```

Or if you want to group by multiple variables:

```{r 04-groupby-02}
flights |> 
  summarize(
    delay = mean(dep_delay, na.rm = TRUE), 
    n = n(),
    .by = c(origin, dest)
  )
```

## Exercises (groups) {-}

### Question 1 {-}

Which carrier has the worst average delays?
Challenge: can you disentangle the effects of bad airports vs. bad carriers?
Why/why not?
(Hint: think about `flights |> group_by(carrier, dest) |> summarize(n())`)

The carrier with worst average delays is `F9`

```{r 04-exercises-groups-01-1}
flights |>
  group_by(carrier) |>
  summarize(
    delay = mean(arr_delay, na.rm = TRUE),
    n = n()
  ) |>
  arrange(desc(delay))
```

To try to disentangle the effects of bad airports vs. bad carriers and using the hint to group and summarize:

```{r 04-exercises-groups-01-2}
# calculate average delay and sort

flights |>
  group_by(carrier, dest) |>
  summarize(
    avg_delay = mean(arr_delay, na.rm = TRUE),
    n = n()
  ) |>
  arrange(desc(avg_delay))

# add total_delay and sort
flights |>
  group_by(carrier, dest) |>
  summarize(
    delay = sum(arr_delay, na.rm = TRUE),
    n = n()
  ) |>
  mutate(total_delay = delay * n) |> 
  arrange(desc(total_delay))
```

In the 1st calculation, UA to STL has a very high average delay, but it only happens twice. 
In the 2nd calculation, DL to ATL has the higher total delay. 

### Question 2 {-}

Find the flights that are most delayed upon departure from each destination.

The top 10 most departure delayed flights from each destination.

```{r 04-exercises-groups-02}
flights |>
  group_by(dest) |>
  slice_max(dep_delay, n = 1) |>
  relocate(dep_delay, dest) |>
  arrange(desc(dep_delay))

# using by in slice_max

flights |>
  slice_max(
    dep_delay, n = 1,
    by = dest
  ) |>
  relocate(dep_delay, dest) |>
  arrange(desc(dep_delay))
```

### Question 3 {-}

How do delays vary over the course of the day. Illustrate your answer with a plot.

```{r 04-exercises-groups-03}
# Calculate the average departure delay for each hour of the day
delays_by_hour <- flights %>%
  group_by(hour) %>%
  summarize(avg_dep_delay = mean(dep_delay, na.rm = TRUE))


# Plot the results
ggplot(delays_by_hour, aes(x = hour, y = avg_dep_delay)) +
  geom_line() +
  ggtitle("Average Departure Delay by Hour of the Day") +
  xlab("Hour of the Day") +
  ylab("Average Departure Delay (minutes)")
```

### Question 4 {-}

What happens if you supply a negative `n` to `slice_min()` and friends?

It doesn't slice the data but it does arrange the values. 

```{r 04-exercises-groups-04}
# only shows the lowest 5 rows
flights |> 
  slice_min(dep_delay, n = 5) |>
  relocate(dep_delay)

# shows all rows
flights |> 
  slice_min(dep_delay, n = -5) |>
  relocate(dep_delay)

```

### Question 5 {-}

Explain what `count()` does in terms of the dplyr verbs you just learned.
What does the `sort` argument to `count()` do?

`count()` is used to count the number of rows in a data frame, grouped by one or more variables. 

The `sort` argument specifies if how the results are sorted (`TRUE` ascending) is the default.


## Case Study {-}

When you summarize data, it is always a good idea to include a count of the number of observations. This helps you to make sure that your conclusions are based on a large enough sample size.

In this example using baseball data, we're comparing how often a player successfully hits the ball (H) to the total number of attempts they made to hit the ball (AB). Including a count ensures our analysis is based on a reasonable amount of data and not just a few instances.

```{r 04-case-study-01}
batters <- Lahman::Batting |> 
  group_by(playerID) |> 
  summarize(
    performance = sum(H, na.rm = TRUE) / sum(AB, na.rm = TRUE),
    n = sum(AB, na.rm = TRUE)
  )
batters
```

When we compare how well baseball players hit the ball (measured by batting average) to how many times they try (measured by times at bat), we notice two things:

1. Players with fewer attempts to hit show more varying results in their performance. This is a common pattern: when you compare averages for different groups, you'll often see less variation as the group size gets larger.

2. Skilled players tend to have more chances to hit. This is because teams prefer to let their best players have more opportunities to bat. So, better players get more chances to hit the ball.

```{r 04-case-study-02}
batters |> 
  filter(n > 100) |> 
  ggplot(aes(x = n, y = performance)) +
  geom_point(alpha = 1 / 10) + 
  geom_smooth(se = FALSE)
```

If you simply rank players by batting average, the players at the top of the list will be those who had very few at-bats and happened to get a hit. These players may not actually be the most skilled players.

```{r 04-case-study-03}
batters |> 
  arrange(desc(performance))
```


## Meeting Videos {-}

### Cohort 5 {-}

`r knitr::include_url("https://www.youtube.com/embed/J6mGn1F1kiA")`

<details>
  <summary> Meeting chat log </summary>
  
```
00:13:48	Jon Harmon (jonthegeek):	"dplyr" as in "data plyer" (tools for working with data)
00:22:03	Ryan Metcalf:	I call the | as “handlebar”…may be my own lingo too.
00:22:20	lucus w:	I like vbar
00:38:30	Ryan Metcalf:	Quick thought, on dat ingestion, does the tidyverse convert null to NA? Or an alternative, does is.na look for null too?
00:39:19	Jon Harmon (jonthegeek):	Null coming in from a database will convert to NA. NULL specifically means "does not exist," and can't be inside a vector of numbers in R. It's its own data type in R.
00:39:24	lucus w:	I believe NA and NULL aren’t the same thing, so I’d guess no
00:39:53	Njoki Njuki Lucy:	can one use filter to remove na?
00:40:01	Ryan Metcalf:	👍🏻
00:40:45	lucus w:	filter(!is.na(x)) wil do the trick
00:40:51	Jon Harmon (jonthegeek):	filter(flights, !is.na(month)) would remove NA rows.
00:41:05	Jon Harmon (jonthegeek):	Lucus beat me to it :D
00:42:22	Njoki Njuki Lucy:	awesome, thank you both :)
00:46:06	Jon Harmon (jonthegeek):	Chapter 14 has more on regular expressions.
00:47:03	Jon Harmon (jonthegeek):	https://regexr.com/
00:58:04	lucus w:	I wish all aggregate functions would have na.rm = TRUE as a default
01:04:21	lucus w:	is magrittr a function or just an operator
01:04:35	lucus w:	%>%
01:04:38	Jon Harmon (jonthegeek):	If you're curious why the pipe package is called magrittr: https://en.wikipedia.org/wiki/The_Treachery_of_Images#/media/File:MagrittePipe.jpg
01:05:02	Jon Harmon (jonthegeek):	magrittr is the package which exports the %>% function (but it's a special kind of function because it can go in the middle of its arguments)
01:16:15	Eileen:	Great presentation
01:16:17	Ryan Metcalf:	Great job!
01:16:34	LG:	Thank you!
01:16:46	Njoki Njuki Lucy:	Thank you!
01:17:40	Eileen:	Thank you!
```
</details>

`r knitr::include_url("https://www.youtube.com/embed/KtDeYCfurz4")`

<details>
  <summary> Meeting chat log </summary>
```
00:04:33	Susan Neilson:	Yes!
00:27:11	shamsuddeen:	https://stackoverflow.com/questions/51901398/what-does-a-tilde-in-front-of-a-single-variable-mean-facet-wrap/53835451#53835451
00:27:30	Federica Gazzelloni:	https://www.rdocumentation.org/packages/tibble/versions/3.1.2/topics/tribble
00:27:54	shamsuddeen:	http://uc-r.github.io/integer_double/
00:28:15	shamsuddeen:	1L, 2L
00:28:50	Federica Gazzelloni:	tribble() creates tibbles using an easier to read row-by-row layout.
00:37:49	Becki R. (she/her):	the audio is out
01:08:20	Susie N.:	I have to head out everyone - thank you all so much for an excellent class and discussion!
01:08:32	Federica Gazzelloni:	info for lealeft: https://rstudio.github.io/leaflet/map_widget.html
01:09:32	Federica Gazzelloni:	the documentation says: ~ x means the variable x in the data object
01:10:21	shamsuddeen:	https://dplyr.tidyverse.org/reference/pull.html
01:10:44	Njoki Njuki Lucy:	thanks you
01:10:51	Njoki Njuki Lucy:	thank*
01:12:24	shamsuddeen:	Strings
01:12:35	shamsuddeen:	Chapter
01:12:38	Federica Gazzelloni:	thanks
01:12:40	shamsuddeen:	I can do
```
</details>


### Cohort 6 {-}

`r knitr::include_url("https://www.youtube.com/embed/DNwioC-SgKQ")`

<details>
  <summary> Meeting chat log </summary>
  
```
00:01:33	Adeyemi Olusola:	good day Daniel
00:01:48	Daniel Adereti:	Hello Adeyemi!
00:03:09	Adeyemi Olusola:	we are experiencing loadshedding as regards electricity in SA. So, If I am unavailable at some point kindly know that its light issues
00:03:26	Daniel Adereti:	Got you!
00:06:42	Daniel Adereti:	Let's give people about 5 minutes to join in
00:07:21	Daniel Adereti:	Also, expecting our volunteer Matthew to join a bit late
00:07:38	Shannon:	Sounds good
00:07:56	Daniel Adereti:	He had to reschedule a number of things due to the timezone issues (CDT vs WAT)
00:09:07	Shannon:	it's a bit crackly
00:09:30	Adeyemi Olusola:	yes...its getting better
00:10:29	Adeyemi Olusola:	yeah
00:10:32	Shannon:	much better!
00:10:45	Adeyemi Olusola:	you decide
00:12:04	Shannon:	yes! seeing your RStudio screen clearly
00:19:39	Vrinda Kalia:	That’s my guess as well
00:21:10	Freya Watkins (she/her):	maybe it's just the order, a=1, b=2 etc? I guess
00:21:25	Daniel Adereti:	seems like @freya
00:21:26	Shannon:	That seems right
00:21:27	Adeyemi Olusola:	my guess...
00:21:36	Adeyemi Olusola:	perhaps we can try z and t
00:22:56	Daniel Adereti:	'z'  > 't' == TRUE, so it appears it is by order
00:23:10	Adeyemi Olusola:	nice one
00:24:25	Adeyemi Olusola:	equal to
00:24:29	Adeyemi Olusola:	a = 1
00:24:40	Vrinda Kalia:	on Mac: option + -
00:24:45	Freya Watkins (she/her):	ALT-minus is a shortcut for on Windows
00:29:26	Adeyemi Olusola:	1 = 5 is false but the negation turns it true
00:32:00	Adeyemi Olusola:	yes, we can
00:32:54	Vrinda Kalia:	Sorry, I don’t see the pop-up. We see the help pane
00:33:23	Adeyemi Olusola:	Yes, I was referring to the help pane, I thought he will be scrolling along the pane
00:36:01	Vrinda Kalia:	flights$month
00:38:15	Shannon:	I haven't seen/used the as.levels() function yet, looks like it can be useful!
00:38:57	Daniel Adereti:	levels(as.factor(flights$month))
00:39:07	Adeyemi Olusola:	exactly
00:39:31	Shannon:	Nice!
00:40:59	Shannon:	oops, as.factor(), not as.levels()
00:41:19	Adeyemi Olusola:	@Shannon exactly
00:41:44	Shannon:	So many functions to learn! :)
00:44:54	Adeyemi Olusola:	its filtering november
00:45:20	Adeyemi Olusola:	across years
00:47:09	Adeyemi Olusola:	the time is also there
00:47:43	Adeyemi Olusola:	so there are many flights in one (1) day
00:52:32	Daniel Adereti:	Let's aim to finish the filter() fxn lesson today, right? it ends at "missing values"
00:52:57	Shannon:	that sounds good
00:58:50	Vrinda Kalia:	That sounds good!
00:58:52	Adeyemi Olusola:	Nice work!!!!
00:59:04	Adeyemi Olusola:	Thanks a lot Mattew!
00:59:04	Aalekhya Reddam:	That’s great! Thank you Matthew :)
00:59:05	Vrinda Kalia:	thank you so much, Matthew!
00:59:23	Shannon:	Thank you, Matthew!
00:59:35	Adeyemi Olusola:	Bye Bye
00:59:42	Freya Watkins (she/her):	Thanks Matthew!
00:59:45	Daniel Adereti:	Thank you!
```
</details>


### Cohort 7 {-}

`r knitr::include_url("https://www.youtube.com/embed/LsU9XZ8iNws")`


### Cohort 8 {-}

`r knitr::include_url("https://www.youtube.com/embed/aJFrAAmTCVE")`

<details>
<summary> Meeting chat log </summary>

```
00:55:33	shamsuddeen:	https://r4ds.github.io/bookclub-r4ds/data-transformation.html
01:09:35	shamsuddeen:	https://juliasilge.com
```
</details>

`r knitr::include_url("https://www.youtube.com/embed/_NbdlBznxIE")`

<details>
<summary> Meeting chat log </summary>

```
00:50:43	shamsuddeen:	https://docs.google.com/spreadsheets/d/1reByMPb5Og3OHbRgplzV5Jkz_o6CkEYz_adBGfHTFwg/edit#gid=0
00:51:50	shamsuddeen:	Ctrl + Shift + M
```
</details>