Wrangling the Data [R]

Today’s Agenda

• The gap between what you have and what you asked
• Four transformations, built one at a time
• Joins, and how they fail silently
• Your wrangling script as a methods section

What We Build Today

Working session

• One analysis-ready table, from the raw chat and stream tables
  • A readable timestamp, message length, a gaming label, a join
• Weeks 12 and 13 compute everything from it
• Your Methods section is largely this script

The least rewarding evening of the term, and where researchers spend most of their hours.

The Gap You Are Closing

• The study asks whether the two kinds of channel differ
  • It operationalizes that through message length
• Neither column exists
  • No message length, no gaming label
• The timestamps are unreadable runs of digits
• The gaming information is in the other table entirely

The Verbs of Data Wrangling

• filter() keeps rows meeting a condition
• select() keeps or drops columns
• mutate() adds or changes a computed column
• arrange() reorders rows by a column’s values
• group_by() with summarize() collapses rows per group

Each takes a data frame and returns one, which is what lets the pipe chain them.

Two Verbs, Seen

chat %>%
  filter(channel == "bobross")

chat %>%
  select(channel, sender, message)

The first keeps only Bob Ross’s messages and drops every other row. The second keeps three columns and drops the rest.

Transformation 1, a Readable Timestamp

chat <- chat %>%
  mutate(
    timestamp = as.POSIXct(
      date / 1000,
      origin = "1970-01-01",
      tz = "UTC"
    )
  )

as.POSIXct() turns a number into a date-time value, counting from the origin you give it.

The Millisecond Trap

• date counts milliseconds since 1 January 1970
  • That moment is the Unix epoch
• as.POSIXct() expects seconds, a thousand times coarser
• Hand it the raw number and everything lands in the future
  • Tens of thousands of years into it

date / 1000 converts first, and skipping it is how this fails.

The Type Change Is the Payoff

# A tibble: 3 × 2
           date timestamp
          <dbl> <dttm>
1 1542578127023 2018-11-18 21:55:27
2 1542578135562 2018-11-18 21:55:35
3 1542578143610 2018-11-18 21:55:43

The type moved from <dbl> to <dttm>, and only from a date-time can you pull the hour, the weekday, or the calendar date.

Transformation 2, Message Length

chat <- chat %>%
  mutate(message_length = str_length(message))

chat %>%
  select(message, message_length) %>%
  head(3)

str_length() counts the characters in a string, and mutate() writes the count into a new column.

What the Lengths Look Like

# A tibble: 3 × 2
  message                                 message_length
  <chr>                                             <int>
1 "???????????????????????"                           23
2 "TriEasy Clap TriEasy Clap TriEasy Cla…"            441
3 "cmonBruh"                                            8

• message_length is ratio: true zero, equal intervals, averageable
• Eight, twenty-three and 441 are all real Twitch chat

That spread is the raw material, and Week 12 looks hard at its shape.

Tidy Data, Defined

“Tidy data is a standard way of mapping the meaning of a dataset to its structure.”

Wickham (2014, p. 4)

• One observation per row, one variable per column, one value per cell
• Not a preference, but the structural assumption every tidyverse function makes about its inputs and produces about its outputs

Step 1, What a Channel Mostly Streams

channel_type <- streams %>%
  filter(!is.na(game)) %>%
  count(channel, game) %>%
  group_by(channel) %>%
  slice_max(n, n = 1, with_ties = FALSE) %>%
  ungroup()

Drop snapshots with no category, count how often each channel streamed each category, then keep each channel’s modal category, one row per channel.

Step 2, Name the Non-Gaming

nongaming_categories <- c(
  "Art", "ASMR", "Beauty & Body Art", "Creative",
  "Food & Drink", "IRL", "Just Chatting",
  "Makers & Crafting", "Music",
  "Music & Performing Arts", "Science & Technology",
  "Sports & Fitness", "Talk Shows & Podcasts",
  "Travel & Outdoors"
)

Listing them states the rule, so the judgment is visible in the code.

Step 3, the Join

channel_type <- channel_type %>%
  mutate(is_gaming = !(game %in% nongaming_categories)) %>%
  select(channel, is_gaming)

chat <- chat %>%
  left_join(channel_type, by = "channel")

left_join() keeps every chat row and attaches the matching is_gaming value, matching on the shared channel key.

Joins Fail Silently on Keys

• A join is only as good as its key match
  • Twitch writes channel names with a leading #
  • So #bobross in one source, bobross in another
• Keys that disagree fail every row, with no error
• The v2v fixture is normalized, and raw data is not

When a join returns nothing, check the keys first.

Always Inspect a Derived Column

chat %>% count(is_gaming)

# A tibble: 3 × 2
  is_gaming     n
  <lgl>     <int>
1 FALSE      3457
2 TRUE      31309
3 NA          501

Counting the new column is one line, and it is the line that catches the problem you did not anticipate.

The NA Is the Correct Outcome

• Two channels never had a category recorded
  • No categories, so no modal category, so no lookup row
  • The join left their messages NA
• The data genuinely does not say for those 501 messages
• An honest NA records that the study cannot classify them

They sit the comparison out rather than being guessed into a side.

The Whole Pipeline in One Place

chat <- v2v::twitch_chat() %>%
  mutate(
    timestamp = as.POSIXct(date / 1000,
                           origin = "1970-01-01",
                           tz = "UTC"),
    message_length = str_length(message)
  ) %>%
  left_join(channel_type, by = "channel")

Four transformations, run in sequence from the raw fixture, rebuild the table every later result depends on.

The Analysis-Ready Table

Rows: 35,267
Columns: 8
$ id             <int> 89, 551, 1033, 2094, 2803, …
$ channel        <chr> "sodapoppin", "xqcow", "forsen", …
$ sender         <chr> "madzee", "prometheanow", …
$ message        <chr> "???????????????????????", …
$ date           <dbl> 1542578127023, 1542578135562, …
$ timestamp      <dttm> 2018-11-18 21:55:27, …
$ message_length <int> 23, 441, 8, 12, 206, …
$ is_gaming      <lgl> TRUE, FALSE, TRUE, TRUE, FALSE, …

Your Script Is a Methods Section

• A documented transformation chain, raw to analysis-ready
  • Every mutate() and filter() is a methodological decision
  • Which cases to exclude, how to treat missing, how to bucket time
• A reviewer will ask how you handled something
  • The answer is in the script, if it is commented and committed

Identify three decisions a reviewer could challenge and comment each.

Never Overwrite Raw Data

• data/raw/ is read-only after the first commit
  • Your script reads from it and writes to data/processed/
• That separation lets you rebuild from immutable raw data
  • Overwriting severs the guarantee, quietly and permanently
• Test it: re-run in a fresh session

If it does not reproduce without intervention, you have a hidden dependency.

Common Errors and Meanings

• Timestamps in the year 51,000: you forgot / 1000
• could not find function "%>%": the tidyverse never loaded
• Every joined column NA: the keys disagree, often a #
• Row count grew after a join: duplicate keys in the lookup
• object 'channel_type' not found: chunks run out of order

Run ?v2v::common_errors for the ones specific to this data.

Checkpoint

You should now have:

• A chat object with 8 columns and 35,267 rows
• A timestamp of type <dttm> and a message_length of type <int>
• An is_gaming with 3,457 FALSE, 31,309 TRUE, and 501 NA
• A commented script that runs top to bottom in a clean session
• The processed dataset written out, and raw data untouched

Before Week 12

• Submit Data Wrangling [R], 50 points
  • The script, its comments, and three justifications
• Read Chapter 12 with the graduate toggle on
  • Plus the assigned article: Lakens (2013) on effect sizes
• Write a journal entry, 450 to 500 words
• Bring the same laptop and environment

Questions?