Wrangling the Data

Today’s Agenda

• Why the loaded table cannot answer your question
• The five verbs that reshape a table
• Readable timestamps, and the thousand that breaks them
• Message length, your first derived variable

From Raw to Analysis-Ready

Lab session

• The chat table cannot answer your question yet
• Today: five verbs, then two transformations
  • Readable timestamps, and message length
  • Thursday brings the gaming label and the join
• Nothing here is analysis, and every later result rests on it

The least rewarding part of the term, and what professionals spend most of their time on.

What the Table Cannot Answer

• The study compares gaming and non-gaming chat
  • It operationalizes that through message length
• Neither column exists
  • No message length, no gaming label
• The timestamps are unreadable runs of digits
  • The gaming information sits in another table, unlinked

Data Wrangling, Defined

• Turning data you collected into data you can analyze
  • Reshaping, cleaning, deriving, joining
• Unglamorous, rarely in the paper, and most of the work
  • It looks like small technical steps
  • It is building the table every result rests on
• The tidyverse does it with five dplyr verbs

The Verbs, Filter and Select

chat %>% filter(channel == "bobross")

filter() keeps the rows meeting a condition and drops the rest, here only Bob Ross’s messages.

chat %>% select(channel, sender, message)

select() keeps or drops columns, here narrowing five to three.

The Verbs, Mutate and the Rest

• mutate() adds or changes a column, computed from others
• arrange() reorders the rows by a column’s values
• group_by() with summarize() collapses rows into one per group
  • That pair is how you get a per-channel figure

Five verbs, and they carry almost all the wrangling you will ever do.

Why the Verbs Chain

• Each one takes a data frame and returns a data frame
  • That shared design is what lets %>% string them together
  • One verb’s output is the next one’s input
• A pipeline reads in order
  • Take the data, then do this, then do that

Four such transformations rebuild the chat table.

The Unreadable Timestamp

• date is a column of very large numbers, typed <dbl>
  • It counts milliseconds since midnight, 1 January 1970
• That moment is the Unix epoch
  • Counting from it makes a timestamp one integer
  • Precise, and unreadable

Chapter 8 called this interval data, with a catch. This is the catch.

Converting It

chat <- chat %>%
  mutate(timestamp = as.POSIXct(date / 1000, origin = "1970-01-01", tz = "UTC"))

as.POSIXct() turns a number into a date-time value, given the origin to count from, and mutate() writes the result into a new timestamp column.

The Thousand That Breaks It

• as.POSIXct() expects seconds
  • date counts milliseconds, a thousand times finer
• Hand it the raw number and every message lands in the future
  • Tens of thousands of years into it
• date / 1000 converts first

A small arithmetic step, and the most common way this conversion fails.

Seeing the Conversion

chat %>% select(date, timestamp) %>% head(3)

           date timestamp
          <dbl> <dttm>
1 1542578127023 2018-11-18 21:55:27
2 1542578135562 2018-11-18 21:55:35
3 1542578143610 2018-11-18 21:55:43

Same instant, now legible: three messages eight seconds apart.

The type changed from <dbl> to <dttm>, and a date-time gives you the hour and the weekday that the raw integer never will.

Measuring Message Length

chat <- chat %>%
  mutate(message_length = str_length(message))

str_length() counts the characters in a string, and mutate() stores that count as a new column your codebook named but the data did not contain.

What the Lengths Show

chat %>% select(message, message_length) %>% head(3)

  message                          message_length
1 "???????????????????????"                    23
2 "TriEasy Clap TriEasy Clap…"                441
3 "cmonBruh"                                    8

A single word at 8, punctuation at 23, copypasta at 441. message_length is ratio: true zero, equal intervals, safe to average.

Common Errors Today

• Every message dated in the year 50000: you forgot / 1000
• could not find function "str_length": load the tidyverse
• object 'message_length' not found: you never assigned back
  • <- is what makes a change stick
• unexpected '=': usually = where == belongs

Run ?v2v::common_errors for the ones specific to this data.

Checkpoint

You should now have:

• A chat object with a working timestamp, typed <dttm>
• A message_length column, typed <int>
• Output showing dates in November 2018
• A saved script, not lines typed into the console

Still missing: the gaming label. That is Thursday.

Before Thursday

• Run today’s transformations on your project and save the script
• Due this week: Sampling Plan and Pilot, and Data Wrangling
• Thursday: the gaming label, the join, and what it leaves behind
• Bring your script, because we pick it up where it stops

Questions?