Cleaning

Goals of this notebook

The steps we’ll take to prepare our data:

  • Download the data
  • Import it into our notebook
  • Clean up data types and columns
  • Export the data for next notebook

Setup

library(tidyverse)
library(janitor)
library(dplyr)

Import data

I am importing the data comes from Climate Data Online, a collection of official weather data from the National Centers for Environmental Information, a division of National Oceanic and Atmospheric Administration. The actual data source we’ll use is called the Global Historical Climate Network “Daily Summaries” which includes daily land surface observations from weather stations around the world.

I pulled the data from the “primary” weather station for downtown Austin, the Camp Mabry station, which can be found here.

weather_raw <- read_csv("data-raw/3806794.csv") |> clean_names()
Warning: One or more parsing issues, call `problems()` on your data frame for details,
e.g.:
  dat <- vroom(...)
  problems(dat)
weather_raw |> head()

Problems check

  • Check the import problem. (This could be done in the console)
  • Open the data and Go To Line.

There are some anomalies around the dates 1981-10-09 and 1981-10-08.

problems(weather_raw)

Remove columns

I am removing the station, name, tavg and tobs columns.

# Remove columns
weather <- weather_raw |> select(-c(station, name, tavg, tobs))

# peak at the data 
weather |> glimpse()
Rows: 31,522
Columns: 6
$ date <date> 1938-06-01, 1938-06-02, 1938-06-03, 1938-06-04, 1938-06-05, 1938…
$ prcp <dbl> 0.00, 0.00, 0.00, 0.40, 0.02, 0.00, 0.00, 0.00, 1.60, 0.01, 0.00,…
$ snow <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ snwd <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ tmax <dbl> 91, 94, 94, 90, 94, 92, 95, 92, 87, 90, 92, 91, 91, 91, 89, 89, 9…
$ tmin <dbl> 72, 67, 70, 68, 68, 70, 70, 76, 64, 76, 75, 71, 70, 68, 71, 70, 7…

I am looking at the table.

weather |> head(10)

I will print summary stats of my data.

weather |> summary()
      date                 prcp              snow               snwd         
 Min.   :1938-06-01   Min.   :0.00000   Min.   :0.000000   Min.   :0.000000  
 1st Qu.:1959-12-28   1st Qu.:0.00000   1st Qu.:0.000000   1st Qu.:0.000000  
 Median :1981-07-25   Median :0.00000   Median :0.000000   Median :0.000000  
 Mean   :1981-07-25   Mean   :0.09111   Mean   :0.002053   Mean   :0.002538  
 3rd Qu.:2003-02-20   3rd Qu.:0.00000   3rd Qu.:0.000000   3rd Qu.:0.000000  
 Max.   :2024-09-18   Max.   :7.55000   Max.   :6.500000   Max.   :6.000000  
                                        NA's   :7          NA's   :4         
      tmax            tmin      
 Min.   : 20.0   Min.   :-2.00  
 1st Qu.: 70.0   1st Qu.:47.00  
 Median : 82.0   Median :61.00  
 Mean   : 79.6   Mean   :58.45  
 3rd Qu.: 92.0   3rd Qu.:72.00  
 Max.   :112.0   Max.   :93.00

Exports

Now, I’ll take my newly cleaned data and put it in a safe place.

weather |>
  write_rds("data-processed/01-weather.rds")