Chapter 7: Fixing Messy Data when Aggregating
Earlier chapters of this book introduced techniques for generating summary statistics on a whole DataFrame. We used methods such as
quantile to do that. This chapter covers more complicated aggregation tasks: aggregating by categorical variables, and using aggregation to change the structure of DataFrames.
After the initial stages of data cleaning, analysts spend a substantial amount of their time doing what Hadley Wickham has called splitting-applying-combining. That is, we subset data by groups, apply some operation to those subsets, and then draw conclusions about a dataset as a whole. In slightly more specific terms, this involves generating descriptive statistics by key categorical variables. For the
nls97 dataset, this might be gender, marital status, and highest degree received. For the COVID-19 data, we might segment the data by country or date.
Often, we need to aggregate data to prepare it for subsequent...