Lab 07 Visualizing in ggplot2
URBAN 520 Quantitative Methods in Urban Design and Planning
- Data Preparation
- Tidy Data
- Missing Values
- First
Use of
ggplot2 - Visualizing Time: A Deeper Example
- More Visualization in R
In this lab, we are going to explore tidy data, start to visualize
data using R, and produce nice diagram using ggplot2.
Please try to understand each step, including the rational, functions,
and arguments. The dataset we are using is residential real
estate rental listing data, from RentHub and can be accessed through Dewey
Data Platform (UW NetID required).
Recommended Reading
Chapter 2, 6, Modern Data Science with R. Baumer et al. 2024.
Tidy Data, The Journal of Statistical Software, Hadley Wickham, 2014.
library('tidyverse')
library('ggplot2') # the major package for visualization in R
library('skimr')Data Preparation
Instead of using API or R data package as we did in previous labs, we
will use data on our local device. We have two datasets in csv file
rent_city_wide.csv and rental_data_wa.csv. Comma-separated
values (CSV) is a text data format that uses commas to separate
delimiter-separated values, and newlines to separate records.
Rental Listing Data
RentHub provides us residential real estate rental listing data, including pricing, property information, location, and more from 2014 to present. The whole dataset is huge (over 100 GB). Therefore, we only select all data for the State of Washington during 2019-01-01 to 2023-12-31. Note: the time refers to when RentHub collected the data, rather than when the listing was originally posted. To keep the file small, we further keep listings posted between 2019 and 2023 and 17 of the 25 columns. You can access the whole data through Dewey Data Platform (UW NetID required).
# load the rental listing data, it may take a while because its size
rental <- read_csv("rental_data_wa.csv")## Rows: 2268590 Columns: 17
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (11): CITY, BUILDING_TYPE, GRANITE, STAINLESS, POOL, GYM, DOORMAN, CLUB...
## dbl (5): ZIP, BEDS, BATHS, SQFT, RENT_PRICE
## date (1): DATE_POSTED
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
There are 2,268,590 observations (rows) and 17 variables (columns) in total. The dataset is already tidy, so no further transformation is required. You can take a look at the meaning of each column.
Click to Read Data Dictionary
- DATE_POSTED: Date listing observation was posted on source
- CITY: City in which listing observation is located
- ZIP: Property zip
- BUILDING_TYPE: Type of property referenced in listing observation
- BEDS: Number of bedrooms in listing observation
- BATHS: Number of bathrooms in listing observation
- SQFT: Approximate square footage in a listing observation
- RENT_PRICE: Advertised rent price for a listing observation
- GRANITE: The term “granite” is mentioned within the marketing description
- STAINLESS: The term “stainless” is mentioned within the marketing description
- POOL: The term “pool” is mentioned within the marketing description
- GYM: The term “gym” or “fitness center” or “health club” is mentioned within the marketing description
- DOORMAN: The terms “doorman” or “attended lobby” is mentioned within the marketing description
- FURNISHED: The term “furnished” is mentioned within the marketing description
- LAUNDRY: The terms “laundry” or “washer” or “washer/dryer” or “W/D” is mentioned within the marketing description
- GARAGE: The term “garage” is mentioned within the marketing description
- CLUBHOUSE: The term “clubhouse” is mentioned within the marketing description
# skim the whole dataset
# skim(rental)Aggregation Rental Listing Data by City
We have organized the average monthly rent of the 10 Washington cities with the most rental listings from 2019 to 2023 to have a smaller dataset.
rent_city <- read_csv("rent_city_wide.csv")## Rows: 10 Columns: 63
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (2): CITY, COUNTY
## dbl (61): LISTINGS, 2019-01-01, 2019-02-01, 2019-03-01, 2019-04-01, 2019-05-...
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
CSV files are just plain text. You can use your text
editor (TextEdit on Mac and Notepad on Windows) to have a look. They do
not store metadata such as column types, units, or formats. So. when we
use read_csv(), it automatically guesses column types by
inspecting the data. Sometimes its guess is wrong, we need to specify
the types using spec().
rent_city <- read_csv("rent_city_wide.csv",
col_types = cols(
CITY = col_character(),
COUNTY = col_character(),
LISTINGS = col_number(),
.default = col_double() # for other columns
)
)
head(rent_city, 3)## # A tibble: 3 × 63
## CITY COUNTY LISTINGS `2019-01-01` `2019-02-01` `2019-03-01` `2019-04-01`
## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Seattle King 585257 2049 2140 2178 2199
## 2 Renton King 126745 1861 1874 1872 1935
## 3 Tacoma Pierce 115160 1557 1584 1549 1587
## # ℹ 56 more variables: `2019-05-01` <dbl>, `2019-06-01` <dbl>,
## # `2019-07-01` <dbl>, `2019-08-01` <dbl>, `2019-09-01` <dbl>,
## # `2019-10-01` <dbl>, `2019-11-01` <dbl>, `2019-12-01` <dbl>,
## # `2020-01-01` <dbl>, `2020-02-01` <dbl>, `2020-03-01` <dbl>,
## # `2020-04-01` <dbl>, `2020-05-01` <dbl>, `2020-06-01` <dbl>,
## # `2020-07-01` <dbl>, `2020-08-01` <dbl>, `2020-09-01` <dbl>,
## # `2020-10-01` <dbl>, `2020-11-01` <dbl>, `2020-12-01` <dbl>, …
We will not see the message if we specified the data types. let’s have a look of the first 3 rows of this data. The data is the average rent of all listings posted in each month. The data includes average rent for 10 cities from 2019 to 2023 by month!
Currently, the rent_city dataset has 10 rows and 63
columns. This presentation of the data has some advantages: we can
easily look through the average rent for single city by date (one row).
But, we have 60 columns for average rent, which treated as separate
variables now. In data science, researchers defined a particular format
of data table: tidy data - A research paper
about tidy data.
Tidy Data
Data Tidying
This section was adopted from Tidy data page, authored by Hadley Wickham.
Happy families are all alike; every unhappy family is unhappy in its own way — Leo Tolstoy
Like families, tidy datasets are all alike but every messy dataset is messy in its own way. The principles of tidy data provide a standard way to organize data values within a dataset. A standard makes initial data cleaning easier because you don’t need to start from scratch and reinvent the wheel every time. The tidy data standard has been designed to facilitate initial exploration and analysis of the data, and to simplify the development of data analysis tools that work well together.
Data Semantics
A dataset is a collection of values, usually either numbers or strings. Values are organised in two ways: every value belongs to a variable and an observation.
- A variable contains all values that measure the same underlying attribute (like height, temperature, duration) across units.
- An observation contains all values measured on the same unit (like a person, or a day, or a race) across attributes.
head(rent_city, 3)## # A tibble: 3 × 63
## CITY COUNTY LISTINGS `2019-01-01` `2019-02-01` `2019-03-01` `2019-04-01`
## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Seattle King 585257 2049 2140 2178 2199
## 2 Renton King 126745 1861 1874 1872 1935
## 3 Tacoma Pierce 115160 1557 1584 1549 1587
## # ℹ 56 more variables: `2019-05-01` <dbl>, `2019-06-01` <dbl>,
## # `2019-07-01` <dbl>, `2019-08-01` <dbl>, `2019-09-01` <dbl>,
## # `2019-10-01` <dbl>, `2019-11-01` <dbl>, `2019-12-01` <dbl>,
## # `2020-01-01` <dbl>, `2020-02-01` <dbl>, `2020-03-01` <dbl>,
## # `2020-04-01` <dbl>, `2020-05-01` <dbl>, `2020-06-01` <dbl>,
## # `2020-07-01` <dbl>, `2020-08-01` <dbl>, `2020-09-01` <dbl>,
## # `2020-10-01` <dbl>, `2020-11-01` <dbl>, `2020-12-01` <dbl>, …
Look at our rent_city dataset, it contains 630 values
representing 63 variables and 10
observations. The variables are:
CITY: the name of 10 citiesCOUNTY: county nameLISTINGS: the number of rental listings in the city from 2019 to 2023- 60 columns representing months: average rent of listings posted in that month
A tidy version of the data looks like this:
rent_city_tidy <- rent_city %>%
pivot_longer( # using `pivot_longer` to reshape wide → long
cols = -c("CITY", "COUNTY", "LISTINGS"), # keep these 3 as identifier variables (don’t pivot them)
names_to = "month", # old column names (like "2019-01-01") become a new column called month
values_to = "mean_rent" # the cell values go into a new column called mean_rent
# values_drop_na = TRUE; default as FALSE; if TRUE, will remove rows with missing mean_rent.
) %>%
mutate(
month = as.Date(month, format = "%Y-%m-%d")
)
head(rent_city_tidy,3)## # A tibble: 3 × 5
## CITY COUNTY LISTINGS month mean_rent
## <chr> <chr> <dbl> <date> <dbl>
## 1 Seattle King 585257 2019-01-01 2049
## 2 Seattle King 585257 2019-02-01 2140
## 3 Seattle King 585257 2019-03-01 2178
Look at our rent_city_tidy dataset again, it contains
3,000 values representing 5 variables and
600 observations. The variables are:
CITY: the name of 10 citiesCOUNTY: county nameLISTINGS: the number of rental listings in the city from 2019 to 2023month: NEW - the month listings were postedmean_rent: NEW - average rent of listings posted in that month
The tidy data frame explicitly tells us the definition of an
observation. Every combination of CITY and
month is a single measured observation
(mean_rent). It’s important to clearly define what
is an observation. In the case of rent, we are interested in
average rent for sure! For an observation, such as 2049, we
need know two information: city and month.
Tidy data
In tidy data:
- Each variable is a column; each column is a variable.
- Each observation is a row; each row is an observation.
- Each value is a cell; each cell is a single value.
Tidy data makes it easy for an analyst or a computer to extract
needed variables because it provides a standard way of structuring a
dataset. Tidy data is particularly well suited for vectorised
programming languages like R, because the layout ensures that values of
different variables from the same observation are always paired. Many R
packages, such as ggplot2 are designed around the tidy data
principles.
From Tidy Data to Wider Data
We can use pivot_wider() to perform the inverse
operation.
rent_city_wide <- rent_city_tidy %>%
pivot_wider(
names_from = month, # expand `month` to multiple columns
values_from = mean_rent # extract `mean_rent` column to those new columns
)
head(rent_city_wide, 3)## # A tibble: 3 × 63
## CITY COUNTY LISTINGS `2019-01-01` `2019-02-01` `2019-03-01` `2019-04-01`
## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Seattle King 585257 2049 2140 2178 2199
## 2 Renton King 126745 1861 1874 1872 1935
## 3 Tacoma Pierce 115160 1557 1584 1549 1587
## # ℹ 56 more variables: `2019-05-01` <dbl>, `2019-06-01` <dbl>,
## # `2019-07-01` <dbl>, `2019-08-01` <dbl>, `2019-09-01` <dbl>,
## # `2019-10-01` <dbl>, `2019-11-01` <dbl>, `2019-12-01` <dbl>,
## # `2020-01-01` <dbl>, `2020-02-01` <dbl>, `2020-03-01` <dbl>,
## # `2020-04-01` <dbl>, `2020-05-01` <dbl>, `2020-06-01` <dbl>,
## # `2020-07-01` <dbl>, `2020-08-01` <dbl>, `2020-09-01` <dbl>,
## # `2020-10-01` <dbl>, `2020-11-01` <dbl>, `2020-12-01` <dbl>, …
Missing Values
In pivot_longer, there is an optional argument called
values_drop_na =, default as FALSE. If setting
as TRUE, it will remove rows with missing
mean_rent. We set as FALSE but is there any
NA (Not Available) values?
We can use is.na() to check and there are 8 missing
value in mean_rent column:
colSums(is.na(rent_city_tidy))## CITY COUNTY LISTINGS month mean_rent
## 0 0 0 0 8
We can remove NA or smooth/impute them based on their neighbors. We will just remove them here because those 8 missing values do not impact too much on our visualization. But, there are many ways you can do data interpolation. It can based on known temporal neighbors, spatial neighbors, group-based neighbors. It goes beyond this course but it’s an important problem for data science. If you are interested in missing data, you can refer to this book: Statistical Analysis with Missing Data.
# remove NA then we can see there are 592 rows afterwards, 8 missing values are removed
rent_city_tidy_NA_remove <- rent_city_tidy %>%
filter(!is.na(mean_rent))
dim(rent_city_tidy_NA_remove)## [1] 592 5
📚 TODO: Handle Missing Values
Instead of omitting those missing values, let’s try filling missing
values in rent_city_tidy using a linear
interpolation method. You can use the zoo package,
which provides a convenient function na.approx() for
handling time-series data.
The documentation and this post may be helpful to you!
# TODOFirst Use of ggplot2
Starting from this lab session, we are going to conduct data
visualization using R and ggplot2. You don’t have to
install ggplot2 separately because it is already a part of
tidyverse. ggplot2 is built on the Grammar of
Graphics — a system that describes plots as combinations of independent
components. This is a typical chunk of ggplot2:
ggplot(data, aes(x = ..., y = ...)) +
geom_*() +
facet_*() +
scale_*() +
coord_*() +
theme()
Let’s use the average rent of Seattle and Tacoma to complete
ggplot2 figure step by step to understand how each
component contributes. We defined one objects with average rent of both
Seattle and Tacoma (rent_sea_tac).
rent_sea_tac <- rent_city_tidy_NA_remove %>%
filter(CITY %in% c("Seattle", "Tacoma"))Starting from ggplot(), we create a plot object. We can
define the data and and aes (or later). We
have argument aes(), which defines how data fields (e.g.,
categorical, ordinal) map to visual properties/channels
(e.g., x, y, color, shape, size).
If we assign this ggplot() object to g, we
can later add anything upon this object using +. Otherwise,
the plot will show here. ggplot() alone initializes a
coordinate system and data mapping, so there is nothing on the
diagram.
ggplot(rent_sea_tac, aes(x = month, y = mean_rent, group = CITY)) # will show below the chunk# what will happen if we do not set group = CITY
g <- ggplot(rent_sea_tac, aes(x = month, y = mean_rent, group = CITY)) # will not show the plotsIn ggplot2, defining geom_*() (geometric
object) is required. What do we want in this plot? Points or lines? We
can stack more than one geom_*() in a plot and each
geom_*() function will return a layer.
Within geom_*() functions, we can modify the visual
attributes of this layer. Note: different geometric objects have
different arguments.
g <- g + geom_line(aes(color = CITY),
alpha = 0.8, # Opacity: 0 means fully transparent (invisible)
linetype = 6, # 6 = twodash, you can check more option using `?linetype`
linewidth = 0.4) # linewidth
gThis is a confusing thing here: we added
aes(color = CITY) within geom_line(). Why did
we use CITY twice, one in ggplot() one in
geom_line()?
ggplot(rent_sea_tac, aes(x = month, y = mean_rent, group = CITY))
defines global mappings: defaults that all layers geom_*()
will inherit. This line just say we have month as X and average rent as
y, and data are organized into two groups based on CITY.
Color is a new aesthetic mapping that we can introduce
to this layer! Also, it’s good to know that if we want
ggplot2 to both group and color by city,
we could directly define (introduce color to the global
setting):
ggplot(rent_sea_tac, aes(x = month, y = mean_rent, color = CITY)) + geom_line().
Visualizing Time: A Deeper Example
Between 2019 and 2024, one of the most significant global events was the COVID-19 pandemic, which profoundly affected housing and rental markets. In this analysis, we are interested in rent resiliency across different groups after the pandemic — that is, how quickly and to what extent rents in various apartment types and amenity levels recovered following the COVID-19 shock.
First, we filter the rental dataset to apartment buildings only and add four useful variables:
month: the posting month of each listing, for time-series analysisbed_group: bedroom categories, for comparing rent trends by unit sizeamenities_sum: the total number of key amenitiesamenity_group: groups of listings based on the values ofamenities_sum(0-1, 2-4, 5-6)
We call it the amenity group rather than quality:
amenities_sum counts words in the marketing description,
not the condition of the unit. In December 2019, 1- and 2-bedroom
listings with 0-1 amenities actually rented for more than those with
2-4.
rental_apt <- rental %>%
filter(BUILDING_TYPE == "apartment building") %>%
mutate(
# converts posting date into the first day of its month for monthly analysis
month = as.Date(paste0(format(as.Date(DATE_POSTED), "%Y-%m"), "-01")),
# categorizes apartments by the number of bedrooms
bed_group = case_when(
BEDS == 0 ~ "Studio",
BEDS == 1 ~ "1 Bedroom",
BEDS == 2 ~ "2 Bedrooms",
BEDS >= 3 ~ "3+ Bedrooms",
TRUE ~ "Unknown"),
# creates a variable counting how many of six amenities each listing offers
amenities_sum = (
(CLUBHOUSE == "Y") +
(GRANITE == "Y") +
(STAINLESS == "Y") +
(POOL == "Y") +
(GYM == "Y") +
(DOORMAN == "Y")),
# groups apartments by how many of the six amenities are mentioned
amenity_group = case_when(
amenities_sum <= 1 ~ "0-1",
amenities_sum %in% 2:4 ~ "2-4",
amenities_sum >= 5 ~ "5-6",
TRUE ~ "Unknown"
),
# convert two variables from categorical to ordinal
bed_group = factor(bed_group,
levels = c("Studio", "1 Bedroom", "2 Bedrooms", "3+ Bedrooms")),
amenity_group = factor(amenity_group,
levels = c("0-1", "2-4", "5-6"))
)Then, let’s calculate median monthly rent broken down by amenity group and bedroom category. We use the median rather than the mean because a few listings above $10,000 pull the mean around (see the histogram and boxplot in More Visualization below). We also count the listings in each group. Notes: It is important to note that decisions about grouping, variable construction, and filtering should ideally be informed by exploratory data analysis and domain knowledge. For simplicity and clarity, these exploratory steps are not shown in this tutorial.
apt_rent_by_month <- rental_apt %>%
# group by three variables: month, amenity_group, bed_group
group_by(month, amenity_group, bed_group) %>%
dplyr::summarise(median_rent = median(RENT_PRICE, na.rm = TRUE),
listings = n(), # number of listings in each group
.groups = "keep")Let’s produce the first plot about monthly rent trends for apartment buildings, separated by bedroom type and amenity group. We have four variables: time, median rent, bedroom group, amenity group. We need to determine the aesthetics (channels/encoding variables).
Recall the Expressiveness + Effectiveness we discussed: positions are always the best, so we put two important variables (time, median rent) as x and y.
Amenity group and bedroom group are ordinal categorical variables. We
need to think about whether we want to preserve this order. If
preserving order is not essential for our analytically purpose, we may
treat them simply as nominal categories. In the current design, amenity
group (amenity_group) is represented by color, while
bedroom type (bed_group) is shown through faceting.
ggplot(apt_rent_by_month, aes(x = month, y = median_rent, color = amenity_group)) +
geom_line() +
facet_wrap(~bed_group)+
labs(
title = "Monthly Apartment Rent Trend 2019–2023",
x = "Time",
y = "Median Rent Price",
color = "Amenities Advertised (of 6)"
) +
theme_minimal() +
theme(legend.position = "bottom")In general, the figure looks quite good after some adjustments if our goal is simply to show overall rent trends. However, the key message I’d like to emphasize is the rent resiliency across different groups after the pandemic—that is, how quickly and strongly rents in various apartment types and amenity levels recovered following the COVID-19 shock. The absolute rent levels are not particularly meaningful on their own, so we can make an adjustment by converting them into relative rents, measured as rent changes compared to December 2019 levels (pre-pandemic).
The lines are also jumpy. Some groups have only 30-50 listings a month (3+ bedrooms with 5-6 amenities, for example), so one unusual month can move a line a lot. We smooth each line with a 3-month moving average (the previous, current and next month) before comparing it with December 2019.
apt_rent_rel <- apt_rent_by_month %>%
# compute relative change within each group
group_by(amenity_group, bed_group) %>%
# put months in order, lag() and lead() use the row order
arrange(month, .by_group = TRUE) %>%
mutate(
# 3-month moving average: previous, this and next month
rent_smooth = (lag(median_rent) + median_rent + lead(median_rent)) / 3,
# smoothed rent in Dec 2019 as baseline
base_rent = rent_smooth[month == as.Date("2019-12-01")],
# rent change (e.g., +10%)
rent_change = (rent_smooth - base_rent)/base_rent
) %>%
ungroup()We are going to create a relatively complicated plot and you can have
a look of the final result at the end To achieve that, we need many
functions in ggplot2 and other software, such PowerPoint
and PhotoShop, to do final edits (like the notes inside).
Basic Structure
Similar as previous plot. We started from the data:
apt_rent_rel. We can do filter here to make sure the plot
will only show the time period we want: between December
2019 (just before the first Covid-19 case in Washington State)
and December 2022.
The we need to determine the aesthetics (channels/encoding variables), we map time to the x-axis, rent changes to the y-axis. There are two additional dimensions in the data: amenity group (amenity_group), which is represented by color, and bedroom type (bed_group), which is shown through faceting.
After that, we are going to add the first geometry:
geom_line. Its location is decided by the global
aes we already defined. We also add the
labels, including title, subtitle, caption (data source
and definitions), x and y axis title, and the label for color group.
g <- ggplot(
filter(apt_rent_rel, month >= as.Date("2019-12-01") & month <= as.Date("2022-12-01")),
aes(x = month, y = rent_change, color = amenity_group)
) +
geom_line(linewidth = 1.2) +
facet_wrap(~bed_group, nrow = 1) +
labs(
title = "Washington State Monthly Apartment Rent Changes Since Covid-19",
subtitle = "Change in median asking rent compared to December 2019, by number of bedrooms and amenities advertised in the listing",
caption = "Data: RentHub listings, December 2019 to December 2022. Lines are 3-month moving averages of the monthly median rent.\nAmenities: clubhouse, granite, stainless, pool, gym, doorman mentioned in the listing text.",
x = "Time",
y = "Relative Rent Change",
color = "Amenities Advertised (of 6)"
)
gGraphical Primitives
We would like emphasize the line \(y=0\) because it means no change compared
to the beginning of the pandemic. Meanwhile, we need to let readers know
the time period of the pandemic. So, we use geom_hline() to
add a horizontal line as another layer. (Remember to check Cheatsheet, R
documentation, and using ?function when you are not
familiar with a function.)
There is another way we can simply draw a line is adding an
annotation layer. But unlike a typical
geom function, the properties of the geoms are
not mapped from variables of a data frame, but are instead passed in as
vectors. This is useful for adding small annotations (such as text
labels or a line). We add a rectangular annotation to highlight a time
period, from the first Covid-19 case in Washington State (January 2020)
to the statewide reopening (June 30, 2021). -Inf and
Inf stretch it to the edges of each panel, and
alpha makes it transparent.
annotate() repeats the same text in every panel. To
label only one panel, we give geom_text() a one-row data
frame whose bed_group is Studio, so the text only shows in
the Studio panel.
# a one-row data frame for the label: bed_group decides the panel
band_label <- tibble(
bed_group = factor("Studio", levels = levels(apt_rent_rel$bed_group)),
month = as.Date("2020-01-20"),
rent_change = 0.57,
label = "Pandemic period\nfirst case to statewide reopening"
)
g_line_rec <- g +
# add a horizontal line across the plot at y = 0
geom_hline(yintercept = 0, linewidth = 0.6) +
annotate("rect", # a rectangular
# we need 4 points to determine the location
xmin = as.Date("2020-01-01"), xmax = as.Date("2021-07-01"),
ymin = -Inf, ymax = Inf, # from the bottom to the top of each panel
fill = "grey50", alpha = 0.12) + # explain below
geom_text(data = band_label, # the label only shows in the Studio panel
aes(label = label),
color = "black",
size = 3.6,
hjust = 0, vjust = 1, # the top-left corner of the text sits at (x, y)
fontface = "italic") # explain below
g_line_recFontface
fontface/face is a catch-all argument that
describes the style of the typeface to use. “plain” is an upright
normal-weight font, “italic” is a slanted normal-weight font, “bold” is
an upright bold-weight font, and “bold.italic” is a slanted bold-weight
font.
Colors
We can use R built-in colors (600+), you can check their names using
colors().
We can also use Hex colors and palette functions.
Hex colors are written as #RRGGBB, where each pair (RR, GG,
BB) represents the intensity of Red, Green, and Blue in hexadecimal
(from 00 to FF, i.e., 0–255). ggplot2 provides built-in palette
functions that automatically generate harmonious color sets for
categorical or continuous variables.
R also supports numeric color systems like RGB, HCL, and HSV.
# colors()
# scale_color_brewer(palette = "Set2")
# scale_color_viridis_d(option = "plasma")Scale
Scale controls how data values are mapped to
aesthetics (channels/encoding variables). For example, we use
color = amenity_group and by default ggplot2
will choose some colors for that. But we can change how color encodes
through scale_color_*.
| Aesthetic | Scale function | Controls |
|---|---|---|
| x-axis | scale_x_*() |
axis range, tick marks, labels |
| y-axis | scale_y_*() |
axis range, tick marks, labels |
| color | scale_color_*() |
how color encodes categories or values |
| fill | scale_fill_*() |
fill colors for bars, areas, etc. |
| size | scale_size_*() |
point or line thickness |
| shape | scale_shape_*() |
point shapes |
| alpha | scale_alpha_*() |
transparency |
Viridis is the most colorblind-friendly, perceptually-uniform color maps used in data visualization. Read more about viridis palettes. You can use those common commands for color or fill:
scale_fill_viridis_d("palette name"): fill discrete variable with viridisscale_fill_viridis_c("palette name"): fill continuous variable with viridisscale_color_viridis_d("palette name"): color discrete variable on the border with viridisscale_color_viridis_c("palette name"): color continuous variable on the border with viridis
g_scale <- g_line_rec +
# change how assigns colors to the variable amenity_group
scale_color_manual(values = c("0-1" = "#ACA3CB", "2-4" = "#685A8E", "5-6" = "#32006E")) +
# how to use an viridis palette here?
scale_x_date(
breaks = as.Date(c("2020-01-01", "2021-01-01", "2022-01-01")),
labels = c("2020", "2021", "2022")
) +
# formats the numeric values into percentages with a plus or minus sign
scale_y_continuous(
breaks = seq(-0.2, 0.6, by = 0.2),
labels = function(x) sprintf("%+.0f%%", x * 100)
)
g_scaleCoordinate System
The coordinate system determines how data coordinates are translated to positions on the plot. We will talk about this more in the future class on mapping and geospatial data.
g_coordinate <- g_scale +
# limits the y-axis to between −0.2 and 0.6, without extra space around the data
coord_cartesian(ylim = c(-0.2, 0.6), expand = FALSE)
g_coordinateTheme
Theme controls all non-data visual components, text, background, grid lines, margins, etc.
Components
First, we need to know how ggplot2 names those
components:
You can see there is a hierarchy among those plot components. We list some here as the example, you can read all components of a theme from its official document.
- For the whole plot
element_text(): text, plot.title, plot.subtitle, plot.captionelement_rect(): plot.background, plot.margin
- For x and y axis
element_text(): axis.title.x/y, axis.text.x/yelement_line(): axis.line.x/y, axis.ticks.x/y
- For grid and panel
element_line(): panel.grid; panel.grid.major, panel.grid.minorelement_rect(): panel.borderunit(): panel.spacing
- For the legend
element_text(): legend.title, legend.textlocation using character/vector: legend.position, legend.justification, legend.margin
- For facet
element_text(): strip.text, strip.text.x, strip.text.yelement_rect(): strip.background
Elements
There are several types of components, ggplot2 calls
them elements, such as text, line, and rectangular. For example, the
line of axis is element_line while the title of axis is
element_text. We remove some components through assigning
them as element_blank().
We change the attributes of those components (such as legend.title,
axis.line) by assigning them an appropriate element_*()
function inside theme(). Please refer to its official
document or ?element_*() for detailed arguments - Theme
elements.
Complete Themes
Beyond those specific theme configuration, we can start with existing
complete themes by + theme_*(). The one I like most is
theme_minimal() because it is super clean. Within
theme_*(), we can set a few argument for the whole plot:
base_family for default font, base_size for
default font size, for example.
Theme Inheritance
It is useful to know theme inheritance feature of
ggplot2: theme elements inherit properties from other theme
elements hierarchically. For example, there is the hierarchy for text
elements:
text
├── plot.title
├── plot.subtitle
├── axis.title
├── axis.title.x / axis.title.y
└── legend.text / legend.title
Modify our Plot using Theme
g_theme <- g_coordinate +
theme_minimal(base_family = "Helvetica", base_size = 13) +
theme(
# ---------------- for the whole plot ----------------
text = element_text(lineheight = 1.1), # set line height for all text
plot.title = element_text(face = "bold", size = 18), # make title bold and large
plot.subtitle = element_text(margin = margin(b = 10)), # space below the subtitle
plot.caption = element_text(hjust = 0, color = "grey30", margin = margin(t = 12)), # left-aligned caption
plot.title.position = "plot", # align title and caption with the whole plot, not the panels
plot.caption.position = "plot",
plot.margin = margin(4, 12, 4, 4), # external margins (top, right, bottom, left)
# ---------------- for x and y axis ----------------
axis.title.x = element_blank(), # remove axis.title.x
axis.title.y = element_blank(),
axis.ticks.x = element_line(linewidth = 0.4), # style tick marks
# ---------------- for grid and panel ----------------
panel.grid.major.x = element_blank(), # remove vertical grid lines
panel.grid.minor = element_blank(), # remove minor grid lines
panel.grid.major.y = element_line(color = "grey85", linewidth = 0.3), # keep light horizontal lines to read values
panel.spacing.x = unit(2, "lines"), # spacing between facets horizontally
# ---------------- for the legend ----------------
legend.position = "top", # place legend above the plot
legend.justification = "left",
legend.title = element_text(face = "bold"),
# ---------------- for facet ----------------
strip.background = element_blank(),
strip.text.x = element_text(face = "bold", # facet labels
size = 14,
hjust = 0, # left-aligned
margin = margin(b = 8)) # bottom margin (t/r/b/l)
)
g_themeThe Complete PLot
ggplot(
filter(apt_rent_rel, month >= as.Date("2019-12-01") & month <= as.Date("2022-12-01")),
aes(x = month, y = rent_change, color = amenity_group)
) +
# 1️⃣ geometry
annotate("rect",
xmin = as.Date("2020-01-01"), xmax = as.Date("2021-07-01"),
ymin = -Inf, ymax = Inf,
fill = "grey50", alpha = 0.12) +
geom_hline(yintercept = 0, linewidth = 0.6) +
geom_line(linewidth = 1.2) +
geom_text(data = band_label, aes(label = label),
color = "black", size = 3.6, hjust = 0, vjust = 1, fontface = "italic") +
# 2️⃣ scale
scale_color_manual(values = c("0-1" = "#ACA3CB", "2-4" = "#685A8E", "5-6" = "#32006E")) +
scale_x_date(
breaks = as.Date(c("2020-01-01", "2021-01-01", "2022-01-01")),
labels = c("2020", "2021", "2022")
) +
scale_y_continuous(
labels = function(x) sprintf("%+.0f%%", x * 100),
breaks = seq(-0.2, 0.6, by = 0.2)
) +
# 3️⃣ facet and coordinate system
facet_wrap(~bed_group, nrow = 1) +
coord_cartesian(clip = "off", ylim = c(-0.2, 0.6), expand = FALSE) +
# 4️⃣ labels
labs(
title = "Washington State Monthly Apartment Rent Changes Since Covid-19",
subtitle = "Change in median asking rent compared to December 2019, by number of bedrooms and amenities advertised in the listing",
caption = "Data: RentHub listings, December 2019 to December 2022. Lines are 3-month moving averages of the monthly median rent.\nAmenities: clubhouse, granite, stainless, pool, gym, doorman mentioned in the listing text.",
x = "Time",
y = "Relative Rent Change",
color = "Amenities Advertised (of 6)"
) +
# 5️⃣ theme
theme_minimal(base_family = "Helvetica", base_size = 13) +
theme(
text = element_text(lineheight = 1.1),
plot.title = element_text(face = "bold", size = 18),
plot.subtitle = element_text(margin = margin(b = 10)),
plot.caption = element_text(hjust = 0, color = "grey30", margin = margin(t = 12)),
plot.title.position = "plot",
plot.caption.position = "plot",
axis.title.x = element_blank(),
axis.ticks.x = element_line(linewidth = 0.4),
axis.title.y = element_blank(),
legend.position = "top",
legend.justification = "left",
legend.title = element_text(face = "bold"),
panel.grid.major.x = element_blank(),
panel.grid.minor = element_blank(),
panel.grid.major.y = element_line(color = "grey85", linewidth = 0.3),
strip.background = element_blank(),
strip.text.x = element_text(face = "bold", size = 14, hjust = 0, margin = margin(b = 8)),
panel.spacing.x = unit(2, "lines"),
plot.margin = margin(4, 12, 4, 4)
)# save it as 14 inch * 7 inch
ggsave("rent_change_plot.png", width = 14, height = 7, dpi = 200)
# save it as 7 inch * 3.5 inch
ggsave("rent_change_plot_735.png", width = 7, height = 3.5, dpi = 200)When working with ggplot2, customizing the theme()
controls visual style — but the final output size and aspect ratio are
just as important. RStudio preview IS NOT the actual
exported figure. I will recommend that you work within a single chunk
and check changes in exported figure all the time. I decided to use 14 *
7 inch at the beginning so it is perfect when I export as 14 * 7 inch.
But the following figure was exported as 7 * 3.5 inch: everything
squeezed together.
Final Edits
Not every figure — or every visual component within a figure — is best created entirely in R.
While ggplot2 is powerful for data-driven visualization,
it’s not always flexible for fine-grained graphic design tasks. We can
always combine R/ggplot2 with other tools, like PowerPoint
and Photoshop. Here, we added an explanation on pandemic and a
directional line using Photoshop.
More Visualization in R
Histograms
A histogram summarizes the distribution of the data by placing observations into intervals (also called classes or bins) and counting the number of observations in each interval.
ggplot(data = rental, aes(x = RENT_PRICE)) +
geom_histogram(
binwidth = 200, # width of each bin, in dollars
color = "white", # border color
alpha = 0.7 # transparency
) +
scale_x_continuous(labels = scales::dollar) +
scale_y_continuous(labels = scales::comma) +
coord_cartesian(xlim = c(0, 8000)) + # zoom in; listings above $8,000 are still counted
labs(title = "Histogram of Rent Price", x = "Rent Price", y = "Count of Listings") +
theme_minimal()Boxplots
A box and whisker plot (also referred to as boxplot) provides a compact summary of the distribution of a variable.
ggplot(
data = rental %>%
filter(CITY %in% rent_city$CITY), # the 10 cities in `rent_city`
aes(x = RENT_PRICE, y = reorder(CITY, RENT_PRICE, FUN = median)) # horizontal boxes, sorted by median rent
) +
geom_boxplot(
outlier.size = 0.2,
fill = "black",
alpha = 0.3, # transparency
width = 0.5 # box width
) +
labs(title = "Boxplot of Rent Price by City", x = "Rent Price", y = NULL) +
scale_x_continuous(labels = scales::dollar) +
coord_cartesian(xlim = c(0, 8000)) + # zoom in; the boxes are still computed from all listings
theme_minimal()Cumulative Distribution Functions (CDF)
The cumulative distribution function CDF is a function \(F(x)\) that is the probability that the observations of a variable are not larger than a specified value. In constructing the CDF, weights (e.g., inclusion probabilities from a probability design) can be used. In this way the probability that a value of the variable in the statistical population is less than a specified value is estimated. Otherwise, for equal weighting of observations, the CDF applies only to the observed values.
\[F(x) = P(X \leq x) = \text{Probability of Random Variable}\space X \space \text{is not greater than} \space x\]
ggplot(data = rental %>%
filter(CITY %in% c("Seattle", "Tacoma", "Spokane"))) +
stat_ecdf(aes(x = RENT_PRICE),) + # stat builds new variables to plot, we build CDF
facet_wrap(~CITY)+ # it is not the best option to use facet
scale_x_continuous(labels = scales::dollar, breaks = c(0, 2000, 4000)) +
coord_cartesian(xlim = c(0, 6000)) + # zoom in on where the curves rise
labs(
title = "CDF of Rent Price in Seattle, Spokane and Tacoma",
subtitle = "F(x) = P(X ≤ x)",
y = "Cumulative Distribution Function (CDF)",
x = "Rent Price"
) +
theme_minimal()Scatter Plot
Scatterplots are graphical displays of matched data plotted with one variable on the horizontal axis and the other variable on the vertical axis. Data are usually plotted with measures of an influential parameter on the horizontal axis (independent variable) and measures of an attribute that may respond to the influential parameter on the vertical axis (dependent variable). Scatterplots are a useful first step in any analysis because they help visualize relationships and identify possible issues (e.g., outliers) that can influence subsequent statistical analyses.
set.seed(520) # set seed, then the random sample below stays the same
ggplot(
data = rental %>%
filter(!is.na(SQFT)) %>%
sample_n(2000), # a random sample of 2,000 listings
aes(x = SQFT, y = RENT_PRICE)
) +
geom_point(size = 0.5, alpha = 0.6) +
geom_smooth( # we use a sample linear regression to model the relationship (trends)
method = "lm",
formula = y ~ x,
se = TRUE, # show confidence interval
color = "#4B2E83"
) +
scale_x_continuous(labels = scales::comma) +
scale_y_continuous(labels = scales::dollar) +
labs(title = "Scatter Plot of Square Footage and Rent Price", x = "Square Footage", y = "Rent Price") +
theme_minimal()2D Histogram
With 2 million listings, the points of a scatter plot cover each
other, which is why we took a sample of 2,000 above. A 2D histogram
(geom_bin_2d()) uses all of them: like a histogram, it cuts
the plane into small rectangles, counts the listings in each, and colors
each rectangle by the count. The color uses a log scale, because the
densest rectangles hold over 10,000 listings and most hold far
fewer.
ggplot(
data = rental %>%
filter(!is.na(SQFT)), # all listings with square footage, no sampling
aes(x = SQFT, y = RENT_PRICE)
) +
geom_bin_2d(binwidth = c(50, 100)) + # each rectangle is 50 sq ft wide and $100 tall
scale_fill_viridis_c(option = "mako", direction = -1,
transform = "log10", # log scale: 1, 10, 100, 1,000, ...
labels = scales::comma) +
scale_x_continuous(labels = scales::comma) +
scale_y_continuous(labels = scales::dollar) +
coord_cartesian(xlim = c(0, 3000), ylim = c(0, 6000)) + # zoom in; all listings are still counted
labs(title = "2D Histogram of Square Footage and Rent Price",
x = "Square Footage", y = "Rent Price", fill = "Listings") +
theme_minimal()Bar Chart
A bar chart shows the count of each category; sorting the bars makes
the ranking easy to read. Here it also finds a data problem: the same
building type is written in different ways
(apartment building, Apartment,
apartment), and many listings are unknown or
missing.
ggplot(
data = rental %>%
mutate(BUILDING_TYPE = replace_na(BUILDING_TYPE, "(missing)")), # name the missing values, so they are sorted too
aes(y = fct_rev(fct_infreq(BUILDING_TYPE))) # sort the bars by count, largest on top
) +
geom_bar(alpha = 0.7) +
scale_x_continuous(labels = scales::comma) +
labs(title = "Listings by Building Type", x = "Count of Listings", y = NULL) +
theme_minimal()Heatmap
A heatmap is a graphical representation of data where the individual values contained in a matrix are represented as colors. It’s like looking at a data table from above, where each cell’s color intensity indicates the magnitude of the value it represents. Heatmaps are especially useful for identifying patterns, trends, and outliers in large datasets.
Here each cell is the median rent of one city and one bedroom group: reading across a row shows how rent rises with size, and reading down a column compares cities. Seattle studios ($1,530) rent for less than studios in Renton or Lynnwood, but Seattle 2-bedrooms are the second most expensive.
rent_city_bed <- rental %>%
filter(CITY %in% rent_city$CITY) %>% # the 10 cities in `rent_city`
mutate(
bed_group = case_when(
BEDS == 0 ~ "Studio",
BEDS == 1 ~ "1 Bedroom",
BEDS == 2 ~ "2 Bedrooms",
BEDS >= 3 ~ "3+ Bedrooms"),
bed_group = factor(bed_group, levels = c("Studio", "1 Bedroom", "2 Bedrooms", "3+ Bedrooms"))
) %>%
group_by(CITY, bed_group) %>%
summarise(median_rent = median(RENT_PRICE), .groups = "drop")
ggplot(rent_city_bed, aes(x = bed_group, y = reorder(CITY, median_rent), fill = median_rent)) + # sort cities by rent
geom_tile(color = "white", linewidth = 0.25) + # white lines between the cells
scale_fill_viridis_c(option = "mako", direction = -1, labels = scales::dollar) +
labs(title = "Median Rent by City and Number of Bedrooms", x = NULL, y = NULL, fill = "Median Rent") +
theme_minimal() +
theme(panel.grid = element_blank(),
legend.key.width = unit(2, "cm"), # a wider color bar
legend.position = "bottom") # Stacked Area Chart
We combine the labels into five groups with case_when()
and use them in the next chart. A stacked area chart shows how a total
and its parts change over time.
rental <- rental %>%
mutate(
building = case_when(
str_to_lower(BUILDING_TYPE) %in% c("apartment building", "apartment") ~ "Apartment",
str_to_lower(BUILDING_TYPE) %in% c("house", "single family house") ~ "House",
str_to_lower(BUILDING_TYPE) == "condo" ~ "Condo",
str_to_lower(BUILDING_TYPE) %in% c("townhouse", "duplex") ~ "Townhouse/Duplex",
TRUE ~ "Other/Unknown" # unknown, missing, Other, RNT, and Multi-family (see the next chart)
),
building = factor(building,
levels = c("Apartment", "House", "Condo", "Townhouse/Duplex", "Other/Unknown"))
)ggplot(
data = rental %>%
count(month = floor_date(DATE_POSTED, "month"), building) %>% # listings per month and group
complete(month, building, fill = list(n = 0)), # a group with no listings in a month gets 0, not a gap
aes(x = month, y = n, fill = building)
) +
geom_area(position = position_stack(reverse = TRUE)) + # the first group (Apartment) at the bottom
scale_fill_viridis_d(option = "mako", begin = 0.15, end = 0.85) + # skip the near-black and near-white ends
scale_x_date(date_breaks = "1 year", date_labels = "%Y") +
scale_y_continuous(labels = scales::comma) +
labs(title = "Listings per Month by Building Type", x = NULL, y = "Count of Listings", fill = NULL) +
theme_minimal() +
theme(legend.position = "bottom")Dumbbell Chart
A dumbbell chart connects two values of the same category, so the length of the line is the message. Among 1-bedroom listings in apartment buildings, those that mention a doorman ask $660 more (median) than those that do not, and those that mention a pool ask $301 less. That does not mean a pool lowers rent: only 9% of these listings in Seattle mention a pool, compared with 45% elsewhere, and Seattle rents are higher. A simple comparison like this mixes the amenity with location.
amenity_gap <- rental %>%
filter(BUILDING_TYPE == "apartment building", BEDS == 1) %>%
pivot_longer(GRANITE:GARAGE, # the nine amenity columns
names_to = "amenity", values_to = "mentioned") %>%
group_by(amenity) %>%
summarise(without = median(RENT_PRICE[mentioned == "N"]),
with = median(RENT_PRICE[mentioned == "Y"])) %>%
mutate(amenity = fct_reorder(str_to_title(amenity), with - without)) # sort by the gap
ggplot(amenity_gap, aes(y = amenity)) +
geom_segment(aes(x = without, xend = with, yend = amenity), color = "grey75", linewidth = 1) +
geom_point(aes(x = without), color = "coral4", size = 3) +
geom_point(aes(x = with), color = "navyblue", size = 3) + # you can use `colors()` to check built-in color names
scale_x_continuous(labels = scales::dollar) +
labs(title = "1-Bedroom Apartment Rent With and Without Each Amenity",
subtitle = "Median rent. Blue: amenity mentioned in the listing; red: not mentioned",
x = NULL, y = NULL) +
theme_minimal()