Lab 09 Hypothesis Testing in R

URBAN 520 Quantitative Methods in Urban Design and Planning


In the previous lab, we estimated single quantities for King County households, such as the share with a high energy burden, and checked whether their confidence intervals contain a benchmark. In this lab, we will ask the similar question as a hypothesis test, and then use tests to compare groups: Do Seattle households pay less for electricity than households in the rest of the county? Are older homes heated differently from newer ones? Do older homes cost more to power? We will still use the 2020-2024 ACS microdata, kc_energy_2024.csv, and treat the 42,178 households as a simple random sample of King County households. The data dictionary is in the previous lab.


library('tidyverse')
energy <- read_csv("kc_energy_2024.csv") %>%
  mutate(era = factor(
    era,
    levels = c("Before 1940", "1940-1979", "1980-1999", "2000 or later")
  )) # in time order, not alphabetical

Which Test?

The type of the outcome and the number of groups decide the usual test.

Outcome Compared across Question Test R function
Numerical A benchmark Does the mean differ from a value? One-sample t-test t.test(y, mu = )
Numerical 2 groups Do the two means differ? Two-sample (Welch) t-test t.test(y ~ group)
Numerical Matched pairs Does the mean difference differ from 0? Paired t-test t.test(y1, y2, paired = TRUE)
Numerical 3 or more groups Do any of the means differ? One-way ANOVA aov(y ~ group) + TukeyHSD()
Skewed numerical or ordinal 2 groups Does one group tend to have larger values? Wilcoxon rank-sum (Mann-Whitney) test wilcox.test(y ~ group)
Skewed numerical or ordinal 3 or more groups Do some groups tend to have larger values? Kruskal-Wallis test kruskal.test(y ~ group) + pairwise.wilcox.test()
Binary A benchmark Does the proportion differ from a value? One-sample proportion z-test prop.test(x, n, p = )
Binary 2 groups Do the two proportions differ? Two-proportion z-test prop.test(c(x1, x2), c(n1, n2))
Categorical Categories of another variable Are the two variables independent? Chi-square test chisq.test(table)

Under the hypothesis testing framework, almost every test follows the same steps:

  1. State \(H_0\) and \(H_1\), and choose a significance level \(\alpha\), commonly 0.05.
  2. Compute a test statistic: that measures how far the data are from what \(H_0\) predicts.
  3. Find the null distribution: the distribution of the test statistic if \(H_0\) were true.
  4. Compute the p-value: the probability, assuming \(H_0\) is true, of a test statistic at least as extreme as the one observed.
  5. If \(p \le \alpha\), reject \(H_0\); the result is statistically significant at level \(\alpha\). If \(p > \alpha\), fail to reject \(H_0\).

In this lab, we use \(\alpha = 0.05\), check the assumptions before each test, and interpret the result in context, with a confidence interval.

One Sample t-test

In the previous lab, the 95% confidence interval for the mean monthly electricity cost in King County ($179.01 - $183.08) is far above the average residential bill in Washington State reported by the EIA ($113.68). Let’s ask the same question as a two-sided test.

elec <- energy %>%
  filter(!is.na(elec_month)) %>%
  pull(elec_month) # pull a vector

# mu = the value in H0; conf.level = 0.95 (default); alternative = "two.sided" (default)
t.test(elec, mu = 113.68) 
## 
##  One Sample t-test
## 
## data:  elec
## t = 64.802, df = 40440, p-value < 2.2e-16
## alternative hypothesis: true mean is not equal to 113.68
## 95 percent confidence interval:
##  179.0055 183.0805
## sample estimates:
## mean of x 
##   181.043

To compare a sample mean with a fixed value \(\mu_0\), we compute the t statistic:

\[t = \frac{\bar{y} - \mu_0}{s / \sqrt{n}}, \qquad \text{df} = n - 1\]

Here \(\mu_0 = 113.68\) is the benchmark (the number we want to compare), and \(t = 64.8\): our observed mean lies 64.8 standard errors above it. The p-value is almost 0 (printed as p-value < 2.2e-16), so we reject \(H_0\). It is hard to believe this is a coincidence.

One-sample proportion z-test

Proportions work the same way with prop.test(). A household is generally considered to have a high energy burden when it spends more than 6% of its income on home energy. A national study found that 25% of U.S. households had a high energy burden (ACEEE, 2020), so we can ask the high energy burden situation in King County:

burden <- energy %>%
  filter(!is.na(energy_burden)) %>%
  pull(energy_burden)
# x households out of n, against p = 0.25
# burden > 0.06 is TRUE or FALSE for each household; sum() counts the TRUEs
prop.test(sum(burden > 0.06), length(burden), p = 0.25) 
## 
##  1-sample proportions test with continuity correction
## 
## data:  sum(burden > 0.06) out of length(burden), null probability 0.25
## X-squared = 4144.5, df = 1, p-value < 2.2e-16
## alternative hypothesis: true p is not equal to 0.25
## 95 percent confidence interval:
##  0.1061363 0.1123366
## sample estimates:
##        p 
## 0.109198

prop.test() reports X-squared, the square of the z statistic for a proportion. The p-value is almost 0, so we reject \(H_0\). The share of households with a high energy burden in King County, 10.9% (95% CI: 10.6% to 11.2%), is far below the national 25%.

Two Means: Welch’s t-Test

We compare one estimate with one benchmark in the previous cases. We can compare more than 1 estimate instead. In King County, electricity comes from different utilities. Seattle City Light, a public utility, supplies electricity in Seattle; most of the rest of King County is served by Puget Sound Energy. Do Seattle households pay less for electricity?

energy %>%
  filter(!is.na(elec_month)) %>%
  group_by(seattle) %>% # `seattle` is a TRUE/FALSE variable
  summarise(n = n(), mean = mean(elec_month), sd = sd(elec_month), median = median(elec_month))
## # A tibble: 2 × 5
##   seattle     n  mean    sd median
##   <lgl>   <int> <dbl> <dbl>  <dbl>
## 1 FALSE   26545  193.  210.   154.
## 2 TRUE    13896  159.  206.   110

The t-test assumes that the observations are independent and that the sampling distribution of each mean is close to normal. Central limit theorem can guarantee it for moderate samples even when the data are skewed. We saw in the previous lab that the costs are right-skewed, but the two groups have 13,896 and 26,545 households.

t.test() with a formula outcome ~ group compares the two groups. Welch’s t-test estimates the two spreads separately and it is R’s default (var.equal = FALSE).

The two SDs in the summary above are close ($210 and $206), so the classical pooled t-test (var.equal = TRUE) would give almost the same result; Welch’s version is the safe default because it does not need the SDs to match.

# rows with NA in elec_month are dropped by default
t.test(elec_month ~ seattle, data = energy) 
## 
##  Welch Two Sample t-test
## 
## data:  elec_month by seattle
## t = 15.78, df = 28722, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  29.96372 38.46285
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            192.7991            158.5858

Can we say Seattle City Light causes the lower bills? No. This is an observational comparison and probably Seattle homes have fewer bedrooms and smaller building area. Always be cautious when using the term “causes”.

Rank Test: Wilcoxon Rank-Sum

In the summary above, each group’s mean is well above its median ($193 vs $154 outside Seattle): a few very high bills pull the mean up.

A rank is a value’s position when all the values are sorted: 1 for the smallest, 2 for the next, and so on. You can imagine that it keeps only the order. A rank test runs on the ranks instead of the data values. The Wilcoxon rank-sum test is the permutation test with ranks in place of the data values; \(H_0\) is that neither group tends to have larger values. It does not require normal sample means, and it works well for ordinal outcomes.

elec_wilcox <- wilcox.test(elec_month ~ seattle, data = energy)
elec_wilcox
## 
##  Wilcoxon rank sum test with continuity correction
## 
## data:  elec_month by seattle
## W = 224910902, p-value < 2.2e-16
## alternative hypothesis: true location shift is not equal to 0
# W / (n1 × n2); statistic takes W out of the result
# n1 and n2 are from the summary above (sample sizes)
as.numeric(elec_wilcox$statistic) / (26545 * 13896)
## [1] 0.6097306

The p-value is almost 0, so we reject \(H_0\). The statistic \(W\) counts, over every pair of one case from each group, how often the first group’s case is the larger (a tie counts as half), so \(W / (n_1 n_2)\) is the chance that a random case from the first group is larger than one from the second, about 0.5 under \(H_0\).

Here the first group is outside Seattle (FALSE): pick one household from each group at random, and the one outside Seattle pays more 61% of the time.

Chi-Square Test: Categorical Variables

Heating a home with fuel oil emits more carbon than heating it with natural gas, and Seattle’s Clean Heat Program pays homeowners to switch from oil heat to electric heat pumps. Is heating fuel related to when a home was built?

table() counts how many rows fall into each combination of categories. With two variables, it makes a contingency table: the first variable gives the rows, the second the columns, and NA values are left out.

table(energy$heat_fuel, energy$era) 
##                           
##                            Before 1940 1940-1979 1980-1999 2000 or later
##   Bottled, tank, or LP gas          56       216       199           192
##   Coal or coke                       0         2         1             1
##   Electricity                     1561      7091      5634          6189
##   Fuel oil or kerosene             225       754        24            26
##   No fuel used                      51        77        62           113
##   Other fuel                        34        48        23            28
##   Solar energy                      20        20        20            21
##   Utility gas                     2621      7345      4625          4602
##   Wood                              42       147        84            24

Expected Counts

The chi-square test asks whether the pattern in a contingency table could have come from chance. Two categorical variables are independent when the distribution of one is the same in every category of the other. Under independence, the expected count in the cell at row \(r\) and column \(c\) is

\[E_{rc} = \frac{(\text{row } r \text{ total}) \times (\text{column } c \text{ total})}{n}\]

and the test statistic adds up how far each observed count \(O\) is from its expected count \(E\):

\[\chi^2 = \sum_{\text{cells}} \frac{(O - E)^2}{E}, \qquad \text{df} = (\text{rows} - 1) \times (\text{columns} - 1)\]

If heating fuel had nothing to do with era, every era would have the county’s overall fuel mix. For example, 1,029 of the 42,178 homes (2.44%) heat with oil, so among the 4,610 homes built before 1940 we would expect 4,610 × 1,029 / 42,178 ≈ 112 oil-heated homes. We observe 225.

The test assumes independent observations and an expected count of at least 5 in every cell. Only 4 households heat with coal, so each era expects only about one coal-heated home, and R warns that the “Chi-squared approximation may be incorrect”:

chisq.test(table(energy$heat_fuel, energy$era))$expected 
## Warning in chisq.test(table(energy$heat_fuel, energy$era)): Chi-squared
## approximation may be incorrect
##                           
##                             Before 1940   1940-1979   1980-1999 2000 or later
##   Bottled, tank, or LP gas   72.4650292  246.789796  167.754185    175.990991
##   Coal or coke                0.4371947    1.488928    1.012092      1.061786
##   Electricity              2237.8906065 7621.449571 5180.643938   5435.015885
##   Fuel oil or kerosene      112.4683484  383.026696  260.360567    273.144388
##   No fuel used               33.1175020  112.786287   76.665940     80.430272
##   Other fuel                 14.5367253   49.506852   33.652046     35.304377
##   Solar energy                8.8531936   30.150790   20.494855     21.501162
##   Utility gas              2097.7696904 7144.248186 4856.268576   5094.713547
##   Wood                       32.4617099  110.552895   75.147802     78.837593

So let’s combine the small categories into “Other or none”:

energy <- energy %>%
  mutate(heat = case_when(
    heat_fuel %in% c("Utility gas", "Electricity", "Fuel oil or kerosene") ~ heat_fuel,
    TRUE ~ "Other or none" # bottled gas, wood, solar, coal, other fuel, and no fuel
  ))
heat_era <- table(energy$era, energy$heat)
heat_era
##                
##                 Electricity Fuel oil or kerosene Other or none Utility gas
##   Before 1940          1561                  225           203        2621
##   1940-1979            7091                  754           510        7345
##   1980-1999            5634                   24           389        4625
##   2000 or later        6189                   26           379        4602

Direction of Percentages

Before testing, look at the percentages. We must distinguish between the explanatory variable (the era) and the response variable (the heating fuel), ensuring our analysis relies on conditional probabilities conditioned on the explanatory variable. The era may explain the fuel, so the percentages run within each era (rows), and we compare them down the columns.

round(prop.table(heat_era, margin = 1) * 100, 1) # margin = 1: each row adds up to 100
##                
##                 Electricity Fuel oil or kerosene Other or none Utility gas
##   Before 1940          33.9                  4.9           4.4        56.9
##   1940-1979            45.2                  4.8           3.2        46.8
##   1980-1999            52.8                  0.2           3.6        43.3
##   2000 or later        55.3                  0.2           3.4        41.1

About 5% of homes built before 1980 heat with fuel oil, against 0.2% of homes built since 1980. Gas heats 57% of the homes built before 1940; electricity heats 55% of those built since 2000. The test itself uses the counts; the percentages show what the relationship looks like.

heat_era_test <- chisq.test(heat_era)
heat_era_test
## 
##  Pearson's Chi-squared test
## 
## data:  heat_era
## X-squared = 1505.4, df = 9, p-value < 2.2e-16
min(heat_era_test$expected) # the smallest expected count is now far above 5
## [1] 112.4683

\(\chi^2 = 1505.4\) with \((4-1) \times (4-1) = 9\) degrees of freedom, p-value < 0.001. We reject \(H_0\): the heating fuel of King County homes relates to when they were built.

Two Proportions

When both variables have only two categories, “independent” means “the two groups have the same proportion”, so the chi-square test compares two proportions. prop.test() does the same thing! Are Seattle homes more likely than other King County homes to heat with fuel oil?

oil <- energy %>%
  group_by(seattle) %>%
  summarise(x = sum(heat_fuel == "Fuel oil or kerosene"), n = n())
oil
## # A tibble: 2 × 3
##   seattle     x     n
##   <lgl>   <int> <int>
## 1 FALSE     511 27298
## 2 TRUE      518 14880
prop.test(x = c(518, 511), n = c(14880, 27298)) # x = homes heating with oil, n = all homes, from the table above; Seattle (TRUE) first
## 
##  2-sample test for equality of proportions with continuity correction
## 
## data:  c(518, 511) out of c(14880, 27298)
## X-squared = 104.11, df = 1, p-value < 2.2e-16
## alternative hypothesis: two.sided
## 95 percent confidence interval:
##  0.01268512 0.01949990
## sample estimates:
##     prop 1     prop 2 
## 0.03481183 0.01871932

3.48% of Seattle homes heat with oil, against 1.87% elsewhere: 1.6 percentage points more (95% CI: 1.27 to 1.95). The same 2 × 2 table gives the same statistic with chisq.test(). What is the difference?

# rows: Seattle or not; columns: oil heat or not
chisq.test(table(energy$seattle, energy$heat_fuel == "Fuel oil or kerosene")) 
## 
##  Pearson's Chi-squared test with Yates' continuity correction
## 
## data:  table(energy$seattle, energy$heat_fuel == "Fuel oil or kerosene")
## X-squared = 104.11, df = 1, p-value < 2.2e-16

Same X-squared and p-value. The difference: prop.test() also reports the two proportions and the confidence interval for their difference, the numbers we put in a sentence.

Several Means: ANOVA

Are older homes more expensive to power? I guess so. Let’s compare the mean monthly energy cost (electricity, gas, and other fuels) across the four eras.

energy %>%
  filter(!is.na(energy_month)) %>%
  group_by(era) %>%
  summarise(n = n(), mean = mean(energy_month), sd = sd(energy_month))
## # A tibble: 4 × 4
##   era               n  mean    sd
##   <fct>         <int> <dbl> <dbl>
## 1 Before 1940    4254  257.  268.
## 2 1940-1979     14966  250.  261.
## 3 1980-1999     10097  228.  245.
## 4 2000 or later 10209  209.  255.

ANOVA assumes:

Here the SDs run from $245 to $268, so the rule holds. Otherwise, we need to use oneway.test() to avoid equal variance assumption. We will not check the normality here because it needs some concepts we will learn in future sections. Luckily, central limit theorem covers moderate skew in moderate groups.

era_anova <- aov(energy_month ~ era, data = energy)
summary(era_anova)
##                Df    Sum Sq Mean Sq F value Pr(>F)    
## era             3 1.285e+07 4282961   65.21 <2e-16 ***
## Residuals   39522 2.596e+09   65676                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 2652 observations deleted due to missingness

How to read the output:

\(F = 65.2\) with 3 and 39,522 degrees of freedom, p-value < 0.001: we reject \(H_0\). But a significant \(F\) says that some group means differ, not which ones.

Tukey’s method (TukeyHSD() in R) is specifically designed for comparing every pair of means after an ANOVA (assuming equal variances). It adjusts the p-values of the six comparisons so that the chance of even one Type I error in the whole family stays at or below \(\alpha\):

TukeyHSD(era_anova)
##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = energy_month ~ era, data = energy)
## 
## $era
##                                 diff       lwr        upr     p adj
## 1940-1979-Before 1940      -6.740091 -18.17937   4.699185 0.4291921
## 1980-1999-Before 1940     -28.800055 -40.83432 -16.765789 0.0000000
## 2000 or later-Before 1940 -47.886273 -59.90096 -35.871591 0.0000000
## 1980-1999-1940-1979       -22.059964 -30.53889 -13.581037 0.0000000
## 2000 or later-1940-1979   -41.146183 -49.59729 -32.695073 0.0000000
## 2000 or later-1980-1999   -19.086218 -28.32677  -9.845667 0.0000007

Each row is one pair, read as the first era minus the second: 2000 or later-1940-1979 is 2000 or later minus 1940-1979 = -$41. lwr and upr are the 95% interval, and p adj is the adjusted p-value. A pair differs when its interval does not contain 0 (p adj < 0.05).

Five of the six pairs differ. The exception is homes built before 1940 and homes built from 1940 to 1979 (p adj = 0.43; the interval runs from -$18 to $5). Costs fall in the homes built since 1980, by $41 a month for the newest homes compared with 1940-1979. Again, this is an observational comparison, so we cannot draw a causal conclusion.

Rank Test: Kruskal-Wallis

The Kruskal-Wallis test is one-way ANOVA on ranks, for long tails or small sample sizes, or an ordinal outcome. Like the rank-sum test for two groups, it ranks all \(n\) cases together, from 1 for the smallest to \(n\) for the largest. Under \(H_0\), no group tends to have larger values. In R, we use kruskal.test(). Like the \(F\) (ANOVA) test, a significant result only says that some groups differ; pairwise.wilcox.test() then runs a rank-sum test for every pair and adjusts the p-values for the six comparisons.

energy %>%
  filter(!is.na(energy_month)) %>%
  group_by(era) %>%
  summarise(median = median(energy_month))
## # A tibble: 4 × 2
##   era           median
##   <fct>          <dbl>
## 1 Before 1940     204.
## 2 1940-1979       200 
## 3 1980-1999       175.
## 4 2000 or later   158.
kruskal.test(energy_month ~ era, data = energy)
## 
##  Kruskal-Wallis rank sum test
## 
## data:  energy_month by era
## Kruskal-Wallis chi-squared = 766.42, df = 3, p-value < 2.2e-16
pairwise.wilcox.test(energy$energy_month, energy$era) 
## 
##  Pairwise comparisons using Wilcoxon rank sum test with continuity correction 
## 
## data:  energy$energy_month and energy$era 
## 
##               Before 1940 1940-1979 1980-1999
## 1940-1979     0.25        -         -        
## 1980-1999     <2e-16      <2e-16    -        
## 2000 or later <2e-16      <2e-16    <2e-16   
## 
## P value adjustment method: holm
# p-values adjusted for six comparisons (Holm's method, the default)

Kruskal-Wallis \(\chi^2 = 766.4\) with 3 degrees of freedom, p-value < 0.001. The table from pairwise.wilcox.test() gives the adjusted p-value for each pair (row vs column). The pairwise tests tell the same story as Tukey’s method: every pair differs except homes built before 1940 and from 1940 to 1979 (p = 0.25).

With 40,000+ households, the two test results are similar.

Notes about Reading a Test Result

Significant Is Not the Same as Large

With 40,000 households, even small differences are significant. A p-value is not an effect size. Each test pairs with a measure of how large the difference or association is:

# this chunk is optional
# sums of squares: between groups, then within
era_ss <- summary(era_anova)[[1]][["Sum Sq"]] 
era_ss[1] / sum(era_ss) # eta squared
## [1] 0.004925769
# Cramér's V; as.numeric() drops the "X-squared" name
as.numeric(sqrt(heat_era_test$statistic / (sum(heat_era) * (min(dim(heat_era)) - 1)))) 
## [1] 0.1090749

The era of the building explains 0.5% of the variation in household energy cost. The rest of the variation comes from everything else (such as the size of the home, the number of people). Cramér’s V for fuel and era is 0.11, a weak association. Both results are highly significant and both are small.

Not Significant Is Not the Same as No Difference

Now suppose we only had 100 households from Seattle and 100 from the rest of King County.

set.seed(520) # keep the same random sample every time we run it
energy_small <- energy %>%
  filter(!is.na(elec_month)) %>%
  group_by(seattle) %>%
  slice_sample(n = 100) %>% # 100 random households from each group
  ungroup()
t.test(elec_month ~ seattle, data = energy_small)
## 
##  Welch Two Sample t-test
## 
## data:  elec_month by seattle
## t = 1.0199, df = 130.73, p-value = 0.3097
## alternative hypothesis: true difference in means between group FALSE and group TRUE is not equal to 0
## 95 percent confidence interval:
##  -28.72171  89.84991
## sample estimates:
## mean in group FALSE  mean in group TRUE 
##            195.4513            164.8872

The p-value is 0.31 > 0.05, so we fail to reject \(H_0\) and the testing cannot support \(H_1\). But we know from all 40,441 households that there is a difference of about $34. Even with 200 households, the 95% interval for the difference runs from -$29 to $90. A large p-value is not evidence of no effect. “Not significant” means that these data cannot reject \(H_0\). The possible reason may only be that the sample size was small.

📚 TODO: Owners and Renters

Do renters pay less for home energy than owners? Use energy_month and keep only Owner and Renter in tenure. State \(H_0\) and \(H_1\), run Welch’s t-test, and interpret the result with the confidence interval. Is the difference large?

# TODO

📚 TODO: High Energy Burden by Tenure

Is a high energy burden (energy_burden above 0.06) more common among renters than among owners? Keep only Owner and Renter in tenure again, make the 2 × 2 table, compute the percentages in the right direction, and test it with prop.test() and chisq.test(). Which households are missing from this comparison, and why might that matter?

# TODO

📚 TODO: Electricity Cost by Heating Fuel

Does the monthly electricity cost (elec_month) differ by heating fuel (heat)? Check the group SDs, run an ANOVA, and use Tukey’s method to find which pairs differ. Then compute \(\eta^2\). Would you get a different answer with log(elec_month), or with kruskal.test()? (heat is created in the chi-square section; run that chunk first.)

# TODO