Module 5: Categorical Data, Part II

Fill-in-the-blank coding exercises for Module 5: Chi-Squared Tests & Fisher’s Exact Test
Author

Haley Grant

Chi-Squared Tests

The chi-squared (\(\chi^2\)) test shows up in three related but distinct situations, all based on comparing observed counts to expected counts:

  1. Goodness-of-fit: does a single categorical variable follow a specified distribution?
  2. Homogeneity: does the distribution of a categorical variable differ across two or more groups?
  3. Independence: are two categorical variables associated with each other?

All three use the same test statistic:

\[\chi^2 = \sum \dfrac{(O - E)^2}{E}\]

where \(O\) is an observed count and \(E\) is the count we’d expect under \(H_0\), summed over all cells of a table.

Goodness-of-Fit

This version tests whether a single categorical variable follows a specified set of proportions.

Motivating example: At a flu clinic, lab testing identified which flu strain (A, B, or C) was responsible for each of 200 confirmed flu cases this season. Historically, strains are distributed as 50% A, 30% B, 20% C. This season, the observed counts were: 80 Strain A, 70 Strain B, 50 Strain C. Does this season’s distribution differ from the historical pattern?

Expected Counts

Fill in the code below to calculate the expected count for each strain, under the historical proportions (50%, 30%, 20%), out of 200 total cases.

# observed counts
obs = c(80, 70, 50)

# expected counts under historical proportions
exp_props = c(, , )
exp_counts = exp_props * 
exp_counts
# observed counts
obs = c(80, 70, 50)

# expected counts under historical proportions
exp_props = c(0.5, 0.3, 0.2)
exp_counts = exp_props * 200
exp_counts

[1] 100 60 40

We’d expect 100 Strain A cases, 60 Strain B, and 40 Strain C under the historical distribution—but we observed 80, 70, and 50. There’s a noticeable shift toward Strains B and C this season.

Calculating the Test Statistic

Fill in the code below to calculate the chi-squared test statistic by hand.

# chi-squared test statistic
sum(( - exp_counts)^2 / )
# chi-squared test statistic
sum((obs - exp_counts)^2 / exp_counts)

[1] 8.166667

With \(k = 3\) categories, this statistic has \(k - 1 = 2\) degrees of freedom.

Using chisq.test()

Fill in the code below to run the chi-squared goodness-of-fit test directly, using chisq.test() with the p argument for the hypothesized proportions.

# chi-squared goodness-of-fit test
chisq.test(, p = c(, , ), correct = F)
# chi-squared goodness-of-fit test
chisq.test(obs, p = c(0.5, 0.3, 0.2), correct = F)

[1] X-squared = 8.1667, df = 2, p-value = 0.01685

With a p-value of about 0.017, we’d reject \(H_0\) and conclude this season’s strain distribution does differ from the historical pattern.

What assumption do we need about the expected counts for a chi-squared test to be valid (regardless of which of the three versions we’re running)? Were those conditions met here?

Test of Homogeneity

This version compares the distribution of a categorical outcome across two or more independent groups—it’s really just a generalization of the two-proportion z-test (for a 2x2 table) to tables with more rows or columns.

Motivating example: Does flu infection rate differ between vaccinated and unvaccinated patients? Among 150 vaccinated patients, 9 got the flu (141 did not). Among 100 unvaccinated patients, 20 got the flu (80 did not).

Setting Up the Table

Here’s our table of counts, with rows for vaccination status and columns for flu status:

# table of counts: rows = vaccination status, columns = flu (yes, no)
tbl = matrix(c(9, 141, 20, 80), nrow = 2, byrow = T)
rownames(tbl) = c("Vaccinated", "Unvaccinated")
colnames(tbl) = c("Flu", "No Flu")
tbl

Checking Expected Counts

Before trusting the chi-squared approximation, let’s check that all the expected cell counts are large enough (the usual rule of thumb is at least 5).

Fill in the code below to extract the expected counts for this table. (You can pull these from chisq.test()’s output, even before looking at the p-value.)

# expected counts
chisq.test(, correct = F)$expected
# expected counts
chisq.test(tbl, correct = F)$expected

All four expected counts (about 17.4, 132.6, 11.6, and 88.4) are comfortably above 5, so the chi-squared approximation is valid here.

Running the Test

Fill in the code below to run the chi-squared test of homogeneity on this table.

# chi-squared test of homogeneity
chisq.test(, correct = F)
# chi-squared test of homogeneity
chisq.test(tbl, correct = F)

[1] X-squared = 11.468, df = 1, p-value = 0.0007080

A p-value of about 0.0007—strong evidence that flu rate differs between the two groups. In fact, for a 2x2 table like this one, \(\chi^2 = z^2\), where \(z\) is the test statistic from the corresponding two-proportion z-test!

Calculate a two-proportion z-test for the same data (prop.test(x = c(9, 20), n = c(150, 100), correct = F)), and square the resulting z-statistic (hint: you can extract it, or just compare $statistic from each test). Does it match the chi-squared statistic above?

Test of Independence

This version asks whether two categorical variables, both measured on the same group of subjects, are associated with each other (rather than comparing pre-defined groups, as in the homogeneity case). Mechanically, the test is identical—only the design and interpretation differ.

Motivating example: Among 300 flu patients (a different, broader sample than our vaccinated/unvaccinated comparison above), is there an association between age group (under 50, 50-65, over 65; 100 patients each) and whether they were hospitalized for the flu? The counts were: under 50 (5 hospitalized, 95 not), 50-65 (10 hospitalized, 90 not), over 65 (15 hospitalized, 85 not).

Here’s our table of counts, with rows for age group and columns for hospitalization status:

# table of counts: rows = age group, columns = hospitalized (yes, no)
tbl2 = matrix(c(5, 95, 10, 90, 15, 85), nrow = 3, byrow = T)
rownames(tbl2) = c("Under 50", "50-65", "Over 65")
colnames(tbl2) = c("Hospitalized", "Not Hospitalized")
tbl2

Before running the test, fill in the code below to check the expected counts for this table.

# expected counts
chisq.test(, correct = F)$expected
# expected counts
chisq.test(tbl2, correct = F)$expected

All six expected counts come out to either 10 or 90—comfortably above 5, so the chi-squared approximation is valid here.

Now fill in the code below to run the chi-squared test of independence on tbl2.

# chi-squared test of independence
chisq.test(, correct = F)
# chi-squared test of independence
chisq.test(tbl2, correct = F)

[1] X-squared = 5.5556, df = 2, p-value = 0.06218

With a p-value of about 0.062, this is just above the conventional 0.05 cutoff—a borderline, non-significant result. We don’t have quite enough evidence to conclude age group and hospitalization are associated, though the pattern (older patients hospitalized more often) might be worth following up with a larger sample.

We checked that all of our expected cell counts were 10 or 90—comfortably above the usual rule-of-thumb minimum of 5. What would you do differently if several expected cell counts came out below 5?

Fisher’s Exact Test

Fisher’s exact test is used for the same kind of question as the chi-squared test of independence/homogeneity (is there an association between two categorical variables?), but it doesn’t rely on the chi-squared approximation—which breaks down when expected cell counts are small. Instead, it calculates an exact p-value based on the hypergeometric distribution.

Rule of thumb: if any expected cell count in your table is below 5, prefer Fisher’s exact test over the chi-squared test.

Motivating example: In a small safety sub-study, researchers compared the rate of a rare severe allergic reaction between two vaccine brands. Out of 25 patients given Brand X, 1 had a severe reaction (24 did not). Out of 20 patients given Brand Y, 5 had a severe reaction (15 did not). Is there an association between vaccine brand and reaction rate?

Setting Up the Table

Here’s our table of counts, with rows for vaccine brand and columns for reaction status:

# table of counts: rows = vaccine brand, columns = reaction (yes, no)
tbl = matrix(c(1, 24, 5, 15), nrow = 2, byrow = T)
rownames(tbl) = c("Brand X", "Brand Y")
colnames(tbl) = c("Reaction", "No Reaction")
tbl

Checking Expected Counts

Fill in the code below to check the expected cell counts using chisq.test()$expected (even though we won’t trust the chi-squared p-value itself, this is still a quick way to check the expected counts).

# expected counts under chi-squared assumptions
chisq.test(, correct = F)$expected
# expected counts under chi-squared assumptions
chisq.test(tbl, correct = F)$expected

Two of the four expected counts (around 3.3 and 2.7) are below 5—the chi-squared test’s assumptions are violated here. This is exactly the situation Fisher’s exact test is designed for.

Why does a small expected cell count cause problems for the chi-squared test specifically? (Hint: think about what the chi-squared distribution is approximating, and how well that approximation works with very few observations in a cell.)

Running Fisher’s Exact Test

Fill in the code below to run Fisher’s exact test on the table.

# Fisher's exact test
fisher.test()
# Fisher's exact test
fisher.test(tbl)

[1] p-value = 0.07409, odds ratio = 0.1253912

With a p-value of about 0.074, we’re just above the conventional 0.05 threshold—not quite enough evidence to conclude the reaction rates differ between brands, though the estimated odds ratio (about 0.13, meaning the odds of a reaction are much lower for Brand X) suggests this might be worth investigating with a larger sample.

Fisher’s exact test gave us a p-value, but we didn’t need to specify a test statistic or distribution (like we would for a t-test or chi-squared test) to get there. In your own words, what is Fisher’s exact test actually calculating to arrive at this p-value?

Comparing to the Chi-Squared Test

For comparison, fill in the code below to run the (technically invalid, given our small expected counts) chi-squared test on the same table, and compare the p-values.

# chi-squared test (for comparison only - not actually valid here!)
chisq.test(, correct = F)
# chi-squared test (for comparison only - not actually valid here!)
chisq.test(tbl, correct = F)

R may give a warning here (something like “Chi-squared approximation may be incorrect”) precisely because of the small expected counts we checked earlier. The resulting p-value may be noticeably different from Fisher’s exact p-value—a good illustration of why checking conditions matters before trusting a test’s output.

Even if R lets you run chisq.test() without an error (just a warning), why is it important not to just use whichever test gives you the smaller p-value?