The JOBS II study offered job-search workshops to randomly assigned unemployed workers. Some people offered the workshops did not attend. The control condition received a booklet of job-search tips. We distinguish assignment to the program from actual participation and use assignment as an instrument for participation.
We keep the outcome used in the original example: job-search self-efficacy, a score from 1 to 5. Higher scores indicate greater confidence in searching for a job. This is not an employment indicator. We estimate assignment effects using methods from Examples 1–4, then use their ratio to estimate a complier average causal effect (CACE), as in Lectures 9–10.
The supplied jobs data contain 899 observations. The dataset
documentation describes them as illustrative data, not a basis for
conclusions about program efficacy. For these teaching calculations, we
treat individuals as independent observations in two randomized
assignment arms, conditioning on their observed sizes. These
calculations do not reconstruct every feature of the original trial
design.
Keep jobs.RData in the same folder as this R Markdown
file. The data come from the authors’ mediation
package repository. All analysis calculations below use base R. To
knit the document, install rmarkdown and knitr
if needed.
load("jobs.RData")
dim(jobs)
## [1] 899 17
data <- data.frame(
Z = jobs$treat, W = jobs$comply, Y = jobs$job_seek,
jobs[c("sex", "age", "marital", "nonwhite", "educ", "income")]
)
stopifnot(!anyNA(data), all(data$Z %in% c(0, 1)),
all(data$W %in% c(0, 1)))
head(data)
| Notation | Data variable | Meaning |
|---|---|---|
| \(Z_i\) | treat |
Assignment: 1 = offered the program, 0 = control |
| \(W_i\) | comply |
Observed receipt: 1 = participated, 0 = did not participate |
| \(Y_i\) | job_seek |
Observed job-search self-efficacy score |
| \(X_i\) | Six baseline variables above | Sex, age, marital status, race, education, and income |
Despite its name, comply records participation,
not everyone’s latent compliance type. As in the lectures,
\(W_i(z)\) denotes participation under
assignment \(z\), and \(Y_i(z)\) denotes the outcome under
assignment \(z\).
Before running the code: Which cell should be empty if control participants cannot receive the workshops?
receipt_table <- with(data, table(Assignment = Z, Receipt = W))
receipt_table
## Receipt
## Assignment 0 1
## 0 299 0
## 1 228 372
prop.table(receipt_table, margin = 1)
## Receipt
## Assignment 0 1
## 0 1.00 0.00
## 1 0.38 0.62
n <- nrow(data)
n1 <- sum(data$Z == 1)
n0 <- sum(data$Z == 0)
c(n = n, assigned_program = n1, assigned_control = n0)
## n assigned_program assigned_control
## 899 600 299
Of 600 people assigned to the program, 372 participated and 228 did not. None of the 299 controls participated. Thus the observed receipt rate is 0.62 in the program arm and 0 in the control arm.
For the causal interpretation, assume one-sided access: \(W_i(0)=0\) for every individual. An empty observed cell is consistent with this assumption, but does not reveal the unobserved \(W_i(0)\) of those assigned to the program. Under one-sided access:
| Observed group | What can we infer about compliance type? |
|---|---|
| \(Z_i=1,W_i=1\) | Complier: \(W_i(1)=1,W_i(0)=0\) |
| \(Z_i=1,W_i=0\) | Never-taker: \(W_i(1)=W_i(0)=0\) |
| \(Z_i=0,W_i=0\) | Could be a complier or a never-taker; \(W_i(1)\) is unobserved |
Check your understanding: Why can we identify compliers in the program arm, but not identify all the compliers in the control arm?
The earlier examples compared groups defined by the randomized treatment indicator. Here \(Z\) is randomized; \(W\) is not. First estimate the effect of assignment on the outcome and on participation:
\[\widehat{ITT}_Y=\bar Y_{Z=1}-\bar Y_{Z=0},\qquad \widehat{ITT}_W=\bar W_{Z=1}-\bar W_{Z=0}.\]
arm_means <- aggregate(cbind(Y, W) ~ Z, data = data, FUN = mean)
arm_means
ITT_Y <- with(data, mean(Y[Z == 1]) - mean(Y[Z == 0]))
ITT_W <- with(data, mean(W[Z == 1]) - mean(W[Z == 0]))
c(ITT_Y = ITT_Y, ITT_W = ITT_W)
## ITT_Y ITT_W
## 0.06745002 0.62000000
Assignment increases the average score by 0.067 points and participation by 62 percentage points in these data. The latter is the first stage. Neither calculation excludes nonparticipants from the program arm.
as_treated <- with(data, mean(Y[W == 1]) - mean(Y[W == 0]))
c(assignment_contrast = ITT_Y, participation_contrast = as_treated)
## assignment_contrast participation_contrast
## 0.06745002 0.12156335
The participation contrast is descriptive. People who attend may differ in motivation or other causes of self-efficacy from people who do not attend. Randomizing the offer does not randomize actual attendance.
Check your understanding: Would restricting the program arm to participants preserve the randomized comparison with all controls? Why not?
For assignment to identify the effect of participation, we need:
We also maintain the usual consistency and no-interference assumptions. The compliers are people who would attend if offered the program and would not attend under control. CACE is the average effect of participation for this group. It is not the conditional average treatment effect often abbreviated CATE.
Under these assumptions, \[ITT_W=\pi_c,\qquad ITT_Y=\pi_c\tau_c,\qquad \tau_c=\frac{ITT_Y}{ITT_W},\] where \(\pi_c\) is the complier proportion.
Before running the code: Here \(0<\widehat{ITT}_W<1\). Will dividing by the first stage increase or decrease the absolute value of the outcome ITT estimate?
tau_IV <- ITT_Y / ITT_W
c(ITT_Y = ITT_Y, ITT_W = ITT_W, CACE = tau_IV)
## ITT_Y ITT_W CACE
## 0.06745002 0.62000000 0.10879036
The CACE estimate is 0.109 self-efficacy points. Under the stated assumptions, it estimates the average effect of participation among people whose participation responds to assignment. Its larger magnitude than the ITT reflects rescaling by the estimated complier proportion; it is not evidence of greater precision.
We use the delta method. Both the numerator and denominator are estimated, and they use the same people. Their covariance therefore matters. The approximate variance is
\[\widehat V(\widehat\tau_{IV})= \frac{\widehat V_Y+\widehat\tau_{IV}^{\,2}\widehat V_W -2\widehat\tau_{IV}\widehat C_{YW}} {\widehat{ITT}_W^{\,2}},\]
where \(\widehat V_Y\) and \(\widehat V_W\) estimate the variances of the two ITT estimates, and \(\widehat C_{YW}\) estimates their covariance.
Use the same two-group variance calculation as in earlier examples:
\[\widehat V_Y=\frac{s^2_{Y,1}}{n_1}+\frac{s^2_{Y,0}}{n_0},\qquad \widehat V_W=\frac{s^2_{W,1}}{n_1}+\frac{s^2_{W,0}}{n_0},\] \[\widehat C_{YW}=\frac{s_{YW,1}}{n_1}+\frac{s_{YW,0}}{n_0}.\]
Subscripts 1 and 0 refer to assignment groups. The control covariance enters with a plus sign because both ITTs subtract their control mean: \(\operatorname{Cov}(-\bar Y_0,-\bar W_0)=\operatorname{Cov}(\bar Y_0,\bar W_0)\). Its numerical value need not be positive in every application.
Y1 <- data$Y[data$Z == 1]
Y0 <- data$Y[data$Z == 0]
W1 <- data$W[data$Z == 1]
W0 <- data$W[data$Z == 0]
V_Y <- var(Y1) / n1 + var(Y0) / n0
V_W <- var(W1) / n1 + var(W0) / n0
C_YW <- cov(Y1, W1) / n1 + cov(Y0, W0) / n0
c(V_ITT_Y = V_Y, V_ITT_W = V_W, Cov_ITT_Y_ITT_W = C_YW)
## V_ITT_Y V_ITT_W Cov_ITT_Y_ITT_W
## 2.530042e-03 3.933222e-04 5.053978e-05
For a finite completely randomized population, these are the usual estimable Neyman components. The joint covariance matrix omits an unidentifiable covariance matrix of individual assignment effects. The resulting delta-method variance is a large-sample conservative approximation, not an exact randomization variance. Under independent sampling within arms, these components estimate the corresponding sampling variances and covariance directly.
V_delta <- (V_Y + tau_IV^2 * V_W - 2 * tau_IV * C_YW) / ITT_W^2
SE_delta <- sqrt(V_delta)
CI_delta <- tau_IV + qnorm(c(0.025, 0.975)) * SE_delta
c(estimate = tau_IV, SE = SE_delta,
lower = CI_delta[1], upper = CI_delta[2])
## estimate SE lower upper
## 0.10879036 0.08102653 -0.05001872 0.26759943
The estimated SE is 0.081 points. The approximate 95% interval is [-0.05, 0.268]. This approximation needs an adequately strong first stage; a ratio with a denominator near zero can behave very differently from a normal estimate.
Since control receipt is always zero, \(s^2_{W,0}=s_{YW,0}=0\). But control outcomes still vary, so \(\widehat V_Y\) still includes \(s^2_{Y,0}/n_0\). Their contribution to the estimated IV variance is \(s^2_{Y,0}/(n_0\widehat{ITT}_W^2)\).
Check your understanding: Does observing no participation in the control arm mean that the control arm contributes no uncertainty to the IV estimate?
For this bootstrap, treat the observed records as an empirical approximation to the distribution within each arm. Resample whole rows with replacement separately within \(Z=0\) and \(Z=1\). This keeps the arm sizes fixed and preserves the association between each person’s receipt, outcome, and covariates.
This is a sampling approximation for inference. It is different from a Fisher randomization test: we resample people, rather than reassigning a fixed set of people under a sharp null.
wald_estimate <- function(d) {
itt_y <- with(d, mean(Y[Z == 1]) - mean(Y[Z == 0]))
itt_w <- with(d, mean(W[Z == 1]) - mean(W[Z == 0]))
if (abs(itt_w) < 1e-10) return(NA_real_)
unname(itt_y / itt_w)
}
# Store the draws so later methods use exactly the same resampled people.
set.seed(13)
B <- 1000
boot_ids <- replicate(B, {
c(sample(which(data$Z == 0), size = n0, replace = TRUE),
sample(which(data$Z == 1), size = n1, replace = TRUE))
}, simplify = FALSE)
head(boot_ids[[1]])
## [1] 763 223 596 780 394 637
table(data$Z[boot_ids[[1]]])
##
## 0 1
## 299 600
Before running the code: Why would independently
resampling Y and W destroy information needed
for the standard error?
tau_boot <- vapply(boot_ids, function(idx) {
wald_estimate(data[idx, , drop = FALSE])
}, numeric(1))
# Do not silently discard ratios with undefined denominators.
stopifnot(all(is.finite(tau_boot)))
SE_boot <- sd(tau_boot)
CI_boot <- tau_IV + qnorm(c(0.025, 0.975)) * SE_boot
c(estimate = tau_IV, bootstrap_SE = SE_boot,
lower = CI_boot[1], upper = CI_boot[2])
## estimate bootstrap_SE lower upper
## 0.10879036 0.08204324 -0.05201144 0.26959216
We use the SD of the bootstrap estimates as the SE and form a
normal interval with a bootstrap SE, matching the
original example. This is not a percentile interval. Increasing
B reduces Monte Carlo error in the bootstrap calculation;
it does not increase the study’s sample size. An undefined ratio is a
reason to investigate the first stage, not to quietly delete the
offending draw.
hist(tau_boot, breaks = 25, col = "grey85", border = "white",
main = "Bootstrap distribution of the IV estimate",
xlab = "Effect on self-efficacy (points)")
abline(v = tau_IV, col = "red", lty = 2)
The dashed line marks the estimate in the original data. The bootstrap and delta-method SEs need not agree exactly. Neither ordinary normal intervals nor this ordinary bootstrap procedure automatically solves weak-instrument inference.
With one binary instrument and an intercept, first-stage fitted values are the two group receipt rates: \[\widehat W_i=\bar W_0+(\bar W_1-\bar W_0)Z_i.\]
first_stage <- lm(W ~ Z, data = data)
data$W_hat <- fitted(first_stage)
aggregate(W_hat ~ Z, data = data, FUN = mean)
Although observed receipt is binary, fitted receipt here equals 0 or 0.62. Use these fitted rates without recoding them to 0 and 1.
second_stage <- lm(Y ~ W_hat, data = data)
tau_2SLS <- unname(coef(second_stage)["W_hat"])
c(ITT_ratio = tau_IV, two_stage_least_squares = tau_2SLS)
## ITT_ratio two_stage_least_squares
## 0.1087904 0.1087904
stopifnot(isTRUE(all.equal(tau_IV, tau_2SLS)))
The second-stage line goes through the two group-mean points \((\bar W_0,\bar Y_0)\) and \((\bar W_1,\bar Y_1)\). Its slope is therefore \[\widehat\tau_{2SLS}=\frac{\bar Y_1-\bar Y_0}{\bar W_1-\bar W_0} =\frac{\widehat{ITT}_Y}{\widehat{ITT}_W}.\]
The ordinary SE from summary(second_stage) is
not the IV standard error. Its regression residual uses fitted
receipt, \(Y_i-\widehat\alpha-\widehat\tau\widehat
W_i\). IV inference instead uses the structural residual based on
observed receipt, \(Y_i-\widehat\alpha-\widehat\tau W_i\), with
the appropriate IV variance calculation. Use the formulas above, a
bootstrap that repeats the full estimation, or an IV software routine
with an appropriate covariance estimator.
In this randomized example, baseline adjustment is used to improve precision. It does not make attendance randomized. Include the same baseline variables in the outcome and receipt regressions:
\[Y_i=\alpha_Y+b_Y Z_i+\gamma_Y^\top X_i+e_{Yi},\qquad W_i=\alpha_W+b_W Z_i+\gamma_W^\top X_i+e_{Wi}.\]
model.matrix() turns factor variables into indicator
columns, as in Example 2. Use their indicators, not arbitrary integer
codes for categories.
X <- model.matrix(~ sex + age + marital + nonwhite + educ + income,
data = data)[, -1, drop = FALSE]
dim(X)
## [1] 899 15
colnames(X)
## [1] "sex" "age" "maritalmarried"
## [4] "maritalsepartd" "maritaldivrcd" "maritalwidowed"
## [7] "nonwhitenon.white1" "educhighsc" "educsomcol"
## [10] "educbach" "educgradwk" "income15t24k"
## [13] "income25t39k" "income40t49k" "income50k+"
fit_Y_add <- lm(Y ~ Z + X, data = data)
fit_W_add <- lm(W ~ Z + X, data = data)
b_Y <- unname(coef(fit_Y_add)["Z"])
b_W <- unname(coef(fit_W_add)["Z"])
tau_add <- b_Y / b_W
c(outcome_Z_coefficient = b_Y, receipt_Z_coefficient = b_W,
adjusted_ratio = tau_add)
## outcome_Z_coefficient receipt_Z_coefficient adjusted_ratio
## 0.07382593 0.61247630 0.12053679
The two coefficients are the regression-adjusted estimates of assignment effects. Under the teaching model of complete randomization and standard large-sample conditions, their ratio targets CACE. The adjusted and unadjusted estimates need not agree exactly in this sample, and adjustment need not lower the estimated SE in every dataset.
Use \(Z\) and \(X\) in the first stage, then include the same \(X\) in the second stage.
W_hat_add <- fitted(fit_W_add)
second_stage_add <- lm(Y ~ W_hat_add + X, data = data)
tau_2SLS_add <- unname(coef(second_stage_add)["W_hat_add"])
c(adjusted_ITT_ratio = tau_add, adjusted_2SLS = tau_2SLS_add)
## adjusted_ITT_ratio adjusted_2SLS
## 0.1205368 0.1205368
stopifnot(isTRUE(all.equal(tau_add, tau_2SLS_add)))
To see why, substitute \(\widehat W_i=\widehat\alpha_W+\widehat b_W Z_i+\widehat\gamma_W^\top X_i\) into the second-stage fitted equation. The coefficient on \(Z_i\) becomes \(\widehat\tau_{2SLS}\widehat b_W\), while the intercept and \(X\) coefficients absorb the remaining terms. This is another parameterization of the regression of \(Y\) on \(Z\) and \(X\), so \[\widehat b_Y=\widehat\tau_{2SLS}\widehat b_W.\] The equality requires the same observations and covariates, an intercept, one instrument for one treatment, and \(\widehat b_W\ne0\).
For an observational instrument, including \(X\) does not by itself establish instrument independence. The required adjustment and model adequacy need justification. Also, when assignment probabilities and effects differ across \(X\), ordinary adjusted 2SLS need not target the population CACE. The algebraic equality of the two procedures still holds.
The original example used assignment-by-covariate interactions. Retain this as an extension of Examples 2–3. Center each covariate column over the full analysis sample, then include interactions with \(Z\) in both regressions:
Xc <- sweep(X, 2, colMeans(X), FUN = "-")
fit_Y_int <- lm(Y ~ Z * Xc, data = data)
fit_W_int <- lm(W ~ Z * Xc, data = data)
stopifnot(!anyNA(coef(fit_Y_int)), !anyNA(coef(fit_W_int)))
ITT_Y_int <- unname(coef(fit_Y_int)["Z"])
ITT_W_int <- unname(coef(fit_W_int)["Z"])
tau_int <- ITT_Y_int / ITT_W_int
c(adjusted_ITT_Y = ITT_Y_int, adjusted_ITT_W = ITT_W_int,
adjusted_ratio = tau_int)
## adjusted_ITT_Y adjusted_ITT_W adjusted_ratio
## 0.07248094 0.61616030 0.11763325
Because the centered columns average to zero, the coefficient of
Z equals the average fitted assignment contrast over the
sample’s covariates. Take the ratio of these two averaged
contrasts. Averaging individual or subgroup ratios would
generally use different weights and would not give the same target.
This interaction-based ratio need not equal the additive 2SLS estimate above. In this randomized setting, both approaches target CACE under standard large-sample conditions. Adding the interactions changes the finite-sample estimator. Do not apply the one-instrument additive 2SLS equivalence to the interacted fits without further justification.
Check your understanding: Without centering, what
covariate values would the Z coefficient in an interaction
regression describe? Why do we average the two fitted contrasts before
taking their ratio?
The helper below repeats the same regressions on a supplied dataset. For the interaction version, it recomputes the centering in each bootstrap sample. Both assignment effects and their ratio are re-estimated; we do not hold the original first-stage predictions fixed.
adjusted_ratio <- function(d, interactions = FALSE) {
X <- model.matrix(~ sex + age + marital + nonwhite + educ + income,
data = d)[, -1, drop = FALSE]
if (interactions) {
Xc <- sweep(X, 2, colMeans(X), FUN = "-")
fit_y <- lm(Y ~ Z * Xc, data = d)
fit_w <- lm(W ~ Z * Xc, data = d)
} else {
fit_y <- lm(Y ~ Z + X, data = d)
fit_w <- lm(W ~ Z + X, data = d)
}
# Flag non-estimable fits rather than interpreting aliased coefficients.
if (anyNA(coef(fit_y)) || anyNA(coef(fit_w))) return(NA_real_)
b_y <- unname(coef(fit_y)["Z"])
b_w <- unname(coef(fit_w)["Z"])
if (abs(b_w) < 1e-10) return(NA_real_)
b_y / b_w
}
boot_adjusted <- vapply(boot_ids, function(idx) {
d <- data[idx, , drop = FALSE]
c(additive = adjusted_ratio(d),
interacted = adjusted_ratio(d, interactions = TRUE))
}, numeric(2))
# A failed fit can occur if a resample loses a sparse factor category
# within an arm. Inspect such failures rather than dropping them silently.
stopifnot(all(is.finite(boot_adjusted)))
SE_add <- sd(boot_adjusted["additive", ])
SE_int <- sd(boot_adjusted["interacted", ])
c(additive_bootstrap_SE = SE_add, interacted_bootstrap_SE = SE_int)
## additive_bootstrap_SE interacted_bootstrap_SE
## 0.08321593 0.08220227
The bootstrap treats people as sampling units within assignment arms. Its interpretation is a sampling approximation under that working model; it is not an exact finite-population randomization calculation. If the design has clusters, pairs, or additional randomization strata, the resampling and variance calculation must reflect them.
The first two rows below use exactly the same point estimator. The remaining two use covariate adjustment. All intervals use a normal cutoff; the bootstrap rows use bootstrap SEs.
interval_row <- function(estimate, SE) {
c(estimate = estimate, SE = SE,
lower = estimate - qnorm(0.975) * SE,
upper = estimate + qnorm(0.975) * SE)
}
results <- rbind(
"ITT ratio: delta method" = interval_row(tau_IV, SE_delta),
"ITT ratio: bootstrap" = interval_row(tau_IV, SE_boot),
"Additive adjustment / 2SLS: bootstrap" = interval_row(tau_add, SE_add),
"Centered interactions: bootstrap" = interval_row(tau_int, SE_int)
)
knitr::kable(results, digits = 3,
col.names = c("Estimate", "SE", "95% lower", "95% upper"))
| Estimate | SE | 95% lower | 95% upper | |
|---|---|---|---|---|
| ITT ratio: delta method | 0.109 | 0.081 | -0.050 | 0.268 |
| ITT ratio: bootstrap | 0.109 | 0.082 | -0.052 | 0.270 |
| Additive adjustment / 2SLS: bootstrap | 0.121 | 0.083 | -0.043 | 0.284 |
| Centered interactions: bootstrap | 0.118 | 0.082 | -0.043 | 0.279 |
op <- par(mar = c(5, 18, 3, 1))
pos <- rev(seq_len(nrow(results)))
plot(results[, "estimate"], pos,
xlim = range(c(0, results[, "lower"], results[, "upper"])),
ylim = c(0.5, nrow(results) + 0.5), yaxt = "n", ylab = "",
xlab = "Effect on self-efficacy (points)", pch = 19,
main = "CACE estimates and approximate 95% intervals")
axis(2, at = pos, labels = rownames(results), las = 1, cex.axis = 0.75)
segments(results[, "lower"], pos, results[, "upper"], pos)
abline(v = 0, col = "red", lty = 2)
par(op)
The intervals include zero in this illustration. That does not establish that the true effect is zero. We also do not know the true CACE, so the smallest estimated SE does not tell us which point estimate is closest to it.
Check your understanding: Why can the delta method and bootstrap give different standard errors even though they use the same point estimate? Does covariate adjustment necessarily produce a narrower interval?
Use the unadjusted first-stage estimate and its two-group SE from Section 4. For the regression with one binary instrument and an intercept, this is also the HC2 robust SE of the coefficient on \(Z\). The squared robust \(t\) statistic supplies a one-instrument robust Wald \(F\) statistic.
SE_ITT_W <- sqrt(V_W)
t_first <- ITT_W / SE_ITT_W
F_first <- t_first^2
c(ITT_W = ITT_W, robust_SE = SE_ITT_W,
robust_t = t_first, robust_F = F_first)
## ITT_W robust_SE robust_t robust_F
## 0.62000000 0.01983235 31.26205031 977.31578947
The first stage is large relative to its SE here. The familiar \(F>10\) guideline is a rough historical diagnostic, not a universal guarantee. With adjustment, diagnose the coefficient on \(Z\) after the same adjustment for \(X\); do not use a test of all regressors jointly as the instrument-strength test.
A strong first stage shows that the instrument predicts participation. It does not establish independence or exclusion. In this one-sided-access example, the access restriction supplies no defiers; a positive first stage alone would not establish no defiers in a general study.
Discuss: Could being offered a workshop affect self-efficacy even without attending? Which assumption would that challenge? Would a very large first-stage \(F\) statistic resolve that concern?