Rfuncs Project
Example 4
Clinical Trial of the Janssen COVID Vaccine (2021)
In February 2021, Janssen BioTech, a pharmaceutical company of Johnson & Johnson (J&J) obtained a US FDA Emergency Use Authorization (EUA) to begin distributing their one-dose vaccine (Ad26.COV2.S) for SARS-CoV-2 ("COVID-19"). This example reanalyzes data provided in the official FDA Briefing Document and in Sadoff, J., et al. (2021). Interim results of a phase 1–2a trial of Ad26.COV2.S Covid-19 vaccine. New England Journal of Medicine. The official protocol (as amended and re-approved 14 December 2020) is here.
​
From the protocol:
​
... [Study 3001] is a multicenter, randomized, double-blind, placebo-controlled, Phase 3, pivotal efficacy and safety study in adults 18 to <60 years of age and >60 years of age. The efficacy, safety, and immunogenicity of Ad26.COV2.S will be evaluated in participants living in, or going to, locations with high risk for acquisition of SARS-CoV-2 infection after administration of study vaccine.
​
The randomization ratio was 1:1.
​
Section 5.2.1 of the Briefing Document:
​
The originally specified primary endpoint was efficacy of the vaccine to prevent centrally confirmed, moderate to severe/critical COVID-19 occurring at least 14 days post-vaccination in SARS-CoV-2 seronegative adults (with “seronegative” defined as negative RT-PCR and negative serology against SARS-CoV-2 nucleocapsid on Day 1). Study protocol amendment 3 (December 14, 2020) added a co-primary endpoint counting COVID-19 cases from 28 days post-vaccination.
​
On 8 March 2012, the US Centers for Disease Control and Prevention (CDC) stated that
​
... people are considered fully vaccinated for COVID-19 ... 2 weeks after they have received a single-dose [of the Janssen] vaccine.
​
Incidence event: first occurrence of moderate to severe/critical COVID. Table 3.1 restates data presented in Table 14 of the Janssen Briefing Document.
**************************************************************************
Table 3.1. Vaccine efficacy of first occurrence of moderate to
severe/critical COVID with onset at least 14 days after
vaccination, negative at baseline for SARS-CoV-2, per protocol
set.
**************************************************************************
Group: Ad26.COV2.S Placebo
Per protocol sample size: 19,514 19,544
Overall surveillance time (person-years): 3114 3089
Overall surveil. time ratio (OSTR = 3114/3089): 1.008
Average surveillance time (years): 0.160 0.158
Number of cases: 173 509
Incidence rate (173/3114, 509/3089): 0.0556 0.1648
Incidence rate ratio (IRR = 0.0556/0.1648): 0.337
Vaccine efficacy (VE = 1 - IRR)): 0.663*
**************************************************************************
*The reported adjusted 95% confidence interval for VE was [0.599, 0.718].
When these results were announced, many in the media noted that this estimated 66% VE was inferior to the VEs reported for the two-shot mRNA-based vaccines developed by Pfizer-BioNTech's and Moderna, and already approved by the US FDA. For example, on 24 February 2021, reporters Carl Zimmer, Noah Weiland, and Sharon LaFraniere of the New York Times stated that "Johnson & Johnson’s vaccine has a lower efficacy rate than the vaccines from Moderna and Pfizer-BioNTech, which are both around 95 percent." On 5 March 2021, CNN and other news organizations reported that Detroit mayor Mike Duggan initially turned down a shipment of the Janssen vaccine, saying "So, Johnson & Johnson is a very good vaccine. Moderna and Pfizer are the best. And I am going to do everything I can to make sure the residents of the city of Detroit get the best."
Ill-considered statements about the efficacy of a vaccine or treatment can only harm our efforts in public health. The Janssen vaccine was tested under more demanding conditions than both the Moderna or Pfizer-BioNTech vaccines. the average surveillance time for the Janssen trial was 25% longer than for the Pfizer-BioNTech trial (8.3 vs. 6.6 weeks), but the incidence rate in the Janssen placebo sample was 226% greater in the Pfizer-BioNTech trial (509/3089.1 = 0.165 versus 0.0729). This confirms that the Janssen trial did indeed recruit more heavily in populations that were at higher risk to acquire a SARS-CoV-2 infection. Reporters and commentators in the press and media should have realized that the Janssen trial differed substantively from the Pfizer-BioNTech and Moderna trials with respect to who was recruited. More than one health professional being interviewed noted that contrasting the results in the Janseen trial to those in the Pfizer-BioNTech and Moderna trials was like "comparing apples and oranges." Unfortunately, like Detroit's Mayor Duggan, millions in the public were set in believing that the Janssen vaccine was markedly inferior with respect to efficacy.
​
Janssen did not use the beta-binomial method, but we do so here with the null diffuse (ND) and null skeptical (NS) priors featured in the Pfizer-BioNTech analysis (Example 2).
> # Using the null diffuse prior:
> OSTR <- 1.008
> irr <- c(0.01, 0.05, 0.10, 0.20, 0.35, 0.50, 0.70, 1.00)
> BetaBinomial(Median=P.IRR(1.0,OSTR),
+ P.Q=P.IRR(0.01,OSTR), Q=0.01,
+ Trans="P/((1-P)*OSTR)", TransName="IRR",
+ CP.points=P.IRR(irr,OSTR),
+ M=c(173, 509))
​
​
> OSTR <- 1.008
> irr <- c(0.01, 0.05, 0.10, 0.20, 0.35, 0.50, 0.70, 1.00)
> BetaBinomial(Median=P.IRR(1.0,OSTR),
+ P.Q=P.IRR(0.70,OSTR), Q=0.05,
+ Trans="P/((1-P)*OSTR)", TransName="IRR",
+ CP.points=P.IRR(irr,OSTR),
+ M=c(173, 509))
​
​
For succinctness, the output is only summarized here: Table 3.2.
Table 3.2. Comparing alternative null priors for applying the
beta-binomial method to the Janssen data. Data: 173 vaccine
cases vs. 509 placebo cases.
***********************************************************************
Prior: Diffuse Skeptical
Prior Median: 0.999 1.000
Prior 95% PI: [0.026, 38.9] [0.653, 1.531]
Prior 95% PI spread: 38.9 0.877
Prob[IRR < 0.35 | no data]: 0.259 <0.0005
Posterior Median: 0.338 0.388
Posterior 95% PI: [0.284, 0.401] [0.331, 0.454]
Prior 95% PI spread: 0.117 0.123
Prob[IRR < 0.35 | 173 vs. 509]: 0.654 0.100
***********************************************************************
The issues here are much like those covered in Example 2.
Incidence event: Hospitalization due to COVID. Major news organizations and even public health leaders stated that results from the Janssen trial indicated that the Ad26.COV2.S vaccine was (quoting from ABC News) "100% effective against hospitalizations and deaths." Simple logic should have squelched such statements. (The American Statistical Association once sold a t-shirt proclaiming that "Being a statistician is never having to say you are certain.")
​
That all said, is there a reasonable case to be made that the vaccine is nearly 100% efficacious against hospitalizations? Operationally, we address this question by estimating Prob[VE > 0.98] = Prob[IRR < 0.02].
The relevant data are in Table 18 of the Briefing Document. 31 subjects were hospitalized with COVID (after at least 14 days on trial), with 2 coming from the vaccine group and 29 from the placebo group. In addition, the groups' total surveillance times (person-years at risk) were virtually identical, 3125.8 vaccine vs. 3125.1 placebo, giving incidence rates of 2/3125.8 = 0.00064 and 29/3125.1 = 0.00928. In terms of IRR = 1 - VE, Janssen's estimate and 95% confidence interval were 0.069 (agreeing with [2/3125.8]/[29/3125.1]) and [0.008, 0.273]. The overall surveillance time ratio was OSTR = 3125.8/3125.1 = 1.000.
​
Here we apply the beta-binomial method using the null diffuse prior.
> OSTR <- 1.000
> irr <- c(0.02, seq(.05,0.30,0.05))
> BetaBinomial(Median=P.IRR(1.0,OSTR),
+ P.Q=P.IRR(0.01,OSTR), Q=0.01,
+ Trans="P/((1-P)*OSTR)", TransName="IRR",
+ CP.points=P.IRR(irr,OSTR),
+ M=c(2, 29))
Transformation: t(P) = IRR(P) = P/((1-P)*OSTR)
Solution for prior: P ~ beta(a0 = 1.0001000, b0 = 1.0001000)
Fit of beta(a0, b0) to requested Median (IRR_0.50) and IRR_0.01.
*****************************************************************
Median IRR_0.01
As requested: 1.000 0.0100
As fit: 1.000 0.0101
*****************************************************************
Prior: P ~ beta(a0 = 1.00, b0 = 1.00)
Posterior: P ~ beta(a1 = 1.00 + 2 = 3.00, b1 = 1.00 + 29 = 30.00)
Median and 95% PI and its spread, IRR(P) = P/((1-P)*OSTR), P ~ beta().
**********************************************************************
Median Prob. Interval PI Spread
Prior 1.000 [0.026, 38.989] 39.0
Posterior 0.090 [0.020, 0.263] 0.243
**********************************************************************
100q% quantiles of IRR(P) = P/((1-P)*OSTR).
*************************************************************************
q: 0.005 0.025 0.050 0.250 0.500 0.750 0.950 0.975 0.995
Prior 0.005 0.026 0.053 0.333 1.000 3.000 18.996 38.989 198.914
Posterior 0.011 0.020 0.027 0.057 0.090 0.135 0.225 0.263 0.349
*************************************************************************
Prior: P ~ beta(1.00, 1.00)
Posterior: P ~ beta(3.00, 30.00)
Prob[IRR < 0.02 | no data] = 0.020 increases a trace to Prob[IRR < 0.02 | 2 vaccine vs. 29 placebo] = 0.024. The posterior median of 0.090 and 95% PI of [0.020, 0.263] summarize succinctly what the data have to say using this approach.
​
Incidence event: Death due to COVID. Again, major news media and public health officials technically misstated that the Janssen trial indicated that this vaccine was "100% effective" in preventing death. But did those data support saying that the vaccine nearly 100% efficacious against death? Table 19 of the Briefing Document lists 7 COVID related deaths, all in the placebo group. However, one case was COVID positive at baseline. Thus, the binomial comparison is: 0 deaths in the vaccine group versus 6 deaths in the placebo. Again, operationally, we estimate Prob[ VE > 0.98] = Prob[IRR < 0.02].
> OSTR <- 1.000
> irr <- c(0.02, seq(.05,0.30,0.05))
> BetaBinomial(Median=P.IRR(1.0,OSTR),
+ P.Q=P.IRR(0.01,OSTR), Q=0.01,
+ Trans="P/((1-P)*OSTR)", TransName="IRR",
+ CP.points=P.IRR(irr,OSTR),
+ M=c(0, 6))
Transformation: t(P) = IRR(P) = P/((1-P)*OSTR)
Solution for prior: P ~ beta(a0 = 1.0001000, b0 = 1.0001000)
Fit of beta(a0, b0) to requested Median (IRR_0.50) and IRR_0.01.
*****************************************************************
Median IRR_0.01
As requested: 1.000 0.0100
As fit: 1.000 0.0101
*****************************************************************
Prior: P ~ beta(a0 = 1.00, b0 = 1.00)
Posterior: P ~ beta(a1 = 1.00 + 0 = 1.00, b1 = 1.00 + 6 = 7.00)
Median and 95% PI and its spread, IRR(P) = P/((1-P)*OSTR), P ~ beta().
**********************************************************************
Median Prob. Interval PI Spread
Prior 1.000 [0.026, 38.989] 39.0
Posterior 0.104 [0.004, 0.694] 0.690
**********************************************************************
100q% quantiles of IRR(P) = P/((1-P)*OSTR).
*************************************************************************
q: 0.005 0.025 0.050 0.250 0.500 0.750 0.950 0.975 0.995
Prior 0.005 0.026 0.053 0.333 1.000 3.000 18.996 38.989 198.914
Posterior 0.001 0.004 0.007 0.042 0.104 0.219 0.534 0.694 1.132
*************************************************************************
Prior: P ~ beta(1.00, 1.00)
Posterior: P ~ beta(1.00, 7.00)
Prob[t(P) < t(p)], where t(P) = IRR(P) = P/((1-P)*OSTR).
************************************************************
t(p): 0.020 0.050 0.100 0.150 0.200 0.250 0.300
Prior 0.020 0.048 0.091 0.130 0.167 0.200 0.231
Posterior 0.129 0.289 0.487 0.624 0.721 0.790 0.841
************************************************************
Prior: P ~ beta(1.00, 1.00)
Posterior: P ~ beta(1.00, 7.00)
Given there were only six deaths in total, this was the strongest possible data supporting the hypothesis that the vaccine saves lives. Moreover, the null diffuse prior gives those data maximum influence. Even so, the posterior median of 0.104, and 95% PI of [0.004, 0.694] speak against concluding that the Janssen vaccine is nearly 100% efficacious against death. Simply and directly, Prob[IRR < 0.02 | 0 vaccine vs. 6 placebo] = 0.129, which is a long way from 1.00 (certainty). These calculations demonstrate the paucity of support for stating that 0 vaccine deaths vs. 6 placebo deaths indicates that the Janssen vaccine is "nearly 100% effective" in preventing death.
​
What if there were 0 deaths in the vaccine group and 10, 20, 40, 80, or 160 deaths in the placebo group?
> deathsP <- c(10,20,40,80,160) # of deaths in placebo group
> OSTR <- 1.000
> irr <- c(0.02, 0.05)
> q50 <- q025 <- q975 <- Pr_LT0.02 <- Pr_LT0.05 <- numeric(5)
> for (i in 1:5) {
+ # deaths.P <- deathsP[i]
+ temp <- BetaBinomial(A=1.0001, B=1.0001, M=c(0, deathsP[i]),
+ Q.points=c(0.025, 0.50, 0.975),
+ CP.points=P.IRR(irr,OSTR),
+ Trans="P/((1-P)*OSTR)", TransName="IRR",Print=F)
+ q025[i] <- round(temp$qntl[2,1],5)
+ q50[i] <- round(temp$qntl[2,2],3)
+ q975[i] <- round(temp$qntl[2,3],3)
+ Pr_LT0.02[i] <- round(temp$cprob[2,1],3)
+ Pr_LT0.05[i] <- round(temp$cprob[2,2],3)
+ }
> data.frame(deathsP,q50, q025, q975, Pr_LT0.02, Pr_LT0.05)
> # Results summarized in Table 3.3.
​
​
​
*************************************************************************
Table 3.3. Posterior quantiles and probabilities for IRR using null
diffuse prior with 0 deaths in the vaccine group and 10,
20, 40, 80, or 160 deaths in the placebo group.
*************************************************************************
Deaths (vaccine:placebo)
0:10 0:20 0:40 0:80 0:160
.........................................................................
Posterior IRR Median 0.065 0.034 0.017 0.009 0.004
Posterior IRR 2.5% quantile 0.00231 0.00121 0.00062 0.00031 0.00016
Posterior IRR 97.5% quantile 0.398 0.192 0.094 0.047 0.023
Posterior Prob[IRR < 0.02] 0.196 0.340 0.556 0.799 0.959
Posterior Prob[IRR < 0.05] 0.415 0.641 0.865 0.981 >0.999
*************************************************************************
Thus, using this benign prior, even with 0 deaths in the vaccine group and 80 in the placebo arm, there would only 80% support for inferring that the vaccine is "nearly 100% efficacious," defined as IRR < 0.02, VE > 0.98.