VAJC, week 1: hepatitis B vaccination in the first month of life among boys born before 1999 (Gallagher and Goodman)

Welcome to the inaugural posting for the Vaccine Autism Journal Club project!

Today we’ll be discussing “Hepatitis B Vaccination of Male Neonates and Autism Diagnosis, NHIS 1997-2002”, by Carolyn M. Gallagher and Melody S. Goodman, Journal of Toxicology and Environmental Health, Part A 73:1665-1677 (November 2010).

Disclaimer:  I am not an epidemiologist or statistician.  If you have expertise in these fields, and I get the analysis completely wrong, please let me know in the comments!

Question this paper is trying to answer: Is there a difference in the risk for autism diagnosis among boys vaccinated against hepatitis B in the first month of life as compared to later- or never-vaccinated boys?

Background:  Each year, between 3,000 and 5,000 people in the United States die from liver damage or liver cancer caused by the virus hepatitis B.  Vaccination against hepatitis B reduces the risk of infection.  Universal vaccination against hepatitis B was first recommended for U.S. newborns in 1991, and is still recommended today.  You can read the CDC’s fact sheet on hepatitis B and the vaccine here.

Thimerosal is a mercury-containing preservative.  Mercury is obviously something you want to limit your exposure to.  On the other hand, bacteria can grow in multi-dose vials of vaccines without preservatives, and vaccines contaminated with bacteria have killed people.  As a precautionary measure, thimerosal was removed from hepatitis B vaccines in the U.S. in 1999, and today the hepatitis B vaccine is given only from single-dose vials.  You can read the FDA’s statement on thimerosal in vaccines here.

Methods:  The authors started with the files of 79,883 children using the National Health Interview Survey’s dataset.  This sounds like a big number.

Of those 79,883 children, the authors looked at children who had vaccination records.  This makes sense, because in order to look for an association of autism risk with vaccination against hepatitis B, we need to be able to say with confidence whether a given child got the hepatitis B vaccine or not, and when.

Of the children with vaccination records, the authors looked at children who were born before 1999.  If the hypothesis is that the mercury in thimerosal might be contributing to any correlation between autism and vaccination, it makes sense to only look at thimerosal-containing vaccines, which is to say, vaccines from before 1999.

Of the children in the sample with vaccination records born before 1999, the authors looked at just boys.  Boys have more than a fourfold risk for autism compared to girls.  Because of this discrepancy in autism risk, it’s not unreasonable to think that the factors influencing autism susceptibility might differ between boys and girls.  In order to get cleaner data, then, it makes sense to look at the sexes separately.

From the original sample, after selecting only the boys born after 1999 with vaccination records, the sample size is now 7,399, which still sounds like a big number.  The incidence of autism is low, however, so of those 7,399 boys, only 31 had autism.¹

The authors then looked at how many of the boys had received their first dose of hepatitis B vaccine during the first month of life.  The authors have a funny way of calculating that: “Birth month and year were equal to vaccination month and year for observations identified as having been vaccinated as neonates,” which means that if you were born at 11:55pm on April 30, and were vaccinated ten minutes later at 12:05am on May 1, you would be scored as not having been vaccinated in the first month of life.  As a result, some of the boys who are scored as not having received the vaccine as newborns actually could have received their vaccines earlier than some of the boys scored as having received the vaccine as newborns.  I can understand the limitations presented if vaccination records have only months and years without dates: but if a later cutoff point had been chosen (vaccination within the first two or three months of life, for example), the data would be cleaner.  I wrote to the authors asking why they chose the first month of life as their cutoff: they haven’t written back to me yet.

If the risk of autism is associated with receiving the first dose of hepatitis B vaccine during the first month of life, then we would predict that the boys without autism will have a lower rate of receiving the first dose of hepatitis B vaccine during the first month of life than the boys with autism.

Results: Of the 7,368 boys born before 1999 with vaccination records without autism, 1,258 (17%) received the first dose of hepatitis B vaccine within the first month of life (using this funny definition).  If there’s no association between autism and vaccination as newborns, we therefore expect 17% of the 31 boys with autism to have received the first dose of hepatitis B vaccine as newborns, which comes out to 5.3 boys.  (Obviously, we can’t observe 5.3 boys: we expect to see either 5 or 6 boys.)  If a significantly higher number of boys with autism received the first dose of hepatitis B vaccine as newborns, then we’ll conclude that autism is associated with receiving the first dose of hepatitis B vaccine within the first month of life.

9 boys with autism received the first dose of hepatitis B vaccine within the first month of life, which comes out to 3.7 unexpected additional cases of autism.

Is this impressive?

If the rate of first-month vaccination among boys with autism is also 17%, we can work out the mathematics of how likely it is that we would randomly pick 31 and have 9 or more of the 31 boys we picked be vaccinated in their first month.  If we did that experiment many, many times, we would observe such a result 3.8% of the time, just by random chance alone (this is what is meant by the p value of 0.038 in Table 2).  Is 3.8% a sufficiently low probability to reject the hypothesis that the rate of first-month vaccination among boys with autism is the same as the rate of first-month vaccination among boys without autism?

The convention among statisticians is that a p value of less than 0.05 (5%) is considered significant, but that’s all it is: a convention.  If we accept the 5% threshold, then we’re also accepting a false positive rate of 5% (because 5% of the time, the sample sorted randomly will give you a p value less than 0.05, even though it does in fact have the same incidence).

Discussion:  One major question I have about this paper is the question of testing multiple hypotheses.  If the probability of any one hypothesis’s appearing statistically significant by random chance alone is 5%, then as you test more and more hypotheses, the odds increase that any one of them appears statistically significant gets higher and higher.  If you test 100 hypotheses, then on average, you’ll expect 5 of them to have a p of less than 0.05, just by random chance alone, so you wouldn’t be able to conclude that there’s actually a statistically significant difference for those five hypotheses.

When I make a quick count of the hypotheses tested in this paper², I come up with at least nine.  The probability that none of the nine hypotheses would have a p of less than 0.05, by random chance alone, is only 63%: 37% of the time (more than one in three), when you test nine hypotheses, you’d expect at least one of your nine hypotheses to have a p of less than 0.05.  I suspect there were even more hypotheses tested in this paper that weren’t mentioned (the authors, for example, look at boys, and also look at girls: I suspect they looked at both boys and girls together, but didn’t mention that explicitly, so I’m not including that in my conservative count): if there were, that would make it even more likely that any one of them appear significant when no correlation exists in reality.

I’m disinclined to question the statistical methods of either the senior author, who earned a PhD in biostatistics from Harvard, or the corresponding author, who at the time was earning a PhD in population health and clinical outcomes research: the authors almost certainly know more about statistics than I do.  As I said, I’m not an epidemiologist.  On the other hand, papers making these kinds of basic statistical errors have made it through peer review before, so this kind of error is always something worth considering.

I wrote to the authors asking whether the correction for testing multiple hypotheses was included in their statistical model used to calculate their p of 0.038: they haven’t written back to me yet.

More seriously: just weeks before this paper was published in November 2010, the journal Pediatrics published a paper also addressing the question of whether autism incidence was associated with vaccination with the hepatitis B vaccination during the first month of life.  The Pediatrics paper found no such correlation.  Their methods were somewhat different: they used a case-control approach rather than a probability sample-based approach; they looked at both boys and girls rather than just boys; they were looking at other thimerosal-containing vaccines in addition to hepatitis B vaccines, etc.  All the same, if early vaccination really did increase the risk of autism by threefold, that should have been apparent in the data shown in the Pediatrics paper, and it wasn’t.  I see no reason to think that the Pediatrics paper should be any less trustworthy than the Journal of Toxicology and Environmental Health, Part A paper.

Was the representation in the blog post fair?  The title of the blog post that sparked this project was “22 Medical Studies That Show Vaccines Can Cause Autism”.  The authors are very clear in their concluding paragraph: “As with all cross-sectional secondary data analyses, causality cannot be determined, and this study is subject to bias from unmeasured or uncontrolled confounding factors” (emphasis mine).  Even if you accept the authors’ conclusion that there’s a correlation between vaccination and autism, correlation is not the same as causation.  Are families of higher socioeconomic status more likely to get their children vaccinated early, and are they also more likely to seek out an autism diagnosis, especially for children on the less severe end of the spectrum?  Autism incidence is correlated with higher maternal education: are families with more educated mothers more likely to get their children vaccinated early?  (I would have appreciated a more detailed discussion of what the “unmeasured or uncontrolled confounding factors” might be in the paper’s discussion section, but I understand that sometimes space is limited.)

Secondly, if you’re going to accept that this paper shows a correlation between early hepatitis B vaccination and autism incidence, then you must also accept that this paper shows no correlation between varicella (chicken pox) vaccination and autism, or measles-mumps-rubella vaccination and autism.

Thirdly, even if you accept the results of this paper, remember that this correlation was shown only for boys born before 1999.  This paper does not provide evidence that vaccination since 1999 is in any way correlated with autism incidence, and should not be cited as evidence to defend a decision not to vaccinate children today.

Conclusion:  I don’t think this study was fundamentally flawed.  They took a reasonable approach to a difficult problem.  They started with a seemingly large sample of children, but because the incidence of autism is low and apparently the rate of keeping vaccination records is low, they ended up with only 31 boys born before 1999 with vaccination records and autism.  Nine of those boys were vaccinated within the first month of life, a few more than expected.  Any argument that there really is a correlation between first-month hepatitis B vaccination and autism incidence would have to account for the results of the Pediatrics paper, which found no such correlation with a larger sample size of autistic children.  I think this study just suffered from bad luck: it happened to be the 3.8% of the time that that difference in vaccination rates among boys with vs. without autism would appear statistically significant without any actual difference between the two populations.  I also think the representation of this paper’s findings in the blog post was misleading, for the reasons discussed in the previous section.

Please leave your comments and questions below, but please also make sure you make your comments in a spirit of honesty, fairness, kindness, respect, and trust.

Next week’s paper will be “Do aluminum vaccine adjuvants contribute to the rising prevalence of autism?”, Tomljenovic and Shaw, Journal of Inorganic Biochemistry 105:1489-1499 (November 2011).  See you then.

Endnotes:

1.  Autism diagnosis was scored as follows: “The outcome variable was a dichotomous (yes/no) variable created in response to the following survey question and presentation of a card with a choice of diagnoses: ‘Looking at this list, has a doctor or other professional ever told you that [sample child’s name] had any of these conditions…(i.e., autism)?”  This approach seems perfectly reasonable to me: however, I think an important limitation of this study is that the authors did not follow up to confirm diagnosis or lack of diagnosis.

2.  Hypotheses tested in this paper:

·      Boys born before 1999 with known vaccination status, with vs. without autism

o   Hepatitis B vaccination

o   Non-Hispanic white

o   Two-parent household

o   Maternal education

o   Varicella vaccination

o   Measles-mumps-rubella vaccination

·      Hep B vaccination

o   Girls born before 1999 with known vaccination status, with vs. without autism

o   Boys born before 1999 with known vaccination status, with vs. without Down’s syndrome, cystic fibrosis, cerebral palsy, congenital or other heart problems

§  (Note: it’s not clear to me whether these were all counted together or individually.  I’m giving the authors the benefit of the doubt here, but it seems likely that these were actually four different hypotheses, one for each condition.)

o   All boys (born both before and after 1999) with known vaccination status, with vs. without autism