Welcome to the
inaugural posting for the Vaccine Autism Journal Club project!
Disclaimer: I
am not an epidemiologist or statistician.
If you have expertise in these fields, and I get the analysis completely
wrong, please let me know in the comments!
Question this paper is trying to answer: Is there a difference in the risk for
autism diagnosis among boys vaccinated against hepatitis B in the first month
of life as compared to later- or never-vaccinated boys?
Background: Each
year, between 3,000 and 5,000 people in the United States die from liver damage
or liver cancer caused by the virus hepatitis B. Vaccination against hepatitis B reduces the
risk of infection. Universal vaccination
against hepatitis B was first recommended for U.S. newborns in 1991, and is
still recommended today. You can read
the CDC’s fact sheet on hepatitis B and the vaccine here.
Thimerosal is a mercury-containing
preservative. Mercury is obviously
something you want to limit your exposure to.
On the other hand, bacteria can grow in multi-dose vials of vaccines
without preservatives, and vaccines
contaminated with bacteria have killed people. As a precautionary measure, thimerosal was
removed from hepatitis B vaccines in the U.S. in 1999, and today the hepatitis
B vaccine is given only from single-dose vials. You can read the FDA’s statement on thimerosal
in vaccines here.
Of those 79,883
children, the authors looked at children who had vaccination records. This makes sense, because in order to look
for an association of autism risk with vaccination against hepatitis B, we need
to be able to say with confidence whether a given child got the hepatitis B
vaccine or not, and when.
Of the children
with vaccination records, the authors looked at children who were born before
1999. If the hypothesis is that the
mercury in thimerosal might be contributing to any correlation between autism
and vaccination, it makes sense to only look at thimerosal-containing vaccines,
which is to say, vaccines from before 1999.
Of the children
in the sample with vaccination records born before 1999, the authors looked at
just boys. Boys have more than a
fourfold risk for autism compared to girls.
Because of this discrepancy in autism risk, it’s not unreasonable to
think that the factors influencing autism susceptibility might differ between
boys and girls. In order to get cleaner
data, then, it makes sense to look at the sexes separately.
From the
original sample, after selecting only the boys born after 1999 with vaccination
records, the sample size is now 7,399, which still sounds like a big
number. The incidence of autism is low,
however, so of those 7,399 boys, only 31 had autism.1
The authors then
looked at how many of the boys had received their first dose of hepatitis B
vaccine during the first month of life. The
authors have a funny way of calculating that: “Birth month and year were equal
to vaccination month and year for observations identified as having been
vaccinated as neonates,” which means that if you were born at 11:55pm on April
30, and were vaccinated ten minutes later at 12:05am on May 1, you would be
scored as not having been vaccinated in the first month of life. As a result, some of the boys who are scored
as not having received the vaccine as newborns actually could have received
their vaccines earlier than some of
the boys scored as having received the vaccine as newborns. I can understand the limitations presented if
vaccination records have only months and years without dates: but if a later
cutoff point had been chosen (vaccination within the first two or three months
of life, for example), the data would be cleaner. I wrote to the authors asking why they chose
the first month of life as their cutoff: they haven’t written back to me yet.
If the risk of
autism is associated with receiving the first dose of hepatitis B vaccine
during the first month of life, then we would predict that the boys without
autism will have a lower rate of receiving the first dose of hepatitis B
vaccine during the first month of life than the boys with autism.
Results: Of the 7,368 boys born before 1999 with vaccination
records without autism, 1,258 (17%) received the first dose of hepatitis B
vaccine within the first month of life (using this funny definition). If there’s no association between autism and
vaccination as newborns, we therefore expect 17% of the 31 boys with autism to
have received the first dose of hepatitis B vaccine as newborns, which comes
out to 5.3 boys. (Obviously, we can’t
observe 5.3 boys: we expect to see either 5 or 6 boys.) If a significantly higher number of boys with
autism received the first dose of hepatitis B vaccine as newborns, then we’ll
conclude that autism is associated with receiving the first dose of hepatitis B
vaccine within the first month of life.
9 boys with
autism received the first dose of hepatitis B vaccine within the first month of
life, which comes out to 3.7 unexpected additional cases of autism.
Is this impressive?
If the rate of
first-month vaccination among boys with autism is also 17%, we can work out the
mathematics of how likely it is that we would randomly pick 31 and have 9 or
more of the 31 boys we picked be vaccinated in their first month. If we did that experiment many, many times, we
would observe such a result 3.8% of the time, just by random chance alone (this
is what is meant by the p value of
0.038 in Table 2). Is 3.8% a
sufficiently low probability to reject the hypothesis that the rate of first-month
vaccination among boys with autism is the same as the rate of first-month vaccination
among boys without autism?
The convention
among statisticians is that a p value
of less than 0.05 (5%) is considered significant, but that’s all it is: a
convention. If we accept the 5%
threshold, then we’re also accepting a false positive rate of 5% (because 5% of
the time, the sample sorted randomly will give you a p value less than 0.05, even though it does in fact have the same
incidence).
Discussion: One major question I have about this
paper is the question of testing
multiple hypotheses. If the
probability of any one hypothesis’s appearing statistically significant by
random chance alone is 5%, then as you test more and more hypotheses, the odds
increase that any one of them appears statistically significant gets higher and
higher. If you test 100 hypotheses, then
on average, you’ll expect 5 of them to have a p of less than 0.05, just by random chance alone, so you wouldn’t
be able to conclude that there’s actually a statistically significant
difference for those five hypotheses.
When I make a
quick count of the hypotheses tested in this paper2, I come up with
at least nine. The probability that none
of the nine hypotheses would have a p
of less than 0.05, by random chance alone, is only 63%: 37% of the time (more
than one in three), when you test nine hypotheses, you’d expect at least one of
your nine hypotheses to have a p of
less than 0.05. I suspect there were
even more hypotheses tested in this paper that weren’t mentioned (the authors,
for example, look at boys, and also look at girls: I suspect they looked at
both boys and girls together, but didn’t mention that explicitly, so I’m not
including that in my conservative count): if there were, that would make it
even more likely that any one of them appear significant when no correlation
exists in reality.
I’m disinclined
to question the statistical methods of either the senior author, who earned a
PhD in biostatistics from Harvard, or the corresponding author, who at the time
was earning a PhD in population health and clinical outcomes research: the
authors almost certainly know more about statistics than I do. As I said, I’m not an epidemiologist. On the other hand, papers making these kinds of basic
statistical errors have made it through peer review before, so this kind of error is always
something worth considering.
I wrote to the
authors asking whether the correction for testing multiple hypotheses was
included in their statistical model used to calculate their p of 0.038: they haven’t written back to
me yet.
More seriously:
just weeks before this paper was published in November 2010, the journal Pediatrics published a paper also addressing the question of whether autism incidence
was associated with vaccination with the hepatitis B vaccination during the
first month of life. The Pediatrics
paper found no such correlation.
Their methods were somewhat different: they used a case-control approach
rather than a probability sample-based approach; they looked at both boys and
girls rather than just boys; they were looking at other thimerosal-containing
vaccines in addition to hepatitis B vaccines, etc. All the same, if early vaccination really did
increase the risk of autism by threefold, that should have been apparent in the
data shown in the Pediatrics paper,
and it wasn’t. I see no reason to think
that the Pediatrics
paper should be any less trustworthy than the Journal of Toxicology and Environmental Health, Part A paper.
Was the representation in the blog post
fair? The title of the blog post that sparked this project was “22
Medical Studies That Show Vaccines Can Cause Autism”. The authors are very clear in their
concluding paragraph: “As with all cross-sectional secondary data analyses, causality cannot be determined, and
this study is subject to bias from unmeasured or uncontrolled confounding
factors” (emphasis mine). Even if you
accept the authors’ conclusion that there’s a correlation between vaccination
and autism, correlation is not the same as causation. Are families of higher socioeconomic status
more likely to get their children vaccinated early, and are they also more
likely to seek out an autism diagnosis, especially for children on the less
severe end of the spectrum? Autism
incidence is correlated with higher maternal education: are families with more
educated mothers more likely to get their children vaccinated early? (I would have appreciated a more detailed
discussion of what the “unmeasured or uncontrolled confounding factors” might
be in the paper’s discussion section, but I understand that sometimes space is
limited.)
Secondly, if
you’re going to accept that this paper shows a correlation between early
hepatitis B vaccination and autism incidence, then you must also accept that
this paper shows no correlation between varicella (chicken pox) vaccination and
autism, or measles-mumps-rubella vaccination and autism.
Thirdly, even if
you accept the results of this paper, remember that this correlation was shown
only for boys born before 1999. This
paper does not provide evidence that
vaccination since 1999 is in any way correlated with autism incidence, and
should not be cited as evidence to defend a decision not to vaccinate children
today.
Conclusion: I
don’t think this study was fundamentally flawed. They took a reasonable approach to a
difficult problem. They started with a
seemingly large sample of children, but because the incidence of autism is low
and apparently the rate of keeping vaccination records is low, they ended up
with only 31 boys born before 1999 with vaccination records and autism. Nine of those boys were vaccinated within the
first month of life, a few more than expected.
Any argument that there really is a correlation between first-month
hepatitis B vaccination and autism incidence would have to account for the
results of the Pediatrics paper,
which found no such correlation with a larger sample size of autistic
children. I think this study just suffered from bad luck: it happened to be
the 3.8% of the time that that difference in vaccination rates among boys with
vs. without autism would appear statistically significant without any actual
difference between the two populations. I also think the representation of this
paper’s findings in the blog post was misleading, for the reasons discussed
in the previous section.
Please leave your comments and questions
below, but please also
make sure you make your comments in a spirit of honesty, fairness, kindness,
respect, and trust.
Endnotes:
1. Autism diagnosis was scored as follows: “The
outcome variable was a dichotomous (yes/no) variable created in response to the
following survey question and presentation of a card with a choice of
diagnoses: ‘Looking at this list, has a doctor or other professional ever told
you that [sample child’s name] had any of these conditions…(i.e.,
autism)?” This approach seems perfectly
reasonable to me: however, I think an important limitation of this study is
that the authors did not follow up to confirm diagnosis or lack of diagnosis.
2. Hypotheses tested in this paper:
·
Boys
born before 1999 with known vaccination status, with vs. without autism
o
Hepatitis
B vaccination
o
Non-Hispanic
white
o
Two-parent
household
o
Maternal
education
o
Varicella
vaccination
o
Measles-mumps-rubella
vaccination
·
Hep
B vaccination
o
Girls
born before 1999 with known vaccination status, with vs. without autism
o
Boys
born before 1999 with known vaccination status, with vs. without Down’s
syndrome, cystic fibrosis, cerebral palsy, congenital or other heart problems
§
(Note:
it’s not clear to me whether these were all counted together or
individually. I’m giving the authors the
benefit of the doubt here, but it seems likely that these were actually four
different hypotheses, one for each condition.)
o
All
boys (born both before and after 1999) with known vaccination status, with vs.
without autism
