I AM A HOUND FOR SCIENCE. I don’t read hundreds of studies a month, I read thousands. A life-long methodologist, I focus first on the study design, then the design of analysis, and THEN the results. In teaching study design, I always emphasize the use of logic and the importance of mapping a critical path to try to insure that given the variables measured and unmeasured, students and scientists do not rigorously fool themselves into an erroneous conclusion.
In this article, I outline the top Ecological and Epidemiological Fallacies in Vaccines Studies.
Fallacy #1. Association or Correlation Tests Causation. Most vaccine studies employ retrospective association studies. The search for association of vaccination (V) with any negative health outcomes (E, for ‘Effect’) could be a fair test of the presence of an association or correlation. This could be functionally due to either a direct causal path (V->E), or an indirect causal path (V->Eo->E). Such tests as applied in vaccine studies fall far short of a test of causality, for a number of reasons. The first is that associations and correlations can also be due to shared causes (V->E, V->Eo). One indirect causal pathway that is unlikely is a cause C leading both the V and E (shared cause), but see Healthy User Bias, below.
It is clear that association and correlation are nowhere near sufficient tests of causality. The second is that the results of analyses are rarely applied symmetrically; that is the weight of the evidence of a negative outcome (no association) is higher than the weight applied to evidence of a positive outcome (association with adverse health outcomes). This is also a type of Interpretation Bias.
Fallacy #2. Accepting Negative Results When Power is Low. Statistical power is the probability of rejecting a null hypothesis when it is, in fact, false. In the case of Vaccine safety science, the null is usually “no difference in rates” of a specific type of adverse event. A flaw of many clinical studies, including those that do not involve vaccines, is the publication and interpretation of negative results from studies that have such low sample sizes that the expected result is to fail to reject the null. Systematic reviews and meta-analyses that cite low powered studies as if they were, in fact, sufficiently power to confidently interpret a negative result as evidence inherit and perpetuate the flaws of those studies.
Fallacy #3. Adjusting for Risk Factors and Interpreting Resulting Negative Results as “No Effect”. Smoking, number of sexual partners, use of a condom are all known risk factors for HPV infection. Studies of the risk of infection by non-vaccine targeted HPV types that use adjusted odds ratios (aOR) after adjusting for known HPV vaccine risk factors often conclude there is no risk of type replacement at the population level. Type replacement is the increase in non-vaccine targeted types following removal of the vaccine-targeted types. In the unadjusted analyses, the rates of infection by non-vaccine targeted types have been shown to increase, and the loss of the significance of the test of association is taken by some authors to mean that vaccination does not contribute as a causal factor to type replacement. The fallacy is made clear by comparing these two sentences:
“Our adjusted odds ratio was not significant, suggesting that vaccines do not contribute to type replacement” vs.
“Our adjusted odds ratio was not significant, suggesting that vaccines do not contribute to type replacement unless you smoke, continue to have multiple partners, and fail to use a condom”.
Population health policies on vaccination that leave those at highest risk of infection due to other factors should not relegate the entire causality of risk to behavior; in fact, the population-wide rate of infection of non-vaccine targeted types is increasing beyond that expected without vaccination because its prevalence is increased. That means, quite simply, if one is going to infected with an HPV, it is becoming less likely that the HPV will be vaccine-targeted. This expectation exists even without ecological niche competition – all that is needed is an increase in the prevalence of HPV-vaccine targeted types. Individuals who are immune to vaccine targeted types are not immune to all vaccine non-targeted HPV types (although some cross-protection has been detected among a few non-targeted and targeted types).
The inherent risk of misinterpreting type replacement studies is that rarer, more dangerous types of HPV could increase in prevalence, leading to more aggressive HPV-linked cancers.
Similarly, studies of the link between vaccines and autism that adjust for birth weight, gestational age, mother’s age, mother’s income, interpret the loss of significance of association as showing there is no risk of autism from vaccines at the population level. Instead, they should conclude that their analyses support that low birth weight or poor nutrition may be co-factors, or risk factors, for the development of autism after exposure to vaccines. The CDC has published a White Paper that mentions correcting for confounders using a path diagram that could identify “back door” influences (Link). The problem with this entire paradigm is that the back door influences actually do exist and might make vaccines riskier for persons with those risk factors.
In reality, any such causal pathway diagram designed to help test hypotheses would have every factor feeding arrows to Vaccination and then to Environmental Exposure (source of antigen/trigger). Fixing the lower diagram for some of these errors, the missing path Vaccine->Person->Environmental Exposure (antigen source)-> asthma diagnosis is seen:
The fact that leadership in vaccine science mis-specifies the causal critical pathway, and then trims out likely co-factors, is unsettling as it represents an example of Dogmatism. The removal of variation introduced by covariates that are, in reality co-factor, and maybe useful as predictors of vaccine adverse events, is not warranted unless the interaction term has been tested, and found to be not significant, and then only if the study was sufficiently powered to test the interaction terms.
Fallacy #4. Affirming the Consequent (Unwarranted Generalization)
MMR vaccines are not associated with autism.
Other vaccines are vaccines.
Vaccines do no cause autism.
Based on what?
Similarly, from the CDC website:
“In fact, a 2004 scientific review by the IOM concluded that “the evidence favors rejection of a causal relationship between thimerosal–containing vaccines and autism.'”
That does not mean that all vaccines do not cause autism. In reality, not all vaccines have been tested for association with autism.
It is therefore fallacious to conclude that VACCINES do not cause autism.
Fallacy #5. Healthy User Bias
Health user bias is a serious limitation of all observational studies that seek to examine the impact of one medical intervention, and the risk is particularly high in prevention medicine.
Many parents stop vaccinating due to early, bad experiences with vaccine (such as HepB) with their child, or with their older sibling. In a study of later vaccines, such as MMR, these individuals would be marked as “unvaccinated” against MMR but would in reality be partly vaccinated. Also, individuals who can tolerate vaccines would tend to have more vaccines and would thus be more likely to receive the MMR. The net result is total confounding of health care choice with both vaccination status and health outcomes. If the MMR was going to tip the toxic scale of a child and push them over their personal limit, other toxins in the environment still accumulating, the “unvaccinated” child might still regress into autism. Due to the careful nature of their parents, however, the child may not have been given further vaccines, and thus they would appear as “unvaccinated” and not regress into autism. The vaccinated population also may include individuals more likely to participate in other health care programs, not studied by a study of the risk of a single vaccine, and the non-vaccinated population may include individuals more likely to avoid other sources of toxins.
The net effect is a non-random tangled web of confounding variables, potentially leading observational studies to biased odds ratios. The key here is that the effects are non-random and impose selection biases. For non-vaccine and vaccine examples of Healthy User Bias, see this review.
Fallacy #6. Argument from Ignorance
In 2012 and in 2014, IOM reviewed studies available (but ignored animal studies) and concluded that there was not sufficient evidence to conclude that vaccines cause autism. They used terminology such as “Evidence is inadequate to accept or reject a causal relationship … The committee does not conclude that the vaccine does not cause the adverse event …”
Some reported this as “IOM found no evidence” to support the hypothesis that vaccines cause autism…. and they leave out the corollary that IOM also found insufficient evidence to say that vaccines do NOT cause autism.
Cherry-picking the literature is an example of argument from ignorance. A good example of Cherry-picking is the exclusion of studies that reported positive results. When AAP sent their list of studies to POTUS, they conveniently left out a number of key studies, such as these provided by JB Handley:
Other studies left out include
One-sided targeted analyses from biased websites are not useful for understanding the full body of literature on the vaccine/autism question. And meta-analyses that ignore studies are also guilty of cherry-picking, and their fallacious conclusions will naturally be suspect.
Fallacy #7. Assuming a Random Population
If a genetic subgroup with higher susceptibility exists, the population under study is non-homogeneous. Studies that ignore genetic or familial risk will tend to be unable to detect an effect at the population level. Enrichment studies are needed that accrue patients to groups based on familial risk and genetics. High-risk vaccinated vs high-risk unvaccinated would be an important comparison. Until we actually test the relevant hypothesis of risk, we cannot rule out that vaccines might be harming a genetic minority of individuals, who have rights of equal protection.
This fallacy leads, in translation to clinical practice, to the Either/Or or Black/White, False Dilemma, or Excluded Middle Fallacy, in which the “sometimes” option is replaced with the binary option (yes/no). The correct hypothesis is not whether vaccines cause autism in everyone (“I was vaccinated, I am fine”), but rather do vaccine cause autism in some people. Importantly, if there is genetic risk to vaccine injury, those at risk are entitled to protection from harm; those who are injured should be heralded, not scoffed at, and we as a society should do everything we can do to insure that people who fall into the at-risk category are protected.
In reality, the use of observational studies to assess vaccine safety is fundamentally flawed. Randomized clinical trials with total health outcome awareness must be used to insure full knowledge of the risks of vaccination. Those who argue that it would be unethical to not vaccinate some child and deny them protection from infections are committing the fallacy of begging the question: they are assuming the truth of the conclusion, instead of conducting Science to test the question of relative safety (circular reasoning): their argument requires that their desired conclusion be true. The real health risks of any individual vaccine, to >1 vaccine per day, or to exposure to the entire CDC health schedule are unknown due to reliance on passive vaccine injury collection via VAERS, which requires users to acknowledge that no entry can be used to assume causality, and due to intentional steps during vaccine studies to exclude adverse events that the investigators have determined to not be due to the vaccine – both addition examples of the begging the question.
I will add additional common fallacies in the vaccine safety science literature over time. In the meantime, post your favorite flawed studies and let the readers know which fallacy – or fallacies – the studies suffer from.
Allison Park, PA