Science and Policy on Vaccine Safety Science:’Absence of Evidence’ Abused as ‘Evidence of Absence’: Part 1

THERE IS A DISTURBING AND CONSISTENT trend in the vaccine safety literature for government-affiliated researchers in the misinterpretation of negative results from studies of specific, serious adverse events from vaccines, such as autism, encephalopathy, ADHD, and other serious neurological injuries.

Here’s the (il)logic flow:

1. Conduct a retrospective epidemiological study (case/control, or cohort) with a relatively small samples size (few patients). Do not publish any power analysis showing that you had sufficient sampling effort to conduct significant effect.
2. Analyze the data (we won’t go into cherry picking results here, but that, as well as “analyzing to result” both happen when a positive association is found, but the researchers wish to keep the result from public view).
3. Find no or weak association.
4. Lament the high confidence interval.
5. Conclude something sciency-sounding, such as: “The relative risk of severe neurologic disease in the 0–7 day risk period after meningococcal C conjugate vaccination was estimated at 1.28 (95% CI, 0.17–9.75). As evidenced by the wide confidence interval, the sample size is not large enough to get a more precise estimate of the relative risk. The authors concluded that administration of meningococcal C conjugate vaccine is not associated with an increased risk of severe neurologic disease within 0 to 7 days of vaccination” [1]
6. Issue an official policy-sounding statement, such as:

   Weight of Epidemiologic Evidence

   The committee has limited confidence in the epidemiologic evidence, based on one study that lacked validity and precision, to assess an association between meningococcal vaccine and encephalitis or encephalopathy.[1]

7. Create a policy of adoption of the vaccine, and give the following rationale: “There is no scientific evidence of serious neurological disease as result of this vaccine”.

If you don’t find the flaw in the logic of steps 1-7, you’ve been duped.

Scientific studies, whether they are prospective studies, or retrospective studies, are supposed to provide proof that the sample size – the number of patients included in the study – was large enough to detect a specific effect if it did indeed exist. This property of a study is called “STATISTICAL POWER” and the analysis they SHOULD be including in their publications is called “POWER ANALYSIS”.   Statistical power is the ability of a test to detect a significant (difference, increase in relative risk, increase in odds ratio) and is a function of

-Sample Size in each sample group (N1, N2)

-Stringency of the test – the p-value required for a result to be considered ‘significant’ – this is generally 5%

-The degree of intrinsic variability in the data within and between the two groups (population variance)

-The effect size (the size of the actual difference in the measure of interest between two groups).

In randomized prospective studies of adverse events of drugs, a power analysis is par for the course.  Showing the no adverse reactions were found is easy with small sample sizes, because the variability estimated within and between the two groups (say, treated vs. untreated) is a mathematical function of sampling effort N1 and N2.

Any qualified data analyst, epidemiologist, statistician, or scientist knows this.

Any study in vaccine safety that demonstrates a negative result for any given adverse event may show a negative result for two reasons:

(1) No difference exists between the two populations under study with respect to rates of the adverse event of interest, or

(2) The study was conducted with a sample size that was too small to ensure detection of a difference in the frequency of adverse events between the two populations under study.

That is, the study had insufficient power.

Unless a power analysis is conducted, no one – NO ONE – interpreting the results has any position to choose between the two reasons as their interpretation of the results.

Any qualified data analyst, epidemiologist, statistician, or scientist knows all of this, too.

It is flabbergasting why, then, so many studies from the CDC and CDC-affiliated scientists make conclusions such as

• “The relative risk of severe neurologic disease in the 0–7 day risk period after meningococcal C conjugate vaccination was estimated at 1.28 (95% CI, 0.17–9.75).  As evidenced by the wide confidence interval, the sample size is not large enough to get a more precise estimate of the relative risk. The authors concluded that administration of meningococcal C conjugate vaccine is not associated with an increased risk of severe neurologic disease within 0 to 7 days of vaccination.”[1]
• And then issue a official policy-sounding statement, such as:

  Weight of Epidemiologic Evidence

  The committee has limited confidence in the epidemiologic evidence, based on one study that lacked validity and precision, to assess an association between meningococcal vaccine and encephalitis or encephalopathy.[1]

• Leading to a  policy of adoption of the vaccine, based on the rationale:”There is no scientific evidence of serious neurological disease as result of this vaccine” or “No scientific study has shown…”

The Safety Assumption

There is an expression that is sometimes a fallacy, and sometimes not.  It goes like this:

“The absence of evidence is not evidence of absence”.

This is largely considered a general fallacy – due in large part to Carl Sagan’s use of it in arguing for consideration of the likelihood that we just have not discovered all of the missing pieces of the evolution of the cosmos, but we can nevertheless deduce from other evidence that those missing pieces – the absence of evidence – nevertheless occurred.  And we can certainly use the absence of evidence to deduce that some events that have not occurred have, in fact, not occurred.

But when it comes to science, especially epidemiological comparisons of rates of biomedical events in populations, there is a set of conditions in which the absence of evidence MAY NOT be used as evidence of absence.

Those conditions are when the study has low statistical power.

According to the committee statement, they concluded no evidence of the suspected increase in neurological adverse events from the vaccine due in part to a lack of precision.

“Lack of precision” means “high variability”, as in that reported in the sentence in the study:

“As evidenced by the wide confidence interval, the sample size is not large enough to get a more precise estimate of the relative risk.”[1]

The specific committee statement is carefully selected so as to allow the errant policy interpretation.  They committee could have, and should have written:

The committee has no ability to rule out an association between meningococcal vaccine and encephalitis or encephalopathy based on the study, due to a lack of precision, resulting from a small sample size used to assess an association between meningococcal vaccine and encephalitis or encephalopathy.

In other words, a firm “We Don’t Know Because the Sample Size was Not Large Enough”.

This policy-like statement has a higher degree of fidelity to the limits of knowledge (LOK) imposed on interpretation of the study due to small sample size.

Policy based on a lack of evidence that results from a lack of statistical power, or destruction of validity of conclusions of studies via cherry-picking results after applying “kitchen-sink” statistics, is dangerous, because it requires the Safety Assumption: that a lack of evidence implies evidence of absence.  The Safety Assumption only applies after a negative result has been found AND a power analysis has demonstrated that a positive result WOULD have been found given the sample sizes (N1, N2) and a priori estimate of the effect size.

Of course, if you control the sample size, you control the power…

There are many parents who know full well that their child suffered seizures, encephalopathy, and autism as a direct result of the vaccine – (aka “Vaccine-Induced Encephalopathy-Mediated Autism, or VIEMA). They know the vaccines cause their child’s autism as sure as they would be able to tell you their child was injured if they saw the child get hit by a moving car. They don’t need a significant result, nor a p-value, nor a power analysis. As the number of these parents in the population explode, an army of misled, misinformed parents are created.  Fairly rapidly, however, these parents are waking up and becoming informed.  When they attend courses to learn about statistical power – and how ridiculously simple is it to execute power analysis with available software, including many free online applications – they will have their time in the sun, their day in court, and they will re-dedicate their lives to righting the wrongs of the CDC and preventing further injuries.  They, unlike scientists in academia, are not dependent on a culture of turning a blind eye to these willful acts of misinterpretation in the name of a ‘common good’ of vaccination.  And, as I pointed out, they are growing in number every day, possibly by as many as 250,000 vaccine injured people/day.

Vaccine Safety Science as an Archery Contest

Imagine an archery contest in which a choice prize – a bag of gold – is given for hitting a bull eye’s on a target.  In each round, a contestant gets a single draw.  Each time a contestant hits the bull’s eye, they get another bag of gold.

A contestant lines up, draws, aims, and lets go. They miss the bull’s eye because their aim is not good.  No bag of gold for them.

Another contestant lines up, draws, aims, and lets go. They hit the bull’s eye because their aim was good, and their pull was strong.  All in witness of the tournament can see that the bag of gold is well-earned.

The CDC lines up, aims, draws a little, and the arrow flies about three feet, falling far short of the target.  “Bullseye!” they claim!

In reality, they neither hit, nor missed the bull’s eye because they did not draw with sufficient power to enable an appropriate assessment of their aim.

The CDC’s widespread abuse of knowledge – because that’s what it is – leads to the situation where it appears as if a positive study has been conducted upon which public policy is based, when, in reality, the study might as well not even ever have been conducted.

In reality, the above analogy is better fit if the goal of the contestants is to MISS the bull’s eye (no positive result), but you get the idea: The amount of empirical information generated by underpowered studies with negative results is zero.

What is even more disturbing is that the practice of Steps 1-7 have been repeated over, and over… and over.  And worse than that – there is a pattern of CDC analysts taking the extraordinary step of changing study designs by omitting specific patients from studies for arbitrary reasons – with the result being a reduction in the sample size, yielding a corresponding drop in statistical power – AFTER finding a positive association with the full sample available for analysis.

In Part 2, I will enumerate examples in which current public policy on vaccines are based on the illogical, unwarranted Safety Assumption.

In Part 3, I will review the evidence that CDC officials and collaborators committed scientific fraud by taking extraordinary steps to corrupt otherwise robust study designs to reduce statistical power, and publish only the final, negative results, with no reference to the initial positive association.

These abuses of science are widely known as ‘heinous crimes’ in the biostatistics literature. So far, the academic community of statisticians have been oddly silent on these issues, but I will be sharing this series of posts with them for their consideration.

 

 

 

 

Citation 

[1] Committee to Review Adverse Effects of Vaccines; Institute of Medicine; Adverse Effects of Vaccines: Evidence and Causality. Stratton K, Ford A, Rusch E, et al., editors. Washington (DC): National Academies Press (US); 2011 Aug 25. Chapter 11. Meningococcal Vaccine. http:// www.ncbi.nlm.nih.gov/books/NBK190008/