The text within is dedicated to helping patients and caregivers better evaluate medical claims - to help distinguish between strong and weak information; how to avoid confusing cause and effect, and how to better recognize hype, fraud, and wishful thinking.  

We can do this by learning to ask questions of the information we receive, and by recognizing the factors that can distort medical claims and scientific findings. These factors include conflict of interest, and bias - including biases common in patients - and by learning about the scientific methods used to evaluate therapies for cancer and other medical conditions.

Biases: different reasons for different groups:

Bias:  according to the Webster dictionary is  "an inclination of temperament or outlook; especially: a personal and sometimes unreasoned judgment." Bias can influence how one looks at outcomes, and how tests are constructed in order to favor the result that is wanted or wished for.  

Conflict of interest occurs when an individual's personal, professional, or financial interests intentionally or unintentionally influence decisions on scientific methods, or how data is interpreted. 

Financial conflict of interest leads to bias: "A financial conflict of interest, I believe, is any financial association that would cause an investigator to prefer one outcome of his research to another. Let me give you an example. If an investigator is comparing drug A with drug B and owns a large amount of stock in the company that makes drug A, he will prefer to find that drug A is better than drug B. That is a conflict of interest."   ~ Marcia Angell, M. D. - dhhs.gov

Anyone that develops new drugs or sells supplements has an inherent financial conflict of interest in respect to evaluating the true value of their product or service. As just one example, this financial bias can lead to selective reporting or so-called hype in order to maximize shareholder confidence and the value of the company as just one example. 

Similarly, scientists who have a financial interest must disclose these relationships, and they are inherently less able to overcome the biases these secondary interests can create. Consider that by investing in a company a scientist demonstrates a belief in the value of its product, perhaps in advance of definitive evidence.  Scientists have an ethical responsibility to avoid arriving at conclusions ahead of time.  The discipline of good science requires that theories are tested in well-controlled studies, and that the outcomes are evaluated objectively before conclusions are made.

Intellectual bias: Investigators and scientists can develop unintended biases about the value of their work, idea, or intellectual property. Investigators may unconsciously see benefit when none exists, or they may set up a study in ways more likely to reveal weaknesses, exaggerate the benefits, overlook unanticipated side effects, and so on.  And sometimes patients will recommend to other patients what they themselves have done as a way of validating it. While this may seem   reasonable, a few cases are not predictive of what will happen to you or to the average patient. See weaknesses of anecdotes and observational studies below.

Denial: In order to cope with living with a life-threatening disease, patients or caregivers may develop a tendency to minimizes the dangers of the the disease, or to inflate the potential of alternative and other less toxic approaches to control it.  Potentially, denial can lead to missed opportunities and delays that can make the disease more difficult to treat.

Fears: Fears are understandable and human, and sometimes can be justified. In patients the fear of the toxicity associated with many standard cancer therapies can form a bias in favor of claims made for safer alternative, or even investigational low toxic therapies. Similarly, the fear of questioning authority may prevent patients from seeking a second opinion or from exploring appropriate investigational therapies.      

The above speaks to the need for a strong independent and impartial agency to evaluate study findings so that the public can have confidence in the medical products we ultimately receive and rely on.  

Confusing Cause & Effect

It's natural to look for connections between one event and another. But it's important to consider that chance or numerous other factors could also explain the "effect." Therefore it can be misleading to form beliefs on observations and anecdotes, even when supported by logical ideas. 

The significance of so-arrived-at beliefs can range from harmless to dangerous.  A harmless but extreme example is when a baseball player hits a home run and observes he had eggs for breakfast that day; who then continues to have eggs on game days. 

But making observations and exploring potential connections between events (forming hypothesis) is the starting point of science --  inspiring further research and well-controlled experiments that prove or disprove the observed possible connections.

Here are some common and potentially misleading assumptions about cause and effect related to indolent lymphoma: 

• My disease is stable, therefore the life style changes I have made are helping.*

• An assay predicted my response to treatment, therefore the assay is proven to be valid.

• My lymph nodes are regressing, therefore the investigational vaccine I took is effective.

• My lymph nodes started growing while I took xyz, therefore xyz causes progression.

• My lymph nodes started regressing while taking xyz, therefore it's effective against the disease.

*NOTE: Patients with indolent lymphomas may be particularly susceptible to confusing cause and effect, because the natural cause of the disease is more variable than for most other cancers. Therefore, if a practitioner prescribes a life style or alternative protocol that 100 patients follow, as many as 30% are likely to do well for a time - not necessarily as a result of the protocol, but because they would have done well anyway. This "effect," that has good probability of being unrelated to the practice, will often result in strong belief and promotions, as in: "How can you argue with success?"

This is not to say some unproven practices are not helpful in some way or degree.  And if the practices are harmless and based on plausible theory we should not condemn them out of hand, so long as the patients are informed of the alternative explanations. In other words, we believe that practitioners have an obligation to state that their ideas and practices have not been proven in controlled studies to provide benefit.  Furthermore, patients should learn to look for potential financial or intellectual biases - such as does the doctor profit from me using his or her protocol - and take this fully into account. 

Baseball and the FDA:

Pitchers want to believe every close pitch is a strike and that the strike zone is too small.  So just as we recognize the need for a neutral party to call balls and strikes in the game of baseball, we should also recognize the need for an impartial agency, such as the FDA, to evaluate claims of clinical benefit.  

In order to test claims for a given drug, the drug sponsor must conduct well-designed studies that minimize bias and demonstrate clinical benefit.  Without such a system we would have a "Wild West" environment with no means of making informed or safe medical decisions, and no good foundation on which advances in clinical science could be made. 

Is the FDA without bias and conflict of interest?  No agency or human being is completely free of bias, but the agency is committed and mandated by law to achieve impartiality. There are strong policies on ethics and conflicts of interest in place, and criminal penalties can be invoked for violations of these regulations.

• See Death by Medicine Paper at http://www.InteliOrg.com/archive/Death_by_Medicine.pdf

About FDA's Ethics Program: "The Agency’s ethics program is administered to help ensure that decisions made by Agency employees are not, nor appear to be, tainted by any question of conflict of interest. The "ethics" laws and regulations were established to promote and strengthen the public’s confidence in the integrity of the Federal Government. The Agency’s Ethics and Integrity Branch provides advice and assistance to FDA employees on a variety of ethics related matters including, but not limited to, financial disclosure, prohibited financial interests, outside activities, co-sponsorship agreements, and post employment."

See FDA.gov/opacom/ethics/ 

Is anyone's perspective inherently more believable?

Perhaps we can learn quickly from the perspectives of doctors and scientists who have cancer, as the threatening nature of the disease is likely to remove any financial biases they may have had in respect to the integrity of the drug evaluation system in America. And if a secret alternative cures exists, won't these professionals have the background and the strong motivation to recognize and validate these approaches?   

Here as an example we offer the perspective of an inspiring individual and cancer survivor, 
Medical Doctor

"Patients who don’t understand the difference between information based on theory, anecdote, historical analysis, or double-blind placebo controlled studies are making ill-informed decisions, believing alternative therapies are safer or more effective when they are not. Even patients who presume that alternative therapies are ineffective may use them. Why? When faced with a life-threatening disease requiring highly toxic treatments with no guarantees, or when dying because there are no effective conventional treatments, it takes guts to reject something or someone claiming to be able to save you, just in case you might be wrong."  -  Wendy S. Harpham, MD (NHL survivor) Full text: amcancersoc.org

Terminology to help you assess clinical data and medical claims:

Abstracts | Theory | Treatment Response | Statistical Significance (p-value & confidence)

• Abstracts: Abstracts are summaries of larger papers and therefore do not contain all the available details of the study methods and data.  Abstract conclusions may not be accepted by experts in the field. Reputable peer-reviewed journals sometimes require modifications to conclusions from the original abstract for this reason -- or they may reject the paper from publication because it was determined that the methods (methodology) or data did not support the conclusions made in the abstract. Therefore, it's important to avoid forming conclusions on the basis of abstracts.  They should be considered only a starting point for discussions with your doctors and perhaps a basis for additional inquires and research.  

Questions to ask: 
 
1) Was the paper published in a respected journal? 
 
2) What types of studies and methods were used to reach the conclusions? 
 
3) Do papers published by other groups support the conclusions?  

Reproducibility is valued in science, especially when a finding comes from a different investigative group. The reason for this is that it reduces the chance that bias, or choice of methods, influenced the findings or the conclusions. 

• Theory is an idea or hypotheses. In clinical science theory is often based on what is known about the disease, and also perhaps related factors, such as the immune system. There have been many theories of disease intervention that have sounded rational, but haven't worked when tested. 
   
  Theories are starting points for experiments and studies. They should not be regarded as a proof.  When someone tells you that this is how a treatment works and that it is therefore desirable, you might ask:  
   
     What clinical data supports the theory?  
   
     Who published the findings, and in what journal?  What are the possible biases?
   

• Treatment response: an often misused term to describe clinical benefit from a treatment based on a clinical change, such as the reduction in size of a lymph node. 
  
  Treatment responses, however, may or may not result in clinical benefit - improved survival or the reduction of symptoms. For example, with lymphatic cancers the lymph nodes can increase and decrease in size because of transient inflammatory reactions, which could lead to false assumptions about the benefit of a treatment or a life style intervention.  
   
  Some questions to ask about response: 
   
  1) How long was the response? 
  2) At what intervals were the responses measured and with what tests? 
  3) Did the measured response correspond to clinical benefits?
  4) Who reported the responses and were the outcomes verified by independent reviewers?
  5) How large was the patient sample, and how were they selected? 
  6) What is the expected clinical course of the disease?
   

• About statistical significance:  In order to draw better conclusions about data we need to know just a little about measures of statistical significance.  

Statistical conclusions, about responses to treatment in a small group of patients for example, are not absolute.  Everything is possible, but some things are very possible, some are less possible and others are very unlikely -- but still possible to occur. We draw conclusions with a certain amount of confidence, conventionally 95% or 99%, but there is still some chance of an error (5% or lower). Source: stanford.edu  

So lets keep is simple.  We're looking for two measures of statistical significance in scientific reports:  

p-value and confidence interval. If the p-value is .05 or less, the findings are statistically significant. The confidence interval (CI) shows a range that a finding (such as response rate) falls within.   
  

P-value or Probability value: A value that results from a calculation that tells you how likely or unlikely the finding (of a difference in treatment response as an example) was due to chance.  Common language for low p-value: Statistically significant - means unlikely due to chance. Caveats about importance of P-value:  Threshold (cap) is arbitrary, therefore, the closer the p-value is to the threshold (.05), the less statistically significant.    Statistical significance does not necessarily = clinical importance.   Chance is rarely the most pressing issue. Biggest threat is systemic error (bias) Therefore more qualitative questions include: Are these the right pts? Are these the right outcomes? Are there measurement biases? Are observed associations confounded by other factors?   P values provide no information on the results' precision-that is, the degree to which they would vary if measured multiple times. Consequently, journals are increasingly emphasizing a second approach: reporting a range of plausible results, better known as the 95% confidence interval (CI). See below. Factors that influence P-values:  Magnitude of the main effect: a larger difference will have a lower p-value·    Number of observations: a difference noted in a study of 500 patients will have a lower p-value than the same differences observed in a 25 pt group.   Spread of the data (standard deviation): if the observed differences in response are unified and less spread out, the p-value will be lower (more statistically significant)  Source: Adapted from American College of Physicians-American Society of Internal Medicine

Formal definition of a 95% Confidence Interval (CI): "The interval computed from the sample data which, were the study repeated multiple times, would contain the true effect 95% of the time." Common language for Cl:  An estimate of important parameters - how precise or "stable" is your estimate.  CI is often reported with a coverage probability of 95% Confidence intervals are more informative than the simple results of hypothesis tests since they provide a range of plausible values for the unknown parameter. The wider the confidence interval the less precise the reported value or point estimate. Simplified example: Response rate are:  New Drug -- 40/50 pts respond (80% respond)  Standard Drug - 20/40 pts respond (50% respond) The difference in response rate is 80-50, or 30% This resulting 95% CI of 30% calculates to: 11%-49% Since both values in range are above zero, the treatments are significantly different.  We might then say that we are 95% confident that the difference in response rate between the new and the standard drug is between 11 and 49%. How do you calculate the CI?  Use calculator: http://faculty.vassar.edu/lowry/prop2_ind.html

Also see:

• Difference between p-value and confidence interval: - musc.edu

Return to top

Types of Clinical Data (most reliable first)

• Randomized controlled clinical trials: (Provides strongest evidence of clinical benefit)
  Participants are assigned randomly (by chance) to separate groups (arms) for the comparison of different treatments -- usually a standard and an investigational treatment. Patient informed consent is required. Neither the investigators nor the patient choose the group in which participants will be placed. 
   
  Using chance to assign people to treatment arms helps to avoid selection bias -- putting pts in better health in the investigational arm, for example. It also helps to ensure that the groups will be similar and that the treatments they receive can be compared objectively. 
  
  At the time of a trial, genuine uncertainty must exist about which of the treatments is best. These trials can be "double-blinded" or "non-blinded." In double-blind studies, neither the investigator nor the participants are informed of which arm the participants have been assigned to. This also reduces  bias and improves confidence in the findings.
   

• Nonrandomized controlled clinical trials (experimental studies): 
  Participants are assigned to a treatment group based on criteria determined by the investigators, such as prognostic indicators, and disease type. This study design makes it possible for  investigator bias to influence the findings, and therefore there is less confidence that the group receiving the treatment under study and the control group are comparable.
   
• Case series (observational studies): 
  Studies that describe results, such as responses, time to progression, etc.) from patients who received the treatment under investigation.  Case series (observational studies) provide weaker evidence than do experimental studies because of the potential for biases such as, but not limited to, who is observed and what outcomes the observer is looking for --  and unknown association between factors and outcomes -- such as not accounting for other reasons that could account for the observed result in a given group of patients. 
   
  The value of these types of studies (e.g., case series, ecologic, case-control, cohort) is that they provide preliminary evidence that can be used as the basis for hypotheses in stronger experimental studies, such as randomized controlled trials. Consider the recent HRT report finding that using estrogens increases the risk of heart disease and cancers. The hypothesis that it might reduce these risks was based on observations that were proven to be incorrect. 
    

• About press releases, and the reports you don't see: 
  Protocols, ethical principles, and a desire to maintain credibility are beneficial forces that encourage responsible public reporting of drug development research and clinical trial outcomes.  But consider that an easy way to put a positive spin on a company's drug development project is to report or leak to the press only the reports with favorable outcomes, and to keep less than stellar results from being released at all.  Ask yourself: How is response being defined?  How were the patients selected? 
   

• Anecdotes (Least reliable) 
  These are informal reports that may or may not come from direct observation.  Anecdotes may be useful as starting points for future inquires, but they are often the source of misleading information as well. It can be dangerous, wasteful, or both to base decisions on anecdotes for many reasons including, but not limited to, the following: 
   

  • The possible biases - and sometimes the unknown or questionable credentials -- of the individuals making the report or claim; 
     

  • Anecdote will lack details about the specifics of the case, such as natural course of the disease, how responses were measured, and information about the ultimate outcome -- such as how the intervention affected survival; 
     

  • This type of report typically uses vague language to report benefit, such as "response."  See above.
     

  • The anecdote will often be based on assessments of response without use of objective tools, and without review from disinterested 3rd parties, such as the FDA. It will also fail to provide details about the duration of the response, and the overall clinical benefit.
     

  • Anecdotes often confuse cause and effect.  For example, "I took zinc and my bad breath increased or decreased."  In this example, there are a host of other factors that could account for the change, therefore the association cannot be made, credibly.

     

  • Drawback of anecdotes (personal accounts), by example:

    Jane recently asked Jim what weather to expect in New York on October 1^st. Jim reports that it was 25 degrees when he went last year on that date. 

    We know that Jim's account does not predict next year's weather on that day in New York, because we have lots of experience with temperature and weather. At best it's a reference point on what's possible. 

    Examining Jim's report closely it may occur to us that Jim didn't mention where he was in New York, at what elevation, and at what time of day he took the reading ...  Therefore, we don't have other important variables that can tell us how likely his experience is to predict ours.

• In-vivo or in-vitro? 
  We frequently read or hear about the anti-cancer properties of this or that supplement based on scientific research findings.  Here are some questions to ask of this kind of information:
   

  • Was the response detected in a test tube (in-vitro)?  

    The human body is infinitely more complex than a test tube. Nevertheless, indications of activity in a test tube often become the basis for product claims about natural supplements.  Be aware that this can only be a starting point for additional experiments. Using such data as the basis of medical claims is irresponsible, and bias should be suspected.  Furthermore, you might ask if the dose used to produce the in-vitro effect possible to achieve in the body, or can it be achieved safely? 

   

• Animal studies?
  Is the claim for the promise of a drug or supplement based solely on animal studies?  While animals are useful for preclinical testing of new drugs, there are many differences between animals and humans; and drugs that show promise in animals do not always work or work safely in humans.

Junk Science: How Politicians, Corporations, and Other Hucksters Betray Us (Hardcover)

Junk Science Judo: Self-Defense against Health Scares and Scams (Hardcover)

unSpun: Finding Facts in a World of Disinformation (Paperback)

Return to top