The claim that “extraordinary claims require extraordinary evidence” (ECREE) is often used in argumentation, usually with regard to claims of a religious, paranormal or pseudo-scientific nature. It's often used axiomatically or as a dictum in debates; its popularity being attributed to Carl Sagan. The purpose of using the statement is to show that the claim in question should not or cannot be accepted as the evidence required to support it is beyond what would normally be acceptable for a more mundane claim.
How robust is this saying as a factual claim though? Does it stand up to scrutiny?
Lack of definitions
There is no definition given, nor is there any general consensus, for what constitutes an extraordinary claim or extraordinary evidence, so a good place to begin is to look at what they may mean.
The first thing to note is that the word “extraordinary” appears twice in the statement, so it is important to examine whether the word is used consistently. One way of doing this is to choose a likely meaning for it and substitute the new word or phrase in its place to see whether the statement still holds. With regard to claims, “extraordinary” is usually taken to mean something along the lines of “paranormal”, “highly unlikely to be true”, “violates the laws of nature”, or “goes against our current understanding”. Substituting these replacements into the statement for “extraordinary” gives us:
- Paranormal claims require paranormal evidence.
- Claims that are highly unlikely to be true require evidence that is highly unlikely to be true.
- Claims that violate the laws of nature require evidence that violates the laws of nature.
- Claims that go against our current understanding require evidence that goes against our current understanding.
These statements clearly do not make sense. This means that the word “extraordinary”, as used in the statement “extraordinary claims require extraordinary evidence”, is used equivocally: the adjective “extraordinary” does not mean the same thing when used to describe “claim” as it does when used to describe "evidence". Rarely, if ever, do proponents of ECREE state what they mean by “extraordinary evidence” – it's usually implied to mean “more” and/or “better quality” but its extraordinary nature isn't addressed. If evidence for any claim can be gathered for scrutiny, such as from observation or scientific experiments, then this is ordinary, standard evidence. The nature of the claim is of no relevance.
Both “extraordinary claims” and “extraordinary evidence” lack precising definitions, so are ambiguous and vague. This is enough to invalidate the statement as it stands as a factual one as the statement can be subjectively interpreted: what one person considers to be an extraordinary claim may be interpreted as a perfectly normal or acceptable claim by someone else.
Is there an underlying principle nonetheless?
This problem of equivocation may have arisen simply because the phrase was designed to be catchy and memorable; but is the “pop science” nature of the phrase (which was probably worded to appeal to a non-scientific audience) based on an underlying scientific principle regarding claims and evidence?
The phrase suggests that there is some directly proportional link between the nature of a claim and the nature, amount and/or quality of the evidence required to support it. But is this true?
Proponents of the phrase will often give examples by using contrasting claims such as “I have a pet dog” versus “I have a pet dragon” or “I have captured a new species of lizard” versus “I have captured an alien” and explaining that the first instance is not extraordinary therefore very easy to accept whereas the second claim is extraordinary and would require more evidence than the first.
In order to establish that someone owned a pet dragon (a fire-breathing one, not a Komodo!) all that would be required is that they produce the specimen - which is no different from them producing a dog. Likewise, producing an alien's body for anatomical examination is no different from producing a newly discovered lizard's body. The finding may be “extraordinary” in some cases, but the evidence required to establish the claim is still quite “ordinary”.
The correct way to test something is not to generate examples of where it is true (i.e. attempting to confirm it) but to generate examples of where it is false (attempting to falsify it). In order to test whether ECREE is true, all that's required is to see whether there''s an example where it is or would be false. An example would be: if someone claimed they could levitate an anvil using their psychokinetic power, something that would qualify as an extraordinary claim for most people, it would be very simple to design a properly controlled test capable of establishing the claim should it be true. There would be nothing extraordinary about the testing or the evidence, even if a positive result would be truly astonishing.
Sometimes arguments using Bayesian statistics are used to defend ECREE. The idea is that a claim with a very low prior probability (i.e. extraordinary) will require evidence that is proportionally substantial enough to overcome the claim's low prior probability. Such evidence is then defined as 'extraordinary'. However, this is simply circular reasoning: it's nothing more than restating ECREE in slightly different terms.
All claims are different and will therefore require different levels, amount and types of evidence in order to establish them. Some simple claims will require minimal evidence and some simple claims will require substantial evidence. Likewise, some unlikely or complex claims will require simple evidence and some will require substantial evidence. The point here is that there is no fixed relationship between a claim and the evidence required to establish it. Each individual claim needs to be assessed and the evidence required to establish it worked out as appropriate.
i.e. Claims require sufficient evidence - and what is sufficient needs to be determined independently from the nature of the claim.
ECREE used in debates
People who oppose or reject claims that they deem to be “extraordinary” will often use ECREE in debates as if it is true or a confirmed and accepted scientific principle. And because they assume this, it is often used to dismiss claims out of hand or in a “moving the goalposts” manner where any evidence that is presented is rejected because it isn't extraordinary enough – although what would be extraordinary enough isn't normally stated.
If, however, ECREE is used as a general, guiding rule (in a similar way to Occam's Razor) as a precaution against being too quick to accept weak evidence as confirming claims that are highly unlikely, then its use may be more reasonable; however, it's still the sufficiency of evidence that's important, not its nature.
Conclusion
The claim “extraordinary claims require extraordinary evidence” lacks definition which makes its meaning ambiguous. Therefore, it is not legitimate as a scientific claim or rule. Even if someone defines its terms before using it, it's still problematic as the underlying assumption - that there is a universal, directly proportional relationship between the nature of a claim and the nature of the evidence required to support/establish it is not true.
Although there are instances where its usage as a precautionary approach to accepting unlikely claims may seem appropriate, caution should always be exercised when ECREE is used or encountered as it is not a universal principle and so can be misapplied or even abused: to dismiss evidence, avoid debate, avoid assessing pertinent claims, etc.
'