The Null Hypothesis
We have seen the basics of the hypothetico-deductive method. However, that is just the bare bones of the "scientific method". In practice, we need to expand out tools to be truly effective.

One of the first tools is called the null hypothesis. Consider that hypotheses are "statements of patterns". But also consider the fact that humans perceive patterns whether they are really there or not! How do we go about dealing with this?

One way is to state a "null hypothesis": generally, the statement that the pattern or relationship we are interested in does NOT exist. For instance, if we have preliminary evidence that a particular treatment X is helpful in curing a disease Y, we can make the null hypothesis "Treatment X does not result in any greater chance of curing disease Y than doing nothing." We can then run experimental studies where patients either recieve treatement X or just a placebo (a "nothing"), and see if there is a statistically significant better cure rate for those who recieved X than otherwise. (We need to frame this as in the form of statistical significance, because many diseases might be cured without any treatment: the hypothesis we are interested is that treatment X is actually a useful treatment!). If we cannot reject the null (that is, there is no statistically significant rate of improvement for those who had X than those who didn't), we have to conclude that present data show that treatment X is not an effective cure.

Or, take this example. Since the 1960s there has been the popular claim of a "Bermuda Triangle" where ships and planes mysteriously vanish. It makes for a compelling story. All sorts of explanations have been proposed, from relatively mundane (unusual local magnetic fields screwing up navigation; undersea volcanoes or methane seeps; etc.) to the bizarre (portals to other dimensions; aliens bent on kidnapping ships and planes; Satan; etc.). But few of the the people who come up with these explanations ever bothered to state the null: "Ships and planes disappear in the Bermuda Triangle at a rate higher than one would expect from the normal background level."

As it turns out, author, research librarian, and pilot Larry Kusche DID bother to ask the null. In his 1975 The Bermuda Triangle Mystery - Solved he investigated the claims that had been made about the disappearing ships and planes. Some were entire fabrications, others weren't mysterious at all (i.e., ships claimed to have disappeared at day in calm seas actually disappeared at night during a hurricane, etc.), and some weren't even in that part of the Atlantic. When the fabrications were removed, it turned out there is NO greater loss of ships or planes in this area of the Atlantic than one finds in a comparably large patch of heavily-travelled sea anywhere in the world.

In other words, we cannot reject the null hypothesis "ships and planes disappear in the Bermuda Triangle at a statistically higher rate one would expect from the normal background level." There IS NO BERMUDA TRIANGLE. There is NO phenomenon requiring explanation.

So keep in mind: if you can't reject the null, you are probably just wasting time going any further. Unless you get new information that allows you to reject the null hypothesis, there is no reason to try to explain or describe things that don't have evidence of existing!

Uncertainty
An important issue in scientific matters that is often misunderstood is the issue of uncertainty. Many people seek absolute and precise answers to scientific questions. However, limitations of observations (for example, the accuracy or resolution of measuring instruments) and of theories often means that the predictions and estimations of scientific issues are presented with error bars, statistical statements of certainty, and the like. In other words, we can generally predict an answer will very likely fall in some small range of values, almost certainly fall within some larger range of values, and almost certainly not fall outside this range. (For example, I can predict that the high temperature for July 4th in College Park next year will likely be between 300 and 311 K; almost certainly between 283 and 316 K, and almost certainly not 100 K or 1000 K!)

Improved technological and mathematical solutions, as well as an increased number of observations, can often greatly reduce the range of possible values and thus increase our resolution of a particular estimation. Nevertheless (TV forensics and spy movie scenarios notwithstanding!) there is nearly always some uncertainty that will remain. You cannot create information that isn't there, although you can (sometimes) identify true signal with the "noise" of extraneous material.

For some people, the fact that Science provides a range of answers of greater or lesser likelihood of accuracy is disconcerting. They want a nice single answer with no error bars. Unfortunately, that is not the Universe we inhabit. Towards this end, in June 2010 a "mantra" started circulating in scientific circles that might be helpful to remember. (It is based on the Serenity Prayer, and has been called the Uncertainty Prayer):
Grant us...
The ability to reduce the uncertainties we can;
The willingness to work with the uncertainties we cannot;
And the scientific knowledge to know the difference.

We strongly encourage our students to adopt this philosophy!

"Just a Theory"

The word "theory" is used in several different--and non-overlapping--ways. Common usage is "theory = guess or conjecture", and as such would be something weaker than an hypothesis (which at least is something that is testable (and ideally already subjected to tests which could falsify it). This is the version of "theory" that people mean when they say such-and-such idea (evolution; the Big Bang; anthropogenic global warming; etc.) is "just a theory."

Please DON'T USE "THEORY" IN THIS WAY! It is sloppy; it is inaccurate; and it doesn't really represent what the word means to scientists, philosophers, or pretty much any academic or professional.

Instead, theory has a much more important academic meaning. In broadest terms, a "theory" is a "a system of ideas intended to explain something" or a "set of underlying rules" or "operational logic" or (most simply of all) "an explanation". This concept is used in many fields beyond the sciences. "Color theory", "music theory", various economic theories ("laissez-faire capitalism" or "Keynesian macroeconomics" or "Stalinist communism" or so on) and political theories ("libertarianism" or "fascism" or "democratic republicanism" or "constitutional monarchism" etc., etc.) are all examples. (And note that just because they are perfectly good theories doesn't mean that they describe phenomena that aren't real: i.e., no one would suggest color, music, economics, and politics aren't real just because they are described by theory!) Because theory in this sense is a set of rules, it can be predictive (i.e., knowing the starting conditions, you could figure out in advance the likely outcome: for example, what happens when you mix red and blue light.) In this broad context, theory is much closer to the definitions of "model", "scheme", or "rule" than it is to "guess" or "conjecture".

However, under this definition there is no part of the definition that requires a theory to be accurate or realistic!! Anyone who has played (or created) complex video or role-playing games knows that they can operate by a set of rules and underlying logic that do not match the way the real world operates. So to with theories. One can come up with economic or political or philosophical or other theories that are internally consistent and logical but still do not match the real world.

An example: acupunture theory (as distinguished from the practice of acupuncture) is premised on the existence of a vital energy field (qi: think "The Force" from the Star Wars universe) that flows in certain patterns in the human body (and in the cosmos around us). The theory is that diseases are caused by disruptions of this energy, and that by manipulating the energy flow by the insertion of needles at key points the acupuncturist can reduce or cure that disease. It is a consistent, self-contained, logical theory; it is predictive; and indeed it fits into a larger theory of the world from traditional Chinese cosmology (feng shui, or "geomancy" [Earth magic]) where the qi is thought to flow through the Universe and that our physical position and orientation in this flow can modify our health and luck. Unfortunately, for all its logic and consistency it is also wrong: there is no qi as such, and diseases are caused by chemical and biological processes.

(An aside: just because the theory of acupuncture is wrong doesn't mean that the practice is necessarily wrong. Sticking needles into the body (either randomly or at the particular traditional points suggested by acupuncturists) could possibly affect some physiological response, but this might well be for entirely different underlying causes.)

In the humanities and philosophy a theory need only be consistent, and thus can be based purely on ideas. In the sciences, however, theories must have an empirical component.

A scientific theory is a thus a subset of the broader category of "theory". A scientific theory can be defined as a "comprehensive framework for describing, explaining, and making falsifiable predictions about related sets of phenomena based on rigorous observation, experimentation, and logic".

Let's consider Merck's example of a very good hypothesis: the Sun always rises in the East. An easily falsifiable hypothesis, and one which has withstood millenia of tests of disconfirmation. The hypothesis answers the "What?" question (or, more precisely in this case, the "Where?" question): "Where does the Sun rise?" However, it does not offer an explanation for why the Sun would always rise in the East. Hypotheses describe patterns, but not necessarily causes.

There are many traditional "theories of eastern sunrise": for example, that the East is where the Sun God's chariot or boat emerges from the underworld, and the West is where it descends back. Given the discovery of a spherical (rather than flat) Earth, new models were proposed. Two prominent scientific theories of eastern sunrise were worked out over time:

At this point we haven't seen how we should choose one theory over another. In fact, we can test and reject theories in part in the same fashion as we do hypotheses: that is, by parsimony and by tests of falsification. For example, it had long been observed that from the point of view of someone on Earth that the planets showed a peculiar retrograde motion: when plotted against the so-called "fixed stars", the wandering stars (planets) moved backwards than forwards again on the scale of months and years. To the geocentrists this required a second set of spheres stuck onto the spheres to produce epicycles. To the heliocentrists, though, it required no change in the motion of any of the planets (including the Earth): instead, it was just an optical illusion produced by relative motion of the different worlds. Furthermore, while no evidence for the crystal spheres themselves came to light, the work of various astronomers and physicists uncovered gravity, a single force that could explain both motions on the Earth and motions in space that followed the heliocentric model. Thus, the heliocentrist model was more parsimonious.

For a graphical portrayal of these different models, see the following:

There are additional ways by which the fitness of a theory is judged. These include:

Again looking at the theories of eastern sunrise, new data was observed and new hypotheses were brought to bear that were inconsistent with geocentrism but consistent with heliocentrism. These included such things as:

Thus, the theory of geocentrism was utterly rejected. But the theory of heliocentrism as originally envisioned was also incorrect! After all, Copernicus considered everything to have orbited the Sun, yet the universe turned out to be far vaster than he had conceived and nothing outside the Solar system orbits the Sun. So the heliocentric theory was modified into a more general theory of gravity-based astrodynamics. Similarly, the discovery of Einsteinian relativity did not utterly reject Newton's laws of motion: instead, the latter became a special case of the former (basically, the case under lower speeds and low gravitation.)

This highlights a difference between hypotheses and theories: while hypotheses (being limited in scope) stand and fall as a whole, theories often have subcomponents rejected, modified, replaced, and added to over time.

Contrary to some statements (even by reputable scientific organizations), theories are not limited to describing only large scale ("universal") phenomena nor ongoing phenomena. Thus, "one time" only events (such as the impact theory of the extinction at the end of the Cretaceous or the Big Bang at the beginning of spacetime) or phenomena limited in scope (such as the theory of thermohaline circulation as the major driver of modern climate, or again of the impact cause of the Cretaceous extinction) are indeed subject to theory, and these theories can be explored and tested via parsimony, fecundity, consilience, auxilary hypotheses, and so forth. Nevertheless, the most important (and most fecund) theories ARE about large-scale ongoing phenomena: atomic theory of matter; periodic theory of the elements; special relativity; the germ theory of disease; the theory of evolution by means of natural selection; plate tectonic theory of geology; and so on.

It is very common to find that in certain spheres of Science there are multiple different (and sometimes mutually exclusive) theories proposed for the same observations. Indeed, this is what a lot of scientific research is about: the creation of and testing of new theories and their auxiliary hypotheses. Hence there are fields like theoretical physics in which new models of the operation of the universe and its various components are proposed and assembled based on previous observations, logic, conjecture, and speculation. Other scientists (experimental physicists, observational astronomers, etc.) themselves look for observations that could in principle reject some or all of the components of these theories.

One last comment: what are scientific laws? Popular accounts of the scientific method suggest a hierarchy of observation → hypothesis → theory → law, but this is not correct. The phrase "scientific law" in the Sciences was largely been abandoned in the 20th Century. Many of the traditional "scientific laws" were simply scientific theories that can be rendered as mathematical equations. As a consequence they tend to deal with relatively simple and more easily measurable phenomena. "Scientific laws" were thus no better nor worse than other scientific theories at withstanding rejection: for example, Bode's law of planetary orbital distance wound up being a coincidence more than a law; Newton's laws of motions only apply to certain gravitational conditions and speeds; and so forth. "Scientific laws" can be useful in some circumstances (e.g., calculating gas pressure, volume, temperature, or number of particles given the other variables using the ideal gas law PV = nRT), but there are many realms of Science where the phenomena are too complicated to be reduced at present to law-like forms. This holds true of much of geology, climatology, biology, and especially behavioral sciences. So be very careful if you hear from someone who proposes some "law" of biology or anthropology or psychology!

There is a subset of theories that have withstood substantial repeated tests and modifications and survived if not unscathed, at least strongly supported and the victor against all challengers so far. These incude (but aren't limited to) the atomic theory of matter; the theory of evolution by means of Natural Selection; plate tectonics theory; the germ theory of disease; etc. We honestly don't have a good term to distinguish these theories from the more run-of-the-mill, still-in-play types. (One might suggest facts, and there is some merit there, but a "fact" might be a better synonym for a well-substantiated observation rather than an entire theory.) The British evolutionary biologist Richard Dawkins proposed (in 2009) using the new word theorum for this class of theories. Time will tell if this nomenclature will catch on.

Regardless of what we call them, we can describe this collection ideas as theories which are to the best of our knowledge "true", but this requires the caveat that absolute "Truth" is empirically impossible to find (although it would be extraordinarily perverse to reject these things as being "true" in a general sense without some extraordinary new evidence to the contrary.) These winning theories can be thought of as the all-time prize-winning champions in their respective fields, having faced and defeated all challengers. In principle (indeed, "in theory"! :-) they could be defeated by a newcomer, but the weight of evidence so far is with the champions.

Some Relevent Videos
An excellent little video by Adam "Mythbuster" Savage about putting together lines of evidence to develop complex scientific ideas:

"How It All Ends: The Nature of Science" (a three-part video series by Greg Craven).

Part 1 (8:51):

Part 2 (8:06):

Part 3 (7:53):