Return to the clickable list of items
200) Scientific methods
Ludwik Kowalski (February 6, 2004)
Department of Mathematical Sciences
Montclair State University, Upper Montclair, NJ, 07043
What follows are comments on the so-called “scientific method” in the context of teaching science.
They appeared (in recent days) on Phys-L, a discussion list of physics teachers. The names of particpants
were replaced by Professor 1, Professor 2, etc.
Professor 1:
. . . Please, let's stop taking things on faith. That phrase refers to believing things in the absence of
evidence. That's the opposite of what science is. For a discussion of scientific methods in general, see
http://www.av8n.com/physics/scientific-methods.htm [See appendix below.]
Professor 2 (myself):
It is not possible to "stop taking things on faith" and to help students to master material in any
science course. The laboratory discovery approach is extremely important but it is used to learn only a small
percentage of what is in a typical textbook. Even scientists do not perform all experiments in their specialties;
they often accept discoveries made (and published in refereed journals) by other scientists. How many high school
teachers had an opportunity to experiment with scattering of alpha particles, as Rutherford and Geiger did in 1911?
How many university teachers had opportunities to perform experiments through which existence of quarks was
discovered in 1960s and 1970s? Not too many. We read about such experiments in reputable journals, and in textbooks.
Then we accept what was discovered by others. And we teach it. Should we feel guilty? We trust that contents of science
textbooks are verified by recognized authorities in relevant fields. Yes, I know, it is a complicated issue.
Professor 3:
P2, you have defined "faith" vary narrowly--as anything that we accept other than that which we have
personally experienced. I see that definition as flawed for two reasons,
First, as you have cogently pointed out, we can't directly experience everything, but we can examine the evidence
presented, as well as other corroborating evidence, and the consequences of the findings, and come to a pretty reliable
conclusion about what is being asserted. Factored into our acceptance or non-acceptance of the claim has to be such
things as the methodology of the experiment reported, the nature of the data presented (size of error bars, number of
data points, etc.), the reputation of the investigator (for honesty, objectivity, integrity, and skill as an investigator),
how it fits in with, or explains other related results, and what it predicts that can be tested and verified, and others
that depend on the nature of the experiment and the methodology of the particular discipline.
And second, personal experience is a notoriously poor indicator of reality. We all know about experiments that showed
the results the investigator wanted to see. N-rays are the classic example of that, but the Vienese laboratory that was
looking at the energy distribution of beta-rays during the 20s and found evidence of definite energies of the electrons
emitted in beta-decay, because that was what they were looking for, is another and somewhat more subtle example (see Andrew
Brown's biography of Chadwick, "The Neutron and the Bomb," for more on this). And of course, there are all forms
of hallucinations, flawed memories, and other non-events that take on a compelling reality to the observer. So to be able
to say, "I saw it with my own eyes," is not necessarily as reliable as looking at the record of data taken and
examining the experimental set-up, so see if there were any missed systematic errors built into it.
One of the things that makes science more reliable now than in past centuries is the existence of permanent, more or less
objective recordings of what happened--chart recorder tapes, photographs, automated data files, etc., instead of just
hand-written collections of manually taken observations. this allows others to see exactly the same thing that the original
investigator saw, and so provide it with a more or less objective analysis.
One can argue that all of that is fine, but the farther one is from the original, the weaker is the connection. So the
textbook or monograph reader pretty much has to accept what book's author says is the truth. And where does that leave
the student?
To an extent, that is true, but that chain also provides much reinforcement of the result, since each person who passes
the result along, has at least given it enough examination to decide that it is of enough importance and reliability to
include, and that it fits into the overall picture the author wants to paint. Of course that doesn't rule out all of the
human fallibilities that befall textbook writers, and which can make almost anything is a textbook wrong. Keeping textbooks
"honest" is a continuing process, and students need to know this.
Of course the original result can still be wrong, or incomplete, or not sufficiently accurate to account for subsequent results,
or any of a number of things that can happen to experimental results over the years. But the big difference between scientific,
or evidence-based conclusions and faith-based conclusion is that they are much more subject to change in the face of new evidence.
The faithful say, "I hear what you have said, and it's good enough for me. I believe." The scientist says, "OK,
that sounds reasonable, and it fits with the accepted theories (or it explains something that current theories don't), so I'll
take it and use it until I see evidence that it wasn't correct, in which case I will change my mind." I think there is a
huge difference between these two stances.
Appendix - Scientific Methods (Professor 1 above):
Many textbooks and web-sites describe “the scientific method” in terms most scientists find objectionable. Here is an
attempt to do better.
1. There is no such thing as "the" scientific method. Science uses many methods. There will never be a pat answer to
the question "what is science". The very notion that there could be a pat answer bespeaks an attachment to rote
learning that is incompatible with scientific thinking.
2. One of the goals of science is to make useful predictions.
3. A scientific prediction does not need to be exact to be useful.
4. Sometimes it is possible to make useful predictions, and sometimes not. If you are asked to predict the exact total shown
on a particular roll of a pair of fair dice, you will be wrong at least 5/6ths of the time. But if you can get into a situation
where the payoff is greater than 6:1, you can make some useful predictions, and you can make money on average.
5. Scientists use words like rule, law, equation, identity, principle, formula, algorithm, etc. almost interchangeably, to
describe the process for making predictions (although there are slight variations in connotations).
6. The word “theory” can be used in two radically different ways. The first usage means something like law or rule,
only much grander, namely a system of rules giving a coherent description and explanation of a topic. The other usage
refers to a a mere speculation. Remarkably, both versions are correct, and both have been in use for over 2000 years. It is
best to avoid the word entirely when talking to non-scientists, and especially when debating with persons who can’t be
trusted, since if you intend one meaning they’ll use the other meaning against you. (It sure would be nice to find a word
that expresses the idea of “coherent description and explanation” without risk of misunderstanding.)
7. Mathematical results are validated by formality and rigor. This gives us logical statements of the form “If A then
B” and suchlike. Physical-science results are sometimes validated by logic, but may also be validated by appeal to
experiment. This gives us statements of the form “We observe A” and suchlike. Generally science is a complex
lattice of facts and rules, combining observations and logic.
8. Predictive rules generally have a limited domain of applicability. To state the rule without stating its limits of validity
is improper.
9. From time to time, an established rule may be refined. It may be supplemented by other rules so as to extend the domain of
validity. It may be supplemented by exceptions to improve the accuracy. However a rule with too many caveats and exceptions is
likely to be not only inconvenient but unreliable. Occam’s razor and all that.
10. From time to time, a rule may be supplanted entirely by a simpler and better rule.
11. It is considered very poor form to gripe about the imperfections in an established rule, unless you’ve got something
better to offer.
12. Creating new rules from scratch is exceedingly difficult. There is an infinite number of possible rules, and you will never
have enough data to decide which of the contenders is best -- unless there is some sort of additional guidance. Sometimes guidance
is taken from intuition and from notions of “simplicity” or “elegance”. This is bordering on metaphysics, but
it is an important part of science.
13. Scientists, like business executives, government leaders, and everyone else, must often make decisions based on highly
incomplete data. The important thing is to be able to change your mind as soon as you get new data that contradicts old hunches.
This requires keeping score on each of the rules, keeping track of which are well-supported by existing data, and which are
least-well-supported and therefore most subject to revision.
14. An important scientific activity (which applies not just to pure science but also to engineering and even farming, etc.) is
designing a series of measurements that will tell you what you need to know, without undue waste. See section 2 for more on this.
15. An important part of scientific thinking is being able to recognize non-scientific thinking. Examples include:
* Elementary logic errors, such as circular reasoning, non sequitur, and many others.
* Selecting the data. (It is not right to select tendentious anecdotes from a mass of data.)
* Other misuses of probability.
* Proof by bold assertion. (It’s OK to assert something, so long as you don’t pretend to have proved anything thereby.)
* Appeal to authority (as discussed in reference 4).
* Ad hominem arguments.
* Improperly weighted voting. (A thousand pieces of weak evidence should not outweigh one piece of strong evidence, as discussed
in reference 4.)
* Et cetera.
2 Design of Experiment
Consider the famous Twelve Coins Puzzle as discussed in reference 6. Suppose you find a casino that is willing to pay you $350
for identifying the odd coin, but makes you pay $100 for each weighing. If you weigh the right combinations of coins, you can do the
job in three weighings, so you make money every time. In contrast, if you follow a sub-optimal strategy that requires four or more
weighings, you will lose money on average.
This scenario is reasonably analogous to many real-world situations. Commonly there’s a significant price for making a measurement,
and you want to maximize the amount of information you get for this price.
I mention this because all too often, people claim that a principle of scientific experimentation is to “change only one variable
at a time”. It’s easy to see that such a claim is hogwash. The Twelve Coins Puzzle suffices as a counterexample. If each
weighing differs from the previous weighing by only one coin, you cannot come anywhere close to an optimal solution.
The suggestion to “change only one variable at a time” might nevertheless be good advice in some special situations.
That’s because the cost of making a measurement is not always the dominant cost in the overall information-gathering process.
For example, imagine a situation where gathering the raw data is very cheap, while just plain thinking about it is expensive.
Then you might want to follow a strategy, such as changing only one variable at a time, that makes the data easy to interpret, even
though you had to do large number of experiments (much larger than theoretically necessary). Consider the contrast:
For young children doing cheap, simple experiments, it might make sense to tell them to change only one thing at a time, because the
rate-limiting step is interpreting and understanding the data, and we want to make that step as easy as possible. For skilled
scientists (and engineers, farmers, etc.) doing complex, expensive experiments, changing only one variable at a time would be an
unnecessary burden, and often a disastrous burden.
Changing only one variable at a time is a crutch, which may partially compensate for the investigator’s lack of skill in interpreting
the data. In contrast, for performers with ordinary ability and training, crutches are harmful, not helpful.
3 Correctness and Modesty
As mentioned above, a major purpose of scientific methods is to make useful predictions and to avoid mistakes. The known scientific
methods are a collection of guidelines that have been found to work reasonably well. One of the most important steps in avoiding mistakes
is to always keep in mind that mistakes are possible. This is so important that this whole section is devoted to emphasizing it and
re-expressing it in assorted ways. James Randi said you should take care not to fool yourself, keeping in mind that “the easiest
person to fool is yourself”.
It is OK to a limited extent to be an advocate for your favorite idea, but you must not get carried away. When you collect data in support
of an idea, you must also look just as diligently for data that conflicts with that idea. Then you must weigh all the data fairly, and
disclose all the data when you discuss your idea. This is what sets science apart from debating and lawyering, where advocacy is carried
to an extreme, and it is considered acceptable to skip or make light of data that tends to support the “opposing side”. Another
word for this is modesty. Being aware of your own fallibility is modest. Pretending you are infallible is immodest.
A related form of modesty, which is also crucial for avoiding mistakes, is to not overstate your results. Scientists use certain figures
of speech that are designed to avoid overstatement. Among other things, this includes recognizing the distinction between data and the
interpretation that you wish to place upon the data. As an illustration, imagine some children go on a field trip to the dairy. Upon their
return, they write a childish report that says “cows are brown” -- or, worse, “all cows are brown”. A more modest,
scientific approach would be to say “the cows we observed were all predominantly brown”. A statement about the observed cows
sticks closely to the data, while a generalization about all cows requires a leap beyond the data.
As mentioned above, practically all scientific results have some limits to their validity, and you must clearly understand and clearly
communicate these limits.
4 For Further Reading
a. Richard Feynman, The Character of Physical Law
b. Thomas Kuhn, The Structure of Scientific Revolutions
c. Richard Feynman, The Pleasure of Finding Things Out especially the chapter Cargo Cult Science.
d. Valid versus Invalid Arguments: Appeal to Authority etc. ./authority.htm
e. “Truth in Contrast to Knowledge and Belief” ./truth.htm
f. “The Twelve Coins Puzzle” ./twelve-coins.htm
[Contents] _
Copyright © 2003 jsd