Return to the clickable list of items
117) Exposing false claims
Ludwik Kowalski (12/11/2003)
Department of Mathematical Sciences
Montclair State University, Upper Montclair, NJ, 07043
After posting the previous unit I felt guilty for writing about something that I did not fully understand. The main idea of Dr. Kirk Shanahan’s paper is simple enough but details are not. The paper is not written for people like me; it is written for cold fusion calorimetrists. This explains why I ignored the paper when I first saw it in August. In item #116, I mostly repeated statements made by Kirk without understanding many details described in the paper. Other non-specialists might find themselves in the same situation. What follows is an essay summarizing my understanding of Kirk’s views. The essay is a dialog between a student, a teacher and an engineer. The content is inspired by numerous e-mail messages received from Kirk in the last ten days; the form is inspired by Galileo Galilei.
The main point of this pedagogical exercise is to address the subject without using electrochemical terminology and concepts. I want to focus on well know aspects of error analysis, not on details which are likely to emerge in real research situations. Individual contributions are numbered to facilitate further discussion.
=================================================
1) Student:
My friend, an electrical engineer, discovered an unusual resistor. It behaves normally when the current is low (below 100 mA); the electric power, Pin, and the heat generation rate, Pout, are identical, as they should be. But at a larger current the Pout exceed the Pin. The engineer thinks that he has discovered a new phenomenon; some unknown exothermic process is triggered by the current in the material from which the resistor is made. Is this possible?
2) Teacher:
We do not know everything about nature; new discoveries are likely to be made anywhere and at any time. Your friend may indeed be making an important discovery but I am not willing to accept his claim on faith.
3) Student:
My friend has a calorimeter and other instruments. He said he is willing to make a demonstration. Would you be interested in seeing it?
4) Teacher:
Yes, I would.
5) Engineer (one week later):
Thanks for allowing me to use the laboratory. Everything is ready. Should we go and start collecting data?
6) Teacher:
I would prefer you to first tell us exactly what you want to do. Where is your “magic resistor?”
7) Engineer (reaching in his pocket):
Here it is. The length of this green cylinder is 7 cm and its diameter is 2 cm. Two wires are soldered to the terminals. I can lower this cylinder into water and pass an electric current through it. A thin insulating coating keeps water away from the electric circuit. I will connect this resistor to a d.c. power supply and show that V/I is essentially constant (Ohm’s law), at least up to I=0.1 A. The R will be close to 160 ohms.
Next I will immerse the resistor in water and pass a current I=250 mA through it. The input power, Pin, will be calculated as V*I, where V is the measured voltage across the resistor.
8) Teacher:
How accurate are your voltmeter and ammeter?
9) Engineer:
According to manufacturer’s specifications the accuracy of each instrument is 0.5%. This means that the accuracy of Pin will be 1%.
10) Teacher:
And what kind of instruments will be used to measure the Pout?
11) Engineer:
For this I have built a flow calorimeter. It is a well isolated container whose capacity is close to one liter. Water enters through the pipe at the bottom (passing through a commercial flowmeter) and exits through the pipe near the top. I will set the flow rate to about 50 cubic centimeters per minute. The temperature of entering water, T1, and the temperature of exiting water, T2, will be measured.
12) Teacher:
How accurate are the thermometers and the flowmeter?
13) Engineer:
Temperatures can be measured with an accuracy of plus or minus 0.1C. The factory-specified accuracy of the flowmeter is 1%. The container is thermally insulated; this means that practically all heat (at least 99.9%) is removed from it by water.
14) Teacher:
Your goal is to determine Pout. How will this be accomplished?
15) Engineer:
I will calculate it from the following equation: Pout=S*F*(T2-T1), where S is the known specific heat of water (4186 J/kg*C), F is the flow rate (in kg/s) and (T2-T1) is the measured difference of temperatures.
16) Teacher:
That makes sense. Will the (T2-T1) remain constant?
17) Engineer:
At the beginning T2 will be the same as T1. Then T2 will start increasing slowly while T1 will remain constant (room temperature). After about two hours T2 will stop changing and (T2-T1) will remain constant. This will indicate that in each minute heat generated and heat removed (by circulating water) are equal.
18) Teacher:
How accurate will be your determination of Pout?
18) Engineer:
The difference of temperatures will be about 10 C. Thus the accuracy of the (T2-T1) term will be slightly less than 1%. To be on the safe side I will assume it is 1%. The accuracy at which F will be measured, as I said before, will be 1%. In other words, Pout will be known at the accuracy level of 2%.
19) Teacher (addressing students):
Let me make a comment. Our guest is using the word "accuracy;" rather than the word "precision." This is correct. The word “precision” should be used when we are referring to random errors while the word "accuracy" should be used when we are referring to systematic errors. Suppose Pout turns out to be 40 W. Then, knowing that the accuracy is 2%, we would be able to say that the "true value" of Pout can be anywhere between 39.2 and 40.8 watts. Unlike random errors, which may also be present, systematic errors can not be reduced by performing the same experiment many times to obtain the average value.
20) Engineer:
Yes, I learned about this recently. In some textbooks the terms “accuracy” and “precision” are still used interchangeably. It is easy to be confused. What else would you like to know about my apparatus?
21) Teacher:
I think we should continue discussing your data. The two numbers you provided (2% accuracy for Pout and 1% accuracy for Pin) are highly important. That is why we focused on them first. How large were the actual values of Pout and Pin?
22) Engineer:
They turned out to be 37.4 W and 36.4 W, respectively. To me it means that the difference between these two numbers, 1 W, represents the "excess power." It is the rate at which thermal energy is generated inside the calorimeter. I think that this happens through a new nuclear process. What else can it be?
23) Teacher:
I can think of many non-nuclear processes able to generate heat at the rate of 1W.
Before addressing the issue of causes, however, I would like to ask a different question. Is it possible that the difference of 1 W is nothing but an experimental error due to the limited accuracy of measurements? We know that the true value of Pout can be anywhere between 36.6 and 38.2 W. That is based on your 2% accuracy. Likewise, the true value of Pin can be anywhere between 36.0 and 36.8 W; this is based on your 1% accuracy. Thus, the difference between Pout and Pin, can be as small as -0.1 W or as large as 2.1 W. Do you understand my reservations?
24) Engineer:
I understand you very well. Instead of accepting my hypothesis (that the excess heat is real) you prefer to accept your own hypothesis (that the excess is an illusion due to ever-present systematic errors). Why should your hypothesis be taken more seriously than my hypothesis?
25) Teacher:
Because we are not in the same situation. You are claiming that something extraordinary is taking place; I am only a defender of status quo. If you are right then the Nobel Prize might be awarded to you; nothing of that kind exists to recognize my modest contribution. The burden of proof is yours, not mine. That is how science works. In our legal system the accused is presumed innocent until proven guilty; in science a radically new idea is presumed wrong until proven correct.
26) Student:
That’s what we learned in another course. A statistician starts with a “null hypothesis” stating just the opposite to what is being claimed. Then he tries to do everything reasonable to justify that hypothesis. Failure to justify the opposite constitutes the basis for acceptance.
27) Teacher:
Yes, such strategy can be used in natural sciences. But keep in mind that statistical methods have been developed to deal with random errors while we are discussing systematic errors. Statistical errors can be reduced by repeating an experiment many times, random errors can not be reduced in that way. Calibration errors are usually categorized as systematic errors. A faulty thermometer, for example, might show that the temperature of boiling water, at standard pressure, is 110 C instead of 100 C. The strange effect (10 C of “excess temperature”) will not go away when the average temperature is calculated after performing 1000 measurements.
28) Student:
But what if the effect is real? Assuming that the 10C of excess temperature is due to an error of calibration one may miss a chance of discovering something important.
29) Teacher:
Yes, such possibility does exist; the observed effect can be due to a hidden variable of some kind, for example, a cosmic event in another galaxy, or wishful thinking of a magician. Before accepting something unusual, however, a scientist should convince himself that the instrument is not faulty, for example, by measuring the temperature with other thermometers. First he would assume that the instrument is faulty then he would reject this hypothesis by showing that the instrument is not faulty. The name of the game is checking and double-checking before accepting something unusual.
30) Engineer:
So what would it take to convince you that my hypothesis is correct?
31) Teacher:
It is not just me; by announcing a radically new idea you are challenging the entire scientific community. You must do two things to be taken seriously. First repeat the experiment using more accurate instruments. What you have built would be fine if the difference between Pout and Pin were larger, for example, 10 W or more. But it is not accurate enough for your 1 W difference.
32) Engineer:
And what is the second thing that I must do for my hypothesis to be acceptable?
33) Teacher:
Once you convince yourself that the observed effect can not possibly be due to systematic errors you must clearly describe the experiment. The description, in the form of a paper, or in the form of a conference presentation, should be sufficiently detailed to allow replications of your work. You are likely to be taken seriously after your results are independently confirmed by others. The exact error analysis should be an essential part of your publication.
34) Student:
I think that claiming a discovery is more fun than defending the status quo.
35) Teacher:
That might be true. But making scientific contributions is a serious matter. Those who are engaged in it must respect established rules of acceptance. Following these rules is very different from what poets or composers do to be successful. Scientists must be objective; they should resist the temptation to have fun by making false claims. They should be guided by experimental facts and reliable theories, not by wishful thinking.
36) Engineer:
I appreciate your comments and I will try to improve the accuracy. I will borrow a more expensive flowmeter; this alone will allow me to lower the level of uncertainty by about 0.8%. I will also reduce the flow rate; this will increase (T2-T1) from about 10 degrees to about 50 degrees.
37) Student:
How will this help you?
38) Engineer:
The relative error will not change for T1 but it will be reduced for T2. The percentage error, as you probably know, is the ratio of the absolute error (0.1 C) over the value of T2. That value will be higher when the flow is reduced.
39) Teacher:
How much can you gain from making (T2-T1) larger?
40) Engineer:
Perhaps another 0.5%. In fact, I can also use more sensitive thermometers. More accurate voltmeters and ammeters are also available. Reducing the overall instrumental uncertainty from 3% to 0.3% is possible, at least in principle. Is it worth trying?
41) Teacher:
Only you can answer this question. It depends on the amount of time and money at your disposal. Personally I would first think about the so-called procedural errors. They are often responsible for unjustified claims.
42) Student:
What are procedural errors?
43) Teacher:
Procedural errors can be anything from “inexcusable blunders,” such as confusing 30 with 300 (using a wrong scale) to unexpected effects of hidden variables. Let me tell you a little story. A group of students was measuring magnetic field along the axis of a wire loop positioned in the middle of a table. In doing this they discovered an unexpected asymmetry; the magnetic field on one side of the loop was different from the field on the other side. The effect was very strong and results were reproducible. The lab assistant was puzzled. It took him a while to find out that the wooden table, on which the instruments were mounted, had iron nails below the plastic cover. Procedural errors, as you can see, are systematic errors caused by hidden variables. Even experienced researchers can be victimized by such effects.
44) Engineer:
Are you telling us that experimental results can never be trusted?
45) Teacher:
Yes, a single publication is never 100% certain. But it becomes more and more trustworthy when findings are confirmed by others. In the final analysis, one may say, scientific research is a matter of consensus. I am not trying to discourage you; I am trying to make you aware of difficulties faced by scientists. Demonstrating that an effect exceeds known instrumental uncertainties is only the first step. What follows is much more difficult. The researcher should identify as many suspected hidden variables as possible and show that they are not responsible for the “new effect.” The burden of proof is on the discoverer. You must convince others that the effect is not due, for example, to viscosity of flowing water, or to chemical reactions. One way to accomplish this could be to show that a common 160 ohms resistor does not generate excess heat under identical conditions.
=================================================
P.S. COMMENT:
1) Dr. Kirk Shanahan saw my essay and suggested that another participant, QC scientist, be allowed to participate in the debate. That “quality consciences” investigator would focus on several nontrivial aspects of scientific research. My preference is to have three participants and to keep things as simple as possible. But Kirk’s suggestion to focus on hidden variables could not be ignored.
2)Kirk wrote :“Your dialog does focus on the simple concept of propagation of errors, but the point of 'QC Scientist' is that that is not what is relevant in the cold fusion excess heat case, based on my studies. He attempts to get the 3 people to realize there is more to life than just the random/systematic debate.”
3) At one point I asked Kirk: “did I summarize your understanding of experimental errors correctly?” And the answer was: “You present the 'old school' paradigm. My efforts in this arena arise from the 'new school', so, you did fine on the old school stuff, but unfortunately that's not what is relevant.” I learned a lot from Kirk’s messages and I hope he will write a good summary of ‘new school’s ideas in the next unit.