I want to suggest a theory about why there should be such resistance, and how it relates to our problems of spreading the Deming Philosophy.
I believe that the three approaches to statistical inference do not come just from differences about statistics. They correspond to three different views of what *science* is about. What follows is not an exact description of what philosophers say (or said) that scientists should do, but based on my own experience in using all three approaches, and observing what other scientists do.
These three views of science correspond to different stages of development. The logical/mathematical view was unquestioned throughout the middle ages, and reached its peak with Descartes. Scientific truths were deduced by strict logic, starting with self-evident axioms, as in Euclidean geometry.
From the time of Isaac Newton, science changed. I believe (I have not checked the originals) that Newton presented his ideas in the old format, as deductions from axioms. So successful was he that some later writers claimed that Newton's physics is true, not because of observed facts, but by pure logic. But for most people, what Newton did was to provide an explanation - the force of gravity.
Explanation need not be exact. As George Box has put it:
"All models are wrong. Some are useful."
At the beginning of the 20th century, both forms of science fell apart.
There were two blows to previous thinking. First of all, many
"self-evident" ideas turned out to be false. An example is Einstein's
demonstration that time is relative. Secondly, the idea of explanation
itself was called into question.
Quantum Theory, in particular, provides no explanation we can understand. But it predicts strange and unbelievable outcomes, and predicts them with amazing accuracy.
Most people are unaware that science has changed. Only those trained in theoretical physics (like Shewhart and Deming) have adopted the new philosophy of science. Others are stuck in the thinking of earlier centuries. And because knowledge is now so specialised and compartmentalised, few scientists are aware that different ideas are taken for granted in fields other than their own. We are dealing, in most cases, with unconscious assumptions, rather than conscious beliefs. That makes them far harder to deal with. It seems that many people cannot face a challenge to what they think is "obvious" - though the System of Profound Knowledge is one challenge after another.
My theory about the three approaches to statistics is as follows.
I started with statistics because the historical record is so striking. But the other examples are also well documented. Semmelweiss demonstrated that hygiene saved lives. But nothing was done until Pasteur explained the reason for it.
I apologise for what may seem to be lengthy theoretical rambling. But the strange thing about the Deming philosophy is that the most abstract ideas turn out to have direct practical applications. It is not surprising that science based on prediction and action is exactly what we need for management.
In my next instalment I hope to show that this helps us understand some of the difficulties we face.
-----------------------------------------------------------------------------------------------
In the previous post I mentioned conflicting models of science, based on either:
The idea of different models of science may seem remote from practical application. But as I expect we have all found for ourselves, in the Deming Philosophy nothing is "too theoretical" to affect our actions.
Walter A Shewhart certainly saw this. In his 1931 book, on
"The Economic Control of Quality of Manufactured Product"
he wrote:
"Progress in modifying our concept of control has been and
will be comparatively slow. In the first place, it requires
the application of certain modern physical concepts......"
He does not say which concepts. But we can reasonably relate
this to the "Theory of Knowledge" dimension of the System
of Profound Knowledge.
It seems to me - and this is purely a personal reflection which others may dispute - that the "Theory of Knowledge" aspect is the one that makes least impact. It may easily sound as if it contains nothing new. After all, most scientists, if asked, would say something similar. They would probably quote Karl Popper, who popularised this view of science, rather than C I Lewis or A N Whitehead, but the message is the same.
The difference is - again a personal opinion - that most people, whether scientists or laymen, pay lip-service to Popper, but continue to think in earlier modes. Most people see the whole point of science as explanation.
I have just been watching a television series that attempts to explain the ideas of "String Theory" for the layman. It showed one group of scientists arguing that "strings" may provide the "theory of everything" that unites Quantum Theory and Relativity, which are at present in conflict. Other scientists say "Strings are not a scientific theory" because they make no testable predictions. I can almost hear Shewhart laughing.
But "see" is the key word here. What we see determines how we act. WED describes the System of Profound Knowledge as a "lens". In other words, it enables us to bring some things into focus, and to see what we could not otherwise see.
What a pure scientist sees may change the whole world in the
long run. What a manager sees changes everything now. To
quote WED's own words:
"My job is not to tell managers what to do. It is to help them
to see things that they could not otherwise be expected to see."
We have all seen how managers react to the Red Bead Experiment.
The idea that it is wrong to look for an explanation of the red
beads produced by a worker is profoundly shocking. Explanation
is the lens through which they see the world. It is very hard to
switch to thinking in terms of prediction.
This creates resistance to what Deming and Shewhart say about SPC, systems, and even psychology. The problem is all the greater because it is unconscious.