Note: In my PhD program, I was prepared to be a researcher. I have conducted research and published research. I have taught research. I have supervised research done by doctoral students. I have evaluated research as a consulting editor for a number of professional journals. In short, I know something about a scientific approach to doing research. Perhaps not everyone qualified to speak to this topic would agree with what follows but it is, in my opinion, a valid description based on my experience. I think I can, at the very least, posit an informed opinion.
Paradigms are conceptual models that serve an umbrella function for theories in diverse areas of study. For example, the current paradigm in science (see What is Science?) is scientific materialism. This paradigm serves an umbrella function for theories about such things as physical process, biological processes and behavioral process. This paradigm has its origins in the scientific revolution inspired by the scientific thinking of Nicolaus Copernicus in the sixteenth century and Isaac Newton in the seventeenth century. Scientific materialism as a paradigm assumes that everything is comprised of physical particles (principle of physicalism; i.e., the root assumption of the paradigm) governed by cause and effect relationships (principle of causal determinism), that change is continuous (principle of continuity), that phenomena occur within a finite space and over finite periods of time (principle of locality), that phenomena have objective existence independent of observation (principle of strong objectivity) and can be understood through reducing phenomena to their essential components (the principle of reductionism), which implies that phenomena are assembled from the bottom up, piece by piece. All theories falling under the umbrella share these basic assumptions. See Goswami’s Quantum Philosophy (Part I) and Goswami’s Philosophical Alternative for more detail.
One tenet of science as a methodology is that it holds to certain principles about the nature of knowledge. One of these principles is that our knowledge consists of models of reality and are not elucidations of reality itself. In other words, what we know is always considered to be an approximation never truth. Another principle is that what we know is held as tentatively valid until shown otherwise. How we know is through creating explanations for what appear to be related observations or facts about phenomena in the world. These explanations (a.k.a. theories) are then used to derive hypotheses that can be experimentally tested. Successful tests of hypotheses derived from theory increase the confidence that we can have in the explanation or theory. Confirmation of an hypothesis is sometimes possible by successful prediction of an outcome, such as the prediction of planetary motion based on a theoretical model or explanation of the forces governing such motion. In other and more confounded cases, confirmation of an hypothesis is sought through statistical testing in which a conclusion is reached based on probability calculations. The typical standard in such cases is a p. <= .05, which means the observed result would be expected by chance only 5 times in 100 or 1 time in 20. Standards such as this can, of course, result in some false positives but is considered an acceptable error rate for theory testing (see “What is Science?“)
The flaw in this system is that a paradigm can come to be so central to the scientific process that it begins to be viewed as Truth. Once this happens its assumptions acquire the status of dogma. When this occurs, theories subsumed by the paradigm become inoculated against accepting results that are contrary to dogma, which also means contrary to theory or theories grounded in the paradigm. Once this happens, science has become scientism. It appears that contemporary science is grappling with the problem of scientism. With the advent of quantum mechanics in the early twentieth century the basic assumptions of scientific materialism were challenged. Experimental evidence refutes or strongly questions the validity of the principles or assumptions of scientific materialism enumerated above. Resistance to this challenge has been evident in a variety of fields that have simply ignored the challenges and continued to act as if nothing had changed. This is especially true in the cases of biological and behavioral sciences. Many physical sciences have found ignoring the shifting paradigm more difficult. However, even in the physical sciences the tendency has been to attempt to limit the shift to effects occurring at the micro level and preserve the paradigm at the macro level. Unfortunately, experimental evidence is accumulating that demonstrates quantum effects can also be detected at and thus have effects at the macro level.
Another source of challenge to scientific materialism that became evident during the twentieth century was the results from psi experiments (e.g., see Spirituality and Religion). One early body of experimentation was that done by J.B. Rhine at Duke University. Rhine produced evidence that certainly should have caused some serious questioning of the adequacy of scientific materialism, he and his results were widely rationalized away because they were inconsistent with the prevailing paradigm suggesting that the assumptions of the paradigm had become dogma. Later in the twentieth century a large body of research was accumulated under the leadership of Robert Jahn at Princeton University in its engineering anomalies laboratory. This work too was rationalized away to maintain the integrity of the paradigm or if you prefer to preserve the dogma of scientism. In both cases, the correct scientific response should have been intense investigation rather than out-of-hand dismissal.
There are of course researchers that continue to pursue investigation into these challenges to scientific materialism (e.g., see Society for Scientific Exploration). A large group of open minded investigators have formed an organization (Academy for the Advancement of Post-materialist Science) dedicated to finding a new and better paradigm. Whether or not a new paradigm is justified, careful investigation of challenges should be applauded, not ridiculed as is often the case from those wedded to scientism.