Interpretation of research data from psychological and educational studies is changing with the advent of meta-analysis. Meta-analysis seeks to evaluate what a whole series of studies on a particular topic can tell us that no one study can.
Frank L. Schmidt, University of Iowa, reports that meta-analysis represents more than merely a change in the methods of data analysis. It requires researchers to change the way they view the research process itself. Schmidt states that traditional methods for data analysis and interpretation in individual studies stress statistical significance to such a degree that they are counterproductive, hampering the development of theories and cumulative knowledge in the social sciences.
Tests of significance are statistical procedures used to reduce the possibility that researchers see relationships or effects in their studies that do not exist. Significance testing is deceptive, Schmidt states, because as researchers study a particular area in depth and gain a better understanding of it, the chance that they are finding a relationship when there is none becomes less and less likely. However, traditional methods of analysis use increasingly stringent controls that further increase the possibility that study results are judged insignificant. Therefore, underlying patterns of results are missed, which leads to errors in interpreting the meaning of research data.
Schmidt believes that meta-analysis can overcome the limits of traditional statistical procedures. He describes meta-analysis as a process of making sense of and cleaning up research data. In contrast to traditional analysis of data, meta-analysis does not realy on tests of statistical significance. Meta-analysis uses only the “effect sizes” from individual studies which indicate the magnitude of the influence of the factor being studied. In this way, meta-analysis uses available data from many studies to identify the underlying patterns and relationships that form the foundation of new theories.
Because of sampling and measurement errors, there is not enough information in any single study to resolve an issue. For this reason, Schmidt believes that meta-analysis has the potential to measurably increase our knowledge. He adds that there is an “overconfident empiricism in the behavioral and social sciences.. an excessive faith in data as the direct source of scientific truths and an in adequate appreciation of how misleading most social science data are when accepted at face value and interpreted naively.” In Schmidt’s view, any individual study must be considered as only a single piece of data to be contributed to future meta-analysis.
“What Do Data Really Mean? Research Findings, Meta-Analysis and Cumulative Knowledge in Psychology”, American Psychologist, October 1992, Volume 47, Number 10, pp. 1173-1181.
Published in ERN January/February 1993, Volume 6, Number 1.