EDUCATION RESEARCH AND EVALUATION AND STUDENT ACHIEVEMENT: QUALITY COUNTS
G. Reid Lyon, Ph.D.
The following is a statement made by Dr. Lyon to the Committee on Education and the Workforce, U.S. House of Representatives, in Washington, D.C. on May 4, 2000.
Good Morning, Mr. Chairman and members of the Committee. I am Dr. Reid Lyon, Chief of the Child Development and Behavior Branch at the National Institute of Child Health and Human Development (NICHD) at the National Institutes of Health (NIH). I am pleased to have been asked to address this Committee on the value and use of education research and evaluation and its benefits to states and local school districts trying very hard to improve student achievement. The NICHD considers that teaching and learning in today’s schools is not only a critical educational and social issue, but a significant public health issue as well. Our research has shown that if children do not learn to use language to communicate ideas and perspectives, read and write, calculate and reason mathematically, and be able to solve problems strategically, their opportunities for a fulfilling and rewarding life are seriously compromised. Specifically, in our NICHD longitudinal studies, we have learned that school failure has devastating consequences with respect to self-esteem, social development, and opportunities for advanced education.
Translating Education Research into Effective Instructional Practices in the Classroom: The Complexity and Scope of the Problem
Mr. Chairman, historically, education research has not had a significant impact on educational policies and classroom instructional practices. The reasons for this persistent gap between the guidance that education research hopefully provides and the teaching practices that teachers use on a day-to-day basis are many, but three stand out.
1. The Trustworthiness Issue.
First, as recently found by the National Reading Panel (NRP), much of the education research published in archival journals and disseminated to researchers, teachers, and policy makers is of uneven, and often not good, quality. It is important to understand that the trustworthiness of any research study is predicated on two major elements: (1) the suitability of the proposed research design or methodology to address the specific question posed by the study; and (2) the scientific rigor of the methodology itself. For the results to be trustworthy, a study must use the appropriate methodology and apply it in a rigorous manner. For example, if the question is one of effectiveness – let’s say, how effective are specific instructional approaches in teaching children to read – then the only type of research design able to specifically address the question of cause and effect is an experimental or quasi-experimental approach. Such studies are quantitative in nature. In fact, this was the type of research approach selected by the National Reading Panel. To quote the NRP Report, “To make a determination that any instructional practice could be or should be adopted widely to improve reading achievement requires that the belief, assumption, or claim supporting the practice can be causally linked to a particular outcome. The highest standard of evidence for such a claim is the experimental study, in which it is shown that treatment can make such changes and effect such outcomes….” (NRP Reports of the Subgroups, p. 1-7).
On the other hand, qualitative research methods and approaches are more exploratory in nature, for example, the intense study in single children of the influence of cultural factors on teaching and learning. Here, the results of qualitative studies can be extremely helpful in generating hypotheses and raising awareness of potential factors that may influence the effectiveness of an intervention. In addition, descriptive, non-experimental research can be useful in building theory, to help shape the design of instructional approaches and interventions, and to help one understand the target or focus for an intervention. The findings of such qualitative- descriptive studies could, in turn, lead to the development and design of more detailed quantitative studies incorporating these observations. Yet it is important to note that qualitative-descriptive research can only suggest instructional approaches or innovative educational strategies to teach students and to lay the groundwork for the development of such strategies.
Qualitative-descriptive research cannot identify generalizable strategies that will likely improve academic achievement among students. Only quantitatively based experimental research can do that. For example, experiments and quasi-experiments have the capability to show us that instructional approaches that seem promising in the descriptive phases of a study do not necessarily help most students learn to read.
The bottom line is that studies that use both types of approaches are important and necessary if we are to develop the fullest and richest understanding of what specific instructional approaches are most effective for which children at which stage of development and under what particular circumstances. But integrating research approaches in a thoughtful and appropriate manner demands a clear understanding of the assumptions underlying each research method and the purposes for which the method is best suited. Over the past years, some educational research may have confused these assumptions and purposes to the detriment to teachers and students.
Research is terribly demanding and is it not sufficient to simply select the most appropriate methodological approach; it is as critical that this methodology be applied rigorously whether it be for quantitative or qualitative studies. For example, in its work examining quantitative studies, the National Reading Panel first established a set of rigorous research methodology standards by which to judge the trustworthiness of each study under review. Again, to quote the Report, “The evidence-based methodological standards adopted by the Panel are essentially those normally used in research studies of the efficacy of interventions in psychological and medical research. These include behaviorally-based interventions, medications or medical procedures proposed for use in the fostering of robust health and psychological development and the prevention or treatment of disease. It is the view of the Panel that the efficacy of materials and methodologies used in the teaching of reading and in the prevention or treatment of reading disabilities should be tested no less rigorously. However, such standards have not been universally accepted or used in reading education research.” (NRP Report, p. 5).
Specifically, in the case of quantitative studies designed to test the effectiveness of different reading instructional approaches published over the past 30 years, less than a third of studies met basic scientific criteria. Many studies did not have even the most rudimentary elements of scientific methodology such as adequate control or contrast groups, many did not define study participants or instructional approaches sufficiently to permit application in the classroom or replication by other studies, and many did not measure student achievement outcomes appropriately. Thus, it is not surprising that many teachers and researchers have lost faith in the ability of quantitative research to inform instructional practices over the years. Something not done well will have little to offer and little to trust. It remains to be seen whether the extensive qualitative and descriptive education research literature used predominantly over the past decade or more to guide instructional practices contains studies that adhere consistently to the basic principles of reliability, validity, and trustworthiness of the data.
The effects of such limitations in quality on educational policy making and teaching practices are insidious and harmful. Information derived from poorly designed and conducted studies will inevitably produce recommendations that are doomed to failure at the system level, the school level, and the classroom level. Teachers, want, above all, to provide instruction that makes a genuine difference in the lives of their children. They want to use every bit of good information that helps them craft and tailor instructional approaches to meet children’s individual learning needs and to elevate the achievement of their students. When teachers turn to research to inform their teaching, they expect and deserve information that is trustworthy. When the information is not, which it typically is, teachers fail, students fail, schools fail, and our Nation fails.
I know first hand the devastating effect that poor quality research has on teaching practices and the trust teachers have in education research. As a young, brand new third grade teacher in the mid 1970s I was responsible for teaching 28 students of varying abilities and backgrounds. Many of my students had not yet learned to read which concerned me greatly, but I was informed in my education courses and via the school philosophy that this was to be expected – children learn at their own pace. My school had also adopted a reading curriculum that was based upon the assumption that reading was a natural process, similar to learning to listen and speak. Following this curriculum, I presented reading concepts to children through exposing them to wonderful literature, and attempted to teach phonics concepts incidentally as they appeared in different stories. I also employed the oral language and writing activities that were suggested in my teacher’s instructional manual. At the beginning of the year, a third of my students could not read well enough to understand what they had read. Their reading was slow and labored and they mispronounced words constantly. Their spelling was lousy. At the end of the year, the same third of my students could not read well. Their reading remained slow and effortful, the time it took to read text was so great that they could not remember what they read, and their spelling was still lousy. The only change that I could discern was that their motivation to learn to read had waned, and their self-esteem had suffered substantially. Likewise, I felt like a failure, I had let down the children I was responsible for, and I left the classroom teaching profession. I attributed my failure to the fact that I was inexperienced, which I clearly was. It was only later that I came to learn in great depth that the reading instructional approach embraced by my school was not only based upon research that was questionable at best, but that the major assumptions upon which the instructional philosophy and recommended teaching interactions rested had never been adequately tested through well designed studies. I mention this anecdote only to provide a personal explanation for why many teachers lose trust in “research” and eschew educational research findings to guide their practice. Those who stay in the profession learn to simply “wait out” the next “research-based” instructional magic bullet.
While the persistent concern about the trustworthiness of educational research is alarming, even more alarming is the seeming resistance within the educational research community to do anything systematically to increase research quality. Many researchers, school administrators, and education policy makers are currently distracted by debates concerning the specific research approach, quantitative versus qualitative, to inform curricular and teaching practices. Many argue that classroom instruction is far too complex to study adequately by standard experimental or quasi-experimental methods and that qualitative ethnographic and descriptive research findings are more relevant to actual teaching of students in classrooms. Many judge qualitative research, because of its descriptive focus and inability to formally test hypotheses, to be “loose” and less rigorous. It has been said that qualitative research is useful because if the findings are looked at long enough, all predetermined perspectives can be supported. It has also been said that experimental research is too “controlled” to accurately reflect the complexities of classroom life.
Of course, both of these representations are shallow and inaccurate. Both quantitative and qualitative research methods are very useful for specific purposes when selected and used appropriately. The question is not which type of research method is best, but which combination of methods is most appropriate and useful to address specific research questions? The issue is not “either-or.” The issue is TRUSTWORTHINESS. Whether specific aspects of a teacher’s instructional armamentarium are informed by quantitative, qualitative, and integrated research methods, the power of the research to practice linkage is dependent upon the care, rigor, and methodological excellence which characterizes the studies from which the information is derived.
2. The Teacher Preparation Issue.
No doubt, there is a good deal of educational research that is trustworthy and has been used to inform instructional practices in a productive manner. For example, Bob Slavin’s “Success for All” school reform model supported by OERI is based upon substantial research of high quality and undergoes constant evaluation to ensure effectiveness. Likewise, the comprehensive research in beginning reading and the prevention, early intervention, and instructional studies supported by NICHD are being carried out in complex school and classroom environments with good success. A common feature of these two examples is that the school culture and the teachers that apply the research have been specifically prepared to clearly understand and use the results of the research.
No matter how trustworthy a set of research findings might be, the relevance and applicability of the findings will be minimal if teachers and administrators: (1) cannot access the data; (2) do not understand and cannot interpret the findings in an accurate and meaningful manner, and (3) are unable to develop plans and strategies to implement the research in everyday practice. Unfortunately, several recent NICHD supported studies and surveys carried out by Virginia Berninger and her colleagues, and Louisa Moats and by myself in the late 1980s indicate that teachers feel unprepared to address the individual learning needs of their students, and they report that they are particularly under prepared to provide adequate reading instruction. This perception is supported by a recent report from the National Center for Educational Statistics indicating that only one in five teachers feel adequately prepared to teach their students. Why is this the case? Drs. Berninger and Moats have reported that teachers receive insufficient instruction in reading development and reading instruction during their undergraduate, and even graduate studies, with the average teacher completing only one or two reading courses. Surveys of teachers taking these courses indicate consistently that they have not observed professors demonstrate instructional reading methods with children, that course work is frequently superficial and unrelated to teaching practice, and that supervision and guidance during student teaching is fragmentary and inconsistent. Many motivated teachers report that they are left to their own devices to obtain specific skills to improve their instructional practices.
Many teachers report that they do not use educational research findings to guide their teaching practices and many report that they do not trust the idea that research can effectively inform their teaching. This is not unexpected given that it is quite difficult to apply research findings when the information is often of poor quality, lacks authority, is not accessible, is communicated in an incomprehensible manner, and is not practical. Of equal concern is that teachers report that they are not specifically trained in even the most basic approaches to interpreting different types of research studies and are not able to make accurate judgments about the trustworthiness of the research that they do read. Until teachers are provided the necessary basic training to understand how to read and interpret research and how to assess the methodological appropriateness and rigor of the research, they will continue to be buffeted by the capricious pendulum swings that characterize educational fads and instructional practices. This is highly demoralizing. Many teachers find themselves today attempting to implement the latest “research- based” instructional practice only to learn, after it fails, that the research upon which it was based was seriously flawed.
3. The Research To Practice Issue.
While research trustworthiness and teacher preparation play significant roles in determining how well research accurately informs educational policies and instructional practices, a critical problem lies in our failure to identify and understand the conditions under which the results of trustworthy research can be implemented and sustained in complex, “real-life” school systems and classrooms. While specific instructional models, approaches and strategies may be found to be effective in relatively controlled settings, there is little detailed knowledge about the factors that foster or impede application of these modalities under varying conditions and contexts, and among diverse populations of students and teachers. We do not yet have a solid grasp of how to “travel” educational innovations, including school reform models and specific classroom management and content instructional practices because our understanding of the cultural, incentive, training, and administrative conditions that will influence this level of “scaling” remains rudimentary. As a Nation, we have only recently begun to invest the necessary human capital, methodological capital, and institutional capital to ensure that large-scale, rigorous, and systematic research can be genuinely translated into improvements in educational practice and student achievement.
COMPLEX, BUT NOT IMPOSSIBLE
Helen Bernstein once said, “If you always do what you’ve always done, you’ll always get what you’ve always gotten.” Educational research can and should play a major role in improving student achievement, but it won’t unless significant dedication, intellectual capital, collaborative problem-identification and problem-solving, and a commitment to a systematic and sustained effort are brought to bear on the issues surrounding the translation of research to practice. Some progress has been made over the past five years, but substantial work remains to be done. Allow me to summarize a number of relatively new initiatives that have attempted to address issues of research trustworthiness, teacher preparation, and research translation and provide selected recommendations for your consideration in each of these areas.
Research Trustworthiness –
We must raise the trustworthiness, that is, appropriateness and rigor, of all education-related research. It will be important to ensure that all Federally-supported research adhere to the highest standards of excellence and we must encourage privately funded research initiatives to embrace these standards as well.
A major first step in this regard is to undertake a comprehensive and systematic analysis of the educational research literature that is relevant to classroom instructional practices, determine the degree to which the research studies meet standard research criteria, and identify the extent to which research of high conceptual and methodological quality converges on particular findings, and determine the readiness of these findings for application in the classroom. Within this context, the National Reading Panel (NRP) has concluded its analysis of the quantitative experimental data-base relevant to reading instruction and has presented its findings and determinations to Congress. It is important to note that this Report was able to identify instructional approaches that are ready for classroom implementation.
Also important was the finding that a substantial portion of the quantitative and experimental instructional reading research is not capable of informing instruction. These findings provide a clear road map of the gaps that continue to exist in the reading research, and the efforts that will be required to improve research quality, particularly from a methodological standpoint. The NICHD and the OERI are working closely together to develop formal strategic plans to ensure the accurate dissemination of the findings of the NRP and the development of specific strategies to actually implement the findings.
The Federal development and support of the Reading Excellence Act (REA) also represents a major step forward in specifying the types and level of methodological rigor of educational research required to make genuine “research-based decisions when selecting and implementing reading approaches and programs.” In addition, Federal support for the Interagency Educational Research Initiative (IERI) is a significant collaborative step toward improving not only the quality of educational research, but the identification of the conditions that need to be in place to translate and scale research findings to the necessary level to improve student achievement in complex educational environments. The NICHD and the OERI are working closely together on the continued development and evaluation of the REA and are working with the National Science Foundation in the development and management of the IERI.
At a more local level, the NICHD and the OERI have been working closely together to develop and implement models of the peer-review process to ensure that grant applications receive the attention they deserve from highly qualified researchers with specific expertise in the scientific and educational domains represented in the grants.
In addition to these ongoing efforts, several additional recommendations are offered:
• We must develop formal mechanisms to synthesize research that is trustworthy and relevant to instructional practices used in classrooms and with children at-risk for academic failure. A major key to developing a solid and trustworthy research base that will ultimately inform practice is to demonstrate how research findings converge on a particular instructional practice or principle. Research syntheses can also serve a much needed and critical role in assessing the validity of various philosophical and theoretical assumptions that have traditionally guided educational practice before they have been formally evaluated. The tendency in education to shift capriciously from one instructional trend to another is clearly influenced by the field’s inability to develop sustained, serious research efforts capable of establishing evidentiary convergence and ensuring replication of findings. Again, the work and the findings of the NRP is a critical step in this process of establishing clear quality standards for research and evaluating and synthesizing existing studies with respect to these criteria. I would like to offer the Report of the National Reading Panel for inclusion in the hearing record.
• We must strive to increase the research-based quality of educational materials and programs that are offered commercially to schools. It is generally not appreciated that more often than not, schools purchase educational and instructional materials and textbooks on the basis of non- scientific factors. Rarely are the instructional methods and procedures recommended in these materials objectively evaluated to determine how effective they are with children of varying abilities and with children in different types of instructional settings. Consumers must ultimately be able to know and understand the strengths and weaknesses of a given educational material or instructional approach and clearly understand the limitations of the research that supports a particular educational product.
• The research community must begin to address the tendency to conduct narrowly focused studies, studies that often adhere to philosophical rather than scientific principles. The polarization of research methodologies into a quantitative-qualitative dichotomy reflects a parochial and value-laden perspective that will not advance our knowledge, and will distract the research community from establishing the most compelling research need – TRUSTWORTHINESS. The complex problems that we hope to solve require an answer to this question: Which combinations of research methodologies and approaches are most appropriate for which specific research questions, and how are the methodologies best integrated? Some aspects of some questions will have to be addressed under controlled conditions while contextual, cultural, and organizational factors that influence teaching and learning will require qualitative and ethnographic strategies. However, it is likely that the most helpful and enduring answers will be derived from a careful integration of these perspectives applied at the highest level of scientific integrity.
• In addition to increasing research efforts to determine how best to integrate research methods in studying complex educational interactions, a significant need also exists to develop measures that are capable of capturing the essence and authenticity of these complex interactions while at the same time ensuring reliability and validity.
Empowering Teachers –
If trustworthy research findings are going to be effectively used to inform instructional practices in classroom settings, we must acknowledge that teachers must be provided the necessary basic knowledge to translate research into effective classroom practices. Teachers, who are the most important consumers of research, have been let down in this regard. At a minimum, these recommendations are offered:
• Systems of accountability must be developed and put into practice to ensure that all teachers have mastery of the content they are teaching, can deliver instruction through a wide range of approaches and methods, and have a clear understanding of individual differences in their students.
• Teachers must be prepared to understand the basic principles underlying the development of the skills that they are teaching and how these principles relate to instructional practice. It is also critical that teachers receive basic training in how to access and interpret the research literature relevant to their instructional responsibilities. It is only in this way that they will be empowered to judge both the quality and applicability of the research findings. All too often, teachers are provided simplistic “magic bullet” solutions to increasing student achievement. Unfortunately and frequently, many instructional approaches have been developed with children in a setting far different from that the teacher now encounters. We must provide teachers with systematic and rigorous training sufficient to develop the ability to evaluate these of claims with confidence.
• Teachers must be included in the planning, design, and conduct of educational research that is expected to influence their instructional practices in the classroom. Both researchers and teachers must have the opportunity to develop genuine research collaborations where constant input and feedback are provided bilaterally. This will require substantial changes in the current training of both researchers and teachers.
Moving Trustworthy Research Findings to Scale –
While the terms “research- based” practices and “translational research” are terms heard frequently today, there continues to be a paucity of knowledge about how best to implement even the best research information into the daily lives of school administrators, teachers, and students. We don’t understand the systemic requirements that are necessary if research is to inform practice in a genuine fashion. We do not yet understand the incentive systems that are critical in helping teachers to modify their belief systems, when appropriate, and incorporate new concepts into their teaching. We don’t yet understand how teachers can best be taught to do this. We don’t yet understand the amount of time, effort, and resources that are required to address teacher learning, adaptability and change, and we certainly don’t yet understand how such things as school district policies and demands, and high stakes assessment influence this process. And most critically, we have not yet developed the fundamental research methods and approaches that can give us a clear view of how different training experiences provided to teachers actually translate into genuine improvements in student achievement. There are long roads to be traveled for both teacher and student. Disentangling and clarifying the multiple influences that can cause positive changes in both teachers and their students will require the thoughtful and sustained integration of rigorous quantitative and qualitative research methods alluded to earlier.
The NSF, the OERI, and the NICHD have, over the past two years, concentrated on thinking about these issues in depth. Together, we are attempting to build the research infrastructure that will address some of these questions in a productive manner. We are hopeful that the IERI will stimulate the research community to engage in the type of planning and interdisciplinary collaboration that will be absolutely critical to changing the ways in which research is translated effectively into practice. We are hopeful that we have designed the IERI initiative in such a way that the complexity of the research tasks that have to be carried out mirror the complexity of the problems that must be understood. We are convinced that the scientific standards demanded by the IERI will move the field forward, and we are likewise convinced that the peer-review structure and standards that are in place to evaluate IERI grant applications will ensure the research quality that is so sorely needed.
It is also imperative that the Federal investment in education research be predicated on and demand quality efforts from the diverse research community. Likewise, the translation of research to practice will not improve unless the relevant funding agencies and the educational community together understand the urgency to adhere to basic principles of trustworthiness in research. The level of complexity of the problem demands the best of our intellectual and conceptual efforts, the best of our collaborative efforts, and the courage to apply these efforts in a sustained and systematic fashion. There is no question that it can be done and we fervently hope that it will be done.
Dr. Lyon is Chief, Child Development and Behavior Branch at the National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD. Dr. Lyon currently serves as an advisor to President George W. Bush on child development and education research and policies. He is a member of the CDL Professional Advisory Board.