«SE 062 649 ED 431 621 Stigler, James W.; Gonzales, Patrick; Kwanaka, Takako; AUTHOR Knoll, Steffen; Serrano, Ana The TIMSS Videotape Classroom Study: ...»
Despite the obvious value of studying classroom instruction, describing and measuring classroom processes, especially on a large scale, is difficult. To date, measures have been largely based on questionnaires in which teachers report on what happens in their own classrooms. Using questionnaires to measure classroom processes has both advantages and disadvantages, as we discuss here. Observations have different advantages and disadvantages. Although observation is a natural way to study classroom processes, it has generally been considered too difficult and labor intensive for large-scale studies. The methods described here, however, present an approach to overcoming this problem.
Advantages and Disadvantages of Questionnaires for Studying Classroom Processes Most attempts to measure classroom processes on a large scale have used teacher questionnaires.
Teachers have been asked, for example, to report on the percentage of time they spend in lecture or discussion, the degree to which problem solving is a focus in their mathematics classrooms, and so on.
Questionnaires have numerous advantages: They are simple to administer to large numbers of respondents and usually can be easily transformed into data files that are ready for statistical analysis.
On the other hand, there are at least three major limitations in using questionnaires to study classroom instruction. First, the words researchers use to describe the complexities of classroom instruction 2 -may not be understood in the same way by teachers or in a consistent way across different teachers. The phrase "problem solving" is a good example. Many reformers of mathematics education call for problem solving to become the focus of the lesson. But different teachers interpret this phrase in different ways. One teacher may believe that working on word problems is synonymous with problem solving, even if the problems are so simple that students can solve one in 15 seconds. Another teacher may believe that a problem that can be solved in less than a full class period is not a real problem but only an exercise. Such inconsistency in the use of terms is common in the United States, where teachers have few opportunities to observe or be observed by other teachers in the classroom. It may be that because teacher training in the United States generally does not engage teachers in discussions of classroom instruction, and because teachers are often isolated from one another by the conditions under which they work, teachers do not develop shared referents for the words used to describe instruction. Thus, when teachers fill in questionnaires about their teaching practices, interpreting their responses is problematic.
A second problem with relying on questionnaire-based indicators of instruction concerns their accuracy in reporting processes that may, at least in part, be outside of their awareness. Teachers may be accurate reporters of what they planned for a lesson (e.g., what kind of demonstration they used to introduce the lesson) but inaccurate when asked to report on the aspects of teaching that can happen too quickly to be under the teacher's conscious control.
A third limitation of questionnaires is their static nature. Teachers can only answer the questions we as researchers thought to ask. An observer might notice something important just by being in the classroom. This problem is more serious in international research, where unfamiliarity with other nations' instructional approaches makes effective questionnaire design difficult.
Advantages and Disadvantages of Live Observations for Studying Classroom Processes Having discussed some of the advantages and disadvantages involved in using questionnaires to study classroom processes, let us now discuss the advantages and disadvantages of using direct observational
techniques. Direct observation overcomes some of the limitations identified for questionnaires:
Observations allow behavioral categories to be defined objectively by the researcher, not independently by each respondent. They also enable researchers to study on-line implementation of instruction as well as the planned, structural aspects. Teachers themselves may be unaware of their behavior in the classroom, yet this same behavior could be easily accessible to the outside observer.
On the other hand, there are clear disadvantages of live observation as well. Just like questionnaires, observational coding schemes can act as blinders and may make it difficult to discover unanticipated aspects of instruction. The use of live observations also introduces significant training problems when used across large samples or, especially, across cultures. A great deal of effort is required to assure that different observers are recording behavior in comparable ways. In fact, when working in different cultures, it may be impossible to achieve high levels of comparability.
THE USE OF VIDEO FOR STUDYING CLASSROOM INSTRUCTIONBearing in mind the limitations of questionnaires and of live observational coding schemes, especially in the context of cross-cultural research, it was decided to use video for the present study. Most researchers, on hearing the word "video," imagine a small-scale qualitative study. This study is anything but small: Large quantities of video were collected on national samples of teachers. In fact, one goal of
BEST COPY AVAILABLE
Enables Study of Complex Processes Classrooms are complex environments, and instruction is a complex process. Live observers are necessarily limited in what they can observe, and this places constraints on the kinds of assessments they can do. Video provides a way to overcome this problem: Observers can code video in multiple passes, coding different dimensions of classroom process on each pass. On the first pass, for example, we coded the organization of the lesson; on the second, the use of instructional materials; and on the third, the patterns of discourse that characterize the classrooms of each country. It would have been impossible for a live observer to code all of these simultaneously.
Not only can coding be done in passes but it also can be done in slow motion. With video, for example, it is possible to watch the same sample of behavior multiple times, enabling coders to describe the behavior in great detail. This makes it possible to conduct far more sophisticated analyses than would be possible with live observers.
Increases Inter-Rater Reliability, Decreases Training Problems Video also resolves problems of inter-rater reliability that are difficult to resolve in the context of live observations. The standard way to establish the reliability of observational measures is to send two observers to observe the same behavior, then compare the results of their coding. This is often inconvenient and is even infeasible for studies that are performed cross-culturally or in geographically distant locations. Using video to establish reliability means that the behavior can be brought to the observers instead of vice versa. Thus, in the context of a cross-cultural study, observers from different cultural and linguistic backgrounds can work collaboratively, in a controlled laboratory setting, to develop codes and establish their reliability using a common set of video data.
Using video also makes it far easier to train observers. With video, inter-rater reliability can be assessed not only between pairs of observers but between all observers and an expert "standard" observer.
Disagreements can be resolved based on re-viewing the video, making such disagreements into a valuable training opportunity. And, the same segments of video can be used for training all observers, increasing the chances that coders will use categories in comparable ways.
Amenable to Post-Hoc Coding, Secondary Analysis Most survey data sets lose their interest over time. Researchers decide what questions to ask and how to categorize responses based on theories that are prevalent at a given time. Video data, because they are "pre-quantitative," can be re-coded and analyzed as theories change over time, giving them a longer shelf life than other kinds of data. Researchers in the future may code videotapes of today for purposes completely different than those for which the tapes were originally collected.
Amenable to Coding from Multiple Perspectives For similar reasons, video data are especially suited for coding from multiple disciplinary perspectives.
Tapes of mathematics classes in different countries, for example, might be independently coded by psychologists, anthropologists, mathematicians, and educators. Not only is this cost effective, but it also facilitates valuable communication across disciplines. The most fruitful interdisciplinary discussions result when researchers from diverse backgrounds compare analyses based on a common, concrete referent.
Facilitates Integration of Qualitative and Quantitative Information Video makes it possible to merge qualitative and quantitative analyses in a way not possible with other kinds of data. With live-observer coding schemes the qualitative and quantitative analyses are done sequentially: First, initial qualitative analyses lead to the construction of the coding scheme; then, implementation of the coding scheme leads to a re-evaluation of the qualitative analysis.
When video is available it is possible to move much more quickly between the two modes of analysis. Once a code is applied and a quantitative indicator produced, the researcher can go back and look again more closely at the video segments that have been categorized together. This kind of focused qualitative observation makes it possible to refine and improve the code, and may even provide the basis for a new code.
Provides Referents for Teachers' Descriptions Mentioned earlier was the problem that teachers lack a set of shared referents for the words they use to describe classroom instruction. Video can, in the long run, provide teachers, as potential consumers of the research, with a set of such referents. Definitions of instructional quality and the indicators developed to assess instructional quality could be linked to a library of video examples that teachers can use in the course of their professional development. In the long run, a shared set of referents can lead to the development of more efficient and valid questionnaire-based indicators of instructional quality.
Facilitates Communication of the Results of Research It is also possible, with video, to use concrete video examples in reporting research results. This gives consumers of the information a richer qualitative sense of what each category in the coding system means and a concrete basis for interpreting the quantitative research findings.
Provides a Source of New Ideas for How to Teach Another advantage of video over other kinds of data is that it becomes a source of new ideas on how to teach. Because these new ideas are concrete and grounded in practice, they have immediate practical potential for teachers. Questionnaires and coding schemes can help us spot trends and relationships, but they can't demonstrate a new way of teaching the Pythagorean theorem.
Disadvantages Despite all its advantages, video also has some disadvantages. At the very least, video raises a number of problematic issues that must be addressed if it is to yield accurate and valid information about classroom processes. In the next section we will discuss some of these issues and challenges.
ISSUES IN VIDEO RESEARCHThis section briefly discusses a number of issues that must be resolved in order to conduct meaningful video research.
Standardization of Camera Procedures Left to their own devices, different videographers will photograph the same classroom lesson in different ways. One may focus on individual students, another may shoot wide shots in order to give the broadest possible picture of what is happening in the classroom. Yet another might focus on the teacher or on the blackboard. Because we want to study classroom instruction, not the videographers' camera habits, it is important to develop standardized procedures for using the camera and then to carefully train videographers to follow these procedures. This study has done so, and the procedures are described in the Methods section of this document.
The Problem of Observer Effects What effect does the camera have on what happens in the classroom? Will students and teachers behave as usual with the camera present, or will we get a view that 'is biased in some way? Might a teacher, knowing that she is to be videotaped, even prepare a special lesson just for the occasion that is unrepresentative of her normal practices?
This problem is not unique to video studies. Questionnaires have the same potential for bias: Teachers' questionnaire responses, as well as their behavior, may be biased toward cultural norms. On the other hand, it may actually be easier to gauge the degree of bias in video studies than in questionnaire studies. Teachers who try to alter their behavior for the videotaping will likely show some evidence that this is the case. Students, for example, may look puzzled or may not be able to follow routines that are clearly new for them.
It also should be noted that changing the way a teacher teaches is notoriously difficult to do, as much of the literature on teacher development suggests. It is highly unlikely that teaching could be improved significantly simply by placing a camera in the room. On the other hand, teachers will obviously try to do an especially good job, and may do some extra preparation, for a lesson that is to be videotaped. We may, therefore, see a somewhat idealized version of what the teacher normally does in the classroom.