«SE 062 649 ED 431 621 Stigler, James W.; Gonzales, Patrick; Kwanaka, Takako; AUTHOR Knoll, Steffen; Serrano, Ana The TIMSS Videotape Classroom Study: ...»
The camera we selected was a Sony EVW-300 three-chip professional Hi-8 camcorder. Each camera was mounted on a Bogen fluid-head tripod. (Tripods that are not fluid head will produce jerky camera movements.) A small LCD monitor was mounted on the camera to help operators view what they were taping. Sound was collected using two microphones, one a radio microphone worn by the teacher, the second a shotgun zoom microphone mounted on the camera. This equipment was used both for data collection and for training videographers.
TEACHER QUESTIONNAIREA complete copy of the English version of the teacher questionnaire can be found in appendix D.
The purpose of the teacher questionnaire was to elicit information that would aid us in analysis and interpretation of the videotapes. Items for the questionnaire were generated by project personnel in consultation with persons working on the main TIMSS questionnaire, questionnaire design specialists from Westat,' mathematics educators, and classroom teachers. Questions were edited and selected to yield a questionnaire that could be completed by teachers in approximately 20-30 minutes.
The questionnaire was translated and back-translated into German and Japanese and then pilot-tested on teachers participating in the field test. German, Japanese, and U.S. collaborators discussed the responses from the field test, and based on these discussions the questionnaire was revised.
The final translation of the questionnaire was painstakingly reviewed, question by question, by a group of German, Japanese, and U.S. researchers, each of whom was bilingual in two of the three languages.
Questions that were judged too difficult to translate accurately were dropped from the questionnaire.
The resulting questionnaire consisted of three parts with a total of 28 questions. In Part A we asked questions about the lesson that we videotaped, and about how the class was constituted and who the students were. In Part B we asked the teachers to compare what happened in the videotaped lesson with what would typically transpire in their classroom. In Part C we asked teachers to describe what they know about current ideas on mathematics teaching and learning, and asked them to evaluate their own teaching in the videotape in light of these current ideas.
The information collected in the questionnaire served three purposes. First, it helped us to assess the quality and comparability of our samples across the three countries. Although teachers were instructed not to prepare in any special way for the videotaping, we were still aware that what we saw on the videotape might not be typical of what normally happens in a given classroom. Teachers thus were asked to rate the typicality of the videotaped lesson, and these ratings were compared across countries. Similarly, we were able to assess the comparability of the samples across the three countries along several other dimensions as well. For example, whether a lesson dealt with new material or review material might be expected to influence the kind of teaching technique used. Knowing the percentage of lessons in each country that were new versus review helped us to judge the comparability of the samples.
' Westat, of Rockville, Maryland, was the general contractor that conducted the U.S. TIMSS and the Videotape Classroom Study.
A second purpose for the questionnaire was to provide coders with information that would help them interpret what they saw on the videotapes. For example, it is often necessary to know the teacher's goal for a lesson in order to make sense of the activities that constitute the lesson, and so we asked the teacher to state his or her goal for the lesson. Similarly, to interpret the meaning a specific question has for students it is often helpful to know whether the question probes new material or reviews previously learned information. Again, teachers were asked to categorize the content of the lesson in this way on the questionnaire.
Third, the questionnaire responses did, in some cases, enter directly into the analysesstatistical and qualitativeof the videotapes. This occurred in several ways. First, questionnaire responses were entered into correlational analyses within each country to help us relate contextual factors to variations in classroom instruction. Second, by asking teachers to comment on the lesson that was videotaped we were able to learn more about how teachers interpret the language of reform in mathematics education. For example, if a teacher told us that her lesson was focused on problem solving, we could look at the video to see what she meant by the term "problem solving."
Response rates on the questionnaire were high: In Germany, 91 percent of the teachers whom we videotaped returned their questionnaires, in Japan, 94 percent, and in the United States, 97.5 percent.
CONSTRUCTING THE MULTIMEDIA DATABASEOnce the tapes were collected, they were sent to project headquarters at University of California, Los Angeles (UCLA) for transcription, coding, and analysis. The first step in this process was to digitize the video and store it in a multimedia database, together with scanned images of supplementary materials.
The videotapes were then transcribed, and the transcript was linked by time codes to the video. This multimedia database was then accessed, coded, and analyzed using the multimedia database software system that we developed for this project.
Digitizing, Compression, and Storage on CD-ROM To facilitate the processing of such large quantities of video data, we decided to digitize all of the video and supplementary materials, which allowed them to be stored, accessed, and analyzed by computer. Each videotape was digitized, compressed, and stored on CD-ROM disks, one lesson per disk. We then designed and built a multimedia database software application that would enable us to organize, transcribe, code, and analyze the digital video.
Digital video offers several advantages over videotape for use in video surveys. First, the resulting files are far more durable and long lasting than videotape. CD-ROM disks are assumed to last for at least 100 years, as opposed to a much shorter lifespan for videotape. Digital video files also can be copied without any loss in quality, which again is not true for videotapes. And, digital files will not wear out or degrade with repeated playing and re-playing of parts of the video. Digital video also enables random, instantaneous access to any location on the video, a feature that makes possible far more sophisticated analyses than are possible with videotape. For example, when coding a category of behavior it is possible to quickly review the actual video segments that have been marked for that category. This rapid retrieval and viewing of coded segments makes it much easier to notice inconsistencies in coding, or to discover new patterns of behavior, that would not be possible without such rapid access.
As videotapes arrived in Los Angeles they were digitized and compressed into MPEG-1 format on a large hard disk. Text pages, worksheets, and other supplementary materials collected by the videographers were digitized on a flatbed scanner and stored in PICT format on the same hard disk drive as the accompanying videotape. All files for each lesson were then burned onto a single CD-ROM disk.
Transcription/Translation of Lessons Transcription of videotapes is essential for coding and analysis. Without a transcript, coders have difficulty hearing, much less interpreting, the complex flow of events that stream past in a classroom lesson. It also is possible to code some aspects of instruction directly from the transcript, without viewing the video at all. We thus transcribed, as accurately as possible, the words spoken by both the teacher and the students in each lesson and, for German and Japanese lessons, translated the transcriptions into English.
We had several reasons for translating the German and Japanese tapes into English. First, translations were used for training coders from different cultures to apply codes in a comparable way, and for establishing independent inter-rater reliability of codes across coders from different cultures. Even though a translation is never perfect, agreement between coders working with a translation can give us a rough estimate of how reliable a code is. A second purpose for the translation was to aid us in multilanguage searches of the database. If we want to locate, for example, times when a teacher discussed the concept of area we can search using the English word "area." Finally, having the lessons translated allows members of the research team not fluent in German or Japanese to view and understand lessons taught in those languages.
Procedures were developed to ensure that all transcriptions/translations were carried out in a standardized manner. For example, transcribers were given rules about how to indicate speakers, how to break speech into turns, how to use punctuation in a standardized manner, and how to translate technical terms in a consistent way. Using the multimedia database software developed for this project, coders had instant access to the video as they worked with the linked transcripts, and so could easily retrieve the context needed for interpreting the transcript. It was therefore not necessary to transcribe the contextual information generally needed for understanding written transcripts. By the same token, translations of the German and Japanese lessons did not have to be perfect, as all coding was done by native speakers of the language being coded. The translations served as a guide, but not as the actual foundation for coding. Coders did not rely on translations to make subtle judgments about the contents of the video.
Videotapes were transcribed and translated by teams of transcribers fluent in the language they were transcribing. Some members of the German and Japanese teams were native speakers of those languages, while others were native speakers of English but fluent in German or Japanese. Each tape was transcribed/translated in two passes. One person worked on the first pass transcription/translation of a tape and then a different person was assigned to review the work. A hard copy of the first pass transcription/translation was printed out, and the reviewer marked any points of disagreement on this copy. The two individuals then met, discussed all the proposed revisions, and came to an agreement about what the final version should be. In the extremely rare cases in which disagreements could not be resolved, a third party was consulted.
The last step in the transcription/translation process was to time code the tapes (i.e., to mark the exact point at which each utterance begins).
DEVELOPING CODESDeciding What to Code In deciding what to code we had to keep two goals in mind: First, we wanted to code aspects of instruction that relate to our developing construct of instructional quality; Second, we wanted the codes we used to provide a valid picture of instruction in three different cultures.
For the first goal, we sought ideas of what to code from the research literature on the teaching and learning of mathematics, and from reform documentssuch as the NCTM Professional Standards for Teaching Mathematicsthat make clear recommendations about how mathematics ought to be taught.
We wanted to code both the structural aspects of instruction (i.e., those things that the teacher most likely planned ahead of time), and the on-line aspects of instruction, (i.e., the processes that unfold as the lesson progresses).
The dimensions of instruction we judged most important included the following:
The nature of the work environment. How many students are in the class? Do they work in groups or individually? How are the desks arranged? Do they have access to books and other materials? Is the class interrupted frequently? Do the lessons stay on course, or do they meander into irrelevant talk?
The nature of the work that students are engaged in. How much time is devoted to skills, problem solving, and deepening of conceptual understanding? How advanced is the curriculum? How coherent is the content across the lesson?
What is the level of mathematics in which students are engaged?
The methods teachers use for engaging students in work. How do teachers structure lessons? How do teachers set up for seatwork, and how do they evaluate the products of seatwork? What is the teacher's role during seatwork? What kinds of discourse do teachers engage in during classwork? What kinds of performance expectations do teachers convey to students about the nature of mathematics?
Our second goal was to accurately portray instruction in Germany, Japan, and the United States.
Toward this end, we were concerned that our descriptions of classrooms in other countries make sense from within those cultures, and not just from the U.S. point of view. One of the major opportunities of this study, after all, is that we may discover approaches to mathematics teaching in other cultures that we would not discover looking in our culture alone. We wanted to be sure that if different cultural scripts underlie instruction in each country we would have a way to discover these scripts.
For this reason, we also sought coding ideas from the tapes themselves. In a field test, in May 1994, we collected nine tapes from each country. We convened a team of six code developerstwo from Germany, two from Japan, and two from the United Statesto spend the summer watching and discussing the contents of the tapes in order to develop a deep understanding of how teachers construct and implement lessons in each country.
The process was a straightforward one: We would watch a tape, discuss it, and then watch another.
As we worked our way through the 27 tapes we began to generate hypotheses about what the key crosscultural differences might be. These hypotheses formed the basis of codes (i.e., objective procedures that could be used to describe the videotapes quantitatively). We also developed some hypotheses about genBEST COPY AVAILA: LE eral scripts that describe the overall process of a lesson and devised ways to validate these scripts against the video data.
Developing Coding Procedures Once we had developed a list of what to code we began developing the specific coding procedures.