# «SE 062 649 ED 431 621 Stigler, James W.; Gonzales, Patrick; Kwanaka, Takako; AUTHOR Knoll, Steffen; Serrano, Ana The TIMSS Videotape Classroom Study: ...»

Coding procedures were developed by a group of four code developers, all of whom had participated in the initial viewing and discussion of the 27 field test tapes. One of the developers was from Germany (Knoll), one from Japan (Kawanaka), and one from the United States (Serrano). Each of these three were doctoral students in either psychology or education, and all had classroom teaching experience. The fourth member of the team was a doctoral student in applied linguistics (Gonzales), also from the United States, who helped us work through the technical issues involved in coding classroom discourse.

The coding development group first viewed field test tapes and a definition of the category to be coded was proposed. Each member of the group then attempted to apply the definition to field-test tapes from their country. Difficulties were brought back to the group and definitions were revised and refined.

This process was repeated until all members of the group were satisfied with the definitions and procedures, and in agreement with the coding of each instance.

Once codes were developed, coders were trained to implement the codes. Coders, like the code developers, came from Germany, Japan, and the United States. In order to reduce the likelihood that subtle contextual cues would be missed or misinterpreted, coders only coded tapes from their country, except for purposes of training or the assessment of inter-rater reliability. Training was comprised of several activities. Code developers used group meetings to present definitions and discuss procedures for coding. Coders then practiced coding on the field test tapes. Results of practice coding were brought back to the group for discussion and any disagreements were resolved. This process was repeated until coders from each country were applying the codes in a consistent way.

Before beginning to code the main study tapes, a formal reliability assessment was conducted to insure independent agreement across coders at a level of at least 80 percent for each judgment. Reliability was assessed by comparing each coder's results with a standard produced by the coding development team.

If reliability could not be established at the 80 percent level, the code was either dropped or sent back for revision. For the reliability assessment, coders worked with tapes from all countries, relying on English translations when necessary. We reasoned that reliability established across coders from different cultural backgrounds would be a low estimate of the actual reliability achieved among coders coding only tapes from their native countries. This also enabled us to make sure that coders from different countries were applying the codes in a comparable way.

Throughout this process we endeavored to be strategic. For example, just having collected 100 hours of video does not mean that all 100 hours must be analyzed. Depending on the frequency of what is being coded, it may be possible to time sample or event sample. It is also important to divide coding tasks into passes through the data in order to lessen the load on coders. This increases reliability and speeds up coding.

Implementation of Codes Using the Software The code module of our software enables coders to view synchronized video and transcript on their computer screen. On-screen controls allow them to move instantly to the point in the video that corresponds to the highlighted transcript record, or to the point in the transcript that is closest in time to the current frame of video. Depending on the code, codes can be marked either as time codes in the video or as highlighting in the transcript.

## FIRST-PASS CODING: THE LESSON TABLES

We have found that it is useful to have an intermediate representation of each lesson that can serve to guide coders as they try to comprehend a lesson, and that can be coded itself. For this purpose, our first step in coding the lessons was to construct a table that maps out the lesson along several dimensions. Each of these is defined in more detail as we present the results of the study, but a general idea**of what they are is useful at this point:**

Organization of class. Each videotape is divided into three segments: pre-lesson activities (Pre-LA), lesson, and post-lesson activities (Post-LA). The lesson needs to be defined in this way because lesson is the basic unit of analysis in the study.

Outside interruption. Interruptions from outside the class that take up time during the lesson (e.g., announcements over the public address system) are marked on the tables as well.

Organization of interaction. The lesson is divided into periods of classwork (CW), periods of seatwork (SW), and periods of mixed organization. Seatwork segments are characterized as being Individual (students working on their own, individually), Group, or Mixed.

Activity segments. Each classwork and seatwork segment is further divided, exhaustively, into activity segments according to changes in pedagogical function.

We defined four major categories of activities: Setting Up, Working On, Sharing, and Teacher Talk/Demonstration. (Each of these was divided into subcategories, which are defined more completely in Chapter 4.) Mathematical content of the lesson. The mathematical content of the lesson is described in detail. Content is marked, for analytical purposes, into units which are noted on the table: Tasks, Situations, Principles/Properties/Definitions, Teacher Alternative Solution Methods [TASM] and Student Generated Solution Methods [ SGSM]. (A more detailed description of each can be found below.) In addition, frames from the video are digitized and included in the table to help illustrate the flow and content of the lesson.

An example of what the resulting tables look like is shown in figure 3, which represents one of the Japanese lessons in our sample (JP-012).2 The table contains five columns. The first column indicates the time code at which each segment begins as well as the corresponding page number from the printed lesson transcript. The second column shows the segmentation of the lesson by organization of interaction, the third by activity. The fourth and fifth columns show the symbolic description of the content and the concrete description of the content, respectively. Rows with lines between them show segment boundaries. Seatwork segments are shaded gray. The acronyms used refer to the coding categories described.

Throughout this report, individual lessons from the sample are referred to by ID numbers, which include country of origin (GR, JP, US) followed by the lesson ID. In addition, all names used in excerpts from lesson tables and transcripts have been changed to pseudonyms.

NOTE: Abbreviations used in the table: Pre-LA=Pre-Lesson Activities; CW=Classwork;

T/S/PPD=Task/Situation/Principle Property Definition; PPD1=first Principle/Property/Definition in the lesson;

Setting Up: M=Setting Up/Mathematical; S1=first situation in the lesson; T1=first task in the lesson;

Ti[A]=initial, key, or target work of Task i; SW:I=Individual Seatwork; SW:G=Seatwork in Groups;

SW:SG=Seatwork in Small Groups; Sharing: T/S=Sharing Task/Situation; SGSM1=first Student Generated Alternative Solution Method in the lesson; Setting Up: Phys/Dir=Setting Up: PhysicaVDirectional; HS1=first Homework Situation in the lesson; HT1=first Homework Task in the lesson; HW=Homework; PostLA=Post-lesson activities.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Third International Mathematics and Science Study, Videotape Classroom Study, 1994-95.

## METHODS FOR DESCRIBING MATHEMATICAL CONTENT

There are many possible ways of describing mathematical content. One can describe content at a general level in terms of topics (e.g., ratio and proportion, or linear functions); in terms of categories such as "concepts" and "applications"; in terms of the specific tasks and situations assigned to students; or in terms of performance expectations. We attempted in this study to use all of these techniques.The bases of our content descriptions are found in the first-pass coding tables we made for each lesson. We presented an example of these tables in the First-Pass Coding section. Here, we will give a more detailed description of how we constructed the content descriptions for these tables.

As coders watched the video, they first produced a written description of the content in concrete terms. The following example (figure 4), excerpted from the lesson table presented in the First-Pass Coding section (JP-012), illustrates the different kinds of information recorded in the content description. The example shows one task and one situation that the teacher presents to the students, a hint provided by the teacher to the students during Seatwork, and a student's solution method to the problem.

(Chalkboard) There is Eda's land. There is Azusa's land. And these two people's border line is bent but we want to make it straight.

Try thinking about the methods of changing this shape without changing the area.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Third International Mathematics and Science Study, Videotape Classroom Study, 1994-95.

Once the concrete description is recorded, coders produced symbolic category descriptions of the content. Categorizing the content serves two functions: First, it helps guide the coders as they struggle to determine the proper level of detail to include in the description; second, it is useful for the analysis of

**content. We used five mutually exclusive categories for describing content:**

Situation (S)The mathematical environment in which tasks are accomplished.

(For example, real-world scenarios, word problems, and equations could all be the situations within which tasks are performed.) Task (T)The mathematical goal or operation to be performed on a situation.

(For example, "Try thinking about the methods of changing this shape without changing the area" is a task performed within the situation defined by the particular shape that is presented.) Teacher Alternative Solution Method (TASM)An alternative method for solving a problem. A first method must be presented within the same lesson in order for there to be an alternative method coded.

Student Generated Solution Method (SGSM)A solution method generated and then presented by a student.

Principles, Properties, and/or Definitions (PPD)Mathematical information that is not contained in tasks and situations.

Each of these codes are numbered in order to keep track of the content as the lesson unfolds. A change in number signifies a shift to a new event. Numbers of tasks and situations are linked. For example, the notation Tl-S1 represents a specific task and situation combination. If the same task is assigned for multiple situations (as might happen, for example, with a worksheet containing a number of similar exercises), the notation might be as follows: T 1, S1-1, S1-2, S1-3. The first number after the "S" refers to the task (T1) that is being performed on each situation. The second number (1, 2, 3) refers to the

**number of different situations. A situation related to more than one task would be expressed similarly:**

SI, T1-1, T1-2, T1-3.

Relatedness of tasks/situations to previous events (e.g., Teacher Alternative Solution Methods) is expressed in symbolic form with parentheses. For example, T3-S3 (TASM2) means that task and situation "3" is directly related to the previous Teacher Alternative Solution Method 2.

## THE MATH CONTENT GROUP

As mentioned in the introduction, one advantage of collecting video data is that they can be analyzed from multiple perspectives. In this report, we include some analyses by an independent group of researchers who are experts in mathematics and mathematics teaching. There were four members of this group (see names and brief biographical descriptions in appendix C). One had taught mathematics primarily at the high school level, one primarily at the college level but with extensive experience also teaching high school students, one primarily at the college level, and one at both college and graduate levels.The group was assigned the task of analyzing the mathematical content of each lesson based only on information contained in the lesson tables. The group worked with a subsample of 90 lessons, 30 from each country In order to reduce the possibility of bias, tables were disguised so that the group would not know the country of origin. This typically required only minor changes in such details as the names of persons and currencies. Lesson tables were identified only by an arbitrarily assigned ID code; relation of this code to country was not revealed until all coding was complete. Thus, although the group was denied the additional information that would have been provided by looking at the videotapes, the goal was to make possible a blind analysis of content. The analytic tools for describing the lesson content were developed by this group and will be described when we present the results of their analyses.

Coding of Discourse Language is one of the key tools teachers and students use for instruction and learning. Consequently, focusing on how language is used in the classroom has the potential to enrich our analysis of instructional processes (see, for example, Be flack, 1966; Cazden, 1988; Mehan, 1979). It is also true that reformers of mathematics education have focused a great deal of attention on changing the kind of discourse that goes on during mathematics lessons. Mathematics teachers and students, it is suggested, should use 51.

;

language in much the same way that mathematicians do: to explain, justify, conjecture, and elaborate on mathematical understandings (see, for example, Hiebert & Wearne, 1993; Lampert, 1991).

Coding of discourse is very labor intensive, and very difficult when working across three languages.

We decided to code discourse in several passes, and to employ a sampling scheme to save resources. We based our coding system on previous work (e.g., see references in previous paragraph), and on analysis of the field-test tapes.