Turning to the Evidence

The last decade has seen a boom in the use of “manipulatives” in K–8 mathematics classrooms. Teachers around the country are using cubes, pattern blocks, tiles, and other hands-on tools to make mathematical concepts more engaging and concrete for students. The trend has spread even without much research that demonstrates why, how, or if manipulatives work.

EDC’s Lynn Goldsmith sees a parallel in the growing popularity of professional development (PD) programs for mathematics teachers that use “classroom artifacts”—including student work, videotapes, and transcripts drawn from the classroom. “The use of classroom artifacts in PD has become a very big thing,” says Goldsmith, a senior scientist at EDC. “There are a lot of theoretical arguments and anecdotal evidence about why it’s effective to use artifacts in PD, but none of it is explicit about the value of using artifacts. It’s similar to what we saw in the classroom with manipulatives: People embraced them as if they provided a magical solution.”

Goldsmith, with co-director Nanette Seago of WestEd and EDC colleagues Mark Driscoll, Johannah Nikula, and Zuzka Blasi, is carrying out a four-year research study called Turning to the Evidence, designed to understand the value of classroom artifacts and the most effective ways to use them in teacher PD programs. The research team is using qualitative and quantitative methods to track the experience of 52 middle and high school teachers from four school districts (two from the East Coast and two from the West Coast) who are participating in one of two PD programs. The programs use materials from the classroom: the Fostering Algebraic Thinking Toolkit, developed at EDC by Driscoll and colleagues, and Learning and Teacher Linear Functions: Video Cases for Mathematics Professional Development, developed by Seago, Nicholas Branca, and Judith Mumme. At each school district, the programs are being facilitated by their designers, which ensures that the programs are being implemented as designed.

The study, funded by the National Science Foundation, uses two paper-and-pencil instruments (administered pre- and postprogram) to assess the impact the PD programs are having on the teachers and their students:

• Mathematics survey: The project has adapted a mathematics survey used in other research projects to assess teachers’ algebra knowledge. It consists of multiple choice and open response questions. In addition to administering the survey to the 52 teachers in the PD programs, the research team gave the same instrument to 32 “comparison” teachers, who were recruited from the same four districts but who are not participating in the PD programs.
  
• Artifact analysis: The project developed an artifact analysis instrument, which consists of teachers’ written answers to questions about a classroom video of students discussing a mathematical task, and their comments on written samples of student work.

In addition, the project is engaged in qualitative analysis of videotapes of the PD sessions and videotapes of pre- and postprogram classroom visits.

The mathematics survey showed gains for the treatment group at the end of the project: The teachers attempted to answer more of the questions on the survey, and they answered more of the questions correctly. However, those gains were found to be no greater than the gains shown by the comparison group. “The comparison group turned out to be very valuable,” says Goldsmith. “Without the comparison group, we’d be talking about all the gains the teachers made in the post-program surveys. But it turns out that the comparison group teachers made the same gains.”

Goldsmith and her colleagues aren’t sure why all of the teachers improved on the survey to the same degree. They suspect that the math survey isn’t fine-tuned enough to show the changes that may be taking place as a result of the PD programs. “Teachers may be developing the kinds of thinking that can’t be measured by the instrument,” says EDC researcher Johanna Nikula.

Some of those changes are showing up, however, in the artifact analysis. In comparing the written and videotaped comments teachers made on the artifacts before and after the PD programs, they see a number of differences between the treatment and the comparison group.

“The research staff coded the artifact analysis responses ‘blind’—we didn’t know which group the teachers were in,” says Goldsmith. “We analyzed their responses for several issues related to their mathematical thinking and the way they approach students’ mathematical thinking. The results are very encouraging.” For example, the researchers found that, relative to the comparison group, the “treatment” teachers did the following:

• Focused more on the mathematical potential of students’ ideas, rather than simply on the correctness or incorrectness of their work
• Backed up their claims with evidence from the artifacts
• Were more focused on the mathematical ideas in the artifacts

In addition, ongoing analysis of the seminar sessions themselves suggests changes in the ways that teachers approach classroom artifacts, for example:

• Developing more sustained focus on understanding the particulars of students’ thinking and the ways that this thinking relates to important mathematical ideas and habits of mind
• Looking beyond the correctness of a student’s answer to see the mathematical potential in that student’s ideas and to create plausible story lines about what students might be thinking mathematically
• Having a more expansive notion of what “counts” as algebraic thinking

“Teachers who have gone through the programs are more aware of different approaches and different mathematical representations, such as diagrams and tables in addition to graphs and formulas,” says researcher Zuzka Blasi. “They get underneath the mathematical symbols and begin to talk about the meaning of the formula— what’s actually happening.”

Those are the kinds of changes that Mark Driscoll hopes to see among the teachers participating in the Fostering Algebraic Thinking Toolkit PD program, which he designed with EDC colleagues. And yet, while the artifact analysis shows clear signs that teachers are beginning to shift their thinking about student work and classroom videos, the research team has not yet seen strong evidence of changes in classroom practice.

“We see some indications of change in the classroom, but mostly what we are seeing is a slow-moving system,” says Goldsmith. “It takes quite a while for teachers to begin to use practices and strategies they learn in PD sessions in the classroom.” Some teachers have reported to the researchers that they plan to make changes in the next academic year based on the things they learned from the PD program.

The research team is planning return visits to the teachers’ classrooms, as well as other measures to assess the longer term effects of the PD programs. They also see Turning to the Evidence as a preliminary study designed to understand the dynamics in play when classroom artifacts are used effectively in PD programs. “A lot of research studies focus on comparing what goes in the box and what comes out of the box after the treatment,” says Johannah Nikula. “Our study focuses on unpacking what’s inside the box. What’s happening, and what makes it happen?”