Effectively Communicating a Complex Validity Argument

smart female coworkers discussing near window in office

In our last blog, we explained how to use a graph and an outline to develop and represent a validity argument for the interpretation and use of scores from the AP World History test. But a validity argument by itself is going to be difficult to understand in either graph or outline form. In this blog, we present the narrative as a tool to help communicate a validity argument supporting the intended score interpretation and use, especially when communicating with a nontechnical audience.

Narrative

A narrative is a written description of events that tells a story. Specifically, a narrative is an account of a succession of events, ordinarily presented in an ideal chronological order (but may not be the order in which they occurred) and presented as a meaningful whole. The narrative is used to persuade the audience that the story told is the most plausible account of what “really happened” that can be constructed from the evidence. The narrative should compress the events into two or three paragraphs. In advocating for or against the intended score interpretation and use, the narrative should include only elements that are consistent with this stance. But all the elements of the narrative must be supported by the evidence.

The graph of the microstructure of the validity argument we discussed in last week’s blog (link to blog) is a useful tool in constructing the narrative intended for a target audience.  The graph helps to identify micro-arguments most likely to be relevant to that audience.  These micro-arguments, or “magnets,” are used to organize the narrative.

For example, the test developer may be developing a narrative intended for an audience of teachers. The research we cited in an earlier blog suggests that teachers would find the following evidence most relevant to their values and concerns: a) local and situational factors such as similar schools, grades, and classrooms to their own; and b) student thinking. Based on this understanding of teachers’ values and concerns, we identified the two micro-arguments, circled in red in Figure 1, as “magnets” for teachers. These two micro-arguments will probably have a powerful attraction for our teacher audience because both micro-arguments are supported by evidence that the teachers used to write items and recommend passing scores had many years of experience in classrooms just like the classrooms of our intended audience of teachers. 

After the “magnet” micro-arguments have been identified for a target audience, the next step in creating a narrative is to document the chain of micro-arguments from the initial data statement, through the magnet micro-arguments, and ending at the ultimate claim.  As we illustrate with a red line in Figure 1, each of the “magnetic” micro-arguments circled in red is connected to the ultimate claim through a different penultimate claim.  So the narrative for the teacher audience will combine both chains of micro-arguments.

After the chain of micro-arguments has been documented, the next step in creating a narrative is to reorganize the micro-arguments along the chain into a series of syllogisms in which the warrant is taken as true and used as the major premise, followed by the earlier claim used as the minor premise, finally followed by the later claim used as the conclusion. 

Figure 1. Graph of the microstructure of the validity argument supporting the interpretation and use of scores from the AP World History test.

For example, the first chain leads through the penultimate claim of Student scores reflect level of students’ historical thinking skills applied to historical content. The first micro-argument, including the initial data statement, in that chain would be reorganized into the following:

  • Major premise: Teachers’ classroom experience develops deep pedagogical knowledge.
  • Minor premise: Teachers who had taught AP World History for at least five years were recruited as item writers for the AP World History Exam.
  • Conclusion: Therefore, teachers who wrote AP History items had deep pedagogical knowledge of historical thinking skills at the targeted grade band.

The syllogism would be the following: Teachers’ classroom experience develops deep pedagogical knowledge. Item writers are teachers who had taught AP World History for at least five years, therefore teachers who wrote AP History items had deep pedagogical knowledge of historical thinking skills at the targeted grade band.

The second micro-argument in the same chain leading through the penultimate claim of Student scores reflect level of students’ historical thinking skills applied to historical content, would be reorganized into the following:

  • Major premise: Teachers’ pedagogical knowledge provides insight into student thinking and problem solving.
  • Minor premise: Teachers who wrote AP History items had deep pedagogical knowledge of historical thinking skills at the targeted grade band.
  • Conclusion: Therefore, teachers who wrote AP History items understood how to manipulate content to elicit the knowledge and skills at the targeted grade band.

The syllogism would be the following: Teachers’ pedagogical knowledge provides insight into student thinking. Teachers who wrote AP History items had this deep pedagogical knowledge of historical thinking at the targeted grade band, therefore teachers who wrote AP History items understood how to manipulate content to elicit the knowledge and skills at the targeted grade band.

The narrative component for the chain leading through the penultimate claim of Student scores reflect level of students’ historical thinking skills applied to historical content is created by combining the sequence of syllogisms, in chronological order, into a description of the chain of reasoning that tells part of the story.  For example, the first two syllogisms in the chain could be combined by dropping the claim common across the two micro-arguments from the second syllogism and doing a little wordsmithing, as shown in Figure 2.

Figure 2. The combination of the first two syllogisms in the chain of reasoning leading through the penultimate claim of Student scores reflect level of students’ historical thinking skills applied to historical content.

After combining the sequence of syllogisms, the narrative component for the chain leading through the penultimate claim of Student scores reflect level of students’ historical thinking skills applied to historical content would read something like this:

Teachers’ classroom experience develops deep pedagogical knowledge. Item writers for the AP World History test had taught AP World History for at least five years and had developed deep pedagogical knowledge of historical thinking skills at the targeted grade band. The deep pedagogical knowledge of these teachers provided insight into student thinking and how to manipulate content to elicit the knowledge and skills at the targeted grade band. Because all AP World History Exam items were written by these talented teachers, AP World History Exam items elicited the targeted historical thinking skills at the targeted grade band. Students’ responses to these items were scored using well-constructed rubrics and scaled using a widely accepted measurement model to create student scores that reflect level of historical thinking skills applied to historical content.

After combining the sequence of syllogisms, the narrative component for the second chain leading through the penultimate claim that The proficient cut score identifies the student score level reflecting the level of historical thinking skills that deserves college credit, would read something like this:

Only teachers with at least five years of experience in the classroom were recruited for and served on the standard setting committee. These teachers used their deep pedagogical knowledge of and insight into student thinking and problem solving to recommend the scores on the AP World History Exam that would qualify students to be awarded college credit.

The overall narrative combines both of these narrative components into a compelling story for the intended teacher audience that supports the intended score interpretation and use. The overall narrative intended for teachers might read something like the following:

Teachers’ classroom experience develops deep pedagogical knowledge. Item writers for the AP World History test had taught AP World History for at least five years and had developed deep pedagogical knowledge of historical thinking skills at the targeted grade band. The deep pedagogical knowledge of these teachers provided insight into student thinking and how to manipulate content to elicit the knowledge and skills at the targeted grade band. Because all AP World History Exam items were written by these talented teachers, AP World History Exam items elicited the targeted historical thinking skills at the targeted grade band. Students’ responses to these items were scored using well-constructed rubrics and scaled using a widely accepted measurement model to create student scores that reflect level of historical thinking skills applied to historical content.

Those items written by teachers were then shown to another group of experienced teachers serving on the standard setting committee. Those teachers used their deep pedagogical knowledge of and insight into student thinking and problem solving to recommend the level of performance on those kinds of items, making up the AP World History Exam, that would qualify students to be labeled Proficient and awarded college credit.

Students who performed well enough on these teacher-written items to score at or above the Proficient score recommended by experienced teachers deserve to be awarded college credit.

A narrative forces the test developer to organize and present the analysis in a familiar form—a story. Placing aside rhetorical devices and dramatic organization, the narrative method is designed to organize the claims into a coherent story, supported by the evidence, which will enable the test developer to assess whether an audience would be likely to understand and maybe be persuaded by the validity argument the story is intended to support.

What’s Next

Please join us next week for the fourth post in our blog series helping practitioners in educational measurement develop, represent, and effectively communicate validity arguments to stakeholder audiences. In the next blog, we will present thesis statements and themes as tools to summarize a validity argument for an audience.  Stay safe everyone—Michelle and Paul

2 thoughts on “Effectively Communicating a Complex Validity Argument

Leave a Reply

%d bloggers like this: