In our last blog, we described how to develop and use narratives as a tool to help the test developer communicate a validity argument to a nontechnical audience. In this blog, we give you two additional tools to help you effectively communicate a validity argument supporting the intended score interpretation and use:
- A thesis statement, and,
When used in coordination with narratives, these tools support an effective strategy to communicate a complex validity argument to even a nontechnical audience.
The thesis statement supporting your validity argument should be the strongest, positive summary of the validity argument for a targeted audience. The thesis statement can be thought of as an elevator pitch for the validity argument. The thesis statement should be a clear and crisp summary of the validity argument in one to four sentences. The intent is to support the ultimate claim by reinforcing the penultimate or interim claims the test developer believes are most relevant from the standpoint of the intended audience.
A way to construct a thesis statement is to create a syllogism. The syllogism uses the ultimate claim followed by the word because, then summarizes the penultimate or interim claims, and finally summarizes an initial claim or claims most relevant to the concerns of your target audience. The developer of the thesis statement works backwards toward the earliest claims in the microstructure argument. The goal is to focus attention on the critical or most relevant penultimate or interim claims for building the validity argument for the given audience.
In Figure 1, we have drawn a red line connecting the ultimate claim to the earliest claims, circled in red, we believe are most relevant to the concerns of our targeted audience of teachers. We explained in an earlier blog why we believe this evidence is important to teachers. To construct the positive thesis statement, we restate the ultimate claim and attach “because …” to the end of the claim. We use the next one or two sentences to summarize the sequence of claims between the ultimate claim and the evidence we believe most important to teachers. Finally, we restate the claims important to the teachers that also serve to support the intermediate claims. (Note that we ignored claims in the validity argument such as the claim about the measurement model or about the rubric, as these claims were not on the critical path of reasoning between the ultimate claim and the evidence most important to our teacher audience.)
Figure 1. Graph of the microstructure of the validity argument supporting the interpretation and use of scores from the AP World History test.
For example, a thesis statement intended for teachers and supporting the intended score interpretation and use of AP History Proficient level classifications might read as follows:
Students who achieve a Proficient level of applying historical thinking skills to historical content deserve to be awarded college credit because … The Proficient level has been established using cut scores developed by experienced AP History teachers and applied to student scores on items written by experienced AP History teachers. Experienced AP History teachers understand the historical thinking skills used by students to respond to historical content. Teachers’ insight into student thinking and problem solving is gained from years in the classroom.
Although we’ve primarily talked about constructing an argument supporting the ultimate claim, the test developer should also construct the strongest thesis statement that challenges the ultimate claim. There are two reasons for doing so. First, the test developer needs to identify the claims most likely to be challenged to identify claims that require stronger rationales or backing for support. Second, the test developer needs to identify the claims most likely to be challenged, to evaluate if the positive thesis statement addresses what is important to critics.
In addition to thesis statements, themes play an important role in communicating the validity argument. Themes are recurring ideas that can serve to emphasize and reinforce certain parts of the validity argument. The test developer has made judgments about the relative plausibility of the narratives and theses that could be developed to reinforce or attack the ultimate claim. With this background, the test developer must decide what themes can be established and used to reinforce the narrative and thesis. In this context, themes make explicit the concept of persuasive presentation. The test developer should identify and state the one or, at most, two best themes for presenting the validity argument. The following are possible themes for particular audiences:
- A test made by teachers for teachers (audience of teachers);
- A test that meets the highest technical standards (audience of psychometricians); and,
- A test developed with fairness in mind (legal audience).
So far in this blog series, we have helped practitioners in educational measurement develop, represent, and effectively communicate validity arguments that were supporting the intended score interpretation and use. Next week we turn that approach on its head and explain how to develop, represent, and effectively communicate arguments skeptical of the validity of the intended score interpretation and use. Please join us next week for the fifth in our blog series to see how we do it!