We, Michelle and Paul, are partnering to offer a tutorial as a series of six blogs—a new blog posted every week starting Tuesday, September 14, 2021—to help practitioners in educational measurement develop, represent, and effectively communicate validity arguments to stakeholder audiences. A validity argument is a valuable tool, as it can build the foundation for score interpretation and use. However, validity arguments are also a complex network of claims supported, more-or-less, by evidence. An extended validity argument can be difficult to effectively communicate and evaluate, especially for a nontechnical audience. In addition, the kind of evidence valued by different stakeholders in the assessment process may be unclear—or different—for each stakeholder group.
We borrow tools from the legal field (Michelle is a psychometrician and a lawyer!) to support the development and effective communication of a validity argument. The legal field has for decades, even centuries, been developing and evaluating different approaches to discover what evidence is important to a particular stakeholder audience and then to effectively communicate complex, evidence-based arguments to that audience. In the series of blogs, we present several tools from the legal field—graphs and outlines—that support the development of a validity argument. These graphs and outlines are then used by additional legal profession tools—narratives, theses, and themes—to effectively communicate the validity argument. These tools can be used both to support and to challenge an intended score interpretation and use. These tools support communicating effectively to different stakeholder audiences for the validity argument including policymakers, teachers, lawyers, parents, students, and psychometricians.
In applying these communication tools—narratives, theses, and themes—we will make the following recommendations for practice:
- Two versions of the communication tools should be drafted: a version supporting the intended score interpretation and use and a version skeptical of the intended score interpretation and use.
- The test developer should craft the supporting versions of these communication tools and a disinterested third party, or a test critic, should craft the skeptical version of these communication tools.
- Both the supporting and skeptical versions of the communication tools should be presented for a balanced approach that allows a stakeholder audience to fairly evaluate the validity argument.
- Empirical research should be implemented to evaluate if this approach is effective in communicating validity arguments to different stakeholder audiences.
Some readers may be thinking this sounds familiar. Some of these ideas have been presented earlier at the annual meeting of the National Council of Measurement in Education and the National Conference on Student Assessment. But we bring all the ideas together in these posts.
This blog series consists of six blogs with a new blog posted every week starting Tuesday, September 14.
- In the first blog, we explain why you should pay attention to stakeholder values and concerns and address those concerns when developing and communicating your validity argument.
- In the second blog, we will describe how to use graphs and outlines to develop and represent a validity argument and the advantages and disadvantages of each approach.
- In the third blog, we will offer instructions for creating narratives as tools to help the test developer, from a perspective supporting the intended score interpretation and use, communicate a validity argument to a nontechnical audience.
- In the fourth blog, we will offer instructions for creating thesis statements and themes to help the test developer summarize the validity argument from a perspective supporting the intended score interpretation and use, i.e., put your best foot forward.
- In the fifth blog, we will describe how these same tools can be used by the test developer in the role of test critic, from a perspective challenging the intended score interpretation and use, to communicate an opposing version of the validity argument.
- Finally, we will conclude our blog series with a post suggesting the test developer should present both the view supporting and the view challenging the validity argument. We are skeptical that a test developer, given the interest the test develop has in supporting test use, can adequately present both views.
Hope you join us next week for our first blog in this series. We will explain how to address stakeholder concerns when developing and communicating your validity argument. Hit subscribe if you want to be notified when the blog is posted.—Michelle and Paul