Why Is Principled Assessment Design Not Sufficient To Build Successful Assessment Products?

crop girl tying shoelaces on playground

As assessment designers and developers, we can do better than we do now building assessments that positively impact our customers’ lives. But using principled assessment design (PAD) is not enough.

PAD is a collection of assessment design approaches including construct-centered measurement, cognitive design system, evidence-centered design, principled design for efficacy (PDE), and assessment engineering. See this wonderful chapter by Steve Ferrara and colleagues for an overview of PAD approaches.

A defining characteristic across PAD is the explicit construct‐centered nature of the approach. Under a construct‐centered approach, assessment design begins with a careful and comprehensive examination of the constructs intended to be assessed – the targets of inference – and all subsequent design decisions cascade from that initial definition of the construct. Assessment designers are compelled to justify, based on the definition of the target of inference, the chain of decisions necessary to implement an assessment program.

For example, you can understand PDE as a set of six concepts that every assessment developer should understand and use to organize the set of more specific issues that must be addressed during assessment design and development. As shown in 1, the five constructs of consequences, content, evidence, communication, and implementation are organized around the central concept of the construct. 

Figure 1. The PDE conceptual framework

Job-to-Be-Done

But PAD is necessary but not sufficient to develop successful assessment products.  This is because PAD overlooks what should be central in design of any assessment product—the job-to-be-done (JTBD) for the customer. The JTBD describes what customers are trying to achieve.  This may be work they are trying to get done, goals they are trying to attain, or challenges they are trying to overcome.  The JTBD combines an understanding of customer needs, problems, and pain points. A well-defined description of the JTBD consists of the end user and the job statement. 

Defining the end user

The identification of the end user answers the question, “For whom is the product intended to do a job?” The end user is experiencing problems unaddressed or poorly addressed by existing solutions. In education, the end user may be found in a state department of education, district office, school building, classroom, and home.

Defining the job statement

The second part of the JTBD description is the job statement. The job statement answers the question, “What problem is the solution attempting to solve?”  A formal job statement always begins with a verb and is followed by the object of the verb (a noun). The description should also include a contextual clarifier. If the end user is a classroom teacher, a potential job statement might be: Improving (verb) student learning (object) for each and every student in the classroom (contextual clarification).  

Innovation Factory

The Innovation Factory is a design approach that does put the JTBD for the customer at the center of assessment product design. In an earlier blog, I described the Innovation Factory, shown in Figure 2, as an organizational system intended to leverage research from the sciences of learning and assessment to engineer education products that generate impact for learners and stakeholders aligned with organizational mission. In the Innovation Factory, PAD concepts and practices are used to systematically link the understanding of the JTBD to creation of an assessment product in a manner that makes explicit the ways the assessment is intended to address the JTBD and positively impact customers’ lives, thus creating value for the customer and the organization. PAD is an element of successful design and development within a system.

Figure 2. The Innovation Factory product design and development system.

Remedy

To do better than we do now building assessments that positively impact our customers’ lives, we need to take a transdisciplinary approach.  This can be uncomfortable. But we owe students, parents, and educators our best effort to break out of current disciplinary boundaries and single-actor views of real-world problems. We owe them the effort to take a systemic view, with the customer and JTBD at the center of this system, when building assessment products.

Leave a Reply

%d bloggers like this: