Of That

Brandt Redd on Education, Technology, Energy, and Trust

22 April 2021

Abundant Assessment

The Assessment for Learning 2021 conference is using a flipped model, much like a flipped classroom. For my session on Abundant Assessment I prepared a five-minute Ignite talk in video form (view here). The live Q&A discussion will be on 6 May 2021 at 12:00pm Pacific time. The post below is a slighly edited transcript of the video.

Figure skating judging form with the "Retry" rating emphasized.

My daughter in law teaches figure skating. To advance, skaters must pass tests in which they trace standard patterns in the ice, skating forward and backward and changing feet at precise locations. While the artistic part of figure skating competition has changed a lot, the “Moves in the Field” tests have changed little in more than 30 years. Skaters know exactly which patterns will be on a test. They practice them hundreds of times. And they are evaluated by their coaches before they go before judges for a formal exam. The lowest result on an exam is “Retry” and there is no shame for receiving that score. Judges give valuable feedback and coaches consider exams to be an important part of the learning experience.

Skating tests are Abundant Assessments because students know what will be on the test, they are designed to support learning, and students can take them as many times as needed at a modest cost.

Assessments foster learning by letting a student demonstrate their skill, and letting them and their teacher, tutor, or coach know how far they have progressed. Many subjects, such as arithmetic, cannot be learned without constant assessment.

Unfortunately, the consequences of many assessments seem to be excessive. If you fail a final exam, you generally must retake the entire course. Getting a low score on the SAT may block you from the university you hope to attend. Simply being sick on the wrong day may result in a student taking remedial mathematics in summer school.

Why is this? Why aren’t there more practice tests? Why can’t you retake a final?

Assessment Scarcity

Unlike figure skating tests, MOST exams are expensive. They cost a lot to create, to score, and to report. In economic terms, this means that they are scarce.

Assessment scarcity provokes many problems in contemporary education. Learning from assessment is impaired because feedback is infrequent. Instructors are reluctant to give early access to tests for fear that students will memorize answers. Parents and the community can’t review exam questions. Assessment groups must monitor social media for leaked questions.

Abundant Assessment

Abundance is the economic term for things that are cheap and plentiful. Abundant Assessment offers students early and frequent feedback. A failed exam is transformed into a learning experience with opportunity to retry. Assessment becomes part of the learning process rather than a distinct event.

Can academic assessments be made abundant? During my tenure at the Gates Foundation, I asked that question. You would have to cut the per-student cost of preparing tests, scoring them, and reporting by about one hundred times while still maintaining high quality.

I postulated that this could be accomplished by pooling the resources of numerous institutions and the strategic application of technology. A shared question bank would spread the cost of writing good questions across many institutions. Technology, including strategic application of AI, can cut the cost of scoring. And for questions that cannot or should not be scored by computer, self-scoring and peer-scoring are proven to be excellent learning activities.

I accepted the role of CTO at the Smarter Balanced Assessment Consortium, in part, to learn as much as I could about assessment and determine whether these theories are supportable. Over five and a half years I helped develop tests that are now used in twelve states and administered to six and a half million students each year.

Along the way I participated in a workshop at an Open Educational Resource conference where we asked the question, “Can an assessment be offered under an open license?” Conventional wisdom is that you can’t openly license assessment questions because students will memorize the answers.

But that wisdom is wrong. For some assessments, memorization is appropriate. That’s what’s happening with the figure skating tests – muscle memorization. For assessments requiring reasoning and problem solving you apply computer-adaptive testing algorithms and a large pool of questions. In that framework, the probability of a student seeing a question they studied is small enough as to not materially affect the test result. So, an assessment can be open if it has a large enough pool of questions from which tests are assembled.

Can Open and Abundant Assessment be achieved? Absolutely! Practice, data, and theory all support the proposition.

The technical requirements are:

  • Question/Interaction types that are authentic to the skills being taught and assessed.
  • An assessment format that is accepted by a broad range of Assessment and Learning Management systems.
  • A very large question/item bank.
  • Metadata and algorithms for automated assessment composition.

The organizational requirements are:

  • Passionate Leadership,
  • Resource sharing among institutions,
  • Strategic application of technology.

The beauty of an Abundant Assessment initiative is that it leverages economic incentives to encourage and enable the desired outcome. When offered inexpensive and easy-to-administer tests and quizzes, students and teachers will naturally take the opportunity to practice, get feedback, and learn.

As the cost of retaking an exam drops its use for learning is amplified and the consequences naturally become milder. “Retry” becomes the standard low score instead of “Fail.” This will literally save lives.

To be sure, some advocacy and training will be helpful but it’s easier to push a boulder downhill than up. Abundant assessment aligns incentives and facilitates the use of assessments for learning.

Thank you.