26 July 2013

Can Consortia Improve Standardized Testing?

The NCLB embodiment of standardized testing has been in place for eleven years now. Unfortunately, it hasn't resulted in substantial improvement in student learning. But there are things the multi-state assessment consortia can due that will improve the situation.

To many of us, the lack of progress has been confusing. In just about every other industry, when you measure performance and report it back, performance improves. Education is proving to be a more difficult problem than most.

Unintended Consequences

Part of the issue is unintended consequences from standardized testing. Consider a teacher who is anticipating the year-end tests. She and her principal are under pressure to achieve Adequate Yearly Progress (AYP) goals. So, they have regular drill and review sessions -- test preparation to make sure the students are ready.

All of this "teaching to the test" takes time and resources away from more enriching and interesting learning experiences. And it doesn't work. The Measures of Effective Teaching project found that teaching to the test is not as effective as a focus on conceptual understanding and applications. (MET Preliminary Findings Page 21)

That shouldn't be surprising. I wrote about the Flow Channel or Zone of Proximal Development a few months ago. In order to keep the student's attention and prevent frustration, the work needs to be new and challenging but not excessively so. If the work is too easy, the student is bored. If too hard, the student is anxious. In either case the student is frustrated and learning  does not occur. Constant drilling in preparation for the test puts students deeply in the boredom zone.

There are lots of important skills that aren't on the exam. Yong Zhao has written that the most important 21st Century Skills are creativity, entrepreneurship and independent learning. These aren't emphasized in the standardized exams. But the basic skills of literacy and numeracy (which do appear) are required for the kind of creativity and entrepreneurship we need. For this reason, I wish that the Common Core State Standards were named the "Common Foundation Standards" because that's the way I see them -- the foundation skills required for creative work.

This is another unintended consequence of standardized testing. Excessive focus on the exam steals classroom time that could be used for creative application of the knowledge or for self-directed learning. The MET study and others show that more of the latter activities results both in better prepared students and concurrently better exam results.

How To Do Better

Noting the minimal progress, many advocates call for abandoning the standards and assessments altogether. They look at the amount of time and money dedicated to assessment and suggest these resources could be spent in better ways. But, in the absence of standards and measurement we wouldn't know if we are succeeding or failing. The best we could hope for is blissful ignorance. In my opinion we must push forward, improving standards, measurement and teaching.

While improvement will require changes throughout the educational system, there are things that can be done with the assessments themselves to support improvement. Here are some of the things being done by the Smarter Balanced Assessment Consortium and by PARCC:

Richer, More Authentic Assessment Items
Both consortia will use computer-delivered assessments that are much closer to real-world activities. An emphasis on constructed response items will require students to compose an answer to a problem, not just select from a set of prewritten answers.  Upon evaluating the consortia's assessments, the  UCLA CRESST center concluded, "Both PARCC and Smarter Balanced summative assessments ... will represent many goals for deeper learning, particularly those related to mastering and being able to apply core academic content and cognitive strategies related to complex thinking, communication, and problem solving."

Teaching to the test becomes less of a problem the closer the exam gets to assessing real and authentic skills.

Guidance from Interim Assessments
In addition to the year-end summative assessments, both consortia are offering voluntary interim assessments that teachers and administrators can use to gauge students' understanding throughout the school year. If students are found to be prepared, less time will be spend on boring and unnecessary drills. Likewise, identification of weak areas can guide teachers in reviewing just the necessary lessons.

Professional Development and Formative Activities
The consortia are developing training materials for teachers. These will include information on how to plan formative assessment activities for the classroom and how to interpret and make use of assessment results.

Precise, Individual Level Reporting
Existing state assessments measure the number of students who have achieved the state competency threshold for their particular grade. These measures are reported to schools, districts and states in hopes of improving education programs at those institutions. Threshold tests can measure whether a student is above or below the expected competency but the further a student is from that level, the less accurately they can indicate the student's actual competency.

Smarter Balanced will use computer adaptive testing to precisely measure each student's skill level. In adaptive tests, questions are selected based on the results of previous assessment items and testing ends once the student's skill level has been measured to a certain level of confidence. Thus, students that are below grade level aren't subjected to a long series of questions that they can't answer and both those above and below the threshold receive accurate measures of their skill levels.

These more precise assessments measures are used to generate clear, individual level reports for students, their parents and teachers. The reports will have sufficient detail to show growth year over year and to optimize instruction to address individual student needs.

~  ~  ~  ~  ~

Standards and associated assessments haven't resulted in the improvements that were hoped for. But that doesn't mean we should give up on them. They offer an important support for the Personalized Learning theory which has been proven. Refinements to standardized tests listed above will reduce unintended consequences and offer the guidance needed to optimize each student's learning experience.

06 July 2013

References Needed to Complete Engelbart's Vision

Douglas Engelbart, a pioneer in personal computing and one of my heroes, died this past Thursday. Many of the concepts he invented are part of our daily personal computing lives. But there are still a few missing pieces, one of which is a referencing system.

Here's a rough outline of how a series of pioneers developed the technologies you find familiar:
  • 1945: Vannevar Bush, Director of the Office of Scientific Research and Development, writes "As We May Think." Writing at the conclusion of World War II, Bush considers how technologies developed for war can be used to further peace. He envisions an electromechanical system based on microfilm and dry photography that can manage all of the data a person needs and help them organize it into knowledge.
  • 1968: Douglas Engelbart, Director of the Augmentation Research Center at SRI, is inspired by Bush's article. He realizes that the concept can be achieved much more readily using digital computers instead of an electromechanical system. In 1968 he demonstrates their oNLine System (NLS) in what we now call the "Mother of all Demos" including a mouse, graphical user interface, collaborative word processing, teleconferencing and a host of other features that would take decades to make it into the mainstream.
  • 1973: Alan Kay, who had attended Engelbart's demo, incorporates many of Engelbart's ideas into the Xerox Alto. Designing the Alto so that it can be used by children, Kay's insight is that the user interface should manifest the functions that are available. Thus, the system itself can teach the individual how to use it.
  • 1984: Steve Jobs, who had seen a demo of the Alto in 1979, incorporates key elements into the Apple Macintosh. Features inherited from NLS and the Alto include the mouse, GUI and computer networking. Jobs' most important contribution is to get these ideas out of the lab and offer them to a mass market.
If you watch Engelbart's Demo you will see many now-familiar ways of using a computer. NLS centered on the creation and management of documents. These documents were indexed for convenient retrieval and sharable with all other NLS users.

But there was a critical feature in NLS that we have not yet replicated. Each document was given a unique ID. Printed NLS documents were easily recognized because they included index numbers in the margins. The document ID and index number allowed individuals to reference any line in any NLS document.

That need for consistent identifiers that can be referenced has yet to be addressed for most texts. Today, the best that citation systems can do is refer to a page number. But page numbers change with different formats (e.g. hardbound vs. paperback) and editions. Suppose, for example, you want to cite a particular quote from Huckleberry Finn. In order to do so, you have to specify the publisher and edition of the book before citing the page number. And the odds of a reader having that same edition is pretty low. Textbook publishers have taken advantage of this. By changing pagination between editions, they deliberately obsolete previous editions.

With the advent of digital books the problem is compounded. Page numbers change according to user preferences like font size and page orientation. As a stopgap, the Amazon Kindle added "real page numbers" so that you can use references derived from the paper version of a book. But the problem of persistently valid references across editions lingers.

As we honor the legacy of Doug Engelbart, it's appropriate to consider one more of his innovations -- a persistent and universal referencing system. We still need it.