Of That

Brandt Redd on Education, Technology, Energy, and Trust

01 November 2019

Themes manifest as iNACOL Becomes Aurora

Arrows representing systems integration.

In 2010 I took on the responsibility of forming an Education Technology Strategy for the Bill & Melinda Gates Foundation. That same year, I also attended the iNACOL Virtual Schools Symposium (VSS). A year later, I presented at the symposium and I've been pleased to present or contribute in some way most years since.

As my colleagues and I at the Gates Foundation worked on a theory of technology and education education, something quickly became clear. Technology doesn't drive educational improvement; it's simply an enabler. In the early part of this decade there were numerous 1:1 student:computer initiatives. Most failed to show measurable improvement and many turned into fiascos as teachers were tasked with finding something useful to do with their new computers or tablets.

At the foundation we turned to personalized learning, a theory that was based on promising evidence and one that has gained more support since then. With that as basis we looked to where technology could help. The result was support for key projects including Common Education Data Standards, the Learning Resource Metadata Initiative, and Profiles of Next-Generation Learning.

The great folks at iNACOL observed the same patterns and so they pivoted. VSS became, simply, the iNACOL Symposium and their emphasis shifted to personalized and competency-based education with online and blended learning as enablers. This year, they completed the transition, renaming the whole organization to The Aurora Institute. In their words:

[Our] organization has evolved significantly to become a leading nonprofit organization with a deep reach into practitioners creating next-generation learning models. Our focus has grown to examine systems change and education innovation, facilitating the future of learning through personalized learning and student-centered approaches to next-generation learning.

Serving Educators and Students

A theme that spontaneously emerged at the symposium this year is that we must do for the educators what we want for the students. It was first expressed by Dr. Brooke Stafford-Brizard in her opening keynote. As she advocated that we care for the mental health of the children she said, "Across all of our partners who have successfully integrated whole child practice, there isn’t one who didn’t start with their adults." She proceeded to show examples where the a school mental health programs were designed to support both staff and students.

With that as precedent, the principle kept reappearing throughout the symposium.

  • If we expect personalized instruction for the students we must offer personalized professional development for their teachers.
  • Establish the competency set we expect of educators and provide opportunities to master those competencies.
  • Actionable feedback to educators is critical to the success of any learning innovation just as actionable feedback to students is critical to their learning.
  • Create an environment of trust and safety among the staff of your institution - then project that to the students.
  • Growth mindset is as important to educators as it is for the students they teach.

Continuous Improvement

Both themes — technology as enabler, and caring for the educators — are simply signposts on a path of continuous improvement. We must follow the evidence and go where it leads us.

07 March 2019

A Support System for High-Performing Schools

Arrows representing systems integration.

Charter schools operated by Charter Management Organizations (CMOs) tend to outperform other charter schools and public schools. The National Study of Charter Management Organization Effectiveness from 2011 was the first rigorous study of CMO effectiveness and it showed that CMO-operated schools were better than other options. A 2017 study by Stanford University's Center for Research on Education Outcomes found that students enrolled in CMO-operated schools in New York City substantially outperformed their peers in conventional public schools and independent charter schools.

This improvement is to be expected. A basic premise of CMO operations is to study what works, and carry successful practices to other schools in the network.

Some conventional public schools are following a similar pattern. Their solution providers don't necessarily manage the school, like a CMO would. Instead, providers offer an integrated set of services backed by an evidence-driven theory of effective teaching. Here is the ecosystem I expect to emerge in the next few years:

  • Component and Curriculum Suppliers
  • Educational Solution Providers
  • Schools (and other learning institutions)

This same basic model applies to primary, secondary, and higher education though large universities and big districts have the capacity to be their own solution providers. Let's look at the components:

Schools, Districts, and other Learning Institutions

The school is where the teaching and learning occurs. It's where the supply chain of standards, curriculum, educational training, assessments, learning science, and everything else finally meets the student.

Many schools are implementing the same kinds of programs as charters: online curricula, blended learning, teacher dashboards, etc. But the complexity of integration grows exponentially with the number of components to combine. Building an integrated whole is beyond the capacity of most schools and all but the largest districts. The same pattern exists in higher education. Large universities can deliver an integrated solution but community colleges have a harder time.

Component and Curriculum Suppliers

On the supply side, there's a rich, complex, and rapidly growing market of component and curriculum suppliers. They include conventional textbook publishers, online curriculum developers, assessment providers, Learning Management Systems (LMS), Student Information Systems (SIS), and more.

Beyond these well-defined categories there's a host of other components, each designed to address a particular need in the educational economy. For example, Learnosity builds tools for creating and embedding high-quality assessments. Gooru offers a learning map, helping students know where they are in their learning progression. EdConnective offers live, virtual coaching for teachers. In 2018, education technology investment grew to a record $5.23 billion in the U.S. and a breathtaking $16.34 billion worldwide. We can expect many more components and materials to be produced from that level of investment.

Many of these components are raw - requiring significant integration effort before they can become part of an integrated learning solution. Despite this, developers of these components attempt to sell them directly to schools, districts, and states.

Educational Solution Providers

Summit Public Schools is a CMO that consistently achieves high rankings. Summit Learning also offers their online curriculum to public schools. But, separating the curriculum from the balance of the solution hasn't been so successful. In November 2018, Brooklyn students held a walkout and parents created a website to protest "Mass Customized Learning." It's not that the materials were bad; they were well-proven in other contexts. But, separated from the balance of the Summit program the student experience suffered.

An important new category in the education supply chain are Educational Solution Providers. CMOs belong to this category but solution providers to conventional schools don't take over management like a CMO would. Rather, they provide an integrated set of services that includes training and coaching for staff and leadership.

The best solution providers start with an evidence-based learning theory. They then assemble a comprehensive solution based on the theory and selected from the rich menu provided by the component market. A complete solution includes:

  • Training and Coaching Services
  • Professional Development
  • Curriculum (conventional or online)
  • Assessment (ideally curriculum-embedded)
  • Secure Student Data Systems with Educator Dashboards
  • Effectiveness Measures
  • Continuous Improvement

An important job for solution providers is to integrate the components so that they work seamlessly together in support of their learning theory. Training and professional development should embody the same theory that is being expressed to the students. LMS, SIS, dashboards, and all other online systems should function together as one solution even if the provider is sourcing the components from an array of suppliers. In order to do this, the solution provider must have their own curriculum experts for the content side and a talented technology staff focused on systems integration.

Players in this nascent category include The Achievement Network, CLI Solutions Group, and The National Institute for Excellence in Teaching. I think we can expect new entrants in the next few years. Successful CMOs may also cross over to providing services to conventional public schools.


The educational component and curriculum market is rich and rapidly growing with record levels of investment. But, schools don't have the capacity to integrate these components effectively and they need a guiding theory to underpin the selection of components and how they are to be integrated. The emerging category of Educational Solution Provider fills an important role in the ecosystem.

Are you aware of other existing or emerging solution providers? Please let me know in the comments!

11 February 2019

Public-Private Partnership for Public Works

SR 99 tunnel cross section visualization

On February 28, 2001 I was at Microsoft Headquarters in Redmond Washington when the Nisqually Earthquake hit. I was using Microsoft's scalability lab to perform tests on Agilix software. I remember standing in the doorway and asking the someone down the hall, "Is this really an earthquake?" It obviously was, but not having experienced one before my mind was still disbelieving.

Nine years later we moved to Seattle where I developed an education technology strategy at the Bill & Melinda Gates Foundation. At the time, politicians were still trying to figure out what should replace the Alaskan Way Viaduct which had been damaged in the earthquake, and which engineers predicted could collapse should another earthquake occur.

Last week, the Washington SR 99 tunnel replaced the viaduct; 18 years after the earthquake threatened its predecessor. Ironically, the tunnel opening was accompanied by a snowstorm that paralyzed the Northwest making the tunnel one of the few clear roads in the area.

Funding of Public Works

Grand Central Terminal

A few years back I visited New York's Grand Central Terminal and wondered at the great investments made in public works in the early 20th century. The terminal building is beautiful, functional, and built to last. It's been going for more than a century and will probably continue for a century or two more. I wondered why it is so hard to find contemporary investments in public works of such grandeur. However, upon doing some research I found that Grand Central was funded entirely by private investors. Even today, the building is privately owned though the railroad it serves has now been merged into the MTA, a public benefit corporation.

When we visited Seoul, Korea in 2015 we spent five days getting around on the excellent Seoul Metropolitan Subway. It is fast, efficient, clean, and among the largest subway systems in the world with more than 200 miles of track. It features wireless internet throughout, most platforms are protected by automated doors greatly improving safety. Yet, the whole network has been built since 1971. The subway is built and operated by Seoul Metro, Korail, and Metro 9. Seoul Metro and Korail are Korean public corporations; these are corporations where the government owns a controlling interest. Metro 9 is a private venture.

This past December we visited Brisbane, Australia. Brisbane traffic has been mediated through the construction of several bypass tunnels including the Airport Link. The tunnels have been built in relatively short time through public-private partnerships.

As I researched these projects I saw a consistent pattern. The most successful public works projects seem to involve some form of cooperation between government and private enterprise. Funding is more easily obtained and project management is better when a private organization participates and stands to benefit from the long-term success of the project. But government support is also needed to represent the public interest, to streamline access to land and permits, and to ensure that profit-taking isn't excessive. Consider the U.S. Transcontinental Railroad. It was built in six years by three companies with a combination of government land grants, private funding, and some government subsidy bonds.

Less-Successful Examples

Less-successful operations seem to be entirely publicly sponsored and managed. Private companies contract to do the work but they aren't invested beyond project completion. For example, the Boston Big Dig was the "most expensive highway project in the US, and was plagued by cost overruns, delays, leaks, design flaws, charges of poor execution and use of substandard materials, criminal arrests, and one death." While the project was built by private contractors, public agencies were exclusively responsible for sponsorship, oversight, funding, and success.

Similarly, the Florida High Speed Corridor was commissioned by a state constitutional amendment, theoretically obligating the state to build the rail system. While still in the planning stages, the project got bogged down in cost overruns, environmental studies, lawsuits, and declining public support. Ultimately, the project was canceled in 2011. In 2018, however, Brightline, launched service between Miami, Fort Lauderdale, and West Palm Beach with an extension to Orlando being planned. Brightline is privately funded and operated.


The same principles seem to apply in education. In the U.S. the biggest challenge to traditional public education are charter schools. Studies, including this one from the Center on Reinventing Public Education show that charter schools managed by Charter Management Organizations (CMOs) perform better than conventional public schools or independently-managed charter schools. Most CMOs are not-for-profit but they still represent a private, non-government entity. Based on the success of CMOs, some school districts are also considering outside management or support firms. In higher education there is a long tradition of government funding for a mix of public and private universities. Like the successful public works, the greatest success seem to occur when public and private interests are combined and aligned toward a common goal. In these successes, government represents the public interest. The worst outcomes seem to occur when government fails to represent public interests and is either corrupted to serve private needs or excessively focused on politics and party issues.

Organizing for Success

I haven't done a comprehensive search of public works projects. My selection of examples is simply based on projects I happen to be aware of. Nevertheless, it seems that the greatest potential for success is achieved when public and private interests are aligned in a partnership that leverages the strengths of both models and ensures that both groups benefit. public-private partnerships, state-owned enterprises, and public benefit corporations are different ways of achieving these ends.

The SR 99 tunnel in Seattle was bored by Bertha which, at the time, was the largest-ever tunnel boring machine. Early in the process, the machine broke down and it took two years to dig a recovery pit and make repairs. At the time, two state senators sponsored a bill to cancel the project. Despite this setback, and significant cost overruns, the project was ultimately a success. So, we can add persistence to see things through as another key to success.

Though the contract with Seattle Tunnel Partners will conclude when the tunnel project is complete, the organization has achieved a high degree of cooperation with the Washington department of transportation. Public-private cooperation and alignment of interests are behind many of the most successful public projects. And the private interest is often the source of the persistence needed to see things through.

10 January 2019

Quality Assessment Part 9: Frontiers

This is part 9 of a 10-part series on building high-quality assessments.


A 2015 survey of US adults indicated that 34% of those surveyed felt that standardized tests were merely fair at measuring students' achievement; 46% think that the way schools use standardized tests has gotten worse; and only 20% are confident that tests have done more good than harm. The same year, the National Education Association surveyed 1500 members (teachers) and found that 70% do not feel that their state test is "developmentally appropriate.".

In the preceding eight parts of this series I described all of the effort that goes into building and deploying a high-quality assessment. Most of these principles are implemented to some degree in the states represented by these surveys. What these opinion polls tell us is that regardless of their quality, these assessments aren't giving valuable insight to two important constituencies: parents and teachers.

The NEA article describes a hypothetical "Most Useful Standardized Test" which, among other things, would "provide feedback to students that helps them learn, and assist educators in setting learning goals." This brings up a central issue in contemporary testing. The annual testing mandated by the Every Student Succeeds Act (ESSA), is focused on school accountability. This was also true of its predecessor, No Child Left Behind (NCLB). Both acts are based on the theory of measuring school performance, reporting that performance, and incentivising better school performance. States and testing consortia also strive to facilitate better performance by reporting individual results to teachers and parents. But facilitation remains a secondary goal of large-scale standardized testing.

The frontiers in assessment I discuss here shift the focus to directly supporting student learning with accountability being a secondary goal.

  • Curriculum-Embedded Assessment
  • Dynamically-Generated Assessments
  • Abundant Assessment

Curriculum-Embedded Assessment

The first model involves embedding assessment directly in the curriculum. Of course, nearly all curricula have embedded assessments of some sort. Math textbooks have daily exercises to apply the principles just taught. English and social studies texts include chapter-end quizzes and study questions. Online curricula intersperse the expository materials with questions, exercises, and quizzes. Some curricula even include pre-built exams. But these existing assessments lack the quality assurance and calibration of a high-quality assessment.

In a true Curriculum-Embedded Assessment, some of the items that appear in the exercises and quizzes would be developed with the same rigor as items on a high-stakes exam. They would be aligned to standards, field tested, and calibrated before appearing in the curriculum. In addition to contributing to the score on the exercise or quiz, the scores of these calibrated items would be aggregated into an overall record of the student's mastery of each skill in the standard.

Since the exercises and quizzes would not be administered in as controlled an environment as a high-stakes exam, the scores would not individually be as reliable as in a high-stakes environment. But by accumulating many more data points, and doing so continuously through the student's learning experience, it's possible to assemble an evaluation that is as reliable or more reliable than a year-end assessment.

Curriculum-Embedded Assessment has several advantages over either a conventional achievement test or the existing exercises and quizzes:

  • Student achievement relative to competency is continuously updated. This can offer much better guidance to students, educators, and parents than existing programs.
  • Student progress and growth can be continuously measured across weeks and months, not just years.
  • Performance relative to each competency can be reliably reported. This information can be used to support personalized learning.
  • Data from calibrated items can be correlated to data from the rest of the items on the exercise or quiz. Over time, these data can be used to calibrate and align the other items, thereby growing the pool of reliable and calibrated assessment items.
  • As Curriculum-Embedded Assessment is proven to offer data as reliable as year-end standardized tests, the standardized tests can be eliminated or reduced in frequency.

Dynamically-Generated Assessments

As described in my post on test blueprints, high-quality assessments begin with a bank of reviewed, field-tested, and calibrated items. Then, a test producer selects from that bank a set of items that match the blueprint of skills to be measured. For Computer-Adaptive Tests, the test is presented to a simulated set of students to determine how well it can measure student skill in the expected range.

In order to provide more frequent and fine-grained measures of student skills, educators prefer shorter interim tests to be used more frequently during the school year. Due to demand from districts and states, the Smarter Balanced Assessment Consortium will more than double the number of interim tests it offers over the next two years. Most of the new tests will be focused on just one or two targets (competencies) and have four to six questions. They will be short enough to be given in a few minutes at the beginning or end of a class period.

But what if you could generate custom tests on-demand to meet specific needs of a student or set of students? A teacher would design a simple blueprint — the skills to be measured and the degree of confidence required on each. Then the system could automatically generate the assessment, the scoring key, and the achievement levels based on the items in the bank and their associated calibration data.

Dynamically-generated assessments like these could target needs specific to a student, cluster of students, or class. With a sufficiently rich item bank, multiple assessments could be generated on the same blueprint thereby allowing multiple tries. And it should reduce the cost of producing all of those short, fine-grained assessments.

Abundant Assessment

Ideally, school should be a place where students are safe to make mistakes. We generally learn more from mistakes than from successes because failure affords us the opportunity to correct misconceptions and gain knowledge whereas success merely confirms existing understanding.

Unfortunately, school isn't like that. Whether primary, secondary, or college; school tends to punish failures. At the college level, a failed assignment is generally is unchangeable and a failed class, or low grade goes on the permanent record. Consider a student that studies hard all semester, gets reasonable grades on homework, but then blows the final exam. Perhaps they were sick on exam day, or perhaps the questions were confusing and different from what they expected, or perhaps the pressure of the exam just messed them up. Their only option is to repeat the whole class — and even then their permanent record will show the class repetition.

Why is this? Why do schools amplify the consequences to such small events? It's because assessments are expensive. They cost a lot to develop, to administer, and to score. In economic terms, assessments are scarce. For schools to offer easy recovery from failure they would have to develop multiple forms for every quiz and exam. They would have to incur the cost of scoring and reporting multiple times. And they would have to select the latest score and ignore all others. To date, such options have been cost-prohibitive.

"Abundant Assessment" is the prospect making assessment inexpensive — "abundant" in economic terms. In such a framework, students would be afforded many tries until they succeed or are satisfied with their performance. Negative consequences to failure would be eliminated and the opportunity to learn from failure would be amplified.

This could be achieved by a combination of collaboration and technology. Presently, most quizzes and exams are written by teachers or professors for their class only. If their efforts were pooled into a common item bank, then you could rapidly achieve a collection large enough to generate multiple exams on each topic area. Technological solutions would provide dynamically-generated assessments (as described in the previous section), online test administration, and automated scoring. All of this would dramatically reduce the labor involved in producing, administering, scoring, and reporting exams and quizzes.

Abundant assessment dramatically changes the cost structure of a school, college, or university. When it is no longer costly to administer assessments then you can encourage students to try early and repeat if they don't achieve the desired score. Each assessment, whether an exercise, quiz, or exam can be a learning experience with students encouraged to learn quickly from errors.


These three frontiers are synergistic. I can imagine a student, let's call her Jane, studying in a blended learning environment. Encountering a topic with which she is already familiar, Jane jumps ahead to the topic quiz. But the questions involve concepts she hasn't yet mastered and she fails. Nevertheless, this is a learning experience. Indeed, it could be reframed as a formative assessment as she now goes back and studies the material knowing what will be demanded of her in the assessment. After studying, and working a number of the exercises, Jane returns to the topic assessment and is presented with a new quiz, equally rigorous, on the same subject. This time she passes.

Outside the frame of Jane's daily work, the data from her assessments and those of her classmates are being accumulated. When the time comes, at the end of the year, to report on school performance, the staff are able to produce reliable evidence of student and school performance without the need for day-long standardized testing.

Most importantly, throughout this experience Jane feels confident and safe. At no point is she nervous that a mistake will have any long-term consequence. Rather, she knows that she can simply persist until she understands the subject matter.