11 August 2018

Quality Assessment Part 2: Standards and Content Specifications

Mountain with Flag on Summit

In Part 1 of this series I introduced the factors that distinguish a high quality assessment from other assessments. The balance of this series will discuss the process of constructing an assessment and the factors that make them high quality. Today I'm writing about Achievement Standards and the Content Specification.

Some years ago my sister was in middle school and I had just finished my freshman year at college. My sister's English teacher kept assigning word search puzzles and she hated them. The family had just purchased an Apple II clone and so I wrote a program to solve word searches for my sister. I'm not sure what skills her teacher was trying to develop with the puzzles; but I developed programming skills and my sister learned to operate a computer. Both skill sets have served us later in life.

Alignment to Standards

The first step in building any assessment, from a quiz to a major exam, should be to determine what you are trying to measure. In the case of academic assessments, we measure skills, also known as competencies. State standards are descriptions of specific competencies that a student should have achieved by the end of the year. They are organized by subject and grade. State summative tests indicate student achievement by measuring how close each student is to the state standards — typically at the close of the school year. Interim tests can be used during the school year to measure progress and to offer more detailed focus on specific skill areas.

At the higher education level, Colleges and Universities set Learning Objectives for each course. A common practice is to use the term, "competencies" as a generic reference to state standards and college learning objectives and I'll follow that pattern here.

The Smarter Balanced Assessment Consortium, where I have been working, measures student performance relative to the Common Core State Standards. Choosing standards that have been adopted by multiple states enables us to write one assessment that meets the needs all of our member states and territories.

The Content Specification

The content specification is a restatement of competencies organized in a way that facilitates assessment. Related skills are clustered together so that performance measures on related tasks may be aggregated. For example, Smarter Balanced collects skill measures associated with "Reading Literary Texts" and "Reading Informational Texts" together into a general evaluation of "Reading". In contrast, a curriculum might cluster "Reading Literary Texts" with "Creative Writing" because synergies occur when you teach those skills together.

The Smarter Balanced content specification follows a hierarchy of Subject, Grade, Claim, and Target. In Mathematics, the four claims are:

  1. Concepts and Procedures
  2. Problem Solving
  3. Communicating Reasoning
  4. Modeling and Data Analysis

In English Language Arts, the four claims are:

  1. Reading
  2. Writing
  3. Speaking & Listening
  4. Research & Inquiry

These same four claims are repeated in each grade but the expected skill level increases. That increase in skill is represented by the targets assigned to the claims at each grade level. In English Reading (Claim 1), the complexity of the text presented to the student increases and the information the student is expected to draw from the text is increasingly demanding. Likewise, in Math Claim 1 (Concepts and Procedures) the targets progress from simple arithmetic in lower grades to Geometry and Trigonometry in High School.

Data Form

Typical practice is for states to publish their standards as documents. When published online they have been published as PDF files. Such documents are human readable but they lack the structure needed for data systems to facilitate access. In many cases they also lack identifiers that are required when referencing standards or content specifications.

Most departments within colleges and universities will develop a set of learning objectives for each course. Often times a state college system will develop statewide objectives. While these objectives are used internally for course design, there's little consistency in publishing the objectives. Some institutions publish all of their objectives while others keep them as internal documents. The Temple University College of Liberal Arts offers an example of publicly published learning objectives in HTML form.

In August 2017, IMS Global published the Competencies & Academic Standards Exchange (CASE) data standard. It is a vendor-independent format for publishing achievement standards suitable for course learning objectives, state standards, content specifications, and many other competency frameworks.

Public Consulting Group, in partnership with a several organizations built OpenSALT, an open source "Standards Alignment Tool" as a reference implementation of CASE.

Here's an example. Smarter Balanced originally published its content specifications in PDF form. The latest versions, from July of 2017, are available on the Development and Design page of their website. These documents have complete information but they do not offer any computer-readable structure.

"Boring" PDF form of Smarter Balanced Content Specifications:

In Spring 2018, Smarter Balanced published the same specifications, in CASE format, using the OpenSALT tool. The structure of the format lets you navigate the hierarchy of the specifications. The CASE format also supports cross-references between publications. In this case, Smarter Balanced also published a rendering of the Common Core State Standards in CASE format to facilitate references from the content specifications to the corresponding Common Core standards.

"Cool" CASE form of Smarter Balanced Content Specifications and CCSS:

I hope you agree that the Standards and Content Specifications are significantly easier to navigate in their structured form. Smarter Balanced is presently working on a "Content Specification Explorer" which will offer a friendlier user interface on the structured CASE data.


Regardless of how they are published, use of standards is greatly facilitated if an identifier is assigned to each competency. There are two general categories of identifiers: Opaque identifiers carry no meaning - they are just a number. Often they are "Univerally Unique IDs" (UUIDs) which are generated using an algorithm to assure that identifier is not used anywhere else in the world. Any meaning of the identifier is by virtue of the record to which it is assigned. "Human Readable" identifiers are constructed to have a meaningful structure to a human reader. There are good justifications each approach.

The Common Core State Standards assigned both types of identifier to each standard. Smarter Balanced has followed a similar practice in the identifiers for our Content Specification.

Common Core State Standards Example:

  • Opaque Identifier: DB7A9168437744809096645140085C00
  • Human Readable Identifier: CCSS.Math.Content.5.OA.A.1
  • URL: http://corestandards.org/Math/Content/5/OA/A/1/
  • Statement: Use parentheses, brackets, or braces in numerical expressions, and evaluate expressions with these symbols.

Smarter Balanced Content Specification Target Example:

You'll notice that the Smarter Balanced Content Specification target is a copy of the corresponding Common Core State Standard. The CASE representation includes an "Exact Match Of" cross-reference from the content specification to the corresponding standard to show that's the case.

Smarter Balanced has published a specification for its human-readable Content Specification Identifiers. Here's the interpretation of "M.G5.C1OA.TA.5.OA.A.1":

  • M Math
  • G5 Grade 5
  • C1 Claim 1
  • OA Domain OA (Operations & Algebraic Thinking)
  • TA Target A
  • 5.OA.A.1 CCSS Standard 5.OA.A.1

Quality Factors

The design of any educational activity should begin with a set of learning objectives. State Standards offer a template for curricula, lesson plans, assessments, supplemental materials, games and more. At the higher education level, Colleges and Universities set learning objectives for each course that serve a similar purpose. The quality of the achievement standards will have a fundamental impact on the quality of the related learning activities.

Factors to consider when selecting or building standards or learning objectives include the following:

  • Are the competencies relevant to the discipline being taught?
  • Are the competencies parallel in construction, describing skills at a similar grain size?
  • Are the skills ordered in a natural learning progression?
  • Are related skills, such as reading and writing, taught together in a coordinated fashion?
  • Is the amount of material covered by the competencies appropriate for the amount of time that will be allocated for learning?

The Development Process and the Standards-Setting Criteria used by the authors of the Common Core State Standards offer some insight into how they sought to develop high quality standards.

Factors to consider when developing an assessment content specification include the following:

  • Does the specification reference an existing standard or competency set?
  • Are the competencies described in such a way that they can be measured?
  • Is the grain size (the amount of knowledge involved) for each competency optimal for construction of test questions?
  • Are the competencies organized so that related skills are clustered together?
  • Does the content standard factor in dependencies between competencies? For example, performing long division is evidence that an individual is also competent at multiplication.
  • Is the organization of the competencies, typically into a hierarchy, consistent and easy to navigate?
  • Does the competency set lend itself to reporting skills at multiple levels? For example, Smarter Balanced reports an overall ELA score and then subscores for each claim: Reading, Writing, Speaking & Listening, and Research & Inquiry.


Compared with curricula, standards and content specifications are relatively short documents. The Common Core State Standards total 160 pages, much less than the textbook for a single grade. But standards have a disproportionate impact on all learning activities within the state, college, or class where they are used. Careful attention to the selection or construction of standards is a high-impact effort.

02 August 2018

Quality Assessment Part 1: Quality Factors


As I wrap up my service at the Smarter Balanced Assessment Consortium I am reflecting on what we've accomplished over the last 5+ years. We've assembled a full suite of assessments; we built an open source platform for assessment delivery; and multiple organizations have endorsed SmarterBalanced as more rigorous and better aligned to state standards than prior state assessments.

So, what are the characteristics of a high-quality assessment? How do you go about constructing such an assessment? And what distinguishes an assessment like Smarter Balanced from a typical quiz or exam that you might have in class?

That will be the subject of this series of posts. Starting from the achievement standards that guide construction of both curriculum and assessment I will walk through the process Smarter Balanced and other organizations use to create standardized assessments and then indicate the extra effort required to make them both standardized and high quality.

But, to start with, we must define what quality means — at least in the context of an assessment.

Goal of a Quality Assessment

Nearly a year ago the Smarter Balanced member states released test scores for 2017. In most states the results were flat — with little or no improvement from 2016. It was a bit disappointing but what surprised me at the time was the criticism directed at the test. "The test must be flawed," certain critics said, "because it didn't show improvement."

This seemed like a strange criticism to direct at the measurement instrument. If you stick your hand in an oven and it doesn't feel warm do you wonder why your hand is numb or do you check the oven to see if it is working? Both are possibilities but I expect you would check the oven first.

The more I thought about it, however, the more I realized that the critics have a point. Our purpose in deploying assessments is to improve student learning, not just to passively measure learning. The assessment is a critical part of the eduational feedback loop.

Smarter Balanced commissioned an independent study and confirmed that the testing instrument is working properly. Nevertheless, there are more things that the assessment system can do support better learning.

Features of a Quality Assessment

So, we define a quality assessment as one that consistently contributes to better student learning. What are the features of an assessment that does this?

  • Valid: The test must measure the skills it is intended to measure. That requires us to start with a taxonomy of skills — typically called achievement standards or state standards. The quality of the standards also matter, of course, but that's the subject of a different blog post.. A valid test should be relatively insensitive to skills or characteristics it is not intended to measure. For example, it should be free of ethnic or cultural bias.
  • Reliable: The test should consistently return the same results for students of the same skill level. Since repeated tests may not be composed of the same questions, the measures must be calibrated to ensure they return consistent results. And the test must accurately measure growth of a student when multiple tests are given over an interval of time.
  • Timely: Assessment results must be provided in time to guide future learning activities. Summative assessments, big tests near the end of the school year, are useful but they must be augmented with interim assessments and formative activities that happen at strategic times during the school year.
  • Informative: If an assessment is to support improved learning, the information it offers must be useful for guiding the next steps in a student's learning journey.
  • Rewarding: Test anxiety has been the downfall of many well-intentioned assessment programs. Not only does anxiety interfere with the reliability of results but inappropriate consequences to teachers can encourage poor instructional practice. By its nature, the testing process is demanding of students. Upon completion, their effort should be rewarded with a feeling that they've achieved something important.

Watch This Space

In the coming weeks, I will describe the processes that go into constructing quality assessments. Because I'm a technology person, I'll include discussions of how data and technology standards support the work.

09 June 2018

A Brief History of Copyright

In the early 2000s I began writing a book titled Frictionless Media. The subject was business models for digital and online media. My thesis was that digital media is naturally frictionless — naturally easy to copy and transmit. Prior media formats had natural friction, they required specialized equipment and significant expense to copy. Traditional media business models are based on that natural friction. In order to preserve business models, publishers have attempted to introduce artificial friction through mechanisms like Digital Rights Management. They would be better off adapting their business models to leverage that frictionlessness to their advantage. My ideas were inspired by experience at Folio Corporation where we had invented a sophisticated Digital Rights Management system for textual publications. We found that the fewer restrictions publishers imposed on their publications the more successful they were.

I didn’t finish the manuscript before the industry caught up with me. Before long, most of my arguments were being made by dozens of pundits. Nevertheless, the second chapter, "A Brief History of Copyright," remains as relevant as ever. In 2018 I updated it to include recent developments such as Creative Commons.

31 July 2017

Why Assessment?

Folio Corporation LogoMost departments or ministries of education state the purposes of assessment. I'm particularly fond of New Zealand's statement:

The primary purpose of assessment is to improve students’ learning and teachers’ teaching as both respond to the information it provides. Assessment for learning is an ongoing process that arises out of the interaction between teaching and learning.

I like this because it captures the feedback process and acknowledges that both students and educators should respond to that feedback. It also encompasses the various goals of assessment — to inform individual student learning, to measure the effectiveness of the learning system, and to serve as evidence of student skills.

Today I'm writing about the purposes of assessment and the value of standardized assessment.

Inform Individual Student Learning

The most important use of assessment is to improve individual learning. When used properly, assessments improve individual learning in three ways. 1) Exercising and demonstrating skills reinforces student understanding and helps retention. 2) Student attention to the assessment results can increase motivation and direct their choice of learning activities. 3) Educator attention to assessment results can direct their assignment of learning activities or inform interventions.

All of these involve educational feedback loops. However these impacts are only achieved if the right assessment is used for the right purpose. For example, many high-stakes assessments were developed primarily to comply with regulations, such as The No Child Left Behind Act (replaced by ESSA). The reports required by these regulations focus on the percentage of students at each institution that achieve standards for grade-level competency. A test focused on that type of report centers on the threshold of competency. It can indicate with great reliability whether a student is above or below the threshold but may not be reliable for other insight. Such a threshold test is a poor choice for informing learning activities, diagnosing areas of weakness, or measuring growth.

More advanced tests include questions at a variety of skill levels centered on the expected competency level. These tests indicate the student's competency on a continuous scale. Accordingly, they can indicate how far ahead or behind the student is, not just whether they are above or below a certain threshold. By comparing scores from successive tests, you can measure student growth over a period of time. Advanced tests include questions designed to get measure greater depths of knowledge. Such tests offer more reliable detail about student skills in specific areas.

One objection to advanced tests is that it takes more questions and more time to measure skills at this level of detail. The use of computer adaptive testing can shorten the test while maintaining reliability.

Measure the Effectiveness of the Learning System

When standardized assessments are used to measure the effectiveness of the learning system, individual student results are aggregated to indicate the fraction that are achieving competency levels. Typically results are compared with previous years to see if schools are improving. This Delaware Press Release is a typical example of the public statements made each year.

Assessments like these are based on the premise that if you measure performance and report on it, then performance will improve. Unfortunately, education has proven to be a stubborn counter example of this premise. Sixteen years after the No Child Left Behind act mandated standardized testing and established remedies for underperforming schools there has been limited progress. This leads some to call for abandonment of standardized assessment altogether. But if we don't measure performance, we will never know if we are succeeding.

These are our contemporary challenges: discovering the factors under that contribute to better learning and investing the resources needed to improve those factors. Continuing to measure performance will support gathering evidence of principles of effective teaching and learning.

Less frequently applied but equal in importance is using assessments to evaluate parts of the learning system. For example, assessment data can be used to compare different curricula or textbooks, for continuous improvement of online learning systems, and to evaluate the effectiveness of professional development programs.

Provide Evidence of Student Achievement

Since 1905, the primary measure of student achievement in the U.S. has been the Carnegie Unit. This measure uses the time a student spends in the classroom as proxy for how much they have learned. A century later, in 2005, New Hampshire began converting to a competency-based system where student skill is measured directly rather than by proxy. Other states have programs that allow competency measures as an alternative to seat time. Such measures depend on high quality assessments that are aligned to specific and relevant standards of achievement.

Standardized and High Quality Assessment

It has become common, in recent years, to complain about achievement standards and the associated standardized assessments. A typical protest might be, "My child is not standardized." To be sure, our goal should not be to achieve sameness among children and this is not the purpose of achievement standards. Rather, we recognize that people need to achieve a basic competency level in language arts and mathematics in order to function and achieve in our society. The standards are intended to reflect that basic competency with the hope that students and educators will build a wide variety of skill and achievement on that core foundation.

All of these uses — informing learning activities, measuring program effectiveness, and providing evidence of achievement — depend on the quality of the assessment. An assessment will provide poor guidance if it is sensitive to the wrong factors, is unreliable, or is tuned to the wrong skill level. I've written before that personalized learning is currently the most promising approach to improving learning. Choosing high quality assessments to inform personalization is essential to the success of such programs.

That should be our demand — that states, districts, and schools give us evidence of the quality of the assessments they use.

30 June 2017

Reducing Income Inequality when Productivity Exceeds Consumption

Prototype apple harvester by Abundant Robotics.Among the last bastions of labor demand is Agricultural Harvesting. Every fall, groups of migrant workers follow the maturation of fruit and vegetables across the different climates. But even those jobs are going away. In one case, Abundant Robotics is developing robots that use image recognition to locate ripe apples and delicate manipulators to harvest fruit without bruising it.

In my last post I described how creatively destructive innovations like these have increased productivity in the United States until it exceeds basic needs by nearly four times. If it weren't for distribution problems, we could eliminate poverty altogether.

The problem is that when production exceeds consumption, prices fall and the economy goes into recession. As I wrote previously the U.S. and most of the word economy have relied on a combination of advertising, increased standards of living, and fiscal policy to stimulate demand sufficiently to keep up with productivity. But these stimulus methods have a side effect of increasing wage disparity.

The impact is mostly on the unskilled labor force as these jobs are the easiest to automate or export. Even though the income pie keeps growing, the slice owned by the lowest quintile shrinks. Free trade exacerbates the spread between skilled and unskilled labor as do attempts to stimulate consumption through government spending, low interest rates, and tax cuts. (Please see that previous post for details on all of this.)

Conventional Solutions and Their Limits

This is a gnarly problem. Potential remedies tend to have side effects that can make the situation worse, or at least limit the positive impact. In this post I'll name some of the conventional solutions and then attempt to frame up the kind of innovation that will be required to properly solve the problem.

Infrastructure Investment

Investments in infrastructure are a good short-term stimulus. Constructing or upgrading roads, bridges, energy, and communications infrastructure employs a lot of people of all skill levels. As most infrastructure is government-funded, it's a convenient fiscal stimulus. Trouble is that in the long run these infrastructure improvements result in greater overall productivity thereby reducing the labor demand.

Progressive Taxes

A common method of wealth redistribution is a progressive tax. Presumably the wealthy can afford to pay a larger fraction of their income than those with lower incomes. Progressive taxes are effective tools but, in the U.S., we have pretty much reached the limit of how much a progressive system can help low income households. Those earning less than the income tax threshold already pay no taxes. For a single parent with one child, the marginal tax threshold for 2016 was approximately $21,000 before accounting for federal and state health care benefits which amount to approximately $9,400 more.

Since the lowest tax bracket is already zero we can consider increasing the tax rate in upper tax brackets. While that may increase the amount paid by the wealthy it doesn't directly benefit the poor. Indeed, the resulting economic slowdown may worsen the situation.

Refundable Tax Credits

Through tax credits you can reduce the lowest effective income tax rate to negative values — paying money rather than collecting it. In the U.S. the Earned Income Tax Credit (ETC) serves this role with progressive benefits for the number of children in the household. Because the ETC is designed as a percentage of income, the benefit increases as the individual earns more money before being phased out at higher income levels. This is intended to incentivize workers to find and improve their employment even while drawing government benefits.

The downside of tax credits is that they are tied in with the very complicated process of filing income tax returns. The Government Accounting Office and IRS indicate that between 15% and 25% of eligible households don't collect credits to which they are entitled. Nevertheless, this is a promising approach because tax credits are unrelated to productivity and they have a direct impact on income inequality.

Increased Minimum Wage

Recently there have been increased calls for a $15 minimum wage. Many cities and some states have already passed laws to increase wages to that level. Critics of the minimum wage point out that increased wages can lead to increased prices — thereby reducing the benefit to low-income workers. And indexing minimum wage to the cost of living, as certain states and municipalities have done, may eliminate entry level jobs and accelerate inflation.

Many are watching to see how experiments in Seattle, New York City, and San Francisco will work out. Two recent studies of Seattle's minimum wage offer early indicators. One, from the University of Washington indicates an increase of 3% in total wages in low-paying jobs (the desired outcome) but a reduction, by 9%, of total hours worked (not so desirable). A study by UC Berkeley confirms the increase in total wages and indicates no reduction in overall employment though it does show a decline in employment of workers by limited-service restaurants (fast food). Hours worked was not among the data in the Berkeley study.

These are preliminary results but they tend to confirm the expected pattern. An increased minimum wage incentivizes greater automation and, thereby reduces the total labor demand. For example, fast food restaurants are deploying self-serve kiosks in place of human cashiers. So, while the employed benefit from higher wages, many jobs may be eliminated.

Idling Portions of the Workforce

A recession naturally reduces the number of workers until production is a closer match to demand. Unfortunately, those who lose their jobs are predominantly low-income workers, people who are least able to tolerate job loss. Better options preserve household income while reducing hours worked. These include mandatory vacation time, a shortened work week, and more holidays. When couples elect to have only one spouse be employed that also reduces the workforce.


Bar graph showing decline in jobs requiring high school or less and increas in jobs requiring a postsecondary degree Increasing educational achievement is the intervention most dear to my heart. Not only does a better education enable better wages, but it also results in better health, greater happiness, more active citizenship, and reduced violence. Those with higher educational attainment make better financial decisions. All of this results in a better quality of life.

As more and more routine jobs are automated, the opportunities for those without a postsecondary degree or certificate will continue to diminish. A recent study at Georgetown University indicates that, by 2020, 65% of jobs in the U.S. will require postsecondary education. That's up from only 28% in 1973.

Certainly greater educational achievement is a part of the solution as it moves more people into higher-wage jobs. But those jobs pay higher wages precisely because they are more productive. So, as more people achieve higher levels of education, productivity will continue to increase.

Working With Market Forces

Nobel Laureate Milton Friedman observed that the communication function of markets is as important as the commerce function. When demand rises then prices rise. Rising prices signal suppliers to produce more goods. On the other hand, excess goods result in falling prices signaling manufacturers to reduce production. No command economy has achieved the signaling efficiency of the market.

The same signaling occurs in labor markets. Higher wages for more-skilled jobs will encourage people to seek the necessary education and training to qualify for those jobs. But the counter-point is our contemporary concern, when there is excess labor available, especially for low-skill jobs, then wages may fall below the point where workers can earn an adequate living. Some workers will retrain, and many programs will help pay for that. But retraining takes time, interest, and an affinity for the new field.

With worker productivity in the U.S. soon to crest 4 times basic needs the natural market signal is for less production – but that would mean unemployment. To date, we have interfered with that signal by artificially propping up demand. Massive advertising, high consumer debt, low interest rates, the housing bubble, planned obsolescence; all are symptoms of the interference. Besides, the overconsumption caused by this stimulus is also damaging to the environment.

As productivity continues to increase we will have to allow the signal to get through – it will inevitably anyway. The challenge is finding ways to match production to demand while sustaining employment and wages especially for the most vulnerable.

Framing the Problem

The innovation we need is this:

A way to distribute abundant resources more equitably; while preserving incentives to learn, work, and make a difference; and allowing market signals to balance production and consumption.

We can't look to the past because the challenge of abundance is different from any faced by previous generations. It's going to require a generous sharing of ideas, some experimentation, and development of greater trust between parties.

I'm optimistic that there is a solution because never before in history has society had such plentiful resources as today.

31 March 2017

Cut Taxes or Increase Spending - Is the Debate Obsolete?

Photo of the U.S. Capital with full moon overhead.

As the Trump administration turns its attention to a tax reform plan debate swirls about the best way to stimulate the economy. Traditionally, Democrats have advocated for increased government spending while Republicans have fought for reduced taxes. Both methods succeed in stimulating and both have their roots in the theories of prominent economists. But it may be that both strategies, and the theories that support them, are obsolete in a day when production is many times basic needs.

Government spending advocates cite the work of John Maynard Keynes. Prior to Keynes, neoclassical economists theorized that free markets should naturally balance the economy toward full employment. Keynes observed that economies tended to swing between boom and bust cycles and advocated government intervention through fiscal policy (government taxing and spending) and monetary policy (central bank regulation of the money supply) to moderate the swings. Keynesian theory was influential in addressing the Great Depression and remained dominant following World War II into the 1970s.

Among the expectations of early Keynesian economics was that high inflation and high unemployment should not co-exist. Economist Milton Friedman challenged that notion and was proven right when "stagflation" emerged in the 1970s. Friedman theorized that stagflation and related poor economic conditions result from excessive or malinformed government intervention. The solution, he said, was to free the market through reduced regulation and lower taxes. This school of thought is generally known as "supply-side" or "monetarist". President Reagan successfully employed that approach early in his presidency launching a sustained period of economic growth that continued through the Bush and Clinton administrations.

Today, Keynesian economics is associated with greater regulation, increased government spending, and with an overall trust in government interventions. Meanwhile, monetarist economics is associated with free markets, reduced taxes, and with an overall trust in the market's ability to self-balance. In fact, both schools of thought are much more nuanced than these broad strokes. On the Keynesian side, it matters a lot where the government spends its money. On the monetarist side, it matters a great deal which taxes are reduced and how regulations are tuned. Earnest theorists on both sides have a healthy respect for the other theory.

But the nuance is quickly lost in the morass of political debate. Indeed, I fear that most political Keynesians choose that theory because it justifies their existing desire to increase government spending. And most monetarists choose supply-side theory for it's justification of reduced taxes and regulation. In each case I think they first choose their preferred intervention and then select a theory to justify it.

Through the latter half of the 20th century U.S. government economic focus was pretty much what Keynes described - moderating the boom and bust cycle toward more stable continuous growth. During slow cycles this meant adding economic stimulus, through increased spending and reduced taxes. When inflation started to get out of hand, government would slow things by increasing taxes, reducing spending, and reduced interest rates. Reagan met the stagflation challenge (high inflation and high unemployment) with an unusual combination of reduced taxes (to stimulate hiring) and increased interest rates (to slow inflation). Nevertheless, Reagonomics still used the same tools, just in different ways.

Our contemporary challenge is a new one. Since roughly 2001 the economy has required continuous stimulation to maintain growth. Radical new stimuli such as Quantitative Easing and zero interest rates have been used. Previously experts avoided those stimuli because of their potential to provoke high inflation yet inflation remains at historically low levels and it seems that, without continuous stimulation, the economy will slow to a crawl.

Production compared to Basic Needs

Output per hour of all persons 1947 to 2010

The new economic challenge is due principally to the rapid increase in workforce productivity. According to the U.S. Bureau of Labor Statistics individual worker productivity has more than quadrupled since World War II. Overall productivity per person in 2012 was 412% that of 1947.

Productivity growth becomes even more striking when compared with basic needs. In 2014 U.S. per capita GDP was $54,539. The basic needs per capita that same year was approximately $13,908. So, per-capita production exceeds basic needs by nearly four times. And while the basic needs side of this equation includes the whole population, the productivity side only accounts for those employed, it doesn't include unemployed workers and people choosing not to seek paid employment. So productive capacity compared with basic needs would be even higher.

If it weren't for problems of distribution this would be a great thing! For the first time in history, society has sufficient capacity to provide comfortable housing, plenty of food, health care, entertainment, and leisure time for all. The challenge is that, in a market economy, productivity increases disproportionally benefit those who are already at the higher end of the wage scale.

Disproportionate Benefits

Creative destruction is the term economists us to describe the transformation of an industry by innovation. It is usually associated with the elimination of jobs due to new technology but any innovation that increases individual productivity qualifies. Some examples: The backhoe replaces the jobs of several ditch diggers with that of a more-skilled heavy equipment operator. Computerized catalogs reduce the demand on librarians. Industrial robots replace factory workers. The common feature of such innovations is that they substantially increase the productivity of individual workers. Frequently, these innovations also result in jobs moving upscale — requiring more skill or training and with correspondingly higher pay.

Creatively destructive innovations have led to enormous productivity increases in recent decades thereby reducing the demand for labor. As with any market, when supply increases or demand declines the value also declines. In this case the value of routine jobs has declined dramatically. Here's how economist Dr. David Autor describes it.

And so the things that are most susceptible to computerization or to automation with computers are things where we have explicit procedures for accomplishing them. Right? They’re what my colleagues and I often call “routine tasks.” I don't mean routine in the sense of mundane. I mean routine in the sense of being codifiable. And so the things that were first automated with computers were military applications like encryption. And then banking and census-taking and insurance, and then things like word processing and office clerical operations. But what you didn’t see computers doing a lot of — and still don't, in fact — are tasks that demand flexibility and don't follow well-understood procedures. I don’t know how to tell someone how do you write a persuasive essay, or come up with a great new hypothesis, or develop an exciting product that no one has seen before. ... What we’ve been very good at doing with computers is substituting them for routine, codifiable tasks. The tasks done by workers on production lines, the tasks done by clerical workers, the tasks done by librarians, the tasks done by kind of para-professionals, like legal assistants who go into the stacks for you. And so we see a big decline in clerical workers. We see a decline in production workers. We see a decline even in lower-level management positions because they’re all kind of information processing tasks that have been codified.

Recent creative destruction has predominantly affected lower-middle-class jobs and manufacturing jobs. While increased productivity has made our nation more wealthy as a whole, large sectors of the labor force have been left behind. This may be the biggest factor behind the slow recovery from the 2008 recession. Automation substituted for jobs that were eliminated during the recession; those jobs are not coming back.

The decline in U.S. manufacturing employment has been balanced, in part, by growth in the service sector. This makes sense; growth in productivity has resulted in greater overall wealth. On average, people in the U.S. have more money to spend on eating out, recreation, vacations, and health care. But again, the benefits are not evenly distributed. As workers displaced from manufacturing have moved into the service sector, wages in that area have stagnated.

Disproportionate Impact of Globalization

Economists have consistently advocated for free trade. The math is incontrovertible; when regions or countries with different costs of production trade goods and services, all communities benefit as each is able to specialize and all benefit from the overall productivity increase.

Only recently have economists begun to study how free trade impacts sectors of the economy rather than the economy as a whole. Unsurprisingly, the impact to the U.S. has disproportionally affected manufacturing and routine labor. Here's another quote from Dr. Autor:

When we import those labor-intensive goods, we’re going to reduce demand for blue-collar workers, who are not doing skill-intensive production.  Now we benefit because we get lower prices on the goods we consume and we sell the things that we're good at making at a higher price to the world. So that raises GDP but simultaneously it tends to make high-skilled and highly educated labor better off, raise their wages, and it tends to make low-skilled manually intensive laborers worse off because there is less demand for their services — so there's going to be fewer of them employed or they're going to be employed at lower wages. So the net effect you can show analytically is going to be positive. But the redistributional consequences are, many of us would view that as adverse because we would rather redistribute from rich to poor than poor to rich. And trade is kind of working in the redistributing from poor to rich direction in the United States. The scale of benefits and harms are rather incommensurate. ...

We would conservatively estimate that more than a million manufacturing jobs in the U.S. were directly eliminated between 2000 and 2007 as a result of China's accelerating trade penetration in the United States. Now that doesn't mean a million jobs total. Maybe some of those workers moved into other sectors. But we've looked at that and as best we can find in that period, you do not see that kind of reallocation. So we estimate that as much as 40 percent of the drop in U.S. manufacturing between 2000 and 2007 is attributable to the trade shock that occurred in that period, which is really following China's ascension to the WTO in 2001.

Manufacturing Output Versus Employment

During the campaign, Donald Trump and Bernie Sanders both advocated rethinking free trade. Perhaps we can use tariffs or government incentives to return manufacturing back to the U.S. As it turns out, that's already happening even without incentives. As labor costs increase in Asia the offshoring advantage is diluted. Many manufacturers are, indeed, opening new U.S. plants. The trouble is that returning manufacturing doesn't result in substantial job or wage growth. These are highly automated plants, employing a fraction of the workers whose jobs were eliminated when manufacturing went overseas. For example, Standard Textile just opened a new plant in Union, SC to make towels for Marriott International. Due to automation, the plant only created 150 new jobs. A generation ago the same plant would have employed more than 1000 people. And many of the new jobs are more highly skilled — designing, operating, and maintaining automated machinery.

Creative destruction and globalization are working together here. Both increase overall GDP, both increase individual worker productivity, both increase total wealth, and both disproportionately benefit skilled upper-middle-class workers over blue collar and middle-management workers. Any benefit from manufacturing returning to the U.S. will be blunted by the increase in automation reducing labor needs and shifting what remains to more skilled jobs.

Demand-Side Economics

So far, we have looked at the supply side of labor. The massive increase in productivity over the last six decades has been driven by innovative technology with global trade as an accelerant. As noted before, when the supply of labor exceeds demand then the value decreases. When supply exceeds demand across the economy as a whole then you get a recession.

From the end of World War II through the rest of the 20th century we succeeded in driving demand to keep up with supply. Advertising grew tremendously as an important demand driver. Television programs established new norms: two cars per family, a large home in the suburbs, annual luxury vacations, and designer clothing labels to name a few. Home appliances like air conditioning and a dishwasher went from luxury to necessity.

Government has participated in driving demand. Housing programs made home ownership much more accessible. So much so that it resulted in the 2007 real estate bubble. Likewise, the Federal Reserve has kept interest rates down ensuring that consumer credit remains accessible and people can buy ahead of income.

In the 21st Century we seem to have reached the limits of demand stimuli to compensate for ever increasing productivity. Smaller cars like the Mini Cooper or Fiat 500 have become stylish. Even the wealthy are choosing to reduce consumption — buying smaller homes or moving into the city. The result is that it takes increasingly strong stimuli to keep the economy moving. For the recession of 2008 the government spent unprecedented amounts of money borrowing directly from the Federal Reserve to do so. Despite this pressure, interest and inflation rates remain at historically low values.

Increase Spending or Cut Taxes?

And so we return to the contemporary debate: Should government increase spending or cut taxes to stimulate the economy? When government cuts taxes, individuals and companies have more disposable income. Presumably they will spend some of that income and save part. When government increases spending, it chooses directly where that money will be spent. Both theories depend on "trickle-down" effects even though that has traditionally been associated with tax cuts. In each case, the direct beneficiary of government policy employs more people and purchases more goods and services; those employees and suppliers also do more business and the impact "trickles" through the economy. The primary differentiator is whether you have greater trust in government (increase spending) or the market (cut taxes) to determine who is at the top of the trickle-down pyramid.

The question is really obsolete. Regardless of which stimulus you choose, demand stimuli are increasingly unable to keep up with increased productive capacity. As a country, we already produce nearly four times basic needs and the multiplier will continue to grow. Meanwhile, the twin pressures of Creative Destruction and Globalization will continue to drive the greater benefit of demand stimulus to those who already earn higher wages. Under either strategy, wage disparity will continue to worsen despite attempts by policymakers to direct tax breaks or government spending toward lower income households.

It seems that we will need a greater economic innovation than either of these 20th century solutions. In my next blog post I will write about some promising ideas. More effective education for all students is, of course, an essential component but insufficient by itself.

Estimating Basic Needs Per Capita: The Self-Sufficiency Standard is an measure of the income necessary to meet basic needs without assistance. Values are in terms of household. National averages aren't published so we have to make an approximation starting with samples of two cities. The cost of living index for Milwaukee, WI is 101.9% of the national average. Rochester, NY is exactly 100.0%. The 2014 Average household size in the U.S. was 2.54. We round up to 3 - two adults and one child. For Milwaukee the 2016 Self-Sufficiency Standard for that household is $43,112 annually. For Rochester, the 2010 Self-Sufficiency Standard for the same family is $40,334. Per-capita values are $14,371 and $13,445 respectively. Averaging the values comes to $13,908 approximate U.S. basic needs per capita in 2014. To be sure, there's a lot of variability across region, household size, medical needs, and so forth. I also mixed figures across 2010-2016. Nevertheless, this is a good enough working figure for comparing to per capita production in the same timeframe.

30 December 2016

The Challenge of Information Democracy

Folio Corporation LogoIn the 1990s I was a co-founder at Folio Corporation, an electronic publishing software company. Correlating with the growth of the internet we produced tools that let average individuals publish their content and search for the information they needed in vast pools. Such tools are common today but it was cutting edge at the time.

"Information Democracy" was the term we used to describe the concept. In previous generations, a select few were able to publish their words to a sizable audience. Likewise, only business leaders and rulers of countries could afford the research staff necessary to stay well-informed. We produced a video featuring James Earl Jones and held conferences anticipating the how greater access to media would spread liberty, increase productivity, and support a more moral society.

We weren't alone in our optimism. In Life After Television George Gilder wrote, "Television is not vulgar because people are vulgar; it is vulgar because people are similar in their prurient interests and sharply differentiated in their civilized concerns." Ever the optimist, Gilder anticipated that greater diversity of media channels would result in a gradual elevation of quality and subject matter.

We have, indeed, achieved a world where any organization can publish to the whole world, where individual citizens can create TV channels on YouTube, and where average researchers have better resources than national leaders had a generation ago. Unfortunately, unfettered access to media hasn't resulted in the utopia many of us expected. Today's challenge is distinguishing reliable information from deliberate deceit and the whole spectrum between.

Unreliable Information

The recent presidential election brought the issue of fake news to the the media's attention. It will probably take years to sort the origin and impact. One source seems to be entrepreneurial Macedonian teenagers making money with fake news sites and Google AdSense. The Washington Post claims it was a coordinated Russian effort to destabilize American democracy.

The Pizzagate episode offers a warning sign of how fake information can provoke violent response. In terms of death toll, Andrew Wakefield's fradulent MMR Vaccine paper was worse. Despite millions of dollars invested in follow-on studies and publicity campaigns, the anti-vaccine movement has contributed to thousands of illnesses and numerous childhood deaths.

In recent decades, most newspapers and magazines have reduced or eliminated their fact-checking departments. Fact-checking of this sort is a cost-center and with declining revenues due to internet media, publishers have sought to reduce costs. The decline in ante hoc (before publication) fact checking has been matched by a growth in post hoc efforts like FactCheck, and PolitiFact as well as fact-checking pages at major news publications. Post hoc fact checking builds a revenue center out of the effort by sensationalizing politician's and other publication's mistakes. The unfortunate result is that post hoc fact checking is selective, biased, and missed or ignored by those who prefer to believe an inaccurate story.


The Oxford English Dictionary (OED) named "Post-Truth" as it's word of the year for 2016. Their definition is "relating to or denoting circumstances in which objective facts are less influential in shaping public opinion than appeals to emotion and personal belief." The OED editors acknowledge that "post-truth" conditions aren't new; only that use of the term increased rapidly in 2016 year in the context of the U.S. Presidential Election and the U.K. Brexit vote.

I prefer the term "Confirmation Bias", defined as "The tendency to interpret new evidence as confirmation of one's existing beliefs or theories. In extreme cases, conspiracy theorists tend to reject any evidence contrary to their opinion as part of the conspiracy while considering all evidence in support of their opinion as true and factual.

Information Democracy

2 by 2 matrix. Horizontal dimension is Access to Publish Media with left being Exclusive and right being Open. Vertical dimension is Trust in Media with top being High and bottom being Low. Upper-left quadrant labeled Information Hegemony. Upper-right, Information Democracy. Lower-Left, Propaganda. Lower-Right, Information Anarchy.

It's time to resort to that old standby - the 2x2 matrix. In the latter half of the 20th century, prior to the advent of the internet and world-wide web, U.S. society was in a state of high media trust but all publishing flowed through a relatively small set of media outlets. Opinion polls identified Walter Cronkite as "the most trusted man in America." This state of high trust and exclusive access is the upper-left quadrant, "Information Hegemony."

We optimists of the 1990s expected free access to media to provoke a shift to Information Democracy. Likewise, we anticipated that totalitarian states would be forced from state-controlled media to an information democracy model.

We didn't understand the importance of trust systems in that shift. As access was generalized, and economic forces pushed media to a more sensationalist orientation, trust declined and we have ended up with Information Anarchy. It's hard to say whether this is superior to the more trustworthy but also restricted hegemony that preceded our day. But this state isn't unprecedented. In 19th and early 20th centuries, there were a greater variety of newspapers and magazines each with strong biases and little distinction between fiction and fact. Journalistic objectivity as a value didn't become prominent until the mid-20th Century.

Tools for Discerning Truth

At present, the main tool most citizens use to judge media is whether a story matches their existing world view and opinions. Confirmation bias turns out to be a pretty good tool so long as one's world view is somewhat close to truth. And, of course, everyone thinks that their biases are the "true" ones. The problem is that when one relies exclusively on confirmation bias, they don't have a tool for correcting their biases - for getting closer to what's really true.

Trust is our second-best tool for judging content. It's also an excellent tool for correcting ones own biases. That's why the decline of trust, and the concurrent decline of trustworthiness, is such a problem today. The sensationalism of most media outlets concentrates on confirmation bias as a way to gain audience. Careful readers have to seek trustworthy journalists rather than organizations - at least until the trend turns.

Critical thinking isn't so much a tool as a discipline. It's something that our schools can and should teach and it's incorporated into good quality language curricula. As students are taught critical thinking they are taught to recognize and use good-quality arguments, to measure the credibility of facts based on origins and citations, and to compare and contrast writings from multiple authors.

Taking Personal Responsibility

I'm afraid that profit motive will prevent the media industry from solving this problem for us. Rather, we need to take individual responsibility for recognizing and tuning our own biases. We must bring the language of critical thinking into our vocabulary; asking about sources, seeking contrasting points of view, looking for supporting evidence, checking the logic of arguments, and discounting emotional appeals.

Clay Johnson wrote in The Information Diet, "The pattern here is simple: seek to get information directly from the sources, and when the information requires you to act, interact directly with those sources. An over-reliance on third party sources for information and action reduces your ability to know the truth about what's happening, and dilutes your ability to cause change." (Page 140)

The Information Age has given us unprecedented access to the original sources. We can take advantage of that. Institutions will follow the people, not the other way around.

10 November 2016

What I Would Tell Donald Trump about Education

I never thought Donald Trump would survive the first primary much less gain the nomination. By the time we reached the general election I gave up making predictions because, where Trump is concerned, I was always wrong. I don't expect this post to ever make it to the Trump transition team. But I could be wrong about that as well. Regardless, I hope it will help some of you in the community.

The Trump Policy Page on Education is pretty spare. During the campaigns, Trump spoke very little about education policy. In the primaries he made a few anti-Common Core remarks that seemed requisite of all Republican candidates. But those quotes date back to February. Mike Pence has been a strong advocate for school choice and that's reflected in the policy page. Their goal is to "provide school choice to every one of the 11 million school aged children living in poverty."

On the prospect that Trump's education strategy is still nascent, here's what I would tell him if I were asked:

Leave Standards to the States

The No Child Left Behind act required states to set educational achievement standards and measure the degree to which students meet those standards. It's successor, the Every Student Succeeds Act (ESSA) was passed in December 2015 with broad bipartisan support. ESSA maintains the emphasis on standards and accountability while returning responsibility to states to decide how to address underperforming schools.

Contrary to popular belief, the Common Core State Standards (CCSS) are not a federal mandate. They were created in a state-led cooperative effort with support from private foundations. The Obama Administration's, Race to the Top grants encouraged adoption of common standards among states without specifying any particular set. Those grants have mostly expired and there is no continuing federal support for the CCSS.

So, for Trump to eliminate the Common Core or to substitute other standards in their place would constitute more federal meddling in education, not less. Leave the development of standards to the states. Some will choose to collaborate on the CCSS, others will go their own way. We're in the third year of Common Core deployment. Within one or two more years we'll know whether it's been effective.

Ensure Title I Funds Really Benefit Economically Disadvantaged Students

This is a gnarly problem loaded with unintended consequences. Title I of ESSA (which is the latest reauthorization of the Elementary and Secondary Education Act) provides extra funding to schools and districts with a high proportion of children from low-income families. The goal is to close the achievement gap by offering more resources to schools that serve children with greater needs.

Unfortunately, as Marguerite Roza observed in Educational Economics the greater the distance between funding decisions and the students, the less effective they are at achieving the intended result. All too often, Title I funds are balanced by other funds being directed toward more mainstream schools and the most challenged schools remain with the fewest funds.

The Trump Campaign's proposal is to have specific money allocated to each economically disadvantaged child and for that money to move with the child to whatever school they choose. It's a promising strategy because it ties the funding decisions directly to the child but the concept won't work if there aren't good quality schools available for parents and their children to choose from.

Base Strategic Initiatives on Reliable Evidence

The theory behind the No Child Left Behind Act was to measure success and incentivize improvement. It's an approach that has worked in other domains but education has proven to be more challenging. That's because we still don't have a good model for effectively educating all students, at scale while preserving initiative, creativity, the arts, and joy.

We're making progress. And there's a growing body of evidence supporting some key strategies. They include:

Choose a Secretary of Education Who Understands the Landscape

Education doesn't need another shakeup right now. There are a lot of experiments underway that will yield great insights into what works. Some of these are at statewide scale like the competency-based New Hampshire High School Transformation or the Rhode Island Education Action Plan. Others are at district or school scale. We are rapidly learning what works and US Ed can shine a light on successful programs.

The Secretary of Education should have an optimistic outlook for US Education. They should have spoken at iNACOL, Educause, and SXSWEdu. They should know the education leaders at the Gates, Hewlett, and Dell foundations. Most of all, they have a humble attitude about the challenges ahead and the limited but important role of the federal government in US education.

26 February 2016

"Growth Mindset" is the Buzzword of 2016 - and That's a Good Thing

I first encountered the Growth Mindset nearly ten years ago in a New York Magazine article titled "How Not to Talk to Your Kids". The central point of the article was that when a child succeeds at a task, it makes a big difference whether you praise them for their effort or praise them for their talent or ability. Praising a child for their effort is associated with a growth mindset. It fosters children's belief that they can overcome obstacles and increase their mental capacity.

The article I read was based on the research of Dr. Carol Dweck. There is a large and growing body of evidence showing that students with a growth mindset achieve more and overcome challenges more consistently. It's also supported by contemporary research in psychology and neurology. "The brain is like a muscle." is a common metaphor, "Giving it a harder workout makes you smarter." Indeed, continuing research shows that IQ is malleable and can be increased.

In recent years, both anecdotal and rigorous evidence for Growth Mindset has increased with books, school programs, and parental training programs. Mindset Works is an advocacy organization dedicate to the concept. The result is an explosion of Growth Mindset interest in late 2015 and 2016.

And here are some recent examples:

Risk of a Buzzword

Growth Mindset is based on solid evidence and sound psychology. But as the buzzword starts trending we risk failure and discreditation of the idea due to enthusiastic but misguided efforts. A colleague recently worried that growth mindset might fall victim to the Self-Esteem fad of the 1990s. To be sure, the right kind of praise is connected with growth mindset. But equally important are fostering the determination to overcome obstacles and the safety to fail.

Some years ago I had the privilege of being a chaperone when my children's school competed in the Utah Shakespearian Festival. It was a small school and the drama team was composed of the majority of the high school - grades 9 through 12. I watched in amazement as these average kids rehearsed dramatic scenes, choreographed their own dance pieces, and performed a breathtakingly creative ensemble scene from Much Ado About Nothing. In the sweepstakes, they took second place against much larger and better-equipped schools. I chatted with teachers and other parents about what qualities enabled our school to perform so well without cherry-picking the best drama students for the team. We decided that an important factor is the emotional safety students had at the school. The cultural climate enabled students to take risks and regularly fail with minimal fear of ridicule. The courage to step out and take risks is especially important in the performing arts. Years later I found corroborating evidence in Brene Brown's research on vulnerability

Growth Mindset has as much or more to do with proper response to failure as it has to do with proper praise for success. Like a scientist performing experiments, students should be encouraged to treat failures as opportunities to learn and gain insight. Indeed, study of a failure can yield new understanding whereas success simply confirms existing knowledge.

Learning Mindsets

The Raikes Foundation considers a broader concept of "Learning Mindsets". This includes growth mindset and adds other skills that help students "actively participate, work through problems, think critically, and approach learning with energy and enthusiasm." Andy Calkins calls this "Agency." Of these skills; which include grit, determination, self-advocacy, and confidence; growth mindset seems to be getting the attention in 2016. If people study the concept and implement it well, that will be a good thing!

30 December 2015

Personalized Learning - More Evidence, More Progress

I've written a lot about Personalized Learning on this blog. The theory has a lot of things going for it. It's intuitive, it's the principle behind the most effective learning factors, and supporting evidence continues to accumulate.

When introducing personalized learning it's useful to contrast with factory-model education. Under a factory model, students with wide variation in personality, interests, skills, and talents are exposed to a consistent educational experience. Unsurprisingly, there is wide variation in the results because the consistent learning activities resonate better with some students than others. So, we grade the students with some portion of the grade attributable to student effort and other parts attributable to evidence of subject mastery. When students with inconsistent backgrounds participate in consistent learning activities, it's not surprising that the results are also inconsistent.

Personalized education applies in two ways. For fundamental subjects like Reading, Writing, and Mathematics, the learning experience should be personalized to meet the diverse needs of individual students. Customizing the experience to each student's individual needs can result in consistent achievement in a diverse population.

With a foundation of core skills in place, the second form of personalization is supporting students as they pursue diverse interests - science, music, art, history, sports, and so forth. The most successful students have always personalized their education. The innovation is for institutions to deliberately participate in the personalization effort.

Accumulating Evidence

Earlier this year, the Bill & Melinda Gates Foundation commissioned a RAND Corporation study of 62 public charter and district schools pursuing a variety of personalized learning practices. The results are promising. Average performance of students in the study schools was below the national average at the beginning of a two-year study period and was above the national average at the conclusion. Growth rates increased in the third year achieving effect sizes exceeding 0.4 in the third year.

Five specific personalization strategies identified and studied are:
  • Increased one-on-one time between student and instructor.
  • Personalized learning paths with students able to choose from a variety of instructional formats.
  • Competency-based learning models that enable individual-pacing with supports tailored to each student's learning level.
  • Flexible learning environments that can be adapted to student needs, particularly when they have conflicting demands on their time.
  • College and career readiness programs.
The authors observe that, "While the concept of personalized learning has been around for some time, advances in technology and digital content have placed personalized learning within reach for an increasing number of schools."

Progress and Public Support

The most significant policy event this year was the reauthorization of the Elementary and Secondary Education Act (ESEA). The previous iteration was known as "No Child Left Behind", this version is titled the "Every Child Succeeds Act". About the new law, iNACOL wrote, "Through ESEA reauthorization, Congress [supports] the shift to new, personalized learning models by redesigning assessments, rethinking accountability, and supporting the modernization of educator and leadership development."

Another important event this year is Education Reimagined. The Convergence Center for Policy Resolution brought together leaders from across the political and educational spectrum to describe a new vision for education. As they describe it, "We were not your typical group -- no two in agreement about how to fix the current system. What we did share, however, was a fundamental commitment for all children to love learning and thrive regardless of their circumstances. We knew it was time to stop debating how to fix the system and start imagining a new system." I had the privilege of hearing Becky Pringle, vice president of the National Educaton Association and Gizele Huff, director of the libertarian Jacquelin Hume Foundation describe their shared vision of student-centered education. It's compelling that, when you get all of the parties to converge on a shared educational vision it focuses on personalization - on meeting the specific needs of each student.

As we head into the new year, I'm optimistic. At this moment, we have progress, evidence, and policy coherently driving toward a better education for all of our students.

15 December 2015

Back to Blogging, Smarter Balanced, and the Importance of Evidence

When I started this blog I set a few guidelines for myself. One was that I wouldn't blog about blogging. I'm violating that rule today; mostly because it's been nearly 11 months since my last post and I want to record my commitment to resume postings here.

Smarter Balanced

The main reason that I haven't been writing is lack of time. The graphic shows my email traffic over the last approximately 18 months. There's a jump around October of 2014. That's when Smarter Balanced converted to it's sustainable form as a unit in the Graduate School of Education at UCLA. A bigger jump occurred in early 2015 as we entered our first operational summative testing season. My workload is slowly improving as I've staffed up the Smarter Balanced technology team with talented set of individuals.

Here are a few of the things we've accomplished at Smarter Balanced since my last post:
  • Released open source for the test delivery system, digital library, and reporting system and proven out the open source solutions in full-scale deployments.
  • Grown the subscriber base of the Smarter Balanced Digital Library to more than 600,000 educators.
  • Administered tens of millions of interim assessments. (Since interim test results remain with states and districts we only have a rough estimate of the number.)
  • Administered summative tests in English Language Arts and Mathematics to more that 6.5 million students.
  • Gained Iowa and the Bureau of Indian Education as members (while, unfortunately, losing Iowa and Maine).
Of course, this hasn't been without challenges. Addressing challenges accounts for most of the growth in my email traffic.

The Importance of Evidence

Finally, as I return to blogging I want to re-assert the importance of evidence. Too many decisions are made based on preconceived notions, confirmation bias, and a charismatic messenger. Recent research into research (meta-research?) has indicated that even rigorous, peer-reviewed, research findings are subject to confirmation bias.

Conveniently for my own opinions, the evidence in favor of personalized learning continues to grow. I have a lot more to write about this in the coming months.

26 January 2015

K-12 Education Funding... and the Strings Attached

In the 2013-2014 fiscal year, California spent $70 billion on K-12 education. To put that in perspective, Bill Gates' net worth is $80.4 billion. So, in a single year, California spends nearly all of Bill Gates' wealth on teaching children. This is a good thing, of course, but it's also an impressive number.

Nationwide, the country spent $632 billion on on public elementary and secondary schools in the 2010-2011 school year (the latest year for which I could find data). That's nearly 4% of the US GDP and 10% of total U.S government spending (including federal, state and local).

Here's where the 2013-2014 California money came from, in billions of dollars. Other states have similar proportions between federal and state/local funds:

Local Funds$21.78031%
State Funds$40.86458%
Federal Funds$7.38211%

For this post I'm going to concentrate on the strings attached to the Federal funds.

The Elementary and Secondary Education Act (ESEA 1965)

Federal funding of education, at least at contemporary rates, centers on the Elementary and Secondary Education Act (ESEA). Passed in 1965 as part of Lyndon Johnson's "War on Poverty," the ESEA was intended to address inequities in education. It had been long observed that students from lower income, urban schools have significantly lower educational achievement than their middle income, suburban contemporaries. ESEA provided supplementary funding to the lowest achieving schools with provisions intended to insure that existing funding is preserved rather than replaced.

The ESEA was set up to require periodic reauthorization by congress – typically every five years. However, due to congressional gridlock on educational ideas, the reauthorizations have often been single-year continuing resolutions that continue funding for another year without changing the provisions of the law. Major updates occurred in 1981 under the Reagan administration and in 1994 under the Clinton administration. But the biggest update was No Child Left Behind, proposed in 2001 and signed by President Bush in January of 2002.

No Child Left Behind (NCLB 2002)

The No Child Left Behind Act (NCLB) is the name given to the 2001/2002 reauthorization of ESEA. It establishes the accountability and reform framework in which state education systems presently operate. In theory, states have the ability to opt out at the expense of federal funding. In practice, no state is willing to give up approximately 11% of their educational budget.

The principle focus of NCLB is on the Standards and Accountability theory of education reform. Here are the main requirements:
  • States must establish state standards (sometimes known as core standards) for achievement in English Language Arts (ELA), Mathematics, and Science. Most states also include standards for Social Studies and other subjects.
  • States must test all students in grades 3 through 8 and again in either grade 11 or 12 to measure progress in ELA and Math. 
  • At a minimum, states must test students in science three times. Once in grades 3-5, once in grades 6-9, and once in grades 10-12.
  • The testing results for each school should show Adequate Yearly Progress (AYP) toward having all students meeting or exceeding state standards by the 2013-2014 school year.

Adequate Yearly Progress (AYP)

Among the most challenging parts of NCLB as been the Adequate Yearly Progress requirement for schools. Schools receiving Title I assistance (those with a large number of low-income students) receive increasingly strident interventions each consecutive year they fail to achieve AYP:
  • Year 1: No intervention.
  • Year 2: Develop an improvement plan, provide students the option to transfer to other schools including paying for the transportation to get there, and prescribed uses of Title I funds.
  • Year 3: Must continue year 2 interventions plus and also provide tutoring and/or after school programs from a state-appointed provider.
  • Year 4: Must continue year 2 and 3 interventions plus one or more of the following: Replace responsible staff'; Implement a new curriculum; Decrease a school's management authority; Appoint an external expert to advise the school; or Restructure the internal organization of the school.
  • Year 5: Shut down or completely restructure the school.
When NCLB was passed, there was an optimistic outlook. Within 12 years, nearly all schools would be meeting state standards for performance with a small number of underperforming schools receiving intervention. It turns out that, as a country, we haven't worked out a formula for consistent school improvement. If the process for meeting AYP standards was well-known, the goals might have been met.

One concern has been that certain states set unreasonably low standards. Prior to adopting the Common Core State Standards, Tennessee had the lowest standards for reading while Massachusetts had the highest.

Despite low and inconsistent standards, so many schools are failing to meet AYP goals that there aren't enough resources to deliver the prescribed remedies. In 2011, 48% of public schools failed to meet AYP goals. In 21 states, more than half of schools didn't meet AYP goals and in 41 states and Washington D.C. more than one fourth of schools didn't make AYP. There aren't enough tutoring organizations, replacement staff, or trained principals to supply the year 4 and 5 remedies for this many schools, not to mention sufficient funds to pay for these interventions.


With so many schools failing to meet AYP goals and the remedies being impractical to implement, congress is way overdue for an ESEA reauthorization that adapts to current circumstances. Unfortunately, no proposed update has made any significant progress. Congress has left us with continuing resolutions that preserve the law as it stands.

To relieve pressure, the Department of Education, under Secretary Arne Duncan has begun granting waivers to NCLB to states that produce an acceptable alternative plan. Not surprisingly, the granting of waivers is controversial. The authority of the executive branch to waive requirements like these seems to have legal precedent. However, it's not clear that alternative requirements can be applied without congressional action.

Nevertheless, every state except Nebraska has applied for a waiver, many have been granted, and even Nebraska has announced plans to apply for a waiver in 2015.

The Way Forward

There's growing hope that congress may finally address ESEA reauthorization in 2015. There are even hints that the reauthorization may include support for competency education. Many organizations are offering wishlists for reauthorization from civil rights groups to advocates of federalist solutions. As in the past, divisions on education don't follow traditional political lines.

Here is my personal wish list for an ESEA reauthorization:
  • Preserve and strengthen state standards, encourage but don't require alignment of standards between states.
  • Preserve regular assessment of student achievement with an increasing emphasis on Depth of Knowledge.
  • Accelerate the shift from seat-time measures to direct measures of competency for the granting of secondary school credit.
  • Encourage the transition from periodic testing events to continuous assessment of student skills (curriculum-embedded assessment) with frequent and rapid feedback to students, teachers and parents.
  • Clarify the difference between standards and curriculum and establish a framework for public review of both standards and curriculum. Require schools to report the origin of curricular materials on public websites and on every worksheet or assignment.
  • Sustain the concept of interventions for schools not achieving AYP goals while shifting to more practical and supportive remedies than those in NCLB.