Theories of Education Reform

My oldest son was a junior in high school when the standardized tests associated with No Child Left Behind were rolled out. One day, shortly before the exams, he asked me, "Why do we have to take these tests anyway?"

I answered truthfully, "They're not evaluating you, they're evaluating your school."

I found out later that with that information, he and his friends challenged each other to get the lowest scores possible. I sometimes use this story to illustrate broken feedback loops. It was nine months later before the scores had impact. When he returned to school the next fall he found he had been enrolled in remedial math despite aceing Pre-Calculus the previous year. He had to meet with the counselor to get into the right class.

Today, however, I want to explore the theories of education reform that drove the deployment of these exams. There are three prominent theories of reform with a few variations. Most contemporary efforts to improve education are based on at least one of these.

Theory: Standards and School Accountability
This is the primary theory represented by No Child Left Behind (NCLB). It's based on the broader theory that measuring something and reporting on those measurements will bring about improvement – especially if improvement is incentivized. It also represents the truism that if you don't measure something, you can't tell whether you've changed it for the better.

In order to bring about accountability, NCLB requires states to define learning objectives for each year or grade. These objectives are commonly referred to as the state core standards and each U.S. state has its own set. Furthermore, any public school receiving federal funding must administer a state-wide standardized test to every student in grades 3-9 and at least once in grades 10-12. Student scores are compared with previous years' results to determine whether they have achieved Adequate Yearly Progress (AYP). Certain consequences are tied to individual schools' success or failure to achieve progress for all students.

The core of the theory is this: If we set higher standards, measure against those standards and report performance then learning will improve. Unfortunately, 11 years into this experiment the quality of U.S. student learning is nearly flat.

There are numerous criticisms of standards and testing; but my personal concern is that by themselves they are a blunt instrument. In the absence of a proven formula for improvement the result is a form of natural selection – schools that underperform are taken out (actually they "receive interventions") while better performers survive. Natural selection is proven to work but it takes many generations and a lot of the population are brutalized before measurable improvement occurs.

Despite the lack of success, it's not time to abandon standards or accountability. Prior to 2002 most states didn't have well-defined core standards nor was student performance consistently measured. Now all states have standards, we are measuring regularly and 45 of the states have recently agreed to the Common Core State Standards. While standards and testing are inadequate remedies by themselves, they are important assets on which to build.

Theory: Highly Qualified Teacher
Where the Standards and Accountability theory focuses on school improvement. This theory focuses on teacher improvement. It's certainly intuitive; most of us have had one or more great teachers and we know they make a big difference. It's also justified by the data. Studies confirm that teacher quality is an important factor in student achievement and that the variation in achievement between classes within the same school is greater than variation between schools.

NCLB includes a mandate for states to supply highly qualified teachers to every student but it leaves it up to states to determine what it means to be highly qualified. And that turns out to be a problem. Studies show that certain teachers are consistently more effective than others; value added measures can identify which ones they are (albeit with a moderate error rate); but individual teachers often don't know what they need to do to improve.

In raw form this becomes another application of natural selection. If we reward teachers who perform well and eliminate those who don't then eventually performance will improve – assuming we don't run out of teachers beforehand. But many generations will be required and a lot of brutal actions will be taken in the meantime. No wonder there's so much controversy around teacher evaluations being tied to wages and promotions.

I'm actually in favor of merit pay for teachers so long as good quality performance measures are used. But those evaluations need to be deployed concurrently with professional development that informs teachers on how they are doing and what they can do to improve. Conveniently, resources are emerging to support that. For example, the Measures of Effective Teaching project used the Danielson Framework for Teaching to identify teacher behaviors that are well-correlated with student performance. These and similar frameworks can be used to inform teachers on how they can do better.

Even so, effective teachers alone are not enough. In our current educational system, teachers account for approximately 8.5% of variation in student achievement. School-, teacher-, and class-level factors combined account for about 21%. Meanwhile, background characteristics such as race, parental achievement and family income combine to account for 60% of variation in achievement levels.

So, if every teacher in the country was equivalent to our very best, it still wouldn't be enough to overcome the cycle of intergenerational poverty. To achieve that dream, we have to increase the influence school has over student achievement. That can be done by adapting the learning experience to the needs of individual students.

Theory: Personalized Learning
There's a pattern to these theories: The Standards and School Accountability theory introduces the concept of measurement and uses it to assess whole schools. The Effective Teachers theory takes those same measures and applies them at the teacher level. For this third theory, feedback is applied at the student level.

Personalized learning leverages the same standards as the other theories. It can also incorporate the same measures. However, annual testing alone is insufficient for personalization. Instead, understanding is measured weekly, daily or, in the best adaptive learning systems, continuously. Measurement must happen soon enough and feedback given quickly enough to affect learning activities. A truly personalized system selects activities according to student needs and also adapts to student behavior within an activity.

Bloom's Two Sigma experiments and the follow up work they inspired make me optimistic about Personalized Learning. These and other studies have shown that personalized learning experiences enabled by immediate feedback consistently deliver one to two standard deviations improvement in learning. We believe that is sufficient to overcome background factors thereby enabling a majority of students become high achievers.

Personalized learning is the natural result of 1:1 tutoring which is why tutoring is so effective. To do personalized learning at classroom scale generally requires 1:1 computers and a role change for the teacher as she shifts from "deliverer of knowledge" to "facilitator of learning." As with the other theories, there's a lot of skepticism and resistance to change. But pilot deployments are showing great promise.

Variation: School Choice
School Choice attempts to bring competitive pressure for schools to perform better. In this way, it's a variation on the Standards and School Accountability theory. Like NCLB, School Choice needs standards to be set and school performance must be measured against those standards. Performance is reported to parents who are expected to make an informed choice of which school their students should attend.

Since allocation of school funds is tied to enrollment, the theory is that schools seeking students will compete, not only on standards and their measures, but also on the basis of any other factor that's important to parents and students.

School Choice efforts include charter schools, magnet schools and voucher programs. The idea is to give public and private schools more freedom to experiment thereby accelerating the identification of viable formulas for improved leaning. Studies have shown this to be the case as the average of charter school outcomes is similar to that of public schools while variation among charter schools is much greater. Therefore, some charter schools are substantially better and should be emulated while others are substantially worse and should be shut down or reorganized. It's exactly this kind of variety and freedom that school choice advocates seek.

School choice can incorporate Highly Qualified Teachers and Personalized Learning. Indeed, since both of these theories have been shown to be effective, the expectation is that schools that incorporate these principles will be the best rated and will attract more students.

Variation: Small Classes
The small classes movement is based on studies showing that students learn better in smaller classes – all other factors being equal. But other factors are not equal. Lowering the student:teacher ratio costs a lot of money and other factors such as teacher skill have a greater impact than class size. For example, when California mandated smaller classes they had to hire many more teachers. For at-risk populations, the impact of less-experienced teachers overcame the benefits of smaller classes resulting in lower performance instead of the expected improvement.

Variation: No Excuses
The No Excuses model centers on maintaining high expectations for student performance without making excuses for external issues such as background, troubles at home and so forth. It's associated with charter management organizations such as KIPP and BES. Proponents emphasize pillars such as college expectations, culture of respect, voluntary participation and high discipline. They also have extended hours and extended school years. A key value is the whole school's commitment to each student's success. If a student is struggling or falling behind, they discover that early and engage counseling, tutoring and other supports to ensure the student succeeds.

No Excuses engages all three theories, overall school performance is measured, they hire and train highly effective teachers and they adapt the learning environment to the needs of individual students, albeit most No Excuses schools do adaptation with limited use of technology. Over the last decade, No Excuses schools have demonstrated that background factors can, indeed, be overcome by a supportive school structure. On the other hand, their high reliance on supportive interventions sometimes leaves students underprepared for the independent learning discipline required in college. Recognizing this, No Excuses organizations are updating their practices to better train students to become independent learners.

~ ~ ~ ~ ~

As with the variations listed here, most reform projects mix two or more of these theories. Even NCLB includes a mandate for Highly Qualified Teachers. Personalized Learning efforts are more common at charter schools than conventional public schools.

Education Reform will remain an important part of our civic dialog for a long time. Unsurprisingly, it means different things to different people. For some it's a moral crusade. To those being asked or forced to reform it's more threatening. All too often arguments about reform neglect the research (which is abundant) and fail to fully express the theories on which they are based. That shouldn't be the case as there are decades worth of data and studies behind each of these theories – sufficient for advocates and policy makers to make informed decisions.

The data tells those of us seeking to eliminate poverty that incremental improvement to existing schools is insufficient. Personalized learning with an eye toward training independent learners seems to be the most promising approach. Deploying this at scale requires whole-school changes to the way programs are funded, to the choices of curriculum and technology, and to the roles of educators.

Winds of Change: Higher Productivity in Higher Education

Note: This first appeared last week as a guest post on the Next Generation Learning Challenges Blog. I highly recommend both the blog and the NGLC website.

My first lecture hall experience was American Heritage at Brigham Young University. The course was required for all freshmen and more than 500 of us at a time attended two lectures a week. In a third “lab” period we met with a TA. The professor was charismatic and the instructional design team supplied him with carousels full of colorful slides. Still, a large fraction of the class was asleep at any given time.

Large lecture hall courses are one common method of increasing productivity in higher education. Another is weed-out courses – those designed to convince students that they should choose another, less expensive major. For me the weed-out subject was Discrete Structures. This Computer Science subject is rich with metaphors like trees, maps, chains and links. It can be taught through story, modeling, manipulatives and real-world application. But our version was deliberately dry with an emphasis on precise vocabulary and obscure notational forms. The pass rate hovered near 50% and hundreds of students were convinced that they weren't capable of understanding computer science.

Higher education in the United States is sandwiched between twin pressures, increasing societal needs and expectations on one side with flat or declining funding on the other. To meet this challenge, institutions will have to dramatically increase productivity. But traditional productivity boosts like large lecture halls, weed-out courses or greater admissions selectivity won’t be enough this time around. What’s required is fundamental change to the way we support learning. We need a more personalized approach.

Societal Needs and Expectations

Employment projection is from the Bureau of Labor
Statistics Job Outlook. Supply is based on National
Center for Education Statistics data on annual
Computer Science BS degrees awarded. Attrition
is based on a 40-year career span.

While the U.S. unemployment rate hovers around 8%, there is a shortage of engineers and technicians. In 2012, the unemployment rate for software developers was only 2.8%. An Association for Computing Machinery study indicates that the United States will need more than 150,000 new computer scientists each year through 2020 yet our collective colleges and universities only produce 40,000 degree holders to fill those jobs. Healthcare workers are also in short supply. In 2012 the unemployment rate for physicians was 0.8%. For Physical Therapists it was 2.0% and for Registered Nurses, 2.6%.

At a recent Technology Alliance conference it was noted that colleges and universities in Washington State produce less than half as many engineers, technicians and software developers as the state’s employers consume. The rest have to be imported from other states or countries. A speaker from the University of Washington pointed out that they have increased introductory Computer Science enrollment from roughly 1200 to over 2000 per year. But the Microsoft representative responded that they have 3,600 engineering and computer science openings and they’re competing with Amazon, Boeing and many others to fill those spots.

Of 4.3 million freshmen who started college in 2004, only 2.2 million (or 51%) graduated within six years. This isn’t a perfectly accurate figure. Because of the way records are kept, it’s hard to count students who transfer and complete at a different institution. But inadequate record keeping is another symptom that institutions haven’t focused enough on ensuring their students are successful. Higher completion rates will save a lot of wasted student time.

As we move into the 21st century the fraction of unskilled jobs continues to diminish while those requiring advanced skills increase. It’s no longer appropriate to sort students by “aptitude.” We must give students the support and guidance they need to master advanced subjects.

The Funding Landscape

Education is the largest item in most state budgets. In California it accounts to between 52% and 55% of the state general fund. With the recession hitting state revenues and the expiration of stimulus supplements, state fiscal support for higher education dropped by 4.7% between 2011 and 2012, remaining flat in 2013. Overall, annual support has dropped by 10.8% since 2008. On a per-student basis, state and local financing dropped 24% in the 10 years preceding 2011.

At the same time, tuition is rising much faster than inflation. Tuition and fees at U.S. public universities rose 4.8% for the 2012 school year to an average of $8,655. At nonprofit private colleges tuition and fees rose 4.2% to $29,956. In addition to drops in public funding, the cost to provide education is increasing and the recession has diminished private endowments.

Total student debt in the U.S. now exceeds $1 trillion making it higher than the nation’s credit card dept. Student loans aren't a big problem if they are correlated with significantly higher earning potential. But loan approval is not connected with choice of academic major or the graduation rate of the institution.

Personalized Learning

The demands on higher education are greater than ever. We need more graduates – especially in certain fields. We need better completion rates. We need to support students in tackling challenging subjects. Moreover, we have to do this with flat or declining budgets.

The Bill & Melinda Gates Foundation has assembled representatives from a dozen colleges and universities that are trying new approaches with promising results. The Personalized Learning Network, as it's called, includes innovators like Western Governors University and American Public University; pioneering programs at Arizona State University and UC Berkeley; and NGLC grantees like the Kentucky Community & Technical College System, Rio Salado College and Southern New Hampshire University.

Recently I had the privilege of meeting with this group. There’s a lot of variation in their personalized learning programs but they share these common features:

• Mastery Learning and Independent Pacing: Students have to master the current topic before moving to the next step. Self-pacing grants this freedom and ensures that there aren't gaps in understanding due to bad days or illness. And students don’t waste time on topics that they already understand.

• High Expectations: The institutions make a commitment to support all students sufficiently so that they can master the material.

• Feedback: Students and instructors are constantly informed about conceptual understanding and progress through the material.

• Adaptive Learning: The learning system adapts according to individual student actions and performance.

• Individual Attention: The programs facilitate abundant 1:1 time between students and faculty.

• Motivation: Systems and attitudes that foster student motivation include interesting activities, student autonomy, recognizing good performance and avoiding frustration either due to anxiety or boredom.

All of this is enabled through strategic use of technology. Most use some form of blended online and in-person learning. The key point is not to simply add technology but to apply technology in the service of personalized learning.

Personalized learning programs should be able to address higher education pressures for better success and completion rates. But can they also help educate more students at lower cost? I believe so. Technology can automate many tasks that cost a lot of educator time. Video lectures are a personalization technology because they allow students to view on demand and replay as needed. Not only do they save the time in class but they also save the instructor time preparing the lecture. Objective assignments can be graded automatically and feedback given instantly to the student. Feedback to instructors can help them optimize their interactions with students. Subjective grading, while still consuming human time, can also be made more efficient. All of these factors help institutions increase capacity and reduce per-student costs.

Equally important are the savings offered to students. Immediate feedback helps students learn concepts more efficiently and avoids time wasted on misconceptions. Students can advance immediately upon understanding a concept and get credit for things they learned previously. And authentic learning activities support a better and more complete understanding of each topic. In one study by Carnegie Mellon’s Open learning Initiative they were able to teach students the same material in half the time with better retention.

Changing higher education is like turning a glacier. Features like accreditation, tenure, financial aid, credit transfer, and faculty autonomy interlock to form a seemingly insurmountable barrier protecting the status quo. But the twin pressures of increased expectations and diminishing funding result in an unprecedented incentive for change. Like the Maginot Line, traditional barriers won’t be overcome but simply bypassed.

The Common Core State Standards for Literacy are Two Dimensional

The ideal school librarian would know every student in the school – what their interests are, what their current reading level is and what their teachers will be teaching next. With this knowledge, she would use her comprehensive knowledge of the school's book collection to suggest books or activities that would be both enjoyable and yet challenging to the student's abilities. That is, books that are in the student's Zone of Proximal Development.

It's not really possible for a librarian to have such a comprehensive view of both students and the book collection. But under Race to the Top grants, several states are developing Instructional Improvement Systems that, among other things, will support recommendations like these. Such systems operate at the intersection of student data and content data. And to support them, inBloom (formerly the Shared Learning Collaborative) is deploying student and content data services.

The Common Core State Standards (CCSS) and the Learning Resource Metadata Initiative (LRMI) work together to support the content data side when teaching reading and writing. The CCSS for ELA-Literacy have two dimensions to their basic structure. The grid below shows one way to view the Common Core Standards for Reading. Making up the horizontal dimension are Anchor Standards 1-9. These describe specific skills that the student should be able to apply when reading. The vertical dimension is Anchor Standard 10, the requirement that the other nine anchor skills should be demonstrated against texts of increasing difficulty as the student advances from Kindergarten to 12th grade. Notably, grades 9 and 10 share a level as do grades 11 and 12.

Common Core State Standards for Reading Literature

Here's an example of how this works: Anchor Standard for Reading number 6 states:

• CCSS.ELA-Literacy.CCRA.R.6 Assess how point of view or purpose shapes the content and style of a text.

On the diagram this is marked with a vertical gridline. One of the horizontal gridlines is Reading Literature Grade 4 Standard 10. It's statement is:

• CCSS.ELA-Literacy.RL.4.10 By the end of the year, read and comprehend literature, including stories, dramas, and poetry, in the grades 4–5 text complexity band proficiently, with scaffolding as needed at the high end of the range.

I've marked the intersection of these two with the identifier, "RL.4.6". The statement for Reading Literature Grade 4 Standard 6 is:

• CCSS.ELA-Literacy.RL.4.6 Compare and contrast the point of view from which different stories are narrated, including the difference between first- and third-person narrations.

Notice how this last statement is a refinement of anchor standard 6 targeted at a Grade 4 skill level. So, a source text or learning activity that satisfies RL.4.6 would have a text complexity level in the grade 4-5 text complexity band and it would at least use a first-person or third-person narration. Ideally the activity would include both narration forms and give the student a chance to contrast the two.

So, what are these text complexity bands and how do we tell whether a text is within a particular band? In other words, how do we place a text or learning activity on the vertical dimension?

Appendix A of the Common Core State Standards for English Language Arts describes a three-factor model for measuring text complexity. The qualitative factor refers to levels of meaning, structure and demands for prior knowledge on the part of the reader. "Reader and Task" considerations involve matching texts to the reader's needs or interests and the learning tasks that will be associated with the text. The quantitative factor is a numerical measure that is calculated (usually by computer) from word length and frequency, sentence length, vocabulary and text cohesion. A supplement to Appendix A lists six approved scales for indicating quantitative text complexity for the Common Core. The table below indicates which levels are appropriate for certain grade ranges.

Common Core Band	ATOS	Degress of Reading Power®	Flesch- Kincaid	The Lexile Framework®	Reading Maturity	SourceRater
2nd-3rd	2.75-5.14	42-54	1.98-5.34	420-820	3.53-6.13	0.05-2.48
4th-5th	4.97-7.03	52-60	4.51-7.73	740-1010	5.42-7.92	0.84-5.75
6th-8th	7.00-9.98	57-67	6.51-10.34	925-1185	7.04-9.57	4.11-10.66
9th-10th	9.67-12.01	62-72	8.32-12.12	1050-1335	8.41-10.81	9.02-13.93
11th-CCR	11.20-14.10	67-74	10.34-14.20	1185-1385	9.57-12.00	12.30-14.50

The grid diagram also includes an example of how a source text might be fully aligned to the common core literacy standards. In this case, To Kill a Mockingbird is shown as an appropriate text for teaching standards 1-7 at grades 9 or 10. So, the LRMI metadata for To Kill a Mockingbird would include alignment to standards RL.9-10.1, RL.9-10.2, RL.9-10.3, RL.9-10.4, RL.9-10.5, RL.9-10.6, and RL.9-10.7.

On the vertical dimension, To Kill a Mockingbird is positioned toward the middle of the grades 9-10 range. So, it would be considered moderately advanced for grade 9 and moderately easy for grade 10. To Kill a Mockingbird is rated an 870 on the Lexile scale. A quick glance at the table shows that 870 is in the 4th-5th grade range. The book is positioned higher on the grid than the raw Lexile number would indicate due to qualitative factors such as the complex moral dilemmas posed by the text.

The LRMI metadata schema is designed to be flexible enough to represent all of these dimensions. The AlignmentObject type represents the relationship between a text or learning activity and a node in a framework or taxonomy. The most obvious and common way this is use is with a alignmentType of "teaches" or "assesses" and the target node being a statement in the Common Core State Standards. In the To Kill a Mockingbird example, the "teaches" alignmentType would be used with targets of the six standards (RL.9-10.1 to RL.9-10.7). Any one of these six standards also implicitly brackets the vertical, text complexity dimension. In order to more finely position a resource, LRMI also defines a "textComplexity" alignmentType. Publishers of at least two of the quantitative frameworks listed above are in the process writing guidelines for their use with LRMI. It's also possible to use LRMI to indicate non-quantitative factors. To do so, we would need to define taxonomies for qualitative and "reader and task" factors with appropriate identifiers.

In these examples I've used Common Core Standards for Reading but the writing standards have a similar two-dimensional structure. Overall, it's a rich framework with great promise for improving student literacy.

We have achievement standards (CCSS) and data standards (LRMI). There are emerging services like inBloom that build on these standards. I expect very soon a combination of CCSS, LRMI, open libraries of content and custom recommendation engines will offer students custom reading lists and writing activities tailored to their individual learning needs.

Personal Rapid Transit and Driverless Cars

An ULTra PRT vehicle on a test track. (Wikimedia Commons)

As a teenager in the 1970s I remember reading about Personal Rapid Transit in a number of places including this Popular Science article. Unlike conventional transit like light rail or bus systems, a PRT system uses small, individually switched cars on a specially designed guideway. Upon entering a station, you select your destination on a console. Within a few seconds a 3-6 passenger car arrives and whisks you directly to your destination. At least that was the dream.

Of the dozens of proposals and prototypes, the Morgantown PRT that links WVU campuses is the only one of that era ever to be deployed at scale. The rest were either cancelled entirely or were defeatured into automated people mover systems like you find at many airports.

In 2011 two new systems opened, ULTra PRT at London's Heathrow Airport and the 2getthere system in Masdar UAE. Both are relatively small systems each with fewer than five passenger stations and fewer than 25 vehicles. But the new systems also represent an important departure from previous PRT designs. Both use battery powered vehicles with autonomous control. They run on rubber tires and steer themselves so there's no switching gear on the guideway. They are powered by batteries that automatically recharge when the cars wait at stations. This contrasts with previous PRT designs that used powered guiderails and a central control and switching system.

The primary barrier to PRT systems has been the cost of the tracks or "guideways." It's estimated that it would cost beween $30 million and $40 million a mile to expand the Morgantown system. That's because the guideway has to incorporate precision guide curbs, power transmission, track switches and even a heating system to melt snow and ice to keep it safe in bad weather.

In contrast, the ULTra guideway is estimated to cost between $7 and $15 million per mile. That's because it's a simple concrete pathway with no active systems.

Which brings me to Driverless Cars.

In essence, the ULTra and 2getthere systems are self-driving electric cars in which the environment has been constrained enough to simplify the self-guidance problem. High curbs make it easier for the cars to center themselves in lanes, dedicated roadways minimize pedestrian and obstacle avoidance. Strategically placed charging stations let them be electrically powered using batteries of modest capacity.

Meanwhile, Google's self-driving cars have driven themselves more than 300,000 miles accident-free, on conventional roads, without special infrastructure. Like many, I've wondered why Google is building such cars. They're in the information business, not transportation. A talk by Big Data guru, Ed Lazowska clued me in. Before the Google people let a car drive a route by itself, they first have a human drive the car over the same route. During the trip, its sensors scan the environment, picking out landmarks and obstacles, measuring road conditions and fine-tuning its GPS map of the roadway. Google is interested in supplying data to enable driverless cars and they're doing research to determine what data is needed.

A recent Freakonomics post on the subject suggested that driverless cars will arrive incrementally starting with the already common cruise control, adding adaptive cruise control, collision avoidance and self-parking before fully driverless operation arrives.

But I'm afraid that calling these "driverless cars" is the 21st Century equivalent of calling automobiles "horseless carriages". In each case the focus is on what's missing (the driver or the horse) instead of what new capacity has been introduced. "Horseless carriage" doesn't exactly describe a vehicle capable of sustaining 65 miles per hour with a range of over 300 miles. Nor does it conjure images of the megacities it enables or the endless parking lots it requires.

Consider this possibility: driverless technology enables the PRT dream on existing infrastructure. Instead of dedicated guideways costing tens to hundreds of millions, a PRT system built on driverless technology would rely on GPS and 3G data networks, both of which are already in place. Initial deployments can be restricted to certain neighborhoods that meet high standards of traffic signals, lane markings and crosswalk protection. Even a system restricted to certain lanes and certain streets would offer PRT of greater scale and capacity than anything yet deployed. Yet the investment to get started is regulatory permission, a few vehicles and some signage.

Fancy stations aren't required – only some curb space. Cars would be summoned using smartphones. And it wouldn't just be a peoplemover. Cargo, also, could be sent unattended. Grocery stores could use the same infrastructure for home (or corner) delivery. In the long run, even mail delivery and garbage collection could be automated.

We can learn something from this:

There are some fundamental principles at work here that can be applied to other large-scale problems:

• Infrastructure is usually the most expensive component. Whenever possible, use infrastructure that's already in place and share infrastructure with other projects.
• Push control (or decision making) as close as possible to the application or beneficiary.
• Inform the distributed control with global data.
• Build systems that can be scaled incrementally; where adding capacity is a matter of buying more of the same rather than periodic large investments to get to the next capacity threshold.

Consider the above principles applied to education. (You know I can't resist.) Existing infrastructure includes the internet, inexpensive computers and tablets, content development tools, video standards and so forth. Personalized learning relies on giving more control to the student and teacher to adapt learning to individual needs while being informed by common standards. And web-scale technologies are required if systems are to grow to support millions of students.

The "Wouldn't it be Cool" Department

Walt Disney World has the most heavily used monorail system in the world. They also have a well-maintained private road system connecting their resorts and theme parks. Wouldn't it be cool if Disney deployed a PRT system (based on driverless car technology) to connect their resorts and parks together? Such an attraction would enhance the Disney experience while proving the viability of the concept to the world.

Bloom's Two Sigma Problem Revisited

Benjamin Bloom's Two Sigma Problem has been both a guiding framework and a challenge to educators for more than a quarter century. A bit more than a year ago I wrote about the problem and some of the ways people are approaching it.

Here's the concise version: Bloom and some of his grad students compared classroom teaching with 1:1 tutoring. In both cases they used a mastery-based curriculum. The tutored students performed two standard deviations (two sigmas) better than their conventionally taught peers. While it would be nice to have a 1:1 student:teacher ratio, Bloom acknowledged that it's not practical and he proceeded to research ways to achieve similar results using more scalable means. He published the study in 1984. Since then, the Two Sigma Problem has served as a benchmark of how well students can learn if given the right supports.

A recent meta-study by Kurt VanLehn of Arizona State University compares no tutoring (conventional classroom), computer-based Intelligent Tutoring Systems (ITS), and human tutoring. VanLehn notes that a number of well-known ITS efforts have shown one-sigma improvements over conventional instruction. So, the conventional hypothesis is that computer tutors achieve one-sigma gains while human tutors achieve two-sigma gains as compared to conventional instruction.

VanLehn set out to test that hypothesis. He selected numerous studies that collectively yielded more than 100 comparisons between conventional instruction, three forms of ITS, and human tutoring. The result is surprising: answer-based ITS achieved an improvement of 0.31 sigma over conventional instruction. Step-based ITS achieved 0.75 sigma and human tutors achieved 0.79 sigma.

This is mixed news. On the one hand, the best computer tutors are almost as good as human tutors. That suggests that we can scale up much more effective learning than is achieved in conventional classrooms. On the other hand, VanLehn found no replication of Bloom's 2 sigma results. Is Bloom's goal out of reach or is there another factor involved?

To find out, VanLehn retrieved the dissertations from Bloom's grad students that contributed to the more famous paper. One key experiment yielded an effect size of 1.95 sigma – the probable source of Bloom's Two Sigma challenge. In that experiment both the conventional classroom and the tutors used a mastery learning technique. Whether in class or being tutored, students took a quiz after studying each unit. If their score achieved the mastery threshold, they advanced to the next unit. If not, they studied the unit more and were assessed again. This process was repeated until the mastery threshold was achieved.

The missing piece is that classroom students were required to achieve mastery threshold of 80% before advancing. Meanwhile, tutored students were required to achieve a threshold of 90%. Could it be that  adjusting the mastery threshold could account for a full standard deviation improvement in achievement? If so, numerous online learning systems should be tuned accordingly.

Oleg Bespalov and Karen Baldeschwieler, with their colleagues at New Charter University, have evidence to confirm this hypothesis. In their ITS system, students receive periodic formative assessments in the form of multiple-choice quizzes and self-graded short answer questions. From these assessments they calculate a "readiness score" to help students know when they're ready to advance. Students aren't constrained by the score – merely informed.

This creates a natural experiment in which they can compare student performance on the final exam against individual readiness scores. They discovered that students with a readiness above 90 achieved a 98% pass rate. But for those with a readiness score in the 81-90 range the pass rate dropped to 69%.

Both of these projects indicate that there's a critical threshold somewhere between 80% and 90%. Clearly this is an area deserving of more experimentation and research. But we can already tell that that tuning the mastery threshold is a critical factor for improving student achievement.

Measures of Effective Teaching

The Measures of Effective Teaching Project (MET) released its final reports last week. It got considerable press coverage as the study strives to inform teacher evaluation programs, a subject of considerable controversy.

Most of the stories, like this one from Reuters, focus on the the study's finding that teacher performance can indeed be predicted by performance measures. The best evaluations involve a weighted average of student test scores, teacher observations and student evaluations. Any one of these by itself is a much less accurate predictor.

There are nuances to this that can be gleaned from the project's Policy and Practitioner Brief:

• The different measures (student testing, teacher observation and student evaluation) have some overlap but mostly they measure different aspects of the teacher's skills.
• Different weightings are better predictors of different outcomes. Unsurprisingly, placing greater weight on test results is a better predictor of future student test results. However, equal weighting models or those that emphasize teacher observations are more reliable year over year.
• Effective teacher observations are more than a periodic visit from the principal. Evaluations require a consistent framework and procedure. The MET project used the Danielson Framework for Teaching as a rubric. The reliability of teacher observations is greatly improved by having at least two evaluators.
• When done properly, student evaluations are very reliable and an important component of teacher evaluation. As with observations, the key is to ask the right questions. The MET project used the Tripod Student Survey.
• The "value added" theory is supported. When student scores are compared with the previous year's performance (a value added score) the result is a more consistent predictor of future teacher performance than just the most recent year's scores.

One problem with exclusively using standardized tests to evaluate teachers or schools is that it's a blunt instrument. These tests offer a measure of performance but they offer limited guidance to a teacher or school on how they can improve. Sure, we can fire ineffective teachers and close ineffective schools. But using natural selection to improve schools is slow and costly not to mention cruel. Basically you're just hoping for those teachers and schools that randomly find the right formula for success.

Among the advantages of teacher observations and student evaluations are that they supply rich feedback to teachers to help them improve their practice. Another MET project report, Feedback for Better Teaching, offers guidelines for using feedback. They placed cameras in classrooms and observers codified the techniques used by the teachers. The same video recordings were used by the teachers themselves to observe their own performance – usually with an instructional coach. Processes like these can continuously improve teacher skills and effectiveness.

I've written before about how immediate feedback can help the student learn more effectively. In that context, it's no surprise that feedback to teachers helps them to be more effective. Moreover, it supports the passion that got them into teaching in the first place.

Enterprise-Scale is not Web-Scale

In the 90s, the IT world was talking about Enterprise-Scale. It's not that enterprise-scale was anything new. But up until then, the enterprise was the domain of mainframe and minicomputers. Upstart microcomputers – those with the whole processor on a chip – had previously not been capable of enterprise-scale operations.

It took a lot to achieve enterprise-scale with microcomputers. The leaders included Sun, Oracle and Cisco with Microsoft, Intel, Compaq and others playing fast-follower. They invented RAID arrays, symmetric multiprocessing, storage area networks, network load balancing and much more in the pursuit of five nines of reliability.

As microprocessor-based computers achieved enterprise-scale, pioneers like Google, Amazon, Yahoo!, SalesForce and others pushed right past enterprise into web-scale. User counts were measured in hundreds of millions, storage capacity was measured in petabytes and server arrays numbered in the thousands. Unlike enterprise-scale where key technologies had already been invented by the mainframe world, there wasn't any precedent for web-scale and the pioneers had to invent their own methods. I happened to work at Ancestry.com in the late 1990s/early 2000s and got to participate in some of that invention. But it wasn't until later in the decade before the pioneers started to share what they had learned and build products like Amazon Web Services, Google App Engine and Windows Azure to support the web-scale developer.

This is a important issue for education technology. The education industry is a bit behind the curve in moving to web-scale. For example, most learning management systems are built for enterprise-scale. They are intended to be installed on dedicated servers at a college or university's data center and they're architected to handle tens of thousands of students and teachers.

What happens when you move to the K-12 market or to community colleges? These organizations don't have the data centers and skilled staff needed to deploy, maintain and backup enterprise servers like these. In the past, their data systems only had to handle a few hundred or maybe a few thousand administrators. Teachers and students didn't directly access the district's data systems.

But districts are rapidly bringing all of their students and teachers online. And that means two orders of magnitude more users. Many districts have student counts numbering in the hundreds of thousands. Some get into the millions. And since their technology staffs are already overburdened, they seek hosted solutions, not enterprise-scale servers they have to manage themselves. Hosting a single district might not reach web-scale but a cost effective provider would serve hundreds of districts. And web-scale technologies can reduce the cost to something that districts can afford.

Here are some of the principles of web-scale architecture. For purchasers of products and services, these are the things you need to look for. For developers of those services, these are the principles you need to incorporate into your design.

Always Available
Web scale services use redundant servers to ensure that the service is always running – even during software upgrades and system maintenance. The term "24/7" was invented to describe services that have no weekly scheduled downtime. (And please don't write 24/7/365.)

Billions of Database Records
Today a district might keep a few dozen data points per student per year. In a district with 50,000 students that amounts to around 150,000 database rows per year. Eventually the database might grow to a few million rows total.

But personalized learning applications can collect thousands of data points per student per year. And an online service might serve a few million students. Thus, a web-scale learning service should be designed to store billions of data elements with provisions for orders of magnitude growth beyond that as clickstream data become more important.

Single Sign-On and Identity Management
Today's schools typically run a Student Information System, a Learning Management System and a few custom learning systems for specific subjects. Most of these applications have their own user database mandating separate logins and making requiring a lot of data entry to provision the sytems.

The low-hanging fruit is a single sign-on system that lets students and teachers use the same login account across all systems. But single sign-on is of limited value without automatic provisioning. So it's more important to have an identity management system that automatically shares demographic and enrollment information between the Student Information System and the various learning systems. The long-term need is to integrate the data among all of the systems so that all student performance data is accumulated in a common database.

Services-Integrated Security
Consider security in a Student Information System. The student, her teacher and her parents should have access to her school records, but no-one else (except maybe the principal or a counselor). Enterprise-scale security manages things like this through access control lists (ACLs). Record or element has an ACL granting certain levels of access to specific individuals. For example, the teacher has permission to view and change grades while the student and parent only have permission to view them.

The ACL approach becomes fragile at web scale. With millions of students and parents and thousands of changes to class enrollment it becomes difficult to maintain correct ACLs even when the process is automated. Roles and groups offer some relief but inevitably permission errors creep in and they become a technical support nightmare. Even worse, with regulations like FERPA in place, permission errors can result in significant liability.

Instead, web-scale applications use policy-based permissions. When a student is enrolled in a class, the policy says the teacher should be able to access that student's records. There's no ACL to be updated and permission naturally disappears if the student changes enrollment. The databases of these systems describe the relationships between elements (students, classes, teachers, parents, etc.) and the policies describe how permissions should be granted according to those relationships.

Services-integrated security also means that permissions are enforced at all levels of the system. The UI will control permissions that are offered to the user and the API enforces policy when read or write attempts are made. Thus a rogue or buggy application is still prevented from violating security policy.

Developer Note: Policy-based security can be processor and database intensive. The query to determine whether to permit a particular operation can easily be more expensive than the operation itself. This isn't a reason to reject the approach. Rather, use multiple levels of permissions caching to reduce the database burden.

Linear or Sub-Linear Cost Curves

If you graph number of users on the horizontal axis vs total cost on the vertical axis enterprise-scale systems have costs that grow exponentially. This is because they achieve scale by using progressively bigger and more complex servers and one 32-processor server costs many times more lot more than 32 single-processor servers.

In contrast, web-scale architectures have a linear or sub-linear cost curve. They achieve this feat by using software, database and hardware architectures that spread the load across many commodity-priced servers. As demand increases you simply add servers so variable costs are linear and fixed costs get diluted over a large number of users.

Developing scale-out software like this is complicated and expensive. Because of this, enterprise-scale architectures can cost less in enterprise-scale deployments. But when user counts get into the hundreds of thousands or millions, web-scale becomes more cost effective.

Mashups
Web-scale applications rarely stand alone. In most cases, they are combined with other applications to create a complete solution. This is certainly true of our vision for personalized education. A complete solution includes at least the following:

• Student Information System
• Student/Parent Portal
• Teacher Portal
• Adaptive Learning System (probably multiple subject-specific ones)
• Content Library
• Assessment Bank
• Analytics (Teacher, Department, Schools)
• Interactive Professional Development

Plus, other innovative applications are likely to emerge. In a realistic system, these components will originate from a variety of sources. For time-constrained teachers and students to use them effectively, they will have to be seamlessly mashedup together.

Tools and Techniques for Web-Scale

At Ancestry.com we had to invent many of the web-scale tools we used. But the toolkit has matured in the last few years. The easiest approach is to build on one of the cloud platforms like Windows Azure, Google AppEngine or Amazon Web Services. The downside is that doing so locks you into that vendor's hosting service.

Here are some of the other tools and techniques that help in web-scale development:

• Caching (lots of it and at different levels)
• NoSQL Databases
• Content Delivery Networks
• Network Load Balancing
• Browser-side Code
• Multi-tier Architecture

Ultimately, however, you can use all of these tools and still not have a web-scale application. You have to architect for web-scale from the very start.

Game Design and the Zone of Proximal Development

It's an experience many parents have had: From time to time my kids invite me to play video games with them. We pick a multiplayer game like MarioCart or Halo and they proceed to beat me silly. I keep trying to show moderately credible performance with little success. And I wonder, "How long would I need to play this game to achieve some degree of proficiency?"

Part of the problem is that parents like me jump right into the multiplayer mode competing against our kids while simultaneously trying to learn the game mechanics. Most of these games have a single-player "campaign" mode. The campaign is designed both to be consistently entertaining and to step-by-step make you better at playing the game. The two primary rewards of the campaign are progressive discovery of the story (narrative) and progressive mastery of new skills.

The Flow Channel in Game Design

In his book, The Art of Game Design, Jesse Schell describes "flow", a concept he adopted from psychologist Mihalyi Csikszentmihalyi. An individual is in a "flow state" when he or she is entirely involved in the task. "The rest of the world seems to fall away and we have no intrusive thoughts." It's a state of sustained focus and enjoyment.

Figure 1: Flow Channel

In figure 1 we see four player states on a graph of player skill vs game challenges. This can apply to a variety of games from physical sports to puzzles to first-person shooters. In this example a player is at state P₁ where the player's skill is balanced with the challenge of playing. With practice, the player increases in skill and advances to state P₂ in which the game becomes easy enough that the player is bored. P₃ represents condition when the game is too difficult; perhaps the opponent is a lot more skilled or the game is too hard. In this case the player becomes anxious about their performance. Both states P₂ and P₃ can be rebalanced. The game can become more challenging (P₂ to P₄) or the player can gain skill (P₃ to P₄). But if the player remains in Anxiety or Boredom for very long they'll abandon the game because both anxiety and boredom lead to frustration.

Figure 2: Growth Path in the Flow Channel

One goal of game design is to keep the player in the "flow channel." Here the player experiences the flow state continuously while both skill and difficulty gradually increase. But it's usually not a straight line centered in the flow channel. It's more a zig-zag with alternating rewards of easier wins and greater challenges. These match up to Dan Pink's "mastery" and "purpose" motivators.

The Zone of Proximal Development

Russian psychologist Lev Vygotsky defined the Zone of Proximal Development (ZPD) as the area between the tasks that a learner can do unaided and the tasks a learner cannot do at all. So, tasks in the ZPD are those that the learner can do with assistance – with scaffolding. Vygotsky claims that all learning occurs within the ZPD.

Figure 3: Zone of Proximal Development (ZPD)

There are many ways to put this into practice. For example, the Lexile Framework and other text complexity measures allow the matching of reading materials to the student's reading ability. Texts that are close to students' abilities increase confidence while more challenging texts increase skill levels. Texts that are too easy (boring) or too hard (anxiety producing) can be avoided. Mathematics is a structured subject where concepts build upon each other. Therefore, concepts in a student's ZPD are those that build on concepts the student already understands.

Integration

The Flow Channel offers a model to maximize player engagement and enjoyment. The Zone of Proximal Development is a model for optimizing learning productivity. Similarity between the two isn't surprising. Csikszentmihalyi studied and built on Vygotsky's work.

One of the most important things that educators can learn from Flow is that boredom contributes just as much to frustration as anxiety does. Conventional schooling does a lot of redundant work to ensure that most students "get" each concept. The boredom that results from such redundancy means that students rarely experience Flow in their schoolwork. It's also inefficient because students spend a lot of time below their ZPD in which case they aren't learning. Staying within the Flow Channel/ZPD can ensure that effective learning occurs and simultaneously keep the student motivated and rewarded.

It brings whole new meaning to, "Go with the Flow!"

As We May Teach

"My education was very similar to that of my parents. Theirs didn't differ a lot from my grandparents'. My children's schooling has been enhanced by media, word processing and the internet but the experience isn't fundamentally different from my own. They still go to a classroom, sit at relatively small desks and try to pay attention to a teacher in front of a board. The transformation of primary and secondary education in the United States is beginning now and will be well underway within a decade."

Four years ago I wrote the essay, "As We May Teach" to the Meridian School board of trustees where I was serving. I recently re-read the essay and it's just as relevant today as it was then; so I've posted it here.

The examples I used remain valid but we now have many more. Since then, "Blended Learning" has emerged as the term to describe the Cirrus High School experience I use in the essay. Rather than attempt to list the numerous new examples, I recommend you check my colleague, Scott Benson's "Running List of Blended Learning Resources."

In the essay I reference Clayton Christen's prediction that 5% of high school teaching would be online by 2012 and 50% by 2018. The 2012 edition of iNACOL's "Keeping Pace with K-12 Online and Blended Learning" report estimates that 5% of US K-12 students are taking part in at least one online course. So, four years after the prediction we seem to be on track.

I've elaborated on several themes from the essay on this blog:

• Personalized Learning
• Feedback
• Bloom's Two Sigma Problem
• Student Motivation

Unique to this essay is the application of Business Process Automation to the education space. It's a useful lens that's compatible with other approaches to educational improvement.

Learning from Data - An Automotive Example

Monday saw me driving 800 miles home from a family Thanksgiving celebration. Due to my wife's change in plans, my only companion for the drive was our small dog (who had a narrow escape the day before). I needed something to keep my attention. So I decided to perform an experiment in data collection. I learned a lot even from a small data sample.

The vehicle I was driving was a 2010 Subaru Forester. Some friends have the same vehicle and have been pleased with getting around 27 MPG on the highway. We typically get only 23-24 MPG on the highway and I had been wondering why.

Among the features of this car is an average gas mileage display that's tied to the trip odometer. So, sampling the gas mileage is as simple as setting the cruise control, resetting the trip odometer, driving a set distance and reading out the result. As I was crossing the relatively flat plains of Idaho (speed limit 75) this seemed to be a good opportunity to gather some data.

Over a period of several hours, I took a bunch of samples following the above method and using my GPS to track altitude changes. I abandoned samples where the altitude change was more than a few hundred feet. The result is 27 good samples. I've posted the raw data here in case you want to play with them. Most of the samples are for 20 mile segments but some are as long as 40 and some are as short as 5 miles.

As you can imagine, the lower-speed samples got a bit tedious. But I was curious enough that I even took a side trip on a remote road (off the freeway) to get samples below 45 MPH. There's considerable variability in those results as you can see in the plot below. Halfway through the trip I refilled with fuel. I switched from regular (87 octane) to premium (92 octane) to see how that might affect mileage.

2010 Subaru Forester Fuel Economy vs. Speed

The results certainly aren't what I expected. EPA city and highway ratings have always led me to expect relatively flat miles per gallon with city being much poorer due to stop-and-go driving. Instead, I got a nearly linear downward slope. Using Excel's curve-fitting feature and found that a polynomial curve worked better than a line. The formula is embedded in the graph above.

More data points would be required to really validate this curve but it certainly fits within the margin of error of my samples. Therefore we can make some cost estimates using this formula. Notable is that peak economy is between 40 and 45 MPH – much slower than I had expected. From this I was able calculate the cost of each hour saved on this long drive.

Here's a table of results for a 800 mile drive in a 2010 Subaru Forester with average fuel cost of $3.759 per gallon. Time saved and additional cost are from a baseline speed of 55 MPH. Note that the distance is cancelled out in the Cost Per Hour Saved so those numbers are accurate regardless of the length of the trip.

Speed	MPG	FuelCost	Time	Hrs Saved	Addl Cost	Cost Per Hr Saved
55	33.1	$90.83	14.55
60	32.1	$93.71	13:33	1.21	$2.89	$2.38
65	30.7	$97.88	12.31	2.24	$7.05	$3.15
70	29.0	$103.65	11.43	3.12	$12.82	$4.11
75	27.0	$111.55	10.67	3.88	$20.72	$5.34
80	24.6	$122.45	10.00	4.55	$31.62	$6.96
84	22.4	$134.31	9.52	5.02	$43.48	$8.66

Here are a few things I've learned from this:

• My friend with the other Forester drives slower on the highway than I do.
• I had not known how sensitive vehicle gas mileage is to speed.
• Everything I have read led me to expect no benefit from higher octane fuel once the vehicle's requirements have been met. In the case of the Forester, higher octane actually reduced fuel economy. This observation is confirmed by the official EPA ratings.
• I would love to see tables like the one above before purchasing my next car.

I learned a lot from this tiny data sample and my future driving habits will be changed accordingly. Now imagine what we could learn if there were a large public database of fuel economy data. Car manufacturers could optimize for specific driving patterns. Consumers would be better informed about fuel economies to expect. There are fleet tracking devices like this one that are already reporting that data but it's locked up in private databases. If anonymous fuel economy data (speed, distance, altitude and MPG) were released there's a lot we could learn about fuel economy under a variety of conditions.

I can't wrap this post up without relating it to education. A relatively small data sample taught me a lot and will impact my future driving behavior. In the same way, it doesn't take a lot of data fed back to students and teachers before they see opportunities to improve. And when we grow from little data to big data, revolutionary changes are on the horizon.

A Post-LMS Framework for Personalized Learning

In the last few weeks I presented at iNACOL VSS and attended Educause. I’ve also met with the Shared Learning Collaborative team and the CEDS Stakeholders group. Educause included meetings with the Next Generation Learning Challenges organizers and grantees. All in all it’s been a concentrated opportunity to meet with vendors, standards developers and visionaries in the personalized, blended and online learning spaces.

There’s a new pattern emerging on how the technical components of a personalized learning system fit together and it’s a departure from the past. This model seems to apply both in K-12 and postsecondary education.

This new framework is being driven by three trends:

• Innovative creators of courseware and learning systems need greater control over the learning environment than can be achieved in a Learning Management System (LMS).
• Student Information Systems (SIS) and Portals are taking over responsibility for student/teacher communications, gradebooks and consolidating analytics into student and teacher dashboards.
• Students and Teachers are seeking a coherent and seamless experience without separate credentials and logins for each of the systems they use.

A New Framework

The figure below shows the interaction of three systems. Each system may be hosted by a different provider but they’re integrated in such a way that the student should browse between them seamlessly.

The Student Information System (SIS) is generally integrated into the school’s portal. This is the site a student browses for consolidated access to all school information. It’s provided and managed by the school. The portal links to courses in which the student is enrolled.

In this new model, courses are an integrated experience delivered by learning systems custom-adapted to the subject matter. At a basic level, a course is a sequence of learning and assessment activities such as exposition (video, audio, text), virtual labs, exercises, quizzes exams and so forth. Key to personalization is that the selection and order of activities is adapted according to individual student needs.

Traditionally, the same learning system that hosts the course also hosts the activities. This is reasonably simple with conventional media types such as text and video. It gets more complicated with interactive media and assessments. The most innovative learning activities may be separately hosted because they are supported by custom services. These could include interactive labs, intelligent tutoring systems, virtual worlds and games.

Conspicuously absent in this new model is the Learning Management System (LMS). For the last decade or so, the framework has been that schools select and deploy an LMS – ideally with single sign-on and data integration with their SIS – but all too often as an independent system. The idea was that courseware publishers and instructional designers would install the course materials into the LMS using content packaging formats like SCORM and Common Cartridge. But this hasn't happened very much – especially with the most innovative courses. Cutting edge learning systems like DreamBox, Aleks or Read 180 can’t be packaged up and installed into an LMS. The environment is too constraining.

While LMSs are capable of much more, most actual LMS use is in support of teacher-student and student-student communications, not for delivery of instruction. And that communication function is being taken over by the SIS and portal. Contemporary SIS systems have expanded beyond enrollment and course-level data to include full gradebook functionality. Meanwhile, portals are including teacher and student dashboards, online forums, chatrooms and other communication features.

So the new model is composed of Portal/SIS, Learning Systems and Activities often supplied by different organizations. And it’s not just three systems that need to be integrated. A single school will likely have many learning systems. A single student is likely to use different learning systems for different subjects. And a single course may integrate activities from a variety of sources.

Student ID

In order for the student and teacher experiences to be coherent there needs to be a clean handoff between these systems. In the diagram I've shown this as Student ID flowing to the right and Student Data flowing both ways. Student ID may include authentication, authorization and/or provisioning.

• Authentication, often provided by Single Sign-On (SSO) is the real-time indication of who the student is.
• Authorization is a real-time assertion that the student should be granted access to a system or resource.
• Provisioning is the transfer of teacher and student enrollment data so that a learning system or activity can grant access and coordinate a cohort of teachers and students. This may on-demand (coordinated with authentication or authorization) or it may be a periodic batch update.

Depending on features of each component, these work together in different ways. For example, an SIS may transfer provisioning data to a learning system. Then, at runtime the SIS uses an SSO protocol to authenticate the student to the learning system. At this point the learning system knows the identity of the student and the provisioning of the classes, therefore it can internally decide whether to authorize access.

On the other hand, the learning system may use an authorization protocol to grant student access to a learning activity without authentication or provisioning. In this case, the activity provider doesn't know the identity of the student, it only knows that a trusted agent (the school) has indicated that the student should be granted access.

Student ID protocols can transfer three levels of information depending on the needs of the systems:

• Personally Identifiable Information (PII): This might include the students name, grade, enrollment information and so forth. It's sensitive information governed by FERPA regulations.
• Persistent Identifier: This is just enough information that a learning system or activity can identify repeat visitors. The system doesn't have any personal information about the student but knows this is the same student as in a previous visit.
• Authorization Ticket: This is just a trusted indication that a student should be able to access content. The learning system or activity is not assisted in coordinating repeat visits.

Student Data

Most of the student data flow is upstream as student activities and performance are reported to the Learning System and the SIS/Portal. Traditionally that data has been simple scores and grades. But systems are beginning to collect richer information like frequency of access, time on task and clickstream data. these are used in analytics such as student and teacher dashboards. This same data can also be reported downstream for the use of adaptive learning systems and custom activities.

Protocols

The difficulty is that there isn't much consistency in the protocols used for Student ID or Student Data. To their credit, the builders of SISs, Learning Systems and Tools all have APIs for integration with other systems. But in most cases APIs are custom to the application. And upstream systems aren't necessarily prepared to collect the rich data that downstream systems are prepared to share.

Here's a survey of what is available or under development:

SAML and OAuth are two commonly-used protocols for authentication and authorization. The SSO subset of SAML has become common due to its use by Google Apps. OAuth is an authorization protocol that can optionally carry personal information or a persistent ID according to needs. Shibboleth is an open source reference implementation of SAML.

IMS Learning Tools Interoperability (LTI) supports the interaction of Learning Systems and Activities. It incorporates OAuth for the authorization step. LTI v1.0 (also known as Basic LTI or BLTI) coordinates the authorization of the activity (called a Learning Tool) seamlessly embedding it in the Learning System. Later versions of LTI support reporting of simple student performance data. Most mainstream LMSs support LTI 1.0 or better.

LearnSprout and Clever are two companies supporting data integration with SISs. This allows builders of Learning Systems to write to one API (either LearnSprout's or Clever's) and gain integration into a number of prominent SISs. However, they are limited to the data types supported by the SIS.

The Shared Learning Collaborative (SLC) is building a web-scale common student data layer that can be used by the SIS, Portal, Analytics, Dashboard, Learning System and Activities. A rich set of data types is pre-supported and applications can store custom data for persistence and sharing. It also supplies a common student identity framework including authentication services. So, in the SLC instead of handing off student data between systems, they all rely on the same underlying service.

The SLC Approach to the New Model

The new model divides the functions once concentrated in the LMS. Today, custom systems integration must be done to achieve a seamless experience. But protocols and services are under development that should simplify that in the future.

Video: Feedback Loops for More Effective and Personalized Learning

Last month I presented at the iNACOL VSS conference. I posted my slides and resources here.
I experimented with using a Bluetooth microphone and PowerPoint's recording feature to generate a voiceover video of the presentation.


I apologize in advance. The audio quality is fairly poor. It's especially bad at the beginning but improves later. I think I was near the range limit of the microphone. And PowerPoint's recording/video feature is still buggy. In a couple of slides, the sound drops out entirely.
Flaws aside, I'm pleased with how the subject came together. Quality feedback loops are a key component in personalized learning solutions. In researching this topic I found a lot of relevant research that can guide the development and selection of products.

Election Technology Update

It's election day in the U.S. and most of us are fatigued by the campaigns and will be relieved to have them over. Barring an electoral college anomaly there will be more voters who are pleased with the result than dismayed by it (it's a tautology).

I wrote about my misgivings with touchscreen direct entry voting systems in 2009. Things haven't improved since then. The big risk is indetectable vote manipulation. Of course all voting systems, whether electronic or paper, are subject to some form of manipulation. The key is to set up protocols so that manipulations leave evidence. For example, paper balloting systems often count the number of ballots cast and compare that with the number of ballots counted. The number of ballots cast is transmitted to the counting location through a different means from the transmission of the ballots themselves.

In 2010 I wrote about King County, Washington's vote-by-mail system. In addition to mailing ballots, voters can deliver them to dropboxes conveniently located around the county. Not only do they save postage, dropboxes appear to be a more secure delivery method as observers from both major parties watch the sealing and collection of ballot boxes. Other observers watch the opening and counting processes.

As a paper and optical scan method, King County's is among the more secure – once the ballot is delivered to a dropbox. The glaring weakness is privacy. Vote-by-mail opens the door to voter coercion because there's no inspector and booth to ensure privacy when the vote is actually cast. It's entirely possible for others to pre-fill the ballot and simply ask the voter to sign – with intimidation if necessary.

Of course, manipulation of this sort doesn't scale well. Sure it can happen in pockets but widespread, coordinated vote manipulation would be hard to achieve as the more voters are intimidated, the greater the likelihood that someone complains. Therefore it's reasonable to assume that deliberate manipulation will be a small fraction of total votes cast.

This leads to an interesting conclusion: Though we aspire to make every vote count, there's some degree of error regardless of the way votes are cast and counted. Sometimes it's deliberate fraud, manipulation or intimidation. Sometimes it's poorly designed ballots, miscalibrated voting machines or natural disasters. There are two ways to deal with this. Our current system presumes that if the difference in votes is within the margin of error, democracy is preserved regardless of which candidate takes office. That presumption was tested in the 2000 U.S. election.

The alternative is to require another election if the vote is within the margin of error. That approach carries a tremendous cost in terms of time, money and extended uncertainty. Despite misgivings, I have to agree with those who wrote the constitution. I may not like the outcome when the vote is close. I may even believe that the count is inaccurate. But I do believe that Democracy is preserved.

Feedback Loops for More Effective and Personalized Learning

Today I'm presenting at the iNACOL Virtual Schools Symposium on "Feedback Loops for More Effective and Personalized Learning." Here's the slide deck and resources related to that presentation:

• Slide Deck
• The Shared Learning Collaborative
• Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement, John Hattie
• "The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring". Benjamin Bloom
• “The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems and Other Tutoring Systems”, Kurt VanLehn
• The Open Learning Initiative
• The United Way Common Good Forecaster
• The National Center for Education Statistics

Related posts on this blog:

• The Perverse Vocabulary of Feedback Loops
• Motivating Students - Opportunity Isn't Enough
• Learning - Everything Works, But How Well?
• Education by the Numbers
• Things Every Education Tech Entrepreneur Should Know

Thanks to everyone who came and to all those whose research inspired this presentation.

Learning Maps, Common IDs and the Common Core

Today we presented at the iNACOL VSS conference on "Learning Maps, Common IDs and the Common Core. Here are primary resources associated with that presentation:

• The Slides from the Presentation
• Common Core State Standards
• Learning Resource Metadata Initiative
• Learning Registry
• Shared Learning Collaborative

Update: 7:55pm: In addition to the above primary sources, I've written the following on the same subject:

• Common Identifiers for the Common Core
• Schema.org, LRMI and the Learning Registry
• CEDS and the Four-Layer Framework for Data Standards
• Tips for the Common Core XML

Also I corrected the link to LearningRegistry.org.

Many thanks to the panelists: Maureen Wentworth, Michael Jay and Sharren Bates.