Bloom's Two Sigma Problem Revisited

Benjamin Bloom's Two Sigma Problem has been both a guiding framework and a challenge to educators for more than a quarter century. A bit more than a year ago I wrote about the problem and some of the ways people are approaching it.

Here's the concise version: Bloom and some of his grad students compared classroom teaching with 1:1 tutoring. In both cases they used a mastery-based curriculum. The tutored students performed two standard deviations (two sigmas) better than their conventionally taught peers. While it would be nice to have a 1:1 student:teacher ratio, Bloom acknowledged that it's not practical and he proceeded to research ways to achieve similar results using more scalable means. He published the study in 1984. Since then, the Two Sigma Problem has served as a benchmark of how well students can learn if given the right supports.

A recent meta-study by Kurt VanLehn of Arizona State University compares no tutoring (conventional classroom), computer-based Intelligent Tutoring Systems (ITS), and human tutoring. VanLehn notes that a number of well-known ITS efforts have shown one-sigma improvements over conventional instruction. So, the conventional hypothesis is that computer tutors achieve one-sigma gains while human tutors achieve two-sigma gains as compared to conventional instruction.

VanLehn set out to test that hypothesis. He selected numerous studies that collectively yielded more than 100 comparisons between conventional instruction, three forms of ITS, and human tutoring. The result is surprising: answer-based ITS achieved an improvement of 0.31 sigma over conventional instruction. Step-based ITS achieved 0.75 sigma and human tutors achieved 0.79 sigma.

This is mixed news. On the one hand, the best computer tutors are almost as good as human tutors. That suggests that we can scale up much more effective learning than is achieved in conventional classrooms. On the other hand, VanLehn found no replication of Bloom's 2 sigma results. Is Bloom's goal out of reach or is there another factor involved?

To find out, VanLehn retrieved the dissertations from Bloom's grad students that contributed to the more famous paper. One key experiment yielded an effect size of 1.95 sigma – the probable source of Bloom's Two Sigma challenge. In that experiment both the conventional classroom and the tutors used a mastery learning technique. Whether in class or being tutored, students took a quiz after studying each unit. If their score achieved the mastery threshold, they advanced to the next unit. If not, they studied the unit more and were assessed again. This process was repeated until the mastery threshold was achieved.

The missing piece is that classroom students were required to achieve mastery threshold of 80% before advancing. Meanwhile, tutored students were required to achieve a threshold of 90%. Could it be that  adjusting the mastery threshold could account for a full standard deviation improvement in achievement? If so, numerous online learning systems should be tuned accordingly.

Oleg Bespalov and Karen Baldeschwieler, with their colleagues at New Charter University, have evidence to confirm this hypothesis. In their ITS system, students receive periodic formative assessments in the form of multiple-choice quizzes and self-graded short answer questions. From these assessments they calculate a "readiness score" to help students know when they're ready to advance. Students aren't constrained by the score – merely informed.

This creates a natural experiment in which they can compare student performance on the final exam against individual readiness scores. They discovered that students with a readiness above 90 achieved a 98% pass rate. But for those with a readiness score in the 81-90 range the pass rate dropped to 69%.

Both of these projects indicate that there's a critical threshold somewhere between 80% and 90%. Clearly this is an area deserving of more experimentation and research. But we can already tell that that tuning the mastery threshold is a critical factor for improving student achievement.

Measures of Effective Teaching

The Measures of Effective Teaching Project (MET) released its final reports last week. It got considerable press coverage as the study strives to inform teacher evaluation programs, a subject of considerable controversy.

Most of the stories, like this one from Reuters, focus on the the study's finding that teacher performance can indeed be predicted by performance measures. The best evaluations involve a weighted average of student test scores, teacher observations and student evaluations. Any one of these by itself is a much less accurate predictor.

There are nuances to this that can be gleaned from the project's Policy and Practitioner Brief:

• The different measures (student testing, teacher observation and student evaluation) have some overlap but mostly they measure different aspects of the teacher's skills.
• Different weightings are better predictors of different outcomes. Unsurprisingly, placing greater weight on test results is a better predictor of future student test results. However, equal weighting models or those that emphasize teacher observations are more reliable year over year.
• Effective teacher observations are more than a periodic visit from the principal. Evaluations require a consistent framework and procedure. The MET project used the Danielson Framework for Teaching as a rubric. The reliability of teacher observations is greatly improved by having at least two evaluators.
• When done properly, student evaluations are very reliable and an important component of teacher evaluation. As with observations, the key is to ask the right questions. The MET project used the Tripod Student Survey.
• The "value added" theory is supported. When student scores are compared with the previous year's performance (a value added score) the result is a more consistent predictor of future teacher performance than just the most recent year's scores.

One problem with exclusively using standardized tests to evaluate teachers or schools is that it's a blunt instrument. These tests offer a measure of performance but they offer limited guidance to a teacher or school on how they can improve. Sure, we can fire ineffective teachers and close ineffective schools. But using natural selection to improve schools is slow and costly not to mention cruel. Basically you're just hoping for those teachers and schools that randomly find the right formula for success.

Among the advantages of teacher observations and student evaluations are that they supply rich feedback to teachers to help them improve their practice. Another MET project report, Feedback for Better Teaching, offers guidelines for using feedback. They placed cameras in classrooms and observers codified the techniques used by the teachers. The same video recordings were used by the teachers themselves to observe their own performance – usually with an instructional coach. Processes like these can continuously improve teacher skills and effectiveness.

I've written before about how immediate feedback can help the student learn more effectively. In that context, it's no surprise that feedback to teachers helps them to be more effective. Moreover, it supports the passion that got them into teaching in the first place.

Enterprise-Scale is not Web-Scale

In the 90s, the IT world was talking about Enterprise-Scale. It's not that enterprise-scale was anything new. But up until then, the enterprise was the domain of mainframe and minicomputers. Upstart microcomputers – those with the whole processor on a chip – had previously not been capable of enterprise-scale operations.

It took a lot to achieve enterprise-scale with microcomputers. The leaders included Sun, Oracle and Cisco with Microsoft, Intel, Compaq and others playing fast-follower. They invented RAID arrays, symmetric multiprocessing, storage area networks, network load balancing and much more in the pursuit of five nines of reliability.

As microprocessor-based computers achieved enterprise-scale, pioneers like Google, Amazon, Yahoo!, SalesForce and others pushed right past enterprise into web-scale. User counts were measured in hundreds of millions, storage capacity was measured in petabytes and server arrays numbered in the thousands. Unlike enterprise-scale where key technologies had already been invented by the mainframe world, there wasn't any precedent for web-scale and the pioneers had to invent their own methods. I happened to work at Ancestry.com in the late 1990s/early 2000s and got to participate in some of that invention. But it wasn't until later in the decade before the pioneers started to share what they had learned and build products like Amazon Web Services, Google App Engine and Windows Azure to support the web-scale developer.

This is a important issue for education technology. The education industry is a bit behind the curve in moving to web-scale. For example, most learning management systems are built for enterprise-scale. They are intended to be installed on dedicated servers at a college or university's data center and they're architected to handle tens of thousands of students and teachers.

What happens when you move to the K-12 market or to community colleges? These organizations don't have the data centers and skilled staff needed to deploy, maintain and backup enterprise servers like these. In the past, their data systems only had to handle a few hundred or maybe a few thousand administrators. Teachers and students didn't directly access the district's data systems.

But districts are rapidly bringing all of their students and teachers online. And that means two orders of magnitude more users. Many districts have student counts numbering in the hundreds of thousands. Some get into the millions. And since their technology staffs are already overburdened, they seek hosted solutions, not enterprise-scale servers they have to manage themselves. Hosting a single district might not reach web-scale but a cost effective provider would serve hundreds of districts. And web-scale technologies can reduce the cost to something that districts can afford.

Here are some of the principles of web-scale architecture. For purchasers of products and services, these are the things you need to look for. For developers of those services, these are the principles you need to incorporate into your design.

Always Available
Web scale services use redundant servers to ensure that the service is always running – even during software upgrades and system maintenance. The term "24/7" was invented to describe services that have no weekly scheduled downtime. (And please don't write 24/7/365.)

Billions of Database Records
Today a district might keep a few dozen data points per student per year. In a district with 50,000 students that amounts to around 150,000 database rows per year. Eventually the database might grow to a few million rows total.

But personalized learning applications can collect thousands of data points per student per year. And an online service might serve a few million students. Thus, a web-scale learning service should be designed to store billions of data elements with provisions for orders of magnitude growth beyond that as clickstream data become more important.

Single Sign-On and Identity Management
Today's schools typically run a Student Information System, a Learning Management System and a few custom learning systems for specific subjects. Most of these applications have their own user database mandating separate logins and making requiring a lot of data entry to provision the sytems.

The low-hanging fruit is a single sign-on system that lets students and teachers use the same login account across all systems. But single sign-on is of limited value without automatic provisioning. So it's more important to have an identity management system that automatically shares demographic and enrollment information between the Student Information System and the various learning systems. The long-term need is to integrate the data among all of the systems so that all student performance data is accumulated in a common database.

Services-Integrated Security
Consider security in a Student Information System. The student, her teacher and her parents should have access to her school records, but no-one else (except maybe the principal or a counselor). Enterprise-scale security manages things like this through access control lists (ACLs). Record or element has an ACL granting certain levels of access to specific individuals. For example, the teacher has permission to view and change grades while the student and parent only have permission to view them.

The ACL approach becomes fragile at web scale. With millions of students and parents and thousands of changes to class enrollment it becomes difficult to maintain correct ACLs even when the process is automated. Roles and groups offer some relief but inevitably permission errors creep in and they become a technical support nightmare. Even worse, with regulations like FERPA in place, permission errors can result in significant liability.

Instead, web-scale applications use policy-based permissions. When a student is enrolled in a class, the policy says the teacher should be able to access that student's records. There's no ACL to be updated and permission naturally disappears if the student changes enrollment. The databases of these systems describe the relationships between elements (students, classes, teachers, parents, etc.) and the policies describe how permissions should be granted according to those relationships.

Services-integrated security also means that permissions are enforced at all levels of the system. The UI will control permissions that are offered to the user and the API enforces policy when read or write attempts are made. Thus a rogue or buggy application is still prevented from violating security policy.

Developer Note: Policy-based security can be processor and database intensive. The query to determine whether to permit a particular operation can easily be more expensive than the operation itself. This isn't a reason to reject the approach. Rather, use multiple levels of permissions caching to reduce the database burden.

Linear or Sub-Linear Cost Curves

If you graph number of users on the horizontal axis vs total cost on the vertical axis enterprise-scale systems have costs that grow exponentially. This is because they achieve scale by using progressively bigger and more complex servers and one 32-processor server costs many times more lot more than 32 single-processor servers.

In contrast, web-scale architectures have a linear or sub-linear cost curve. They achieve this feat by using software, database and hardware architectures that spread the load across many commodity-priced servers. As demand increases you simply add servers so variable costs are linear and fixed costs get diluted over a large number of users.

Developing scale-out software like this is complicated and expensive. Because of this, enterprise-scale architectures can cost less in enterprise-scale deployments. But when user counts get into the hundreds of thousands or millions, web-scale becomes more cost effective.

Mashups
Web-scale applications rarely stand alone. In most cases, they are combined with other applications to create a complete solution. This is certainly true of our vision for personalized education. A complete solution includes at least the following:

• Student Information System
• Student/Parent Portal
• Teacher Portal
• Adaptive Learning System (probably multiple subject-specific ones)
• Content Library
• Assessment Bank
• Analytics (Teacher, Department, Schools)
• Interactive Professional Development

Plus, other innovative applications are likely to emerge. In a realistic system, these components will originate from a variety of sources. For time-constrained teachers and students to use them effectively, they will have to be seamlessly mashedup together.

Tools and Techniques for Web-Scale

At Ancestry.com we had to invent many of the web-scale tools we used. But the toolkit has matured in the last few years. The easiest approach is to build on one of the cloud platforms like Windows Azure, Google AppEngine or Amazon Web Services. The downside is that doing so locks you into that vendor's hosting service.

Here are some of the other tools and techniques that help in web-scale development:

• Caching (lots of it and at different levels)
• NoSQL Databases
• Content Delivery Networks
• Network Load Balancing
• Browser-side Code
• Multi-tier Architecture

Ultimately, however, you can use all of these tools and still not have a web-scale application. You have to architect for web-scale from the very start.

Game Design and the Zone of Proximal Development

It's an experience many parents have had: From time to time my kids invite me to play video games with them. We pick a multiplayer game like MarioCart or Halo and they proceed to beat me silly. I keep trying to show moderately credible performance with little success. And I wonder, "How long would I need to play this game to achieve some degree of proficiency?"

Part of the problem is that parents like me jump right into the multiplayer mode competing against our kids while simultaneously trying to learn the game mechanics. Most of these games have a single-player "campaign" mode. The campaign is designed both to be consistently entertaining and to step-by-step make you better at playing the game. The two primary rewards of the campaign are progressive discovery of the story (narrative) and progressive mastery of new skills.

The Flow Channel in Game Design

In his book, The Art of Game Design, Jesse Schell describes "flow", a concept he adopted from psychologist Mihalyi Csikszentmihalyi. An individual is in a "flow state" when he or she is entirely involved in the task. "The rest of the world seems to fall away and we have no intrusive thoughts." It's a state of sustained focus and enjoyment.

Figure 1: Flow Channel

In figure 1 we see four player states on a graph of player skill vs game challenges. This can apply to a variety of games from physical sports to puzzles to first-person shooters. In this example a player is at state P₁ where the player's skill is balanced with the challenge of playing. With practice, the player increases in skill and advances to state P₂ in which the game becomes easy enough that the player is bored. P₃ represents condition when the game is too difficult; perhaps the opponent is a lot more skilled or the game is too hard. In this case the player becomes anxious about their performance. Both states P₂ and P₃ can be rebalanced. The game can become more challenging (P₂ to P₄) or the player can gain skill (P₃ to P₄). But if the player remains in Anxiety or Boredom for very long they'll abandon the game because both anxiety and boredom lead to frustration.

Figure 2: Growth Path in the Flow Channel

One goal of game design is to keep the player in the "flow channel." Here the player experiences the flow state continuously while both skill and difficulty gradually increase. But it's usually not a straight line centered in the flow channel. It's more a zig-zag with alternating rewards of easier wins and greater challenges. These match up to Dan Pink's "mastery" and "purpose" motivators.

The Zone of Proximal Development

Russian psychologist Lev Vygotsky defined the Zone of Proximal Development (ZPD) as the area between the tasks that a learner can do unaided and the tasks a learner cannot do at all. So, tasks in the ZPD are those that the learner can do with assistance – with scaffolding. Vygotsky claims that all learning occurs within the ZPD.

Figure 3: Zone of Proximal Development (ZPD)

There are many ways to put this into practice. For example, the Lexile Framework and other text complexity measures allow the matching of reading materials to the student's reading ability. Texts that are close to students' abilities increase confidence while more challenging texts increase skill levels. Texts that are too easy (boring) or too hard (anxiety producing) can be avoided. Mathematics is a structured subject where concepts build upon each other. Therefore, concepts in a student's ZPD are those that build on concepts the student already understands.

Integration

The Flow Channel offers a model to maximize player engagement and enjoyment. The Zone of Proximal Development is a model for optimizing learning productivity. Similarity between the two isn't surprising. Csikszentmihalyi studied and built on Vygotsky's work.

One of the most important things that educators can learn from Flow is that boredom contributes just as much to frustration as anxiety does. Conventional schooling does a lot of redundant work to ensure that most students "get" each concept. The boredom that results from such redundancy means that students rarely experience Flow in their schoolwork. It's also inefficient because students spend a lot of time below their ZPD in which case they aren't learning. Staying within the Flow Channel/ZPD can ensure that effective learning occurs and simultaneously keep the student motivated and rewarded.

It brings whole new meaning to, "Go with the Flow!"

As We May Teach

"My education was very similar to that of my parents. Theirs didn't differ a lot from my grandparents'. My children's schooling has been enhanced by media, word processing and the internet but the experience isn't fundamentally different from my own. They still go to a classroom, sit at relatively small desks and try to pay attention to a teacher in front of a board. The transformation of primary and secondary education in the United States is beginning now and will be well underway within a decade."

Four years ago I wrote the essay, "As We May Teach" to the Meridian School board of trustees where I was serving. I recently re-read the essay and it's just as relevant today as it was then; so I've posted it here.

The examples I used remain valid but we now have many more. Since then, "Blended Learning" has emerged as the term to describe the Cirrus High School experience I use in the essay. Rather than attempt to list the numerous new examples, I recommend you check my colleague, Scott Benson's "Running List of Blended Learning Resources."

In the essay I reference Clayton Christen's prediction that 5% of high school teaching would be online by 2012 and 50% by 2018. The 2012 edition of iNACOL's "Keeping Pace with K-12 Online and Blended Learning" report estimates that 5% of US K-12 students are taking part in at least one online course. So, four years after the prediction we seem to be on track.

I've elaborated on several themes from the essay on this blog:

• Personalized Learning
• Feedback
• Bloom's Two Sigma Problem
• Student Motivation

Unique to this essay is the application of Business Process Automation to the education space. It's a useful lens that's compatible with other approaches to educational improvement.

Learning from Data - An Automotive Example

Monday saw me driving 800 miles home from a family Thanksgiving celebration. Due to my wife's change in plans, my only companion for the drive was our small dog (who had a narrow escape the day before). I needed something to keep my attention. So I decided to perform an experiment in data collection. I learned a lot even from a small data sample.

The vehicle I was driving was a 2010 Subaru Forester. Some friends have the same vehicle and have been pleased with getting around 27 MPG on the highway. We typically get only 23-24 MPG on the highway and I had been wondering why.

Among the features of this car is an average gas mileage display that's tied to the trip odometer. So, sampling the gas mileage is as simple as setting the cruise control, resetting the trip odometer, driving a set distance and reading out the result. As I was crossing the relatively flat plains of Idaho (speed limit 75) this seemed to be a good opportunity to gather some data.

Over a period of several hours, I took a bunch of samples following the above method and using my GPS to track altitude changes. I abandoned samples where the altitude change was more than a few hundred feet. The result is 27 good samples. I've posted the raw data here in case you want to play with them. Most of the samples are for 20 mile segments but some are as long as 40 and some are as short as 5 miles.

As you can imagine, the lower-speed samples got a bit tedious. But I was curious enough that I even took a side trip on a remote road (off the freeway) to get samples below 45 MPH. There's considerable variability in those results as you can see in the plot below. Halfway through the trip I refilled with fuel. I switched from regular (87 octane) to premium (92 octane) to see how that might affect mileage.

2010 Subaru Forester Fuel Economy vs. Speed

The results certainly aren't what I expected. EPA city and highway ratings have always led me to expect relatively flat miles per gallon with city being much poorer due to stop-and-go driving. Instead, I got a nearly linear downward slope. Using Excel's curve-fitting feature and found that a polynomial curve worked better than a line. The formula is embedded in the graph above.

More data points would be required to really validate this curve but it certainly fits within the margin of error of my samples. Therefore we can make some cost estimates using this formula. Notable is that peak economy is between 40 and 45 MPH – much slower than I had expected. From this I was able calculate the cost of each hour saved on this long drive.

Here's a table of results for a 800 mile drive in a 2010 Subaru Forester with average fuel cost of $3.759 per gallon. Time saved and additional cost are from a baseline speed of 55 MPH. Note that the distance is cancelled out in the Cost Per Hour Saved so those numbers are accurate regardless of the length of the trip.

Speed	MPG	FuelCost	Time	Hrs Saved	Addl Cost	Cost Per Hr Saved
55	33.1	$90.83	14.55
60	32.1	$93.71	13:33	1.21	$2.89	$2.38
65	30.7	$97.88	12.31	2.24	$7.05	$3.15
70	29.0	$103.65	11.43	3.12	$12.82	$4.11
75	27.0	$111.55	10.67	3.88	$20.72	$5.34
80	24.6	$122.45	10.00	4.55	$31.62	$6.96
84	22.4	$134.31	9.52	5.02	$43.48	$8.66

Here are a few things I've learned from this:

• My friend with the other Forester drives slower on the highway than I do.
• I had not known how sensitive vehicle gas mileage is to speed.
• Everything I have read led me to expect no benefit from higher octane fuel once the vehicle's requirements have been met. In the case of the Forester, higher octane actually reduced fuel economy. This observation is confirmed by the official EPA ratings.
• I would love to see tables like the one above before purchasing my next car.

I learned a lot from this tiny data sample and my future driving habits will be changed accordingly. Now imagine what we could learn if there were a large public database of fuel economy data. Car manufacturers could optimize for specific driving patterns. Consumers would be better informed about fuel economies to expect. There are fleet tracking devices like this one that are already reporting that data but it's locked up in private databases. If anonymous fuel economy data (speed, distance, altitude and MPG) were released there's a lot we could learn about fuel economy under a variety of conditions.

I can't wrap this post up without relating it to education. A relatively small data sample taught me a lot and will impact my future driving behavior. In the same way, it doesn't take a lot of data fed back to students and teachers before they see opportunities to improve. And when we grow from little data to big data, revolutionary changes are on the horizon.

A Post-LMS Framework for Personalized Learning

In the last few weeks I presented at iNACOL VSS and attended Educause. I’ve also met with the Shared Learning Collaborative team and the CEDS Stakeholders group. Educause included meetings with the Next Generation Learning Challenges organizers and grantees. All in all it’s been a concentrated opportunity to meet with vendors, standards developers and visionaries in the personalized, blended and online learning spaces.

There’s a new pattern emerging on how the technical components of a personalized learning system fit together and it’s a departure from the past. This model seems to apply both in K-12 and postsecondary education.

This new framework is being driven by three trends:

• Innovative creators of courseware and learning systems need greater control over the learning environment than can be achieved in a Learning Management System (LMS).
• Student Information Systems (SIS) and Portals are taking over responsibility for student/teacher communications, gradebooks and consolidating analytics into student and teacher dashboards.
• Students and Teachers are seeking a coherent and seamless experience without separate credentials and logins for each of the systems they use.

A New Framework

The figure below shows the interaction of three systems. Each system may be hosted by a different provider but they’re integrated in such a way that the student should browse between them seamlessly.

The Student Information System (SIS) is generally integrated into the school’s portal. This is the site a student browses for consolidated access to all school information. It’s provided and managed by the school. The portal links to courses in which the student is enrolled.

In this new model, courses are an integrated experience delivered by learning systems custom-adapted to the subject matter. At a basic level, a course is a sequence of learning and assessment activities such as exposition (video, audio, text), virtual labs, exercises, quizzes exams and so forth. Key to personalization is that the selection and order of activities is adapted according to individual student needs.

Traditionally, the same learning system that hosts the course also hosts the activities. This is reasonably simple with conventional media types such as text and video. It gets more complicated with interactive media and assessments. The most innovative learning activities may be separately hosted because they are supported by custom services. These could include interactive labs, intelligent tutoring systems, virtual worlds and games.

Conspicuously absent in this new model is the Learning Management System (LMS). For the last decade or so, the framework has been that schools select and deploy an LMS – ideally with single sign-on and data integration with their SIS – but all too often as an independent system. The idea was that courseware publishers and instructional designers would install the course materials into the LMS using content packaging formats like SCORM and Common Cartridge. But this hasn't happened very much – especially with the most innovative courses. Cutting edge learning systems like DreamBox, Aleks or Read 180 can’t be packaged up and installed into an LMS. The environment is too constraining.

While LMSs are capable of much more, most actual LMS use is in support of teacher-student and student-student communications, not for delivery of instruction. And that communication function is being taken over by the SIS and portal. Contemporary SIS systems have expanded beyond enrollment and course-level data to include full gradebook functionality. Meanwhile, portals are including teacher and student dashboards, online forums, chatrooms and other communication features.

So the new model is composed of Portal/SIS, Learning Systems and Activities often supplied by different organizations. And it’s not just three systems that need to be integrated. A single school will likely have many learning systems. A single student is likely to use different learning systems for different subjects. And a single course may integrate activities from a variety of sources.

Student ID

In order for the student and teacher experiences to be coherent there needs to be a clean handoff between these systems. In the diagram I've shown this as Student ID flowing to the right and Student Data flowing both ways. Student ID may include authentication, authorization and/or provisioning.

• Authentication, often provided by Single Sign-On (SSO) is the real-time indication of who the student is.
• Authorization is a real-time assertion that the student should be granted access to a system or resource.
• Provisioning is the transfer of teacher and student enrollment data so that a learning system or activity can grant access and coordinate a cohort of teachers and students. This may on-demand (coordinated with authentication or authorization) or it may be a periodic batch update.

Depending on features of each component, these work together in different ways. For example, an SIS may transfer provisioning data to a learning system. Then, at runtime the SIS uses an SSO protocol to authenticate the student to the learning system. At this point the learning system knows the identity of the student and the provisioning of the classes, therefore it can internally decide whether to authorize access.

On the other hand, the learning system may use an authorization protocol to grant student access to a learning activity without authentication or provisioning. In this case, the activity provider doesn't know the identity of the student, it only knows that a trusted agent (the school) has indicated that the student should be granted access.

Student ID protocols can transfer three levels of information depending on the needs of the systems:

• Personally Identifiable Information (PII): This might include the students name, grade, enrollment information and so forth. It's sensitive information governed by FERPA regulations.
• Persistent Identifier: This is just enough information that a learning system or activity can identify repeat visitors. The system doesn't have any personal information about the student but knows this is the same student as in a previous visit.
• Authorization Ticket: This is just a trusted indication that a student should be able to access content. The learning system or activity is not assisted in coordinating repeat visits.

Student Data

Most of the student data flow is upstream as student activities and performance are reported to the Learning System and the SIS/Portal. Traditionally that data has been simple scores and grades. But systems are beginning to collect richer information like frequency of access, time on task and clickstream data. these are used in analytics such as student and teacher dashboards. This same data can also be reported downstream for the use of adaptive learning systems and custom activities.

Protocols

The difficulty is that there isn't much consistency in the protocols used for Student ID or Student Data. To their credit, the builders of SISs, Learning Systems and Tools all have APIs for integration with other systems. But in most cases APIs are custom to the application. And upstream systems aren't necessarily prepared to collect the rich data that downstream systems are prepared to share.

Here's a survey of what is available or under development:

SAML and OAuth are two commonly-used protocols for authentication and authorization. The SSO subset of SAML has become common due to its use by Google Apps. OAuth is an authorization protocol that can optionally carry personal information or a persistent ID according to needs. Shibboleth is an open source reference implementation of SAML.

IMS Learning Tools Interoperability (LTI) supports the interaction of Learning Systems and Activities. It incorporates OAuth for the authorization step. LTI v1.0 (also known as Basic LTI or BLTI) coordinates the authorization of the activity (called a Learning Tool) seamlessly embedding it in the Learning System. Later versions of LTI support reporting of simple student performance data. Most mainstream LMSs support LTI 1.0 or better.

LearnSprout and Clever are two companies supporting data integration with SISs. This allows builders of Learning Systems to write to one API (either LearnSprout's or Clever's) and gain integration into a number of prominent SISs. However, they are limited to the data types supported by the SIS.

The Shared Learning Collaborative (SLC) is building a web-scale common student data layer that can be used by the SIS, Portal, Analytics, Dashboard, Learning System and Activities. A rich set of data types is pre-supported and applications can store custom data for persistence and sharing. It also supplies a common student identity framework including authentication services. So, in the SLC instead of handing off student data between systems, they all rely on the same underlying service.

The SLC Approach to the New Model

The new model divides the functions once concentrated in the LMS. Today, custom systems integration must be done to achieve a seamless experience. But protocols and services are under development that should simplify that in the future.

Video: Feedback Loops for More Effective and Personalized Learning

Last month I presented at the iNACOL VSS conference. I posted my slides and resources here.
I experimented with using a Bluetooth microphone and PowerPoint's recording feature to generate a voiceover video of the presentation.


I apologize in advance. The audio quality is fairly poor. It's especially bad at the beginning but improves later. I think I was near the range limit of the microphone. And PowerPoint's recording/video feature is still buggy. In a couple of slides, the sound drops out entirely.
Flaws aside, I'm pleased with how the subject came together. Quality feedback loops are a key component in personalized learning solutions. In researching this topic I found a lot of relevant research that can guide the development and selection of products.

Election Technology Update

It's election day in the U.S. and most of us are fatigued by the campaigns and will be relieved to have them over. Barring an electoral college anomaly there will be more voters who are pleased with the result than dismayed by it (it's a tautology).

I wrote about my misgivings with touchscreen direct entry voting systems in 2009. Things haven't improved since then. The big risk is indetectable vote manipulation. Of course all voting systems, whether electronic or paper, are subject to some form of manipulation. The key is to set up protocols so that manipulations leave evidence. For example, paper balloting systems often count the number of ballots cast and compare that with the number of ballots counted. The number of ballots cast is transmitted to the counting location through a different means from the transmission of the ballots themselves.

In 2010 I wrote about King County, Washington's vote-by-mail system. In addition to mailing ballots, voters can deliver them to dropboxes conveniently located around the county. Not only do they save postage, dropboxes appear to be a more secure delivery method as observers from both major parties watch the sealing and collection of ballot boxes. Other observers watch the opening and counting processes.

As a paper and optical scan method, King County's is among the more secure – once the ballot is delivered to a dropbox. The glaring weakness is privacy. Vote-by-mail opens the door to voter coercion because there's no inspector and booth to ensure privacy when the vote is actually cast. It's entirely possible for others to pre-fill the ballot and simply ask the voter to sign – with intimidation if necessary.

Of course, manipulation of this sort doesn't scale well. Sure it can happen in pockets but widespread, coordinated vote manipulation would be hard to achieve as the more voters are intimidated, the greater the likelihood that someone complains. Therefore it's reasonable to assume that deliberate manipulation will be a small fraction of total votes cast.

This leads to an interesting conclusion: Though we aspire to make every vote count, there's some degree of error regardless of the way votes are cast and counted. Sometimes it's deliberate fraud, manipulation or intimidation. Sometimes it's poorly designed ballots, miscalibrated voting machines or natural disasters. There are two ways to deal with this. Our current system presumes that if the difference in votes is within the margin of error, democracy is preserved regardless of which candidate takes office. That presumption was tested in the 2000 U.S. election.

The alternative is to require another election if the vote is within the margin of error. That approach carries a tremendous cost in terms of time, money and extended uncertainty. Despite misgivings, I have to agree with those who wrote the constitution. I may not like the outcome when the vote is close. I may even believe that the count is inaccurate. But I do believe that Democracy is preserved.

Feedback Loops for More Effective and Personalized Learning

Today I'm presenting at the iNACOL Virtual Schools Symposium on "Feedback Loops for More Effective and Personalized Learning." Here's the slide deck and resources related to that presentation:

• Slide Deck
• The Shared Learning Collaborative
• Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement, John Hattie
• "The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring". Benjamin Bloom
• “The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems and Other Tutoring Systems”, Kurt VanLehn
• The Open Learning Initiative
• The United Way Common Good Forecaster
• The National Center for Education Statistics

Related posts on this blog:

• The Perverse Vocabulary of Feedback Loops
• Motivating Students - Opportunity Isn't Enough
• Learning - Everything Works, But How Well?
• Education by the Numbers
• Things Every Education Tech Entrepreneur Should Know

Thanks to everyone who came and to all those whose research inspired this presentation.

Learning Maps, Common IDs and the Common Core

Today we presented at the iNACOL VSS conference on "Learning Maps, Common IDs and the Common Core. Here are primary resources associated with that presentation:

• The Slides from the Presentation
• Common Core State Standards
• Learning Resource Metadata Initiative
• Learning Registry
• Shared Learning Collaborative

Update: 7:55pm: In addition to the above primary sources, I've written the following on the same subject:

• Common Identifiers for the Common Core
• Schema.org, LRMI and the Learning Registry
• CEDS and the Four-Layer Framework for Data Standards
• Tips for the Common Core XML

Also I corrected the link to LearningRegistry.org.

Many thanks to the panelists: Maureen Wentworth, Michael Jay and Sharren Bates.

Things Every Education Tech Entrepreneur Should Know

This weekend I'm volunteering as a coach for Startup Weekend Edu in Seattle. Preparing for this got me to thinking about things people building education technology should know. The following list isn't comprehensive but it's a good starting point. Follow the links to learn more about these topics.

Theories of Change
You need to have a good theory of how your technology will improve education. There's a lot of money to be made in record keeping and ERP-type applications. But the things that interest me and I hope interest you are those that directly improve student learning. And you need to be specific about the expected improvement. Do you want students to learn more in the same amount of time or take less time to learn a skill? Are you seeking better comprehension and retention? What about "deeper learning" – getting beyond recall and demonstrating the ability to apply concepts or solve problems.

Most ed tech theories of change start with Bloom's Two Sigma Problem. In a 1984 paper, Benjamin Bloom discussed how they had achieved two standard deviations improvement in student learning through a combination of Mastery Learning and one-on-one tutoring. Noting that 1:1 student-teacher ratios are impractical, Bloom's challenge is to find scalable ways to achieve the same results.

The following resources should stimulate your theoretical juices:

• The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Benjamin Bloom's 1984 Two Sigma Paper. Here's an earlier introduction I wrote earlier on this blog.

• A 2011 Metastudy by Kurt VanLehn gives a progress report of Intelligent Tutoring Systems and an update on progress toward Bloom's Two Sigma Problem. In particular, see page 210 (the 15th of the paper) in which VanLehn explains that about half of Bloom's two sigma gains were due to changes in Mastery Learning parameters.

• Personalized Learning is "instruction that is paced to learning needs, tailored to learning preferences, and tailored to the specific interests of different learners. In an environment that is fully personalized, the learning objectives and content as well as the method and pace may all vary." This definition is from the National Education Technology Plan which is an excellent read so long as you skip the executive summary.

• Higher Ed Disruption, Not So New. Alexandra Logue gives a history of Personalized Learning theories dating back to the early 20th century.

• Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. The ultimate meta analysis, John Hattie reports the effect size of 138 factors on student learning along with nuance explanations of the what is known about each factor. I summarized some of the high points in a recent blog post. You should seriously check this book first before running with some theory of educational improvement.

• Let's Use Video to Reinvent Education. Salman Khan speaking at TED. Notably, at 14:30 he talks about measuring individual teacher-student time as opposed to teacher to student ratios. He also introduces the flipped classroom concept.

• As We May Teach. My own essay from a couple of years back in which I apply the IT principles of Business Process Automation to teaching.

• Curing Baumol’s Disease: In Search of Productivity Gains in K-12 Schooling. Paul Hill and Marguerite Roza discuss the long delay from the introduction of automation to industry and measurable productivity gains. Then they apply the concepts to public education.

• Changing the Rules to the Game of School. My presentation at VSS 2011. Games are composed of a goal, rules, feedback and voluntary participation. So is school. What can we learn from this?

• Cognitive scientists talk about the Zone of Proximal Development. Game designers talk about Gameplay Progression. They are similar concepts and they both involve motivation and increasing skill levels. In fact, the motivational reward from this form of gameplay is achievement of greater skill.

• Feedback loops are an essential component of Personalized Learning. (From an earlier post in this blog.)

• Blended Learning, using a combination of online and in-person learning, is a prominent thread thread of research. That's based on studies (such as this one by the Open Learning Initiative) that show online and conventional teaching to be roughly equal but blended learning to be superior to both. An open question is which model to use when blending? My colleague, Scott Benson posts a periodic list of blended learning resources.

• The Puzzle of Motivation: Dan Pink explains the growing science of motivation without which, even the best instruction may fail.

Building Blocks
A number of organizations including the federal government, technology standards groups, associations and foundations have assembled building blocks to support innovative education technology. Some of these can improve time-to-delivery, some help interoperability between applications and some ensure that your application is based on tested learning theories:

• The Personalized Learning Model is a framework that some of us at the Gates Foundation have used to talk about how key components in a learning system work together. It's very similar to frameworks used by others in the community.

• The Common Core State Standards (CCSS) are a common set of learning objectives that have been adopted by 45 states and DC. For primary and secondary education this gives a framework to which learning resources and assessments can be aligned. Recently, the CCSS have been published in machine-readable form with canonical identifiers.

• The Partnership for Assessment of Readiness for College and Careers (PARCC) and the Smarter Balanced Assessment Consortium (SBAC) are two multi-state consortia developing standardized assessments for the Common Core. As with the CCSS, these are expected to be a generational improvement on existing assessment practice.

• Common Education Data Standards (CEDS) is a coordinated effort to define a Data Dictionary and Logical Data Model for the education industry. The standards span primary, secondary and postsecondary education. Version 3, presently in public comment, adds early childhood, workforce and adult learning to the mix. The Four Layer Framework for Data Standards helps understand where CEDS fits into the mix of other standards.

• The Learning Resource Metadata Initiative is a metadata schema for identifying learning resources (text, video, virtual labs, assessments, etc.) and aligning them to education standards like the Common Core.

• The Learning Registry is a system for sharing metadata about learning resources. It's synergistic with LRMI and other metadata formats.

• MyData Button is a federal government initiative to allow students or their parents to download their student data so that it can be used by other systems.

• The Schools Information Framework (SIF) is a standard protocol for exchanging educational data among applications and institutions. SIF recently announced plans to incorporate all of the CEDS standard into SIF v3.

• The Postsecondary Electronic Standards Council (PESC) defines data models and protocols for exchanging data among postsecondary institutions. PESC standards cover admissions applications, test score reporting, student aid applications and reporting, digital transcripts and more.

• IMS Global defines educational content standards (where SIF and PESC concentrate on student and institutional data). IMS standards like QTI and Common Cartridge define how to package assessment items and courseware for exchange between systems. My favorite IMS standard is Learning Tools Interoperability which is a protocol that allows rich, custom learning tools to be integrated into other learning environments.
• Ed-Fi is a data model and set of tools to support teacher and student dashboards indicating student progress.

• The Shared Learning Collaborative (SLC) "is an alliance of states, foundations, educators, content providers, developers and vendors who are passionate about using technology to improve education." It's an ambitious multistate project that leverages many of the technologies listed above into a coherent whole. Vendor outreach programs are at dev.slcedu.org

Jim Goodell has posted his recent presentation on how several of these initiatives – SLC, LRMI, Learning Registry and MyData Button – all fit together coherently.

Product and Service Categories

There are a handful of existing education technology product and service categories with new ones emerging. Here are key categories with some examples. Note that the examples I've listed just happen to be well-known systems. It's far from a comprehensive list and I don't necessarily endorse these products. In each category there are emerging products that may be more innovative than the ones I name.

• Student Information Systems (SIS) maintain student records including admissions, current enrollment and historical achievement. For K-12 attendance information is also maintained. Newer systems include a gradebook – at least for current enrollments. Examples include PowerSchool (K-12), Ellucian (Postsecondary), PeopleSoft Campus Solutions (Postsecondary) and Kuali (Open Source).

• Learning Management Systems (LMS) manage class interactions such as syllabus, assignments, learning materials, quizzes, forums, gradebook and so forth. While LMSs are capable of delivering a rich online learning experience, most deployments are supplementary to conventional classroom learning and only a fraction of their capabilities are used. Well-known examples include BlackBoard, Desire2Learn, Moodle, eCollege, Sakai, BrainHoney and Canvas but there are numerous others.

• Instructional Improvement Systems are an emerging concept. Like an LMS, an IIS manages student learning. However, an IIS uses accumulated student data as well as effectiveness data about learning resources to customize the learning experience to individual student needs. To support continuous improvement, the IIS should place equal emphasis on data collection and data use. Most action in the IIS space is being driven by state-level RFPs often with Race to the Top funding.

• Learning Object Repositories (LOR) are collections of learning resources. They come in many forms. Some incorporate a Content Management System (CMS) to support authoring and curation of learning resources. Some, like CK12 or Khan Academy, are composed of internally authored components. Others, like Curriki or TheGateway collect user-generated content. Commercial vendors like Learning.com and NetTrekker build custom-curated repositories for schools, districts and states. Khan Academy augments their LOR with assessments and learning management. An Open Educational Resource (OER) Repository is a LOR that exclusively includes open-licensed resources. Prominent OER Repositories include Connexions, Hippocampus, Curriki, Merlot, OERCommons, CK12, TheGateway and Khan Academy. Commercial efforts include Learning.com, NetTrekker, TeachersPayTeachers and xpLOR. CoreSpring is an emerging OER repository for assessment items.

• Public Education Datasets are available from the National Center for Education Statistics and other federal and state education agencies. The Digest of Education Statistics is a compilation of many government and privately-sourced datasets. Other public datasets include EdFacts and IPEDS. There some interesting opportunities to consuming existing public data and analyzing it in new ways.

There's much more that could be added but I think I've reached the point of diminishing returns. Please use the comments to point at other important theories, building blocks or initiatives.

Tips For Using the Common Core XML

The Common Core State Standards (CCSS) have been out for a couple of years now and adoption efforts are progressing. To facilitate use of the CCSS in learning applications and with metadata frameworks like LRMI, they have recently posted canonical identifiers and machine-readable XML for the standards. I wrote about that in a recent post.

Update: 1 September 2014
The CCSSO has updated the CoreStandards.org website and most of the links in this post no longer work. However, the Common Core XML files still exist and can be found with the developer information here. Sometime in the near future, I'll re-write this post to describe the new provisions that the CCSSO has made for the techie crowd.

Today I'm going into the nuts and bolts of how a developer can make use of the XML. There are some useful features that aren't obvious at first glance. For background, I recommend that you read the announcement memo that accompanied the release of the XML on the corestandards.org website.

Canonical Identifiers

The canonical identifiers for the common core state standards are available in .csv form here: http://corestandards.org/assets/E0607_ccss_identifiers.csv. The first column lists the URLs that were formerly on the Corestandards.org website. They are included to support conversions for legacy applications.

Notable is there is an exact 1:1 mapping between the three ID types and there are 1844 IDs in the table. So if two applications are using different forms of IDs (e.g. one uses GUIDs and another uses URIs) the translation is a deterministic table lookup. A closer examination will show that there's a simple algorithmic conversion between the "dot notation" identifiers and the corresponding URIs. I've written functions in c# to do the translation and posted them here. It should be easy to port them to Java or any other language.

Hierarchy

The standards for Mathematics and ELA/Literacy follow different hierarchies that are suited to the way the standards are written and are intended to be used. The Dot Notation and URL forms of the identifiers can be parsed into the corresponding hierarchies as shown in the following examples.

Math Example

Dot Notation: CCSS.Math.Content.HSA-SSE.A.1b
URL: http://corestandards.org/Math/Content/HSA/SSE/A/1/b

Initiative	CCSS	(Common Core State Standards)
Framework	Math
Set	Content	(Options are 'Content' and 'Practice')
Grade	HSA	(High School Algebra)
Domain	SSE	(Seeing Structure in Expressions)
Cluster	A
Standard	1
Component	b

You can reference the math standards at the Component, Standard and Cluster levels. Thus, the following are all valid CCSS URI Identifiers:

• http://corestandards.org/Math/Content/HSA/SSE/A/1/b
• http://corestandards.org/Math/Content/HSA/SSE/A/1
• http://corestandards.org/Math/Content/HSA/SSE/A

If you add an ".xml" suffix to the URL then you get the computer-readable XML version of each:

• http://corestandards.org/Math/Content/HSA/SSE/A/1/b.xml
• http://corestandards.org/Math/Content/HSA/SSE/A/1.xml
• http://corestandards.org/Math/Content/HSA/SSE/A.xml

The XML at the cluster and standard levels includes all child items. So the cluster includes all standards in that cluster and all components within that standard.

While the canonical IDs don't include Domain or Grade levels, you can retrieve all standards within a domain or grade by hacking the URL as follows:

• http://corestandards.org/Math/Content/HSA/SSE.xml
• http://corestandards.org/Math/Content/HSA.xml

And, you can retrieve all of the math standards in one XML document with this URL:

• http://corestandards.org/Math.xml

ELA/Literacy Example

Dot Notation: CCSS.ELA-Literacy.W.9-10.3d
URL: http://www.corestandards.org/ELA-Literacy/W/9-10/3/d

Initiative	CCSS	(Common Core State Standards)
Framework	ELA-Literacy
Set		(Optional, not used in this example)
Strand+Domain	W	(Writing)
Grade	9-10
Standard	3
Component	d

You can reference the literacy standards at the Component, Standard and Grade levels. Thus, the following are all valid CCSS URI Identifiers:

• http://www.corestandards.org/ELA-Literacy/W/9-10/3/d
• http://www.corestandards.org/ELA-Literacy/W/9-10/3
• http://www.corestandards.org/ELA-Literacy/W/9-10

If you add an ".xml" suffix to the URL then you get the computer-readable XML version of each:

• http://www.corestandards.org/ELA-Literacy/W/9-10/3/d.xml
• http://www.corestandards.org/ELA-Literacy/W/9-10/3.xml
• http://www.corestandards.org/ELA-Literacy/W/9-10.xml

And, you can retrieve all of the ELA/Literacy standards in one XML document with this URL:

• http://www.corestandards.org/ELA-Literacy.xml

CEDS and the Four-Layer Framework for Data Standards - Updated

About a year ago I posted a Four-Layer Framework for Data Standards. It was developed as Common Education Data Standards (CEDS) working groups were discussing the space in which CEDS operates and what makes its contribution unique. Today I'm updating the framework document – adding some clarity but mostly reconciling terminology with that used by CEDS.

In the June CEDS stakeholders' meeting the group emphasized that CEDS works strictly at layers 1 and 2 (Data Dictionary and Logical Data Model) leaving serialization and protocol to other standards organizations. This leads a unique approach (at least unique to the education standards space) in which the focus is on alignment instead of compliance.

To support this strategy, CEDS has posted the Align and Connect tools. The Align tool allows State Education Agencies, software vendors and other organizations to post their data models and show how their elements align to CEDS. Organizations can choose to make their data models public; in which case Align can be used to report the degree of alignment between two data models. The new Connect tool addresses the sharing of metric definitions like graduation rate, student financial aid repayment or college-going rate. Metrics like these are not in the data model, they are derived from that data. And different organizations may combine the data in different ways. Connect supports the sharing and eventual standardization of these metric definitions.

Another question I've gotten is how the four-layer framework overlaps with the OSI 7-layer model. Layers 1-3 (Data Dictionary, Logical Data Model and Serialization) in the four layer model map to the Application layer (layer 7) which is at the top of the OSI model. All other layers in OSI are combined into the Protocol layer in the four-layer model.

The latest four-layer document is here. It's released into the public domain under a CC0 disclaimer.

Schema.org, LRMI and the Learning Registry

Try this:


     1. Browse to google.com
     2. Search for "potato salad"
     3. Experiment with the recipe search tool that appears on the left.

Most people are aware of Google Shopping – Google set up a way for merchants to list the details of things they have for sale. More recently, Google published a way to mark up recipes so that the search engine can tell what's the title, what are the ingredients, what's a photo of the recipe and so forth. We call this, "metadata" or data about the data.

Get ready for a lot more of this kind of thing. The developers of, Bing, Google, Yahoo! and Yandex have cooperated on a common metadata vocabulary at Schema.org. It's a perfect example of coopetition – the search engines are cooperating how metadata should be embedded in web pages. That way webmasters don't have to code four different kinds of metadata. Meanwhile, the search providers will compete on what they do with that metadata.

The results of this are already emerging, try searching for a movie title, for example, or for a type of restaurant. Any of the major search engines will give you a nicely structured result.

We wanted to do the same for learning resources – videos, exercises, simulations, learning games and so forth. Wouldn't it be great if a teacher or student could search for "fractions" and get a search tool that allows the results to be filtered by age, grade level, subject or learning objective? Conveniently, the Schema.org folks had expressed their interest in new submissions, so long as they represent an industry consortium. So, a bunch of organizations got together and launched LRMI. You can learn much more about the co-sponsors, advisory groups and the specification itself on LRMI website.

Schema.org embeds the metadata right in the webpage (using HTML microdata). That makes sense for search engines, but it means that only the publisher of the webpage can post metadata about it. Yes, there's such a thing as third-party microdata but the search engines don't pay attention to it. Plus there are other kinds of data that need to be shared between learning solutions. Conveniently, there's a complementary alternative.

The Learning Registry is a peer network of LR servers that exchange metadata similar to the way email providers exchange mail messages. In the LR architecture, metadata consists of assertions. Here's are some example assertions rendered into plain English:

• "CCSS.Math.Content.K.CC.A.1" is a "Learning Objective" defined as "Count to 100 by ones and by tens."
• "http://www.youtube.com/watch?v=rG_uUws1oC0" is a "Video" that "Teaches" learning objective "CCSS.Math.Content.K.CC.A.1" and is targeted at "ages 2-4".

The first of these is one of the common-core standards. That data is now available in XML form on the web. The second of these is an LRMI statement. Of course, both are plain-English renditions of what could be machine-formatted.

LR assertions also include their provenance, that is the name of the organization making the assertion, when the assertion was made and a digital signature. This lets users of the LR have confidence in the origin of statements and filter for reliable sources.

You might have noticed that in describing the Learning Registry I used an LRMI example! That's because these are compatible technologies and the groups are coordinating with each other. This diagram helps show the relationship between the efforts:

You'll see that "Schema.org" appears twice in the diagram. That's because Schema.org defines both a vocabulary (a set of metadata tags) and a way of sharing that metadata. LRMI is an addition to the Schema.org vocabulary that enhances descriptions of educational content.

Schema.org and the Learning Registry offer two complementary ways of distributing metadata. They can even be bridged – there are experimental web crawlers that will extract HTML microdata from a page and inject it as assertions into the Learning Registry.

I could write a whole lot more about both of these efforts but far better to link to existing resources:

• Recording of last Monday's SIIA webinar on LRMI.
• Official LRMI website.
• Video explaining the value of LRMI.
• Official Learning Registry website.
• Official Schema.org website.
• Schema.org blog post regarding pending incorporation of LRMI.
• Shared Learning Collaborative metadata efforts.