Learning Maps, Common IDs and the Common Core

Today we presented at the iNACOL VSS conference on "Learning Maps, Common IDs and the Common Core. Here are primary resources associated with that presentation:

• The Slides from the Presentation
• Common Core State Standards
• Learning Resource Metadata Initiative
• Learning Registry
• Shared Learning Collaborative

Update: 7:55pm: In addition to the above primary sources, I've written the following on the same subject:

• Common Identifiers for the Common Core
• Schema.org, LRMI and the Learning Registry
• CEDS and the Four-Layer Framework for Data Standards
• Tips for the Common Core XML

Also I corrected the link to LearningRegistry.org.

Many thanks to the panelists: Maureen Wentworth, Michael Jay and Sharren Bates.

Things Every Education Tech Entrepreneur Should Know

This weekend I'm volunteering as a coach for Startup Weekend Edu in Seattle. Preparing for this got me to thinking about things people building education technology should know. The following list isn't comprehensive but it's a good starting point. Follow the links to learn more about these topics.

Theories of Change
You need to have a good theory of how your technology will improve education. There's a lot of money to be made in record keeping and ERP-type applications. But the things that interest me and I hope interest you are those that directly improve student learning. And you need to be specific about the expected improvement. Do you want students to learn more in the same amount of time or take less time to learn a skill? Are you seeking better comprehension and retention? What about "deeper learning" – getting beyond recall and demonstrating the ability to apply concepts or solve problems.

Most ed tech theories of change start with Bloom's Two Sigma Problem. In a 1984 paper, Benjamin Bloom discussed how they had achieved two standard deviations improvement in student learning through a combination of Mastery Learning and one-on-one tutoring. Noting that 1:1 student-teacher ratios are impractical, Bloom's challenge is to find scalable ways to achieve the same results.

The following resources should stimulate your theoretical juices:

• The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Benjamin Bloom's 1984 Two Sigma Paper. Here's an earlier introduction I wrote earlier on this blog.

• A 2011 Metastudy by Kurt VanLehn gives a progress report of Intelligent Tutoring Systems and an update on progress toward Bloom's Two Sigma Problem. In particular, see page 210 (the 15th of the paper) in which VanLehn explains that about half of Bloom's two sigma gains were due to changes in Mastery Learning parameters.

• Personalized Learning is "instruction that is paced to learning needs, tailored to learning preferences, and tailored to the specific interests of different learners. In an environment that is fully personalized, the learning objectives and content as well as the method and pace may all vary." This definition is from the National Education Technology Plan which is an excellent read so long as you skip the executive summary.

• Higher Ed Disruption, Not So New. Alexandra Logue gives a history of Personalized Learning theories dating back to the early 20th century.

• Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. The ultimate meta analysis, John Hattie reports the effect size of 138 factors on student learning along with nuance explanations of the what is known about each factor. I summarized some of the high points in a recent blog post. You should seriously check this book first before running with some theory of educational improvement.

• Let's Use Video to Reinvent Education. Salman Khan speaking at TED. Notably, at 14:30 he talks about measuring individual teacher-student time as opposed to teacher to student ratios. He also introduces the flipped classroom concept.

• As We May Teach. My own essay from a couple of years back in which I apply the IT principles of Business Process Automation to teaching.

• Curing Baumol’s Disease: In Search of Productivity Gains in K-12 Schooling. Paul Hill and Marguerite Roza discuss the long delay from the introduction of automation to industry and measurable productivity gains. Then they apply the concepts to public education.

• Changing the Rules to the Game of School. My presentation at VSS 2011. Games are composed of a goal, rules, feedback and voluntary participation. So is school. What can we learn from this?

• Cognitive scientists talk about the Zone of Proximal Development. Game designers talk about Gameplay Progression. They are similar concepts and they both involve motivation and increasing skill levels. In fact, the motivational reward from this form of gameplay is achievement of greater skill.

• Feedback loops are an essential component of Personalized Learning. (From an earlier post in this blog.)

• Blended Learning, using a combination of online and in-person learning, is a prominent thread thread of research. That's based on studies (such as this one by the Open Learning Initiative) that show online and conventional teaching to be roughly equal but blended learning to be superior to both. An open question is which model to use when blending? My colleague, Scott Benson posts a periodic list of blended learning resources.

• The Puzzle of Motivation: Dan Pink explains the growing science of motivation without which, even the best instruction may fail.

Building Blocks
A number of organizations including the federal government, technology standards groups, associations and foundations have assembled building blocks to support innovative education technology. Some of these can improve time-to-delivery, some help interoperability between applications and some ensure that your application is based on tested learning theories:

• The Personalized Learning Model is a framework that some of us at the Gates Foundation have used to talk about how key components in a learning system work together. It's very similar to frameworks used by others in the community.

• The Common Core State Standards (CCSS) are a common set of learning objectives that have been adopted by 45 states and DC. For primary and secondary education this gives a framework to which learning resources and assessments can be aligned. Recently, the CCSS have been published in machine-readable form with canonical identifiers.

• The Partnership for Assessment of Readiness for College and Careers (PARCC) and the Smarter Balanced Assessment Consortium (SBAC) are two multi-state consortia developing standardized assessments for the Common Core. As with the CCSS, these are expected to be a generational improvement on existing assessment practice.

• Common Education Data Standards (CEDS) is a coordinated effort to define a Data Dictionary and Logical Data Model for the education industry. The standards span primary, secondary and postsecondary education. Version 3, presently in public comment, adds early childhood, workforce and adult learning to the mix. The Four Layer Framework for Data Standards helps understand where CEDS fits into the mix of other standards.

• The Learning Resource Metadata Initiative is a metadata schema for identifying learning resources (text, video, virtual labs, assessments, etc.) and aligning them to education standards like the Common Core.

• The Learning Registry is a system for sharing metadata about learning resources. It's synergistic with LRMI and other metadata formats.

• MyData Button is a federal government initiative to allow students or their parents to download their student data so that it can be used by other systems.

• The Schools Information Framework (SIF) is a standard protocol for exchanging educational data among applications and institutions. SIF recently announced plans to incorporate all of the CEDS standard into SIF v3.

• The Postsecondary Electronic Standards Council (PESC) defines data models and protocols for exchanging data among postsecondary institutions. PESC standards cover admissions applications, test score reporting, student aid applications and reporting, digital transcripts and more.

• IMS Global defines educational content standards (where SIF and PESC concentrate on student and institutional data). IMS standards like QTI and Common Cartridge define how to package assessment items and courseware for exchange between systems. My favorite IMS standard is Learning Tools Interoperability which is a protocol that allows rich, custom learning tools to be integrated into other learning environments.
• Ed-Fi is a data model and set of tools to support teacher and student dashboards indicating student progress.

• The Shared Learning Collaborative (SLC) "is an alliance of states, foundations, educators, content providers, developers and vendors who are passionate about using technology to improve education." It's an ambitious multistate project that leverages many of the technologies listed above into a coherent whole. Vendor outreach programs are at dev.slcedu.org

Jim Goodell has posted his recent presentation on how several of these initiatives – SLC, LRMI, Learning Registry and MyData Button – all fit together coherently.

Product and Service Categories

There are a handful of existing education technology product and service categories with new ones emerging. Here are key categories with some examples. Note that the examples I've listed just happen to be well-known systems. It's far from a comprehensive list and I don't necessarily endorse these products. In each category there are emerging products that may be more innovative than the ones I name.

• Student Information Systems (SIS) maintain student records including admissions, current enrollment and historical achievement. For K-12 attendance information is also maintained. Newer systems include a gradebook – at least for current enrollments. Examples include PowerSchool (K-12), Ellucian (Postsecondary), PeopleSoft Campus Solutions (Postsecondary) and Kuali (Open Source).

• Learning Management Systems (LMS) manage class interactions such as syllabus, assignments, learning materials, quizzes, forums, gradebook and so forth. While LMSs are capable of delivering a rich online learning experience, most deployments are supplementary to conventional classroom learning and only a fraction of their capabilities are used. Well-known examples include BlackBoard, Desire2Learn, Moodle, eCollege, Sakai, BrainHoney and Canvas but there are numerous others.

• Instructional Improvement Systems are an emerging concept. Like an LMS, an IIS manages student learning. However, an IIS uses accumulated student data as well as effectiveness data about learning resources to customize the learning experience to individual student needs. To support continuous improvement, the IIS should place equal emphasis on data collection and data use. Most action in the IIS space is being driven by state-level RFPs often with Race to the Top funding.

• Learning Object Repositories (LOR) are collections of learning resources. They come in many forms. Some incorporate a Content Management System (CMS) to support authoring and curation of learning resources. Some, like CK12 or Khan Academy, are composed of internally authored components. Others, like Curriki or TheGateway collect user-generated content. Commercial vendors like Learning.com and NetTrekker build custom-curated repositories for schools, districts and states. Khan Academy augments their LOR with assessments and learning management. An Open Educational Resource (OER) Repository is a LOR that exclusively includes open-licensed resources. Prominent OER Repositories include Connexions, Hippocampus, Curriki, Merlot, OERCommons, CK12, TheGateway and Khan Academy. Commercial efforts include Learning.com, NetTrekker, TeachersPayTeachers and xpLOR. CoreSpring is an emerging OER repository for assessment items.

• Public Education Datasets are available from the National Center for Education Statistics and other federal and state education agencies. The Digest of Education Statistics is a compilation of many government and privately-sourced datasets. Other public datasets include EdFacts and IPEDS. There some interesting opportunities to consuming existing public data and analyzing it in new ways.

There's much more that could be added but I think I've reached the point of diminishing returns. Please use the comments to point at other important theories, building blocks or initiatives.

Tips For Using the Common Core XML

The Common Core State Standards (CCSS) have been out for a couple of years now and adoption efforts are progressing. To facilitate use of the CCSS in learning applications and with metadata frameworks like LRMI, they have recently posted canonical identifiers and machine-readable XML for the standards. I wrote about that in a recent post.

Update: 1 September 2014
The CCSSO has updated the CoreStandards.org website and most of the links in this post no longer work. However, the Common Core XML files still exist and can be found with the developer information here. Sometime in the near future, I'll re-write this post to describe the new provisions that the CCSSO has made for the techie crowd.

Today I'm going into the nuts and bolts of how a developer can make use of the XML. There are some useful features that aren't obvious at first glance. For background, I recommend that you read the announcement memo that accompanied the release of the XML on the corestandards.org website.

Canonical Identifiers

The canonical identifiers for the common core state standards are available in .csv form here: http://corestandards.org/assets/E0607_ccss_identifiers.csv. The first column lists the URLs that were formerly on the Corestandards.org website. They are included to support conversions for legacy applications.

Notable is there is an exact 1:1 mapping between the three ID types and there are 1844 IDs in the table. So if two applications are using different forms of IDs (e.g. one uses GUIDs and another uses URIs) the translation is a deterministic table lookup. A closer examination will show that there's a simple algorithmic conversion between the "dot notation" identifiers and the corresponding URIs. I've written functions in c# to do the translation and posted them here. It should be easy to port them to Java or any other language.

Hierarchy

The standards for Mathematics and ELA/Literacy follow different hierarchies that are suited to the way the standards are written and are intended to be used. The Dot Notation and URL forms of the identifiers can be parsed into the corresponding hierarchies as shown in the following examples.

Math Example

Dot Notation: CCSS.Math.Content.HSA-SSE.A.1b
URL: http://corestandards.org/Math/Content/HSA/SSE/A/1/b

Initiative	CCSS	(Common Core State Standards)
Framework	Math
Set	Content	(Options are 'Content' and 'Practice')
Grade	HSA	(High School Algebra)
Domain	SSE	(Seeing Structure in Expressions)
Cluster	A
Standard	1
Component	b

You can reference the math standards at the Component, Standard and Cluster levels. Thus, the following are all valid CCSS URI Identifiers:

• http://corestandards.org/Math/Content/HSA/SSE/A/1/b
• http://corestandards.org/Math/Content/HSA/SSE/A/1
• http://corestandards.org/Math/Content/HSA/SSE/A

If you add an ".xml" suffix to the URL then you get the computer-readable XML version of each:

• http://corestandards.org/Math/Content/HSA/SSE/A/1/b.xml
• http://corestandards.org/Math/Content/HSA/SSE/A/1.xml
• http://corestandards.org/Math/Content/HSA/SSE/A.xml

The XML at the cluster and standard levels includes all child items. So the cluster includes all standards in that cluster and all components within that standard.

While the canonical IDs don't include Domain or Grade levels, you can retrieve all standards within a domain or grade by hacking the URL as follows:

• http://corestandards.org/Math/Content/HSA/SSE.xml
• http://corestandards.org/Math/Content/HSA.xml

And, you can retrieve all of the math standards in one XML document with this URL:

• http://corestandards.org/Math.xml

ELA/Literacy Example

Dot Notation: CCSS.ELA-Literacy.W.9-10.3d
URL: http://www.corestandards.org/ELA-Literacy/W/9-10/3/d

Initiative	CCSS	(Common Core State Standards)
Framework	ELA-Literacy
Set		(Optional, not used in this example)
Strand+Domain	W	(Writing)
Grade	9-10
Standard	3
Component	d

You can reference the literacy standards at the Component, Standard and Grade levels. Thus, the following are all valid CCSS URI Identifiers:

• http://www.corestandards.org/ELA-Literacy/W/9-10/3/d
• http://www.corestandards.org/ELA-Literacy/W/9-10/3
• http://www.corestandards.org/ELA-Literacy/W/9-10

If you add an ".xml" suffix to the URL then you get the computer-readable XML version of each:

• http://www.corestandards.org/ELA-Literacy/W/9-10/3/d.xml
• http://www.corestandards.org/ELA-Literacy/W/9-10/3.xml
• http://www.corestandards.org/ELA-Literacy/W/9-10.xml

And, you can retrieve all of the ELA/Literacy standards in one XML document with this URL:

• http://www.corestandards.org/ELA-Literacy.xml

CEDS and the Four-Layer Framework for Data Standards - Updated

About a year ago I posted a Four-Layer Framework for Data Standards. It was developed as Common Education Data Standards (CEDS) working groups were discussing the space in which CEDS operates and what makes its contribution unique. Today I'm updating the framework document – adding some clarity but mostly reconciling terminology with that used by CEDS.

In the June CEDS stakeholders' meeting the group emphasized that CEDS works strictly at layers 1 and 2 (Data Dictionary and Logical Data Model) leaving serialization and protocol to other standards organizations. This leads a unique approach (at least unique to the education standards space) in which the focus is on alignment instead of compliance.

To support this strategy, CEDS has posted the Align and Connect tools. The Align tool allows State Education Agencies, software vendors and other organizations to post their data models and show how their elements align to CEDS. Organizations can choose to make their data models public; in which case Align can be used to report the degree of alignment between two data models. The new Connect tool addresses the sharing of metric definitions like graduation rate, student financial aid repayment or college-going rate. Metrics like these are not in the data model, they are derived from that data. And different organizations may combine the data in different ways. Connect supports the sharing and eventual standardization of these metric definitions.

Another question I've gotten is how the four-layer framework overlaps with the OSI 7-layer model. Layers 1-3 (Data Dictionary, Logical Data Model and Serialization) in the four layer model map to the Application layer (layer 7) which is at the top of the OSI model. All other layers in OSI are combined into the Protocol layer in the four-layer model.

The latest four-layer document is here. It's released into the public domain under a CC0 disclaimer.

Schema.org, LRMI and the Learning Registry

Try this:


     1. Browse to google.com
     2. Search for "potato salad"
     3. Experiment with the recipe search tool that appears on the left.

Most people are aware of Google Shopping – Google set up a way for merchants to list the details of things they have for sale. More recently, Google published a way to mark up recipes so that the search engine can tell what's the title, what are the ingredients, what's a photo of the recipe and so forth. We call this, "metadata" or data about the data.

Get ready for a lot more of this kind of thing. The developers of, Bing, Google, Yahoo! and Yandex have cooperated on a common metadata vocabulary at Schema.org. It's a perfect example of coopetition – the search engines are cooperating how metadata should be embedded in web pages. That way webmasters don't have to code four different kinds of metadata. Meanwhile, the search providers will compete on what they do with that metadata.

The results of this are already emerging, try searching for a movie title, for example, or for a type of restaurant. Any of the major search engines will give you a nicely structured result.

We wanted to do the same for learning resources – videos, exercises, simulations, learning games and so forth. Wouldn't it be great if a teacher or student could search for "fractions" and get a search tool that allows the results to be filtered by age, grade level, subject or learning objective? Conveniently, the Schema.org folks had expressed their interest in new submissions, so long as they represent an industry consortium. So, a bunch of organizations got together and launched LRMI. You can learn much more about the co-sponsors, advisory groups and the specification itself on LRMI website.

Schema.org embeds the metadata right in the webpage (using HTML microdata). That makes sense for search engines, but it means that only the publisher of the webpage can post metadata about it. Yes, there's such a thing as third-party microdata but the search engines don't pay attention to it. Plus there are other kinds of data that need to be shared between learning solutions. Conveniently, there's a complementary alternative.

The Learning Registry is a peer network of LR servers that exchange metadata similar to the way email providers exchange mail messages. In the LR architecture, metadata consists of assertions. Here's are some example assertions rendered into plain English:

• "CCSS.Math.Content.K.CC.A.1" is a "Learning Objective" defined as "Count to 100 by ones and by tens."
• "http://www.youtube.com/watch?v=rG_uUws1oC0" is a "Video" that "Teaches" learning objective "CCSS.Math.Content.K.CC.A.1" and is targeted at "ages 2-4".

The first of these is one of the common-core standards. That data is now available in XML form on the web. The second of these is an LRMI statement. Of course, both are plain-English renditions of what could be machine-formatted.

LR assertions also include their provenance, that is the name of the organization making the assertion, when the assertion was made and a digital signature. This lets users of the LR have confidence in the origin of statements and filter for reliable sources.

You might have noticed that in describing the Learning Registry I used an LRMI example! That's because these are compatible technologies and the groups are coordinating with each other. This diagram helps show the relationship between the efforts:

You'll see that "Schema.org" appears twice in the diagram. That's because Schema.org defines both a vocabulary (a set of metadata tags) and a way of sharing that metadata. LRMI is an addition to the Schema.org vocabulary that enhances descriptions of educational content.

Schema.org and the Learning Registry offer two complementary ways of distributing metadata. They can even be bridged – there are experimental web crawlers that will extract HTML microdata from a page and inject it as assertions into the Learning Registry.

I could write a whole lot more about both of these efforts but far better to link to existing resources:

• Recording of last Monday's SIIA webinar on LRMI.
• Official LRMI website.
• Video explaining the value of LRMI.
• Official Learning Registry website.
• Official Schema.org website.
• Schema.org blog post regarding pending incorporation of LRMI.
• Shared Learning Collaborative metadata efforts.

Common Identifiers for the Common Core

In order to do personalized learning at scale, with a mix activities and assessments from a variety of sources, we need to agree upon a common set of learning objectives. The Gates Foundation and the Shared Learning Collaborative, have endorsed the Common Core State Standards.

Side note: As we are in the election season, there is a lot of rhetoric around national curriculum and federal mandates. The Common Core is only a set of commonly agreed upon learning objectives. It's not a curriculum (national or otherwise) and it was developed by a voluntary cooperation among states with the federal government staying clear.

As we were developing the LRMI project, we anticipated the need to be able to reference the Common Core as well as other learning objectives. Unfortunately, the Common Core didn't specify a standard set of references. At the time (approximately 12 months ago) there were at least five different and incompatible ways to reference the Common Core. So, we turned to the coordinators – the National Governor's Association (NGA) and Council of Chief State School Officers (CCSSO), and with the help of Student Achievement Partners they developed a consistent set of identifiers for the Common Core.

In the next few weeks, they will be updating the corestandards.org website so that the URL identifiers will link directly the the specified standards. Also, the standards will be available in machine-readable XML format to facilitate a variety of learning applications.

It's a great step and will make a big difference. But in the process we identified another issue. Frequently a particular standard in the common core will include more than one learning objective. Here's an example:

CCSS.Math.Content.6.NS.B.3
Fluently add, subtract, multiply, and divide multi-digit decimals using the standard algorithm for each operation.

There are at least four learning objectives in this standard. The developers of assessments (such as PARCC and SBAC) have a problem with that. A typical assessment item will only test one one of these skills. Without finer-grained identifiers, they can't show complete coverage of the standards. Similar problems exist for learning activities and student records.

A few weeks ago, the NGA, CCSSO, SBAC, PARCC and SETDA announced a collaborative project to address this issue. By the end of the year, they expect to publish an open set of learning objectives based on a fine-grained parsing of the common core. They will also define a standard data format for publishing standards like these. That will be based on the data models proposed for Common Education Data Standards 3.0.

Of course, this still isn't the whole picture. Not all states are adopting the Common Core. The Common Core only addresses Mathematics and English/Literacy, the 50 states still have standards for other subjects. Other countries have their own standard learning objectives.

There needs to be a way for developers of educational standards to publish and share those standards.

The Achievement Standards Network, operated by JES & Co., maintains an open database of all 50 states' existing standards, plus the Common Core, plus those published by the American Association for the Advancement of Science and many others. Under a new grant, they will be enhancing the database to accept the new standardized identifiers and they'll incorporate the learning objectives defined by the granularity project.

Concurrently, the Learning Registry is emerging as a distributed system for sharing Achievement Standards Data, Learning Objectives, Cross-References between standards and an index of learning activities that are aligned to standard objectives.

It may seem like chaos but this is more like a dance. And very shortly we should have a coherent foundation for developers of learning tools and instructional systems.

There are a lot of links up there. Here are repeat links to the three most important announcements:

• Common Core Identifiers
• Granularity Project
• Achievement Standards Network Grant

ACT Scores: Most HS Graduates Aren't Prepared for College

The ACT just released it's annual report on "The Condition of College & Career Readiness." Curious to me is how different news outlets spin the results:

• Associated Press: ACT Scores Show Slight Progress in U.S. High School Seniors' College Readiness
• Inside Higher Ed: ACT Scores Are Flat
• Education Week: ACT Finds Most Students Still Not Ready for College
• Forbes: High School Grads Even Less Prepared but At Least Doing Better in Math and Science

The scores are indeed flat and have been for five years. The AP achieves a positive spin by noting that the number of students taking the test has increased by 17% in those five years. If you assume that those who take the test are the top students, then the 17% addition represents the lowest performers on the exam and things have improved. However, the 17% addition could also indicate that more students are taking both the ACT and SAT rather than selecting one or the other.

Of primary concern to us at the Gates Foundation is the low rate of college readiness. Of those who take the exam (a subset of all high school students) only 25% are prepared for college in all four subject areas (English, Reading, Mathematics and Science). The goals of our U.S. College Ready team are to elevate the standard of high school graduation to mean college ready and to increase the graduation rate beyond 80%. The ACT report reminds us just how far away we are from that goal.

Is that a worthy goal? In an earlier post I noted that society is turning to education as the solution to poverty. I offer two additional facts to support this argument:

• In the State of Washington, we import 43% of our bachelors' degree employees and 55% of graduate degree employees. Many of these people are immigrants because our school systems aren't preparing students for the high-tech jobs that are available.
• In December 2010, U.S. unemployment of those with a Bachelor's degree was approximately 5.1% while unemployment of those with just a high school diploma was nearly double at 9.8%. Those without a degree of any sort endured an unemployment rate of over 15%.

(Edited 2012-08-23 to include the Forbes headline)

Education by the Numbers

If a picture is worth 1,000 words, this post is equivalent to a 5,000 word essay:

With the exception of the last image, I generated these from NCES statistics using Excel. I'm releasing them under a CC0 waiver. Use them at will.

The last image was generated by the Center on Budget and Policy Priorities and posted to Flickr. I'm linking to that copy. You can do the same.

Learning - Everything Works, But How Well?

In a recent Freakonomics Post, Roger Pielke Jr. writes about the perils of "False Positive Science." We constantly fight the fallacy of equating correlation with causation. But false positive science involves a more subtle error. In the search to find statistically significant results, researchers often try many different analytical alternatives. Their papers rarely list all of the failed models, only the one that achieves statistical significance is used. Joseph Simmons and colleagues write, "It is unacceptably easy to publish 'statistically significant' evidence consistent with any hypothesis." And this mistake is more difficult for the reader to detect than the correlation/causation fallacy.

Credit: Randall Munroe - xkcd.com

When it comes to research into educational achievement, another issue comes into play. Since humans are natural learners, just about everything works. In his book, Visible Learning, John Hattie gives this rigorous treatment. Over 15 years, Hattie and his staff studied over 800 meta-analyses representing hundreds of thousands of studies into what affects student learning. For every study, they converted the results into a common effect size scale.

Roughly speaking, the effect sizes used in Visible Learning are the amount of improvement a student would make in a year scaled to one standard deviation on a standardized test. By mapping all effects onto a common effect size scale you can compare the relative value of different techniques and theories.

Among Hattie's observations is the following:

Almost everything works. Ninety percent of all effect sizes in education are positive. Of the ten percent that are negative, about half are "expected" (e.g., effects of disruptive students); thus about 95 percent of all things we do have a positive influence on achievement. When teachers claim that they are having a positive effect on achievement or when a policy improves achievement this is almost a trivial claim: virtually everything works. One only needs a pulse and we can improve achievement. (Hattie, Visible Learning, p. 15)

On Hattie's scale, a child simply living for a year with no schooling achieves an effect size of 0.15. "Maturation alone can account for much of the enhancement of learning." Being present in a classroom with a teacher results in effect sizes between 0.15 and 0.40. So, for an innovation to be interesting, it must result in an effect size substantially higher than 0.40. (Hattie, p. 16).

From the book, here are some selected influences with their rank and effect sizes.

Rank	Domain	Influence	Effect Size
1	Student	Self-report grades	1.44
2	Student	Piagetian programs	1.28
3	Teaching	Providing formative evaluation	0.90
4	Teacher	Micro teaching	0.88
5	School	Acceleration	0.88
6	School	Classroom behavioral	0.80
7	Teaching	Comprehensive interventions for learning disabled	0.77
8	Teacher	Teacher clarity	0.75
9	Teaching	Reciprocal teaching	0.74
10	Teaching	Feedback	0.73
11	Teacher	Teacher-student relationships	0.72
22	Curricula	Phonics instruction	0.60
25	Teaching	Study skills	0.59
29	Teaching	Mastery learning	0.58
31	Home	Home environment	0.57
32	Home	Socioeconomic status	0.57
42	School	Classroom management	0.52
45	Home	Parental involvement	0.51
51	Student	Motivation	0.48
56	Teacher	Quality of teaching	0.44
59	School	School size	0.43
62	Teaching	Matching style of learning	0.41
81	Student	Drugs (e.g. for ADHD)	0.33
91	School	Desegregation	0.28
92	School	Mainstreaming	0.28
100	Teaching	Individualized instruction	0.23
106	School	Class size	0.21
107	School	Charter schools	0.20
129	Curricula	Whole language	0.06
133	School	Open vs. traditional	0.01
134	School	Summer vacation	-0.09
135	Home	Welfare policies	-0.12
136	School	Retention	-0.16
137	Home	Television	-0.18
138	School	Mobility	-0.34

There's a ton of stuff to chew on here. Far more than I can do justice in a blog post. Hattie has between one half and five pages for each of the 138 effects and there is nuance that the numbers don't capture. I'll just make a few observations:

• The top five influences all involve adapting the experience according to individual student needs.
• Charter schools, something I favor, have an unimpressive effect size of 0.20. But charters were intended to enable experimentation. So we should expect them to average similar to conventional public schools but with a much larger standard deviation. Recent studies seem to confirm that expectation. And studies are starting to identify what factors distinguish the high-performing charters from other schools.
• Smaller schools help somewhat while the impact of smaller classes is minimal. That's probably because most small-class initiatives dilute their impact by with a consequential reduction in teacher experience.
• Feedback loops, among my favorite topics, appear at #10 with an effect size of 0.73.
• Home and socioeconomic status have a huge impact. But other factors are bigger so it should be possible to overcome the achievement gap in the school.
• Phonics Instruction has an effect size of 0.60 while Whole Language has one tenth that effect. There's much to be said for Whole Language and I tend to agree with its constructivist roots but not at the expense of phonics.

Of course, the observation that nearly everything works doesn't eliminate the other perils of false positive science and the correlation/causation fallacy. All three of these make it possible to latch on to ones's favorite intervention while claiming to be evidence driven. To defend against this, we must seek 2-5 times improvement in learning performance and replicable results. It also helps to be careful, honest and humble.

Motivating Students - Opportunity Isn't Enough

In his book, Disrupting Class, Clayton Christensen and his co-authors identify four objectives that U.S. society has asked of public education. Each one is incremental; that is, each adds new expectations while still retaining the previous objectives.

• 1840: Preserve the Democracy (Advent of Public Schooling)
• 1890: Prepare Everyone for Vocations (Industrial Revolution)
• 1980: Keep America Competitive ("A Nation at Risk")
• 2000: Eliminate Poverty ("No Child Left Behind")

This latest goal is both transformative and controversial. There's little doubt that society is looking to our educational programs to relieve poverty. But many educators are skeptical about their ability or responsibility to address the poverty problem. Regardless, schools are no longer judged exclusively by their top achievers or even average scores. Frequently, the focus is on the bottom performers -- often at the expense of average or high-achieving students. 

The new goal is based on three important observations:

• Educational attainment predicts financial prosperity.
• Financial prosperity of parents predicts children's educational attainment.
• The most important predictor of educational attainment is the educational attainment of the parents.
  (source here)

When a school fully embraces the "Eliminate Poverty" goal it must accept responsibility for student motivation. Previously schools were expected to offer opportunity. If a student didn't take advantage of that opportunity it was their problem -- or the family's. Now, schools must motivate students to achieve, not just give them the opportunity. The trouble is, schools don't know a lot about motivation.

Dan Pink has been studying what motivates us. There's a body of research into motivation that goes back to the 1960s and earlier. But organizations have been slow to apply this knowledge because it's counter-cultural. It turns out that carrot and stick type motivators like financial incentives, fines, privileges and so forth are effective for repetitive, mechanical type work. But when it comes to cognitively demanding tasks, bigger incentives actually impair performance.

The candle problem, studied by Sam Gluksburg is an early study that showed this effect. More recently, the Federal Reserve Bank sponsored a study of incentives which concluded:

As long as the task involved only mechanical skill, bonuses worked as they would be expected: the higher the pay, the better the performance. But, once the task called for ‘even rudimentary cognitive skill,’ a larger reward led to poorer performance.
(Dan Pink, Drive, pg 60 quoting a study by Dan Ariely et. al.)

For cognitively demanding tasks, Dan Pink has identified three motivators that are effective:

• Autonomy
• Mastery
• Purpose

This has all kinds of implications. For example, in order to place greater emphasis on student achievement, states and districts are considering merit-based pay systems for teachers. But teaching and learning are cognitively demanding activities so merit pay is unlikely to be a functional motivator. Meanwhile teachers are complaining about reduced freedom in the classroom -- a loss of autonomy.

But, the subject of this post is student motivation. Since the advent of the Carnegie Unit, students have received academic credit for seat time. Keeping students in seats is a mechanical task so schools and vice principals are compensated according to attendance rates. We even have truancy laws that require that children spend a certain number of hours in school and punish them when they don't comply. As with teachers, we're using traditional incentives to motivate a mechanical task -- simply being present.

Of course, presence doesn't equate to learning. So how can we use autonomy, mastery and purpose to motivate student learning? There are numerous opportunities and great teachers naturally apply them. Here are three examples:

Automony: In studies of whether changing instruction to match learning styles helps students learn better, researchers find that simply offering students a choice of activities resulted in better performance. This makes intuitive sense because when students choose their own activities, they're invested in the outcome and should perform better regardless of whether the activity is a better match to an individual's learning style.

Mastery: Khan Academy includes a learning map that graphically displays the topics that students have mastered and their progress toward achievement goals. It grants badges for particularly important achievements.

Purpose: How often do parents and teachers hear the question, "Why do I have to learn this?" Lack of purpose is a strong demotivator. Educurious encourages authentic learning by posing real problems to the students and connecting them with real-world experts.

We have more than 50 years of research indicating that motivations are much more complex than carrots and sticks. Yet we keep resorting to these familiar tools despite their ineffectiveness against cognitively-demanding tasks. As schools take greater responsibility achievement, not just opportunity, it will be important to apply the right motivators.

Federated Permissions - Post Facebook

The early web was strictly a public place. Everything out there was visible to everyone. Not too long afterward, private services appeared. Between the extremes of private services, like online banking, and public services, like Twitter, is a spectrum of permissions management.

Whenever permissions are restricted to some degree a credential is needed -- usually a username/password combination. Even public services like Wikipedia use credentials so that contributors can be identified. The result is that anyone that spends more than a minimum amount of time on the internet has a bunch of credentials to keep track of. This has resulted in a call for federated identity systems that let you log into multiple services using the same credentials. 

The most well-known federated ID systems are OpenID and SAML. Despite OpenID being supported by Google, Yahoo, Flickr, MySpace, AOL and many other popular sites, it remains largely unused by individuals. The trouble is that most people have solved the problem by simply using the same two or three passwords across all of their services. To them, the convenience of this approach outweighs the security risk.

In computer security we talk about two components -- authentication and authorization. Authentication is determining who someone is. Authorization determines whether that individual should have access to particular data or services. Federated ID solves the authentication problem but the much more complicated authorization issue is left to the individual websites. Not only does federated ID not solve the authorization problem -- the current solutions can make it more complicated.

For example, in order to preserve privacy, SAML issues each service a different identifier for the same person. This is intended to prevent sites from correlating user behavior from site-to-site. But it makes it really hard to grant privileges because the person granting rights doesn't know the ID that will be randomly assigned to the individual. Meanwhile, there are other ways to correlate user behavior anyway.

Which brings us to Facebook.

Facebook is really a privacy manager. Sure it supplies features like a profile, wall, forums, messaging, photo sharing, games and so forth. But better versions of most of these features are available elsewhere on the web. What Facebook lets you do is selectively grant access to your private (or semiprivate) information that is stored in those services. And Facebook's advantage is that they have accounts for a majority of internet users in North America. That lets them remain dominant despite being pretty bad about managing privacy.

What's next? With competitors envious of their position and users wishing they could jump ship it seems unreasonable that Facebook will remain dominant. I think the next step is Federated Permissions. This would be a system that lets me share private information with select individuals or organizations -- friends, family, business associates, care providers, etc. -- regardless of their identity provider.

To help bring this about. I'll start with some definitions, two Use Cases and a set of Requirements.

Definitions:

• Identity Provider: The service that "logs you in" and tells other services that you are who you claim to. In the use cases below, I suggest that your email provider is also your identity provider. That would be a convenient option though not the only one.
• Content Provider: A service that trusts your identity provider and supplies certain content and services to you.
• Public ID: An identifier assigned to an individual that others can use when granting permissions. In the use cases below, I assume that your public ID is also your email address -- also a convenient option but not the only one.

These definitions are commonly used by existing federated identity systems.

Use Case #1

Sara Smith is organizing a student exchange trip to Germany. In order to share the group's experiences with people back home she wants to share photos and an online journal. All of the students should be able to contribute material and, because the students are minors, only family and select friends should be able to access the content.

First, Sara logs into "genericmail.com" which is both her mail provider and her identity provider. She creates two new groups called "germany2012crew@genericmail.com" and  "germany2012family@genericmail.com". She loads these with the email addresses (public IDs) of the people going on the trip (crew) and family and friends remaining home.

Next, Sara creates a new blog using "semiprivateblog.org." She grants write access to "germany2012crew@genericmail.com" and read access to "germany2012family@genericmail.com". She goes to "sharemyphotos.org", creates a photo album and grants equivalent access.

Finally, she composes an email describing these services and sends it out to the two groups. Since these groups double as email groups the messages get delivered to all of the right people.

While on the trip, lots of stories, messages and photos are shared with the people back at home.

Use Case #2

Max Jones has been attending Notable Community College for two years. He's applying to Potential University where he intends to complete a BS.

First, Max accesses the application page on Potential University's website. So far, he's unknown to Potential University but he's able to log into the site using his public ID, "maxtowin@genericmail.com". Upon logging in, the Potential University site requests access to Max's personal profile. This causes "genericmail.com" to pop up a window asking Max permission to deliver the info. Max clicks "yes" and re-enters his password for verification. The application form is pre-populated with Max's name, address, birthdate, etc.

Next, the application form asks about previous education. Max enters "Notable.edu." Potential U. requests access to Max's transcripts from Notable. As with the profile, Notable pops up a form asking for Max to grant permission to share the transcript. Max grants permission and the information is shared.

Moments later Max is informed that he has met the basic acceptance criteria.

Requirements

A federated permissions system would be based on a set of standard protocols and a network of trusted services. Here are some basic requirements. OpenID and related initiatives fulfill some but not all of these.

• Public Identifiers that are well-known so that permissions can be granted as easily and conveniently as addressing an email message.
• Public Group Identifiers that can be used to grant permissions to groups of people with ongoing management of membership. (Notably, the membership of groups doesn't necessarily have to be public.)
• Revocable Trust relationships between identity providers and content providers.
• Revocable Permissions for individuals and groups to access content and services.
• Standard Policies for relationships between identity and service providers.
• Audit Trails to monitor compliance with policies and regulations.

Today, Facebook's IPO received a lukewarm reception. Could it be that the market is already anticipating a post-facebook option for managing privacy?

When Will Education Productivity Improve?

Recent studies of 1:1 laptop:student initiatives show minimal benefit to supplying every student with their own laptop. This shouldn't be surprising.

Studies from the mid-1990s showed that, twenty years after the introduction of the personal computer, they didn't substantially improve personal productivity. Indeed, annual productivity growth in the US averaged around 1.2-1.5% between the mid-70s and the mid-90s. Economists at the time had several explanations: Perhaps the benefits were mostly in quality of life improvements rather than personal productivity. Maybe inflation had hidden the benefits. Increased government regulation was consuming resources. Or perhaps productivity gains were lost to solitaire and spam (even before Farmville and World of Warcraft).

Source: Bureau of Labor Statistics

While these may be factors, two important theories were confirmed when the productivity gains finally arrived in the late 1990s and the 2000s.

First is the Network Effect, the idea that the value of a good or service is increased by the number of others that are using it. The classic example is the telephone. It's not terribly useful to have the only telephone in the world but the value of your telephone increases with each other telephone that someone buys. Network effects related to the internet really started to take off in the late 90s. These included email, the world-wide-web, e-commerce and much more. Obsession with the network effect contributed to the Dot-Com Bubble but the productivity value of the effect continued even after the crash.

Second and related is Organizational Change. If you introduce new technology to an organization but practice continues as usual, there's increased cost (for the technology) with limited benefit. The structure of an organization must change in order to take advantage of new technologies. In other words, the way goods are produced or services are delivered has to be redesigned with the new technology in mind.

Organizational change is hard. An entire discipline of Change Management has been developed around it. As a result, technology is frequently deployed in hopes that organizational change will spontaneously follow. Technologists like myself are often guilty of taking this approach. But external pressure is is usually the real catalyst for change. In the case of business productivity, that pressure came in the form of economic recessions.

The recessions of 1990-1991, 2001 and 2007-2009 have all ended in so-called Jobless Recoveries. In each case, while GDP rose -- indicating an end to the depression -- employment remained low. Many economists believe that jobless recoveries occurred  because the organizational changes were finally made that took advantage of available technology. Erik Brynjolfsson, an MIT economist, says, "It's as if the economy had a pent-up potential for labor savings that hadn't been harvested until the recession." If you look at the chart above, you'll see a correlation between productivity growth and the end of recent recessions.

Returning to education: Teaching and learning are today where industry was in the early 1990s. We've had computers in classrooms for decades. Learning Management Systems are installed at nearly every college and university in the US and in many high schools. Yet, educational achievement remains flat.

As with other industries, educational institutions will have to take advantage of network effects and make organizational changes before the potential of education technology can be realized. Unlike most industries where productivity is measured by the dollar value of goods and services delivered divided labor invested, I believe educational productivity should be measured by student achievement divided by the dollars invested in their education. So far, we have a few examples of institutions achieving superior results by this measure. Each operates very differently from traditional education. They include:

• Western Governor's University
• Rocketship Education
• Rio Salado College
• School of One / New Classrooms
• Carpe Diem Schools

A number of other examples are coming online and we're excited to see the results. I'll report on them here.

(As with most of my posts, I link to a lot of background information here. If you're interested in this topic or seek evidence to support these claims, I encourage you to follow the links.)

The Perverse Vocabulary of Feedback Loops

I come from a family of engineers. Naturally, we cannot talk about feedback loops without getting into control theory. Conveniently, engineering control theory can be adapted to education reasonably well as soon as you get over some vocabulary hurdles.

An open-loop control system.

Here is a basic control system. One example might be a cruise control for your car. In that case, the reference is the speed you want to go, e.g. 65 miles per hour. The controller translates that reference to an input value – the throttle position . Then the system (the engine, transmission, drive train, tires) produces an output – the actual speed of the car.

In control theory, this is an "open-loop" control system meaning that it has no feedback. The controller must have a very good mathematical model of the system and the system itself must be very precise to get a predictable output. Open loop systems are used when there's a large acceptable margin of error. A cooling fan, for example.

A closed-loop control system.

A car with an open-loop cruise control would slow down as it climbed hills and speed up when it descended. For more precise control, real cruise controls use a feedback loop. A sensor detects the output value (the speed of the car) and returns it to the controller. The controller takes the difference between the feedback and the reference values and adjusts the input (throttle) accordingly.

Mathematically, taking the difference is subtraction so we call this "negative feedback." In contrast, "positive feedback" sends systems wildly out of control. A familiar example is pointing a microphone at a speaker. The terrific squeal that you hear is a "positive feedback loop."

So, when talking about control systems, "closed loop" is better than "open loop" and "negative feedback" is good while "positive feedback" is bad. The precise engineering terminology is the opposite of what we might use in casual conversation. Let's now apply this to education.

A personalized learning system.

With a few word substitutions we get a personalized learning system. For a moment, let's ignore the feedback loop. In an open-loop learning system a set of educational standards (such as the various state or common core standards) are translated into instruction in the form of textbooks, lectures and exercises. These activities are delivered to the student resulting in skills. Conventional schooling is much like this. In order to handle a class of a dozen or more students, all students perform the same activities. But, remember that open-loop systems require a very precise system to get predictable results. Students come to us with different personalities, talents, preferences and backgrounds. The result is a wide margin of error as illustrated by the spectrum of grades assigned at the end of of a course.

Good teachers do better than this. They find ways to adapt their teaching to the needs of individual students. Good students, being intelligent, can also use feedback to adapt their learning to match the instructional methods being used. Both kinds of adaptation require feedback from assessment. And to do this well, the feedback should be compared with the standards for what is intended to be taught.

For feedback to be really effective, it must be frequent, fast and rich. Feedback must come often so that course corrections are made frequently. It must be fast, ideally immediate; otherwise it's too late to affect the learning process. And it should be rich. "Incorrect" is not nearly so meaningful as, "You misplaced the negative sign in step 3."

Frequent, fast and rich feedback depends on frequent, fast and rich assessments. In conventional schooling, assessment is expensive. It costs a lot of teacher time to compose, administer and score assessments. This is an excellent opportunity to apply technology. Computer supports can minimize the effort required to compose, administer and score assessments. And computers can tabulate the results into effective teacher and student dashboards. The result is a more personalized and effective learning experience. Or, as an engineer might say, "more predictable output with a smaller margin of error."

FETC Reminds Me How Hard it is to Change

Last week I attended the FETC conference. This picture from the show floor inadvertantly captures the de facto theme: "21st Century Technology that does not change the way you work.

The audience was composed of about 50% classroom teachers, 25% other school officials the the balance of technologists, vendors, district and state people and so forth. This an education technology conference and the teachers who attend are tech savvy and forward thinking. But the framework remains a conventional classroom and the attitude seems to be "Take school, add technology, mix well and serve."

In my presentation I argued for a different perspective. Digital abundance is nice but we've been adding technology to classrooms for decades with little impact on student learning. In order to the big improvements that society is demanding of schools, we have to change our education systems. Teachers will be more important than ever -- i'm under no illusions that computers can replace them. But just as the practice of business had to change to take advantage of new technology, so must the practice of education.

Resources for Today's Speech at FETC

I'm speaking today at the FETC conference. Here are links to some of the resources I reference in my talk:

• Jane McGonigal: Reality Is Broken http://www.janemcgonigal.com 
• Clayton Christensen – Disrupting Class http://www.innosightinstitute.org 
• Duncker’s Candle Problem http://anthonyogrady.wordpress.com/2010/05/02/carrots-and-sticks/ 
• Dan Pink: The Surprising Science of Motivation http://www.ted.com/talks/dan_pink_on_motivation.html 
• Learning Resource Metadata Initiative: http://lrmi.net
• Shared Learning Collaborative: http://slcedu.org

These previous posts on this blog discuss some of the same subjects:

Breaking the Tyranny of the Bell Curve
Tackling Bloom's 2 Sigma Problem
The Personalized Learning Model
The Learning Resource Metadata Initiative