Of That

Brandt Redd on Education, Technology, Energy, and Trust

11 October 2013

Things Engineers Can Teach Us About Feedback

Some time ago I wrote about feedback loops – how they are part of the engineering discipline of Control Theory and how, by substituting a few words, the principles apply surprisingly well to education.  Here's a diagram of a feedback loop according to control theory:
A closed-loop control system.
And here's the same diagram substituting educational terms for the engineering ones.

A personalized learning system.

In that post I noted that the feedback loop needs to be "closed" in the sense that we use feedback to influence the direction instruction should take. Also that feedback needs to be "negative" in the mathematical sense. That is, feedback should reflect the difference between skills the student demonstrates and standards that are to be taught. Both of these concepts, closed feedback loops and negative feedback, are derived from engineering control theory.

In this post, we'll consider three more insights we can gain from control theory: the role of a transfer function, speed and frequency of feedback, and sensitivity to what is being measured.

Transfer Functions

A transfer function is a mathematical description of the relationship between the input and the output of a system. Let's use the car example from my previous post. In that case, the input is the position of the gas pedal (more correctly called the "accelerator pedal" as we'll see in a moment). The output is the speed of the car. Pressing the pedal to a certain position doesn't make the car go a corresponding speed. Rather, pressing the pedal causes the car to accelerate at a rate proportional to the pedal position. If you keep the pedal pressed to the floor, the car will continue to accelerate to higher speeds until it reaches the limits of its construction. (For calculus fans, this means that the transfer function of a car's drive train is an integral.)

The transfer function is important because it's incorporated into the design of the controller. When an engineer designs a controller they use the transfer function to anticipate what will be the result of a particular input. Typically they take the inverse of the function to determine what input is required to achieve the desired output.

The educational equivalent of a transfer function is a learning theory – a description of how people learn. Learning theories help us select activities that will effectively help a student learn a particular skill. Descriptions of the various theories and their strengths and weaknesses are beyond the scope of this post (or my skills for that matter). I recommend the Wikipedia article on the subject. But we can derive two important insights from this:
  • A personalized learning system will inevitably express some learning theory in the selection of activities. It would be best to deliberately select the theory and design the system accordingly.
  • There are personal differences in the way each student learns. In engineering terms, this means that each student has their own personal transfer function. Therefore, the selection of activities should be tuned to the student's individual interests and affinities.

Speed and Frequency of Feedback

The time from the moment an output is measured to a resulting change in the input is called a propagation delay. In educational terms, this is the time from when a student's skill is assessed until moment a student's activity is affected by that. In a traditional math class the student does homework one day, submits it the next day and receives graded homework back the next day. Thus, the propagation delay is two days (or two class periods). Fast feedback means a shorter propagation delay. Many online learning systems offer near instantaneous feedback. Measurements that require human grading will naturally be slower.

Frequency of feedback is a measure of how often the output (or skill) is measured and feedback generated. In the traditional mathematics example, feedback is daily (or once per class period). Some traditionally taught courses may only have two or three graded activities in the entire course. However, this may be a pessimistic way to measure frequency. For example, if students can check their answers in the back of the book then feedback is both faster and more frequent

A third component of educational feedback is richness. In math, this might be the difference between being told than an answer is wrong and being informed about exactly what mistake was made. In English it might be the difference between a simple score and detailed feedback about how the student might improve their paper.

Students can influence all three feedback factors. For example, if English students seek help at a writing lab then they will be getting faster, more frequent and richer feedback than students that don't make use of the resource.

Control theory tells us that faster and more frequent feedback compensates for inaccurate measurements and poorer transfer functions. In education language, this means that if we can make feedback faster and more frequent we can compensate for a less-than-perfect learning theory and suboptimal assessments.

Of course it would be nice to have everything -- fast, frequent and rich feedback, good quality assessments and a solid learning theory. But it's useful to know that there are real tradeoffs among these factors.

Sensitivity to What's Being Measured

Feedback loops are a very effective tool; so effective that if the wrong thing is being measured or the wrong feedback is offered then the wrong skill will be optimized. A recent manifestation of this are complaints of "teaching to the test." The concern is that since summative tests are used to evaluate schools then the only skills that will be taught are those that are on the test. While this outcome is common, it's unfortunate since studies have shown that focus on conceptual understanding results in better test performance than test-focused instruction.

It's also manifest in the combination of skills that a particular problem might require. For example, a mathematics story problem might require reading, visualization, and problem solving skills in addition to the ability to solve the resulting mathematical equation. In order to offer feedback to a wrong answer, the system (whether human or automated) must be able to detect which of these skills was not applied properly. In most cases, this requires interacting with the student to discover the steps followed in answering the question.

It's tempting to try and isolate skills and only assess one at a time. There are two reasons why this won't work. First, it's very likely that you're seeking the student's ability to use multiple skills together. Second, the demand for some skills simply can't be eliminated. For example, nearly every assessment requires the skill, "Can read and follow directions."

Applying Feedback Loops

To summarize, engineering offers us the following insights about using feedback in education:
  • Choose your learning theory deliberately and measure its effectiveness.
  • Adapt not only to what the student has and has not mastered but to the individual learning patterns and affinities of each student.
  • Fast and frequent feedback can compensate for lower quality in other areas of the system. This is a two edged sword; you may think you have a good learning theory when, in fact, it's fast feedback that's making the difference. But it's also an opportunity to make deliberate trade-offs.
  • Be sure you're measuring what you think you're measuring. And don't forget that every assessment measures multiple skills.