New angle on math education: lessons from product design

Or a user interface for learning math

High school is when many students typically first encounter some of the greatest difficulties in learning what forms the basis for learning post-secondary level math. Those are, therefore, some very critical years in a student's math educational career. If it is important to get more students to consider entering a STEM career (and I'd argue it is [1]), then it is important to consider how math can be taught to these students so they'd be more interested in it, and be better at it later on.

The following is a long piece discussing the problems facing designing a better math educational experience, complete with the principles of design that can be used to solve such problems, closing with some guidelines for how to create a better course.  In principle, all of the following can apply equally well to other subjects, like computing science, or English, but math education is tough, so I’m picking on the hardest as an example.

Math course as a product

Just how do students experience of learning math get formed? Rather than looking at it in a traditional teacher's mindset, let's look at it in terms of a commercial product designer's mindset. That means we see students as interacting with a complete, full-fledged math education product that promises students that they'd get what they want.

This product is, of course, government regulated. There is a government curriculum that must be followed, but it is rather abstract in its prescription of what students must learn. The curriculum is like a product specification where the implementation is left as a detail to the product designer.

The product designer has a great deal of freedom in how to implement the specification, i.e., the curriculum. To help the designer, there are government approved textbooks with a book publisher created implementation of the curriculum. The textbook is, however, not a full-fledge end-product that most students can use.

Most students are just not autodidact, and they cannot learn effectively from reading the book alone. Instead, most students enroll in a course where they attend class and use the teacher, the textbook, and whatever else the teacher deems necessary, to get what they want. Most of these courses are also traditional textbook driven courses --- the kind you'd see in the vast majority of high school classrooms in Canada and the USA.

"To get what they want" — I keep repeating that phrase rather than say to learn the math they need to know. That's because we're treating the student as a user of an end-product. Vaguely speaking, the product the student is using is the math course they can enroll in. They are given the opportunity to use the product, and they get to use it however they want for whatever purpose they want. That means they have a choice to not enroll in the math course, for example.

For students younger than high school grade, the choice to enroll or not is actually essentially non-existent since they are required by their parents and by law to enroll. When it comes to high school and university students, the choice is usually not whether to enroll or not, but rather is between choosing math courses of varying levels of difficulty.  That's because if a student intends to gain a high school diploma or a university degree, they often have no choice but to enroll in some math courses in order to graduate.

Think about that again. If a student intends to graduate school or to move up the academic totem pole, they essentially have no choice but to enroll in some math courses. That means when we look into many math classrooms, especially those in a high school, students there essentially have no choice but to be there — or else not graduate.

I bemoan this trivial point because it easily leads us astray. It's easy to now say that students only enroll in a math course in order to get a good grade, and to leave with enough knowledge to continue to do well in the next math course they enroll in; so that they can finally graduate. No doubt some students are motivated merely to get good grades, but for most students, it is only but one rationale for enrolling in a math course — and enrollment is essentially mandatory.

In order to design a better product, we have to understand who the users will be and what purpose they will be using our product for.

Notes:

[1] I've argued as much in the past. See Why push math education onto students? and Where are the Canadian STEM students?.

---------------------------------------------------------------------------

Knowing the market: or what math students want?

There are many students, and each can have many desires in taking a math course.  The various motivations include:

1. get good grades (for whatever definition of "good" is to the student)
2. get good enough grades to graduate
3. learn enough math to do well in the next math course
4. learn enough math to do well in many subsequent courses, not necessarily all math, to attain some higher degree or career
5. impress the teacher by doing what the teacher wants
6. impress the teacher with what they know
7. impress the teacher with their grades
8. impress friends with what they know
9. impress friends with their grades
10. impress parents with what they know
11. impress parents with their grades
12. impress the parents by doing what the parents wants
13. learn interesting ways to solve problems presented in the course
14. learn interesting ways to solve problems and puzzles beyond what's prescribed by the course
15. learn interesting ways to see the world
16. learn to do something deemed "cool" by whoever has such authority in the student's view
17. just to have fun
18. to face a challenge and succeed
19. to be intellectually stimulated
20. to enjoy a sense of adventure
21. to experience the thrill of understanding some eternal truths to the universe
22. to appreciate a certain kind of beauty

These are just some of the possibilities, and there are certainly other motivations. Some of these may seem redundant, but are actually not, and students can have more than several of these and other motivations at the same time.

Unlike designers of products to be sold into a free market, educators do not have the luxury of defining their market however they see fit. Educators cannot just decide to design and sell into a niche market. The product must have universal appeal (or not — it is a captive audience, but let's not be cynical).

Certainly, educators can't just out of the blue decide only to appeal to students who want a leg-up in learning steps to solving grade-school math problems; who wants to be led along essentially traditional blackboard written explanations but through pre-recorded, replayable, ten minute videos; who wants those instructional videos to be randomly accessible and can be used in the comfort of their own internet devices at a time of their choosing; who may enjoy the social networking aspect of the web site where the videos are hosted; and who has the motivation to self-select themselves into using these videos in good faith in the first place (hello, Khan Academy).

The "need" (captive audience notwithstanding) for universal appeal is true of core courses like math and science (versus complimentary or "options" courses, which have more freedom to define their intended market), and is especially true of public schools (versus private schools, which can define their market to some generic extent). Having said that, a single math course in a private school still cannot be designed and sold into just any niche market — most likely all students in that grade at that school (who, by virtue of being there, has parents who could afford the tuition) must be taught in any case without exception — hence needing universal appeal.

Since educators essentially cannot ignore any part of the market, the kind of students we have in a course is all kinds of students. They'll have diverse backgrounds, varying math abilities, and hold many different and sometimes contradictory motivations and reasons for being there. They'll want all kinds of various things out of the course. A teacher of such a course will have to account for all kinds of students.

Designing a better product, i.e., a better math course, is clearly not easy. Not only are the students in a high school math classroom essentially there without a real choice to not be there, but their motivations and desires while being in that course is diverse and possibly contradictory. On top of that, the teacher doesn't really have a choice as to who gets to be in the room.

If the teacher is the product designer, the job is to design a course that has, as much as possible, universal appeal and value.

---------------------------------------------------------------------------

Who's minding the shop?

Let's back up for a moment. Earlier I mentioned that many math courses are traditional textbook driven ones, and that a course is the end-product a student uses. Can you figure out who is the product designer of a traditional textbook driven math course?

If you thought it's the classroom teacher, you're being very reasonable, but wrong. After all, the curriculum was designed by a government committee, the textbook by a publisher [1], the teacher may be strong-armed by the school or the department to doing things a certain way, and just the normal schooling cultural tradition of Canada and the USA dictates a certain kind of classroom activity [2]. In other words, it was designed by a super-committee of all those people and tradition.

“I would rather deal with a tyrant any day than with a committee. Committees, as a general rule, aren't willing to take chances, which is why you have a committee in the first place — so you can share the blame.”

Hal Riney, Art and Copy (2009)

If you're into design, you know design by committee is an invitation for bad designs. Does that mean the traditional textbook driven math course is a broken product?

I hate to call broken anything as complex as a whole tradition and profession for teaching a subject as complex and as old as math. After all, high schools keep churning out able-bodied men and women, many who go on to university; in turn, universities keep churning out graduates in all the various STEM fields. If traditional math education is broken, that's the kind of broken we can say is "good enough".

Good enough for now, however, is sometimes just not good enough when moving forward. At a societal public policy level, I think it's important we improve math education [3]. At a more local level, that means improving the math course as an end-product that students use.

So let's ask the product designers to do a better job. Oh wait, the product design super-committee isn't in session!

I jest, but government committees on curriculum development are political, slow, and essentially cannot be influenced by any single teacher, parent, or student. Publishers are interested in creating a product that sells: their product is the textbook, and their customers are school boards and government committees — and we've already said those committees are political, slow, and essentially cannot be influenced.

Who's left? The lone classroom teacher [4]: assuming they can avoid being strong-armed by the school or the department into doing things their way or the highway. We'll make this assumption moving forward in our discussion, but it's important to recognize that many teachers, especially young, new teachers, really have no chance of not being strong-armed [5].

It's not that government committees on curriculum development are unimportant, by the way. It's just that market disruptions don't usually come about from playing the game the incumbents are playing, and the incumbents here play a hard game of politics, lobbying, and influencing of educational experts in academia and various trade associations.

For those of us who value effective solutions in the real-world, and in the spirit of hackers and startups, our mindset is better focused on quick-to-market implementations, on fearlessness of iterative and incremental improvements to the implementations, and on creating a better product for the end-users.

The only people who can do any of that for traditional schools are the classroom teachers in them [6].

Notes:

[1] The publisher may even be one that is used to churning out bad textbooks. See "Afraid of Your Child's Math Textbook? You Should Be."

[2] See "The Teaching Gap" for a vivid description of how local cultural expectations affects how students are taught.

[3] I've argued as much in the past. See Why push math education onto students? and Where are the Canadian STEM students?.

[4] As an aside, it's interesting to see the mainstream rhetoric regarding teachers, especially in the USA, but also prevalent elsewhere on the internet. The rhetoric is so bad that Jon Stewart lampoons it. The comments pages in many news articles are filled with negativity and calls for teachers to work harder, or accept less compensation: e.g., see Ofsted chief angers unions with 'work harder' comments for a UK example, and B.C. school boards removed from teachers' contract negotiations for a Canadian example.

It seems many people don't realize that the front-line classroom teachers are the only actors in the education product design super-committee who can respond to the needs of the particular students in their classroom, and who can be innovative in creating a compelling product through iterative and incremental development. Of course, teachers have to be given the freedom and the tools to be innovative, but that's an issue of education (meta-education? No, it's called requiring advanced teacher education and continuous professional development, and keeping politicians and non-professionals from micro-managing what teachers do).

The government curriculum design is also important, but for the most part, they've actually essentially been regulatory captured by the education industry participants (trade associations, publishers, and, for better or worse, some politically connected or prominent educators) and various lobby groups (often with political agendas). Historically, it seems these participants are often not innovative, or wishes to promote silver bullet solutions that prove ineffective — hence all the bandwagons that's come and go: "mastery learning, portfolio assessment, cooperative classroom structures, technology integration, backward design, multimedia projects, personal learning paths, authentic task development and, most recently, differentiated instruction and integrated curriculum" (Trends in Education: How They Come and Go).

Recently and especially in parts of the USA, government curriculum design has been majorly influenced by lobby groups, often with political agendas. See Science education vs. high-profile ignorance. Political issues require political solutions, and I'm going to leave that one alone.

[5] Like many medical residents and doctoral students, many beginning teachers are treated as though they are only just starting to learn the real art of the field, as though they must still be forged in the cauldron of pressure, rapid fire pace, and abuse — because that's how the old and experienced hands in the job were themselves raised (Didn't everyone back in their day walk to school and work, uphill, both ways, in the snow and rain?). It is a recognized issue, and unions and trade associations have tried to take the edge off of this practice, but it's still there, for better or worse. But that's a whole different discussion entirely.

[6] People who want to create the next Khan Academy, please go and create it. Go and do it. Code wins arguments; Show us your implementation. In the mean time, the vast majority of kids are going to school, and learning at school — there's no reason to write off schools instead of improving it.

---------------------------------------------------------------------------

What's a classroom teacher to do?

Classroom teachers are really hemmed in from all sides in the struggle to designing a better product.

• From the government, there is a mandated curriculum.
• From students and parents, the normal schooling cultural tradition of Canada and the USA dictates a certain kind of classroom activity.
• From peers, the school or the department may expect certain ways of doing things.
• From resource materials, textbook publishers [1] are more interested in ticking off boxes to hit all the latest buzzwords and to fulfill all the government regulations, rather than being interested in creating a product that actually improves classrooms in a measurable way.
• From life, there are only so many hours in a day, and the demand on a teacher's time has only grown more and more (e.g., multiple extracurricular clubs, school committees, unpaid work for the board, professional associations, fixing computers and photocopiers).

What's a classroom teacher to do?

Let's start with the textbook

The job usually starts with the textbook because that's where most traditional textbook driven courses (TDC) start, at least in Canada and the USA. It usually start with whatever textbook the school has in stock, since the teacher can't just dictate spending a large part of a non-existent budget to buy their favorite textbook. Whichever textbook is used though, so long as it is government approved, the textbook usually guarantees coverage of the government mandated curriculum. So the job in a traditional TDC is simple: teach everything in the book, test the student against what's taught in the book, and we're done!

Maybe the teacher feels more comfortable skipping around the textbook a bit, using the chapters in a different order. Maybe the teacher thinks one part of the textbook isn't so hot, and substitutes or, more than likely, supplements that part with some other photocopied material. Maybe the teacher thinks one part of the book requires more than than another and schedules the tests accordingly. Maybe the teacher spices some things up with some "technology" [2]. Finally, the teacher decides every day how to present the material to be taught in that day's topic or chapter, usually despite the fact that the material is also presented textually in the textbook pages prior to the exercises for that topic, then the teacher presents in essentially a lecture format (with limited student interaction). Those are the kinds of choices a teacher makes in preparing for a traditional TDC [3].

And there is nothing "broken" in this picture. After all, high schools and universities keep churning out graduates, year after year.

If our goal is to increase students' appreciation and consideration of STEM careers, however, then perhaps classroom teachers want or need to make some changes to how math is taught [4]. What's a teacher to do?

Whatever the teacher does, it must fit the curriculum, must fit the school budget, must respect students' and parents' expectations, mustn't burn the teacher out, and must respect the experience of the teacher's department and school peers. The latter easily leads into a discussion about politics, so I'm avoiding it altogether, but for the sake of our discussion, we'll assume the classroom teacher has the confidence of their peers to do whatever is necessary to get the job done. Oh, and part of the job is to get students to become better at math, and to like it more.

Still, what's a teacher to do? Many teachers and education experts would, at this point, parrot all kinds of bandwagon solutions: mastery learning, portfolio assessment, technology integration, personal learning pathways, and learning through authentic tasks. I've sat in meetings where the buzzword to sentence ratio is greater than one.

A big problem with many of the bandwagon solutions that gets regularly trotted out in discussions about improving education in the classroom is that they are either not actionable, or cannot be proven to work (worse, some are so vague as to be unfalsifiable). Part of the problem is that education research is hard to conduct, since any experiment you try to implement will be an experiment on kids (sound the alarms!). Another part of the problem is that a lot of education research isn't even scientific, but rather literary in tradition.

That just won't do.

Notes:

[1] Sadly, many companies providing resources to teachers are motivated in the same way as textbook publishers. E.g., the kind of product a digital whiteboard manufacturer have to demo and sell to the purchasing decision makers in a school board may very well not be the kind of product that actually helps teachers and students in real classrooms.

[2] "Technology" in education is a funny word. I often wonder if mechanical pencils count, and whether students in "Educational Technology" (EdTech) programs in University education departments are taught how to use and fix them.

[3] It's interesting to note that even the "revolutionary" Khan Academy essentially does the same thing, as essentially admitted to by some of its own employees working on its math section. They take the latest Common Core math standards, and then they push themselves to "cover" it with video recorded lectures and exercise questions generated by machine or "experienced content creators" — and "With each new exercise, [they're] making complementary videos that explain the exact concepts the exercise covers, so [their] videos are also aligned to the Common Core standards".

Khan Academy is revolutionary: it's a revolutionary textbook publisher taking the textbook into a new medium, and publishing it for free. Their customers are not the students, students' parents, or teachers, but are its benefactors like the Bill and Melinda Gates Foundation, and Google.

This is not a criticism of Khan Academy, but just an honest observation of who they are and what they do. I've embraced and used Khan Academy content in my own classroom in the past, but stopped when I found it wanting in actual, real-world usage. At the time of writing, their content hasn't improved enough for me to re-embrace it.

[4] I've argued as much in the past. See Why push math education onto students? and Where are the Canadian STEM students?.

---------------------------------------------------------------------------

Inspired by Product Design

Product design principles are interesting because, although some of it is scientific, much of it is not. They are, however, proven in the marketplace. If a set of design principles leads to the creation of products that don't sell, those principles are abandoned by necessity (on pain of the company going bankrupt, or the designers getting fired) [1].

Rather than looking at designing a math course in a traditional teacher's mindset, let's look at it in terms of a commercial product designer's mindset. That means we see students as interacting with a complete, full-fledged math education product that promises students that they'd get what they want.

As the product designer, we're well aware of the regulations and requirements we've already painstakingly discussed before. We're aware of at least some of what the users want from the product. We're aware of the pressures on costs of what goes into the product. We know the job is to design a course that has universal appeal and value for the students who essentially have no choice but to be there.

We're also aware of some design principles.

What design principles?

The word "design" is used to mean many things, e.g., graphics, arts, manufactured products. By "design", I mean the design of a product the user can use to do something to get what they want. Think of it as the design of the user interface and experience of computer programs like Photoshop or any large numbers of mobile apps. It's not really about pretty graphics and art, but rather the focus is on the product's usability.

There are entire books dedicated to talking about design, so what I want to do here is just point out some very interesting and inspirational sources. They include:

Kathy Sierra's writings about user interface design in Creating Passionate Users [2] and Serious Pony, and her talk at Business of Software 2012 on "Building the minimum badass user", are very interesting. In fact, what she talks about makes for more actionable advice to "student-centered education" than anything I've read about "student-centered education" ever has.

Daniel Pink's books "Drive" and "To Sell is Human" are excellent, including his "accompanying" talks at Ted Global 2009 on The Puzzle of motivation and at Business of Software 2012 on "To sell is human. The surprising truth about moving others". Daniel Pink should be required reading by teachers everywhere, since teachers deal with motivating and selling to students on a daily basis as a huge part of the job.

For similar reasons, I'd recommend Robert B. Cialdini's book "Influence: The Psychology of Persuasion".

Alan Kay, the famous computing scientist, has a very interesting talk on "Doing with Images Makes Symbols" (parts 1 and 2). The talk has parts that are essentially about teaching and learning, but mainly concerns itself with how a certain products are designed (computers, computer user interface, and programming languages).

Alan Kay's talk also raises some interesting cognitive processing issues that have much better scientific underpinnings elsewhere. The part where Kay talks about a mind that uses in parallel a number of different mentalities is much better explored, at least from a linguistic understanding point of view, in Ray Jackendoff's book "Foundations of Language: Brain, Meaning, Grammar, Evolution". The issue about doing with images being an essential underpinning of symbolizing is brilliantly discussed in Ronald W. Langacker's book "Cognitive Grammar: A Basic Introduction". A computational model of a part of what might be happening when these views are combined is explored in the thesis "Cognitive modeling of sentence meaning acquisition using a hybrid connectionist computational model inspired by Cognitive Grammar".

W. Timothy Gallwey's book on "The Inner Game Of Tennis" makes for a fascinating read as well and is squarely about education. Specifically, tennis education, which is itself just a discretionary product as practically everyone who enrolls into tennis lessons do so voluntarily, and would quit as soon as it's no longer useful for getting what the user wants.

James W. Stigler's and James Hiebert's book "The Teaching Gap" is a must read for all math teachers interested in understanding the cultural tradition (or baggage, depending on how you look at it) in the math classrooms of Japan, Germany, and the USA (and Canada is largely not too different from the USA in terms of math pedagogy, especially because of the influence of the National Council of Teachers of Mathematics).

There are many more sources to begin with for looking at building a math course from the point of view of designing a product, but those are some pretty amazing ones to start with.

Notes:

[1] Obviously, the same cannot be said of education research. Many teachers are tenured and union protected. Public schools essentially cannot go bankrupt. Education professors are tenured, and their theories need not necessarily be tested against reality the way theories in experimental physics are. Like memes, a lot of pedagogical principles succeed or fail only in terms of whether it spreads in the minds of the community, not necessarily whether it works in the objective reality of a classroom.

That's not to say that teachers are not experts in pedagogy, or that teacher education in university is worthless. Many teachers admit that they learned most about teaching during their internships and in the first few years of teaching, that is, whilst immersed in field work. Like medical residents, for whom the reality of what goes on in the E.R. just cannot be vividly captured by classroom lectures or textbook descriptions, it turns out teaching similarly involves knowledge that cannot be so captured. The art of healing is, however, backed up by a strong basis of medical and biological sciences. The art of teaching has very little high-quality scientific research to back it up, so individual teachers are left to figure it out in the field, hopefully with feedback from field experts whilst attending university teacher education programs.

That is also not an argument against teacher tenure or public schools. Maintaining tenure and public funding is a political decision with important and valuable political and public policy consequences. That's better left for another discussion entirely.

[2] There were some dramatic and unfortunate events that put an end to Creating Passionate Users.

---------------------------------------------------------------------------

Math Course as a Desirable Product

If you haven't, you really must watch Kathy Sierra's talk, "Building the minimum badass user" (Business of Software 2012) [1]. I'm going to discuss building a math course as a product beginning largely with what Sierra discussed in that talk. What that means is that we're treating the student as a user of an end-product. Vaguely speaking, the product the student is using is the math course they can enroll in. They are given the opportunity to use the product, and they get to use it however they want for whatever purpose they want for the duration of the subscription (oops, I meant course term).

As designers, we want the "product to be REALLY REALLY desirable. Sooooooo desirable that [students] MUST have it... more sustainably desirable than the competition" [1], which may include other courses the student is enrolled in, but also includes at various times of the day (e.g., parent moderated "homework" time during the evenings), such competition as: YouTube, Reddit, Xbox, "TV" programs (not necessarily on a TV set), etc.

A common answer from educators is that we must enhance and increase student engagement. But like commercial products that Sierra talks about, our product also does "not have a problem for which 'more engagement' is the answer" [1]. After all, remember that students in our math classrooms have practically no choice but to be there anyway (on pain of not graduating or going to a university program of their choice). Save lapses in concentration and focus due to the depletion of the very same pool of cognitive resources from which their willpower, cognitive processing, and self-control draws from (Sierra, Your app makes me fat), students are as engaged as they need to be within a traditional textbook-driven course within the cultural tradition of a Canadian or American classroom. In fact, any competent teacher will have had a lot of skills, knowledge, and experience in getting students reasonably engaged with just about anything at all, within reason.

The gamification that Sierra talks about in her Business of Software talk is also very old-school compared to what happens in the typical high school setting. Sure, commercial products these days usually execute on gamification in their products in more effective and fun ways than what schools do.  At the same time, high schools' common overweighted focus on grades and marks (no less cheered on by many mainstream commentaries) is no less a form of gamification for, as commonly used, grades and marks is less about effective assessment and more about operant conditioning.

Every time a teacher incentivises a behaviour, like handing work in on time or getting an extra question correct, by awarding bonus points is gamifying the product. In elementary schools, the same thing happens but instead of marks, some uses stickers and gold stars. Even non-bonus marks can work in gamifying the product: to see this, instead of calling a students grade their grade, call it their "hit points". There's a lot that goes on in a classroom that will give a student the good feelings of a dopamine hit when a desired behaviour is displayed, and a lot of what goes on is teacher driven. Praise from the teacher when a student gets good marks, the aforementioned bonus points for small discrete pieces of work, school-based awards and medals for achieving marks over a certain level, and stickers for good grades (yes, some still use them even in high school).

A teacher once told me students essentially have a feeling towards their grades not unlike a feeling towards money. Especially in high school, marks are very often used as a tool to incentivize or coerce students into doing certain things. It doesn't help when many parents award material bonuses based also on marks (An iPhone for a 90% average, ok?).

Authenticity, Engagement, and Motivation

If we want students to engage with our product at a higher level, to think more effectively at a conceptual level (because that's what good math work requires), then "We need to be sustainably desirable WITHOUT having to bribe, incentivize, coerce them" [1]. The research backs this up.

It turns out that contingent, extrinsic motivators — the "if you do this, then you get that" [2] motivators — "dulls thinking and blocks creativity" [2]. "Once the task called for 'even rudimentary cognitive skill,' a larger reward 'led to poorer performance" [2]. This is a well known result amongst educators, and one of the reasons high-stakes testing is often frowned upon, but how do we translate this result into practice? One common answer amongst educators is to find "authentic" learning tasks at an appropriate level of difficulty that are "meaningful" for students to do, but because the terms "authentic" and "meaningful" are themselves poorly understood, it translates into little to no effective and actionable advice.

"Authentic" and "meaningful" often gets translated to mean "real-world" or "experiential". So we get students hacking away at metal frames for a robot body to ostensibly learn some physical principles authentically, but who come away never having touched how Newtonian force calculations could've or should've been done. We get students working on solving "real-world" math problems about how Sally could calculate the cost of buying wallpaper given she measured the perimeter of the room and the wall height in centimeters, but the wallpaper is sold per square inches. By the way, I'm not making these illustrative examples up either.

There is also a set of common responses to engendering intrinsic motivation that revolves around building positive relationships between a student, their peers, and the teacher. Those are good advice, but says little about what learning activities students should engage in. Similarly, the advice to allow students greater autonomy is good, but also provides little clarity regarding learning activities. Unfortunately, some teachers use relationships and autonomy as part of a carrots-and-sticks pattern of extrinsic motivation, so these elements, while important, do not guarantee that the motivation built will be intrinsic in nature.

So you can see how from knowing intrinsic motivation is good, educators easily got side-tracked into promoting "multimedia projects", which should go into a student's "portfolio assessment", of "real world learning" that revolve around problem-solving math "word problems" that promote "experiential learning", which helps students "socially construct knowledge" for themselves. It's easy to get side-tracked when the answer to, amongst other things, needing intrinsic motivation is something as vague and non-actionable as providing "authentic learning" (see 1, 2, 3 if you want to learn about what counts as authentic learning), while the more actionable advice doesn't speak to what learning activities are required. It's not that authentic learning is bad, by the way, it's just that it's not very actionable advice, and the explanations of what makes a learning task authentic is so vague that even tasks that doesn't feel or seem authentic can be argued as being authentic [3].

Case in point, gamification.  To the non-expert, gamification has something to do with computer games and how hooked kids are to them.  Game makers somehow figured out a way to get kids ultra-engaged.  The fact kids are hooked must mean they find it meaningful, and thus authentic.  If only teachers can use the same tricks to get students engaged in math.

It's no surprise then that, in the name of authenticity and motivation, well-meaning teachers go around extolling the virtues of "gamification" and learning math and logic through mobile game apps without seeing how that easily dries up students' intrinsic motivation for learning something meaningful and authentic.  Just because children enjoy the externally provided rewards achieved within a system of "gamification" more so than the externally provided rewards achieved within a traditional system of grades and parental and teacher provided social recognition doesn't mean the child's motivation is any less extrinsically triggered.

Gamification, as Kath Sierra noted in the context of commercial products, has very few narrow niches areas where it can be deployed to positive effects; the rest of the time, it's harmful.  It is a trendy application of operant conditioning; it is not a metaphor for, but is literally treating the students as rats in a Skinner box, hoping the very same behavioural conditioning responsible for making "successful" slot machines and cocaine sales will somehow make students better people.  Sure, we'll get students who provides behaviour we may say we want to see, but it doesn't produce a sustainable business for the "sale" of math education.  As soon as students leave the high school math classroom casino palace, as soon as doing a few more math questions no longer awards them an artificially created gameified reward that gives them a hit of dopamine, they will quit math like a bad habit.  That is the outcome of a bad product pushed on students through a poor business model.

Maybe some teachers mean "gamification" only as the use of some computer games, on the desktop or on mobile devices, because they are non-experts and didn't understand the difference between "gamification" and computer games.  Yet that is still problematic.  Older children like high school teenagers are not so stupid as to believe that computer games have some higher order function or meaning in the world.  They know computer games do not end wars, alleviate hunger, or cure cancer.  They know the difference between fantasy and reality, and computer games are purely a fiction to entertain them, to make the time pass with less boredom.  They know this so well that there are entire organizations that tries to tap into the youths' desire for making a meaningful difference on the world while they're still young, organizations like: Amnesty Youth & Student Group, Challenge Day, Do Something.org, and We Day, an initiative of the Free the Children organization.  Some of these have, in the past, been heavily advertised by the likes of MTV, targeting a very young demographic.

So in the context of some very savvy teenagers, what do some teachers do?  Show students trivial applications of mathematics to meaningless computer games.  Graphing quadratic equations to the path of the birds flown in the Angry Birds game app [6].  Talking about the use of logic to play games like Candy Crush [7].  If students see math applied only to meaningless problems invented on top of meaningless entertainment, guess what the logical inference might be as to the meaningfulness of math: that math is in fact less fun, and even more meaningless than computer games.

That doesn't mean students won't do the math problems teachers ask of them.  They'll play any game [8] the teacher sets up as long as they need to in order to get what they want.  We've already analyzed what math students want and even created a list of possible motivations in a previous section above.  So of course students will engage within a gamified math classroom, because as already mentioned, we don't have an engagement problem.  Students may even be happier with the engagement given that games are fun to play.  However, let's not blind ourselves to the fact that students in such a classroom are not being taught to value math as meaningful, their intrinsic motivation isn't being engendered, and that classroom is not helping make the students sustainably desire the learning and using of math for years to come (hopefully well into post-secondary studies and into STEM careers).

How do we engender intrinsic motivation then? To start, we do what many teachers already do, which is building good teacher-student relationships as well as building good relationships amongst students in the class. Furthermore, according to Daniel Pink, how we motivate people with intrinsic motivators revolves around three elements [2]:

1. "Autonomy: the urge to direct our own lives.
2. "Mastery: the desire to get better and better at something that matters.
3. "Purpose: the yearning to do what we do in the service of something larger than ourselves."

Building a product that promotes these three elements in addition to promoting positive social relationships may help engender intrinsic motivation, but these elements don't promise that students will learn the right stuff better. In fact, they don't even promise that users will sustainably desire the product more. They only help users be better motivated to continue working on, and be more engaged with, tasks that they may not have a choice but to do. That's important, but incomplete, and that's okay, because what activities students learn with, and how students' desire for the product can be increased, are different topics entirely.

Sustained Desirability

Sierra argues that one strong aspect of what makes a product sustainably desirable is word of mouth: "Word of Mouth (WOM) drives sustained desirability" [1]. "For something to be really desirable, we need, and for us to have users who potentially want to get it, we need other users telling potential users, 'you need to get this.' " [1]. Word of Mouth is people saying good things about your product, and people say good things because they perceive the product as being awesome.

The desire for word of mouth drives many companies into optimizing for the perception of awesomeness rather than optimizing for the users' awesomeness [1]. That's why we get some educators creating courses that are buzzword compliant with the latest bandwagons in education. They want their course to be perceived as awesome, and they themselves perceive the latest bandwagon and buzzwords as what makes a course awesome. It doesn't have to actually make their students awesome, because that's not their focus. They're optimizing on the perceptions around their product, not the students' abilities.

Worse is that these bandwagon buzzwords compliant courses are optimizing on the perceptions of awesomeness, not as perceived by students, but as perceived by fellow educators in the industry. Most students can't tell the difference between properly created formative assessment rubrics versus a summative marking scale masquerading as a self-assessment rubric anyway, and quite frankly, they mostly don't care or don't know why it's important. So when an educator emphasizes a course as having all these great buzzword features, they're really selling to ignorance or else selling to fellow educators.

The math course product doesn't have an engagement problem, as we've already seen. It has a word of mouth problem, not amongst educators, but amongst students (the users). This is quite obvious when we realize how socially acceptable it is for people to say things like "I can't do math" and be proud of it, but that's not true of "I can't read" [5]. Sierra argues that to get the right kind of word of mouth, we have to focus on user awesomeness.

We want to optimize for user awesomeness "because having the users view [the product] as awesome is the natural side effect of doing the right things... if we’re going to reverse-engineer a really sustainable desirable thing, we shouldn’t be looking at the thing, because the key attributes live in the user, not the thing. So we have to look at what makes those users successful, that drives them to talk to other people. So desirability is really about the user getting results" [1].

We want the users — the students — to get results. And by that, we mean that we want users to be awesome. And by that, to use Sierra's terminology again, what we mean is we want users to be badass. Because "People don't use your app [or product] because they like the app. They use your app because they like themselves" [1]. This doesn't mean giving the students the means to pretend they're badasses (say with fluff projects and non-consequential extrinsic and highly visible rewards), but to help them actually be badasses.

Badass at what?

In a math course? I don't know off hand. Sierra, however, urges us to ask: "What does the user do with or because of you? What bigger things do you enable? Nobody's goal is 'badass at your tool'" [1]. For example, "Killer at Final Cut Pro vs. Maker of killer videos" [1]. What is the meaningful context for our product?

This distinction is important because a lot of excellent teachers of traditional textbook-based courses, and a lot of teachers who "teach to the test", end up helping students become badasses at the tool: the techniques taught to solve certain patterns of problems that only exist within the context of the course. No wonder students don't think math is relevant to the real world [4].

Some examples of meaningful context may be: video editing (not Final Cut Pro), cooking (not recipe app), being a prediction, forecasting, and business modeling master (not spreadsheet master) [1].

So what bigger cooler thing(s) is the math we're required to cover (per the government mandated curriculum) a part of?

Sierra [1] also asks us to imagine what a user would write in our ideal Amazon review of our product, but points are deducted anytime we imagine that the review starts saying anything great about the designer or the product; so the review can really only be talking about the user in some way.

So if a student using our product were to write our ideal Amazon review, what would they write? They might write (complete the blanks): "I was able to do ___", or "I now can understand that ____"

Notes:

[1] I'm going to be referring a lot to Sierra's talk, "Building the minimum badass user" (Business of Software 2012).

[2] See Daniel Pink's book "Drive", and his "accompanying" talk at Ted Global 2009 on The Puzzle of motivation.

[3] The "10 Characteristics of Authentic Learning Activities" slide deck provides a pretty good summary of the points commonly used to explain what authentic learning activities are. I personally have an idea or feeling of what effective authentic learning involves, and the characteristics and points in that slide deck certainly agrees with my belief. Having said that, I went through the characteristics and points, and with a little mental and linguistic gymnastics, even many traditional textbook driven courses that I've seen, no doubt some of which are similar to those described in the book "The Teaching Gap", can be argued as involving a lot of authentic learning activities. For other teachers who may have a slightly different idea or feeling of what authentic learning involves, no mental or linguistic gymnastics would even be involved.

[4] I have a friend who swears that all the math she's ever learned in most of high school and university have been completely useless to her. She had great marks, and is now a credentialed and working computer engineer! It's no wonder that so many people think math is useless when even some engineers think that.

Our conversation may stray and apply down to the upper elementary grades (grades 4 and up), and even up to post-secondary levels, but we'll maintain

[5] A quick Google search finds lots of references to this issue. In particular, Why Is It Socially Acceptable To Be Bad At Math?  The perception is so acceptable that a senior official, Lois Lerner, with the IRS (the tax agency in the USA) thought it was okay to admit "I'm not good at math" when confronted with a reporter talking about a quarter of 300 being about 75 or so, then giving a non-apology apology by saying "But I'm a lawyer, not an accountant, sorry!" (IRS Official: 'I'm Not Good at Math'), as if dividing 300 by 4 requires a Chartered Accountant.

Imagine a senior librarian saying the equivalent: "I'm not good at reading... But I'm a lawyer, not a novelist, sorry!" The issue isn't with whether every IRA official should be good at accounting math. The issue is that an otherwise ordinary person speaking publicly to a reporter thought the perception of being bad at rudimentary junior high, if not elementary, school math is socially, if not professionally, acceptable. The public backlash that occurred afterwards confirmed this by focusing the outrage on the fact that someone senior working at the tax agency isn't good at math, rather than an ordinary adult with an advanced western education isn't good at math.  The perception is there because math courses have a word of mouth problem.

[6] I use the word "app" to try to capture the sense of "wonder" that many people have of what amounts to software programs no different than the software applications created and sold since digital time immemorial.  Apparently though, because it's an "app" now, it's special amongst the non-computer-experts.

[7] Using games like Candy Crush to learn the application of logic is something I heard recently from a group of high school (grades 10-12) teachers in 2013 September.  It sounded so cutting edge that I almost forgot that I did exactly the same thing with Microsoft Minesweeper almost 5 years ago with a class of grade 9 math students.  Maybe the opinion that Candy Crush is more fun than Minesweeper, and maybe the "fact" that Minesweeper isn't an "app" [6] makes a difference?

[8] By "game", I don't necessarily mean computer games here.  Teachers often sets up various socially constructed games to get students to do what they ought to do too.

---------------------------------------------------------------------------

How users get superpowers

If we want users to sustainably desire our product, we want our users to be awesome from using our product. And by that, we mean we want users to be badass. And by that, according to Sierra [1], we mean we want users to be experts from using our product.

Expertise is defined as "Given a representative task, experts perform in a superior way, more reliably than experienced non-experts" [1]. Given a representative task, experts perform reliably in a superior manner than experienced non-experts; experts consistently perform better actions and make better choices reliably: one single act of brilliance isn't enough [1].

How to build expertise?

There is a myth that expertise comes merely from having more knowledge, experience, or natural talent: "Experts DO know more, but knowledge alone doesn't make them experts. Experts are not what they know, but what they do" [1].

Building expertise requires three things, in order of importance:

1. Models: repeated exposure to high-quality, high-quantity examples of the performance, process, results of badass users.
2. Edge Practice: a progressive series of [deliberate practice] exercises designed to build precise, measurable, fine-grained skills within 1 to 3 sessions.
3. Forward Flow: a clear, believable map of progression from novice to mastery.  A motivational 'GPS' to keep users making forward progress, especially when the going gets tough. [1]

"Deliberate practice is something designed to build a skill within one to three sessions" [1]. Roughly, "if you can help people go from totally unreliable at this thing to 85 to 90% reliable, that’s a great metric, because that forces you to think about the granularity of that skill. If in three days or three practice sessions, somebody can’t really become more reliable, the skill is not at the right chunk, it wasn’t at their level" [1].

The funny thing is that the traditional textbook-driven course actually provides great Models and Edge Practice for students who "buy in" to that tradition of classroom activity, so long as the teacher is an expert at the math skills and techniques that students are mandated by the curriculum to learn. Unfortunately, the expertise that's built in the traditional manner is expertise in doing grade school basic-skill questions and word problems that will be tested on the exams. In the short term, students can get great marks on the tests, but in the long term, the lack of conceptual depth may well sabotage students' ability to do well in future math courses [2].

Again, this is evidence that the traditional textbook-driven course isn't broken, but there is hopefully a more effective product for getting more students into STEM fields. After all, thinking of the bigger cooler thing(s) that the math we're mandated to cover are a part of, is that bigger cooler thing really: solver of textbook word problems? Do students, when dreaming of how awesome they can be, really want to be badasses at solving problems like:

“7:30am an express train traveling 60 miles per hour leaves Santa Fe bound for Phoenix, 520 miles away. At the same time, a local train traveling 30 miles an hour carrying 40 passengers leaves Phoenix bound for Santa Fe. It's 8 cars long and always carries the same number of passengers in each car. An hour later, the number of passengers equal to half the number of minutes past the hour get off, but three times as many plus six get on. At the second stop, half the passengers plus two get off but twice as many get on as got on at the first stop…”

From "The Simpsons".

Students want to be programmers of insane video games, medical researchers of awesome cancer treatments, algorithmic traders making a killing in financial markets, engineers of badass rockets that fly to outer f'ing space, etc.

Again! Badass at what?

Recall that Sierra urges us to ask: "What does the user do with or because of you? What bigger things do you enable? Nobody's goal is 'badass at your tool'" [1].  We finally have a partial answer that is better than "solver of textbook word problems".

The bigger thing that this math course product can enable the user to do, in fact, is to be programmers of insane video games, researchers of awesome cancer treatments, algorithmic traders making a killing, or engineers of badass rockets!  All these, and many more, are things a user may want to do, and be enabled to do, with the tools from our math course product.

"But those are just applications of math!", you say.  So math should focus on applications and real world, authentic, student-centered, blah blah blah.  That misses the point of making users badasses at what they want, and distracts with bandwagons that has done nothing but advocate for making students good at tools they would rather care less about.

Tennis newbies don't really want to be great at the forehand, they really want to play a good game of tennis.  Certainly the forehand may find applications in a game of tennis, but building a product around the forehand will naturally lead to something very different from building a product around the game of tennis!

The difference is, a user who is now an awesome badass expert might write for our math course product an "Amazon review" like "I was able to build a rocket! Buy this now!", or "I now can understand how my trading strategy can turn against me.  Saved me a bunch of money!  Highly recommend this product".

Versus a student who just gets real good at the math tools might instead write "Wow, I can now solve a hard word problem in our course textbook every 1.5 minutes! Got great marks."

It's not that real world, authentic, student-centered learning is bad.  It's that it doesn't change the basic fact that, except one or two students destined to be pure mathematicians, students don't care to be experts at the tools of math.  Most would rather be experts in those other more interesting fields!

Even if a case can be made that either way, students who make it to the end would still get the knowledge and skills to do all those awesome things, it misses out that survivor bias makes the focus on becoming experts at the tools of math look like a better path than it actually is.  That's because we end up only hearing from researchers and programmers who had jumped through all those hoops, hoops to become good at the math tools, to eventually become awesome themselves.  All the ones who couldn't (or didn't want to!) jump through those hoops don't get a say, even if they could've become awesome had they been offered a better math course as a product to begin with!

Remember that if we want users to sustainably desire our product, we want our users to be awesome experts from using our product [1].  To expect our users to be masters of our product, all 12 years of it, before they can be awesome badasses themselves is perversely putting it backwards.

Putting what backwards?  Well look, the value of our product is in helping the users become experts.  It's not the users role to help make our products valuable for attaining expertise!

It's not that math is not necessary in engineering, for example.  It's that our math course product may not be very useful for students wanting to be experts in engineering, especially if they have access to better products elsewhere.

Naturally, this makes a real difference for how we structure our math course product.  A key question to building this product then is just how do we get students from where they are, closer to where they want to be, starting from a set of mandatory curricular topics.

Notes:

[1] I'm going to be referring a lot to Sierra's talk, "Building the minimum badass user" (Business of Software 2012).

[2] there is related evidence that supports this contention. See Do the Best Professors Get the Worst Ratings?.

---------------------------------------------------------------------------

Math Course as a Lotion to Soothe an Itch

Model, Edge practice, and Forward flow helps learning expertise, but in what?  In rocket building?  Actually, in anything other than pure math tools.  It’s a kind of demand creation, to create an itch for needing certain math tools that helps them become badass experts at, e.g., rocket building.  We teach just enough math tools to soothe that itch, as a kind of calamine lotion that sells itself.  The focus is in helping students become badass experts, to create that itch, and not on the lotion itself.

The simpler the lotion, i.e. the simpler the product to soothe the itch, the better.  If one lotion can soothe more kinds of itches than another lotion, then the former more powerful lotion is better.

That’s because we have to “Recognize that cognitive resources are scarce and depletable.  Design to optimize use of those resources” [1].  “The best mentor says, ‘I know what the books say, but forget that… here's what REALLY matters…’  Your job is to cut through all the noise” [1].

Most math curriculum is but a grab bag of tricks and thematic techniques, reverse engineered as a neutral un-opinionated description of some small section of what is an organically grown math field.  It's a disconnected grocery list of items without a vision of what to bake or, indeed, what the restaurant menu should feel.

We have to do better than that.

Personalization personalized

What if they don't want to be rocket builders.  Personalization is great, but in a big classroom, that isn’t going to be happening much.  We don't give up though, but let students find their own itch.

A learning task (e.g. a project) that creates an itch in rocket building, but that students can also easily see could parlay into other fields of expertise, helps students imagination run into other fields of expertise that they may now have an itch for as well.  In other words, a good project is one that not just creates an itch in some specific topic, but creates an itch that easily spreads to other topics students may be interested in even without the student working too hard at their imagination.

Students also need the time and space to explore those other itches, to soothe those other itches with the same lotion.  To see for themselves that it really does work, that it's not just advertising.

So what does that mean in practice for us in designing learning tasks?

Notes:

[1] I'm going to be referring a lot to Sierra's talk, "Building the minimum badass user" (Business of Software 2012).

---------------------------------------------------------------------------

Learning Tasks Design Considerations

Engender intrinsic motivation:

These principles Daniel Pink addresses are important at the task and project design level. Ultimately, it's important at the level of defining what badass expertise means for the product too, but for the most part, focusing on helping users become real badasses already filters out what is not intrinsically motivating. Most people wouldn't even think you're a legitimate badass if you're just an expert at solving math textbook word problems [1].

According to Daniel Pink [2], how we motivate people with intrinsic motivators revolves around three elements:

1. "Autonomy: the urge to direct our own lives.
2. "Mastery: the desire to get better and better at something that matters.
3. "Purpose: the yearning to do what we do in the service of something larger than ourselves."

Here’s how we can apply the above design principles to designing a math course.

Step 1: Define badass for the product. What bigger, meaningful context, do we want our users to become badass in?

What bigger cooler thing(s) is the math we're required to cover (per the government mandated curriculum) a part of?

If a student using our product were to write our ideal Amazon review, what would they write? They might write (complete the blanks): "I was able to do ___", or "I now can understand that ____"

In general, students want to be programmers of killer video games, discoverers of awesome cancer treatments, engineers of badass rockets that fly to outer f'ing space, etc.

The wrong path is to take the math tools and ask how each is interesting.

Step 2: Define what representative tasks an expert would reliably perform better at and reliably make better choices in, within the bigger, meaningful context for the product.

And make sure it covers the required curriculum. If it doesn’t, find a different or bigger, meaningful context for the product.

Step 3: Schedule tasks for learning the skills required to perform the representative tasks

Step 3a: Considering the whole schedule of tasks, keep in mind to:

• Leverage students' desire for mastery. Give them representative tasks to master that contributes to something that matters, and then help them get progressively better at those tasks. Something that matters doesn't have to be a particular and singular project or task, but could be a large field of study, a career, a cause, or a community (think "sales" vs "cold-calling": the field/career/community, vs. a technique/task) (based on Pink)

• Help students feel secure that what they're doing is in the service of something larger than themselves. This might largely be a sales job in teaching. We can explain rationally why we're doing this, but that's probably largely ineffective if it's too pie-in-the-sky, abstract, and requires an aesthetic appreciation that they don't yet have. Regularly pitching them that what they're about to do is important in an emotional way is probably better (based on Pink)

Pitching them can take a number of forms. Here’s two:

1. Elevator pitch: product, capability, impact, proof, and cost.
   
   It’s based on rationality, but at least it's short.  E.g. some possible impacts of math: We learn this math stuff to pick up some mathematical superpowers that can help us reduce how much gas is used by the airlines fleets of planes, help us get better coverage of cellular or satellite signals for worldwide communication, help us figure out how to prevent diseases from spreading in countries that need our help.
   
2. Question pitch, rhyming pitch.
   
   It’s emotional, and plays on cognitive tricks.  E.g. The “Pixar Pitch”: So, once upon a time there was this problem. Every day people were frustrated by this problem. One day, this thing came along. Because of that. Because of that. Until finally... (Narrative hero story telling).

Further, keep also in mind to:

• Provide elbow room for students' urge to direct their own lives and make their own decisions. Give them opportunities to make choices that are personally significant for them. (based on Pink)

• Optimize the use of depletable cognitive resources for getting users towards being badasses as quickly as possible.

• Provide Forward Flow: add a clear, believable path forward in the schedule of learning and representative tasks towards users becoming badasses as quickly as possible.

Step 3b: For each learning task for acquiring skills required to perform the representative tasks, do the following:

1. provide Models: find examples of expert performances of those representative tasks to show students:

• Ask users to notice the subtleties. Help users get better at seeing in higher resolution. (based on Gallwey, and Sierra).

• Sierra's focus on showing lots and lots of examples to users is less effective where an instructor can ask students to do things. Gallwey (c.f. in a video of Gallwey teaching tennis) shows students more doing than just watching, although it starts from observing before going on to doing.
  
  It's more like the 3Is: Intro, Isolate, Integrate [3], where “Introduce” should take only several minutes, then the rest takes the vast majority of the training time. With physical tasks, mirror neurons make perceiving very effective, whereas less physical tasks require more doing and introspection (but not reasoning about introspection!) in place of perceiving.

2. make learning tasks that are Edge / Deliberate Practice to train for representative tasks:

• Don't worry so much about transferring procedural or declarative knowledge. Focus on helping students perceive what experts do, and then doing what's perceived, without linguistically symbolizing and rationalizing everything (see Gallwey, and Alan Kay’s work). E.g. note what Gallwey says about the wrist stiffness, too stiff or not enough is no good, but declarative statements ("keep it stiff!") just doesn’t communicate enough to do the right thing! (based on Gallwey)

• counteract people's habit to parrot mediocre examples they've been exposed to, which prevent them from performing the representative tasks, by building new grooves, new habits, without trying so hard at deliberately breaking old ones (based on Gallwey)

Classroom social-relational dynamics:

I left this item last because it’s super important, should be ever present, but it’s basically the most obvious thing that’s first taught to aspiring teachers.

Social-relational dynamics do not directly impinge on learning task or project designs, or what expertise students are to work towards. You can teach within a traditional textbook-driven course and these social-relational issues would still be important, and the way to address these issues would be similar.

• Build positive relationships amongst students, and also between student and teacher.
• Establish a sense of community and belonging
• Be supportive
• Discuss personally with struggling students on learning strategies
• etc.

Notes:

[1] “No one ever talks about brave men and their proud simulators”, says fictional submarine Captain Dodge in Down Periscope.

[2] See Daniel Pink's book "Drive", and his "accompanying" talk at Ted Global 2009 on The Puzzle of motivation.

[3] Notes on drilling

---------------------------------------------------------------------------

Afterword

I wrote the above years ago, intermixed with notes from the referenced material, and only now edited it lightly to share here.  I hope I got all the references and quotations correct.  These ideas here were essential and had proven to work extraordinarily well over the years.