Teachers are scientists, they’ve always experimented. Most of the time it’s informal, “Let’s try a new behavior management routine.” Or, “Watch this video tonight and we’ll discuss it in the morning.” Or, “Let’s try a really hard problem.”
It’s been hard to run formal experiments because measurement systems in education are so bad–a year end test won’t help with a reading experiment this week.
What if we gave teachers, and whole schools, the chance to run experiments of their own or to join larger trials? What if they had access to better measures and powerful analytics?
This sort of experimentation is common in business where a number of big vendors (SAS, SAP, Microsoft, IBM) provide business analytics to support quick experiments. For example, APT, acquired earlier this year by MasterCard has, “cloud-based software that determines cause-and-effect relationships between tested programs and key performance metrics. Beyond determining the overall impact of each initiative, by rapidly sifting through huge data-sets, the software helps companies target their actions to the customers predicted to respond most profitably.”
In software development it’s common to run A/B tests that quickly test user experience (UX) features. Rapid feedback is critical to iterative development.
Ian Eslick, who went back to MIT for a mid-career PhD, has studied self-experimentation. He’s interested in efficacy not validity. He’s willing to be wrong for a while but wants to make quick practical progress. He wants to help people find treatments that work for them, fast. “As individuals we make the best progress when our hypotheses are well founded and the conclusions drawn from our experiments are accurate. As a community we make the best progress when we can share our ideas in ways that accurately predict how someone else will fare.”
We need more of this in education, and it’s getting easier to manage. Powerful and easy to use formative assessment systems (from Edmodo, MasteryConnect, and OpenEd, for example) make it easy for a school, district, or network to deploy common assessments around an quick A/B test.
Intelligence platforms like BrightBytes improve hypothesis generation, as well as measurement of results over time. In business, this is typically thought of as Return on Investment, or ROI. BrightBytes thinks of it as ROL, Return on Learning.
Test what? If teachers and schools had the ability to run short trials, what experiments would they run? Some examples include comparing:
- Reading strategies and materials
- Blended learning models (e.g., lab rotation vs station rotation)
- Math software
- Behavior management systems
Like projects, experiments can be assessed by a mixture of quantitative and qualitative factors. A variety of measurement systems can be used including formative assessment, teacher and student surveys, and observations.
Why not just rely on national studies? Results in education are highly context specific– particularly implementation efficacy. As a result, national studies may be suggestive but they won’t tell you how something will work at your school. And they’re not available for a new app or strategy.
The first head of the Institute of Education Sciences, Russ Whitehurst set a gold standard for research particularly around the use of randomized controlled trials (RCT). There is obvious benefit to setting a high bar for evidence but we often need better evidence faster and cheaper than is possible through RCT.
I’ve complained before that many randomized trials (like this 2013 small school study) providing a blinding flash of the obvious a decade late. They sacrificed efficacy for validity.
We need well designed large scale controlled studies but there is a growing need to support an iterative change strategy in education.
Path forward. I’m optimistic about the shift to personalized learning not only for the customization and motivation students will experience, but for the tools that it will provide to teachers. Big Data will allow interested teachers to become data scientists and participate in large scale experiments in real time.
The next step in helping teachers as scientists is an education intelligence platform that combines information about best practices, easy to use formative assessment, templates for setting up experiments, and social features for sharing.