Policy wonks, not getting the results they hoped for (and being a touch lazy), invented a new research method called the “intraocular traumatic test.” What that means is look at the data and if the result hits you between the eyes, it’s significant. Naturally, this kind of eyeballing lends itself to people seeing what they want to see. Talking heads and blog commentators are the most virulent users of the method. Educational policymakers and call-in radio shows are also malignant carriers of the malady.
The latest outbreak is in the release of the National Assessment of Educational Progress (NAEP) scores. Over the past 30 years, the invariable reaction is Chicken Little crying, “The scores are falling! The schools are failing!” Innocent bystanders spot the use of the trauma test by the intense volume, lack of proofs, vaporous claims – all garnished with a plug for a favored solution.
How big are the actual losses? In the instant case, we need to take a close look. Since 1990, NAEP scores show steady increases until the recession in 2008. Then the national progress slows down. The biggest national loss was three points. That does not sound like much – because it isn’t much — a meager three points on a 500-point scale. This is less than 1 percent over two years. If we go back to 1990, the results show continuous gains across the board. Once the test error is considered, the “drop” is less than 1 percent over two years.
The threshold issue: Poverty – The threshold issue is poverty. In a recent Stanford study by prominent researcher Sean Reardon, he examined 350 million test scores from every public school in the nation. He found poverty is the key predictor of test scores. The analysis included social, as well as school, factors.
As fundamental as Maslow’s hierarchy if food, shelter and nurturing are not available, the chances of the child learning quadratic equations is remote. Only then can we effectively address the tangled web of needs. Beyond addressing poverty, there is no single in-school or out-of-school factor that explains the scores. Many just blame “the schools” and offer tired and vaporous generalities. As Brookings points out, some states are still not back to pre-recession financial levels. The effects of “the lost decade” still reverberate in unique and unpredictable ways.
Nevertheless, some potentially illuminating considerations are worth exploring.
Are we evaluating the schools correctly? Recent commentators look at one test given to only a sample and conclude the schools are failing. The blind eye of the intraocular trauma test simply doesn’t have the necessary breadth of vision. It is like recruiting baseball players on their batting average. Whether they are good fielders, base-runners, or teammates is not considered. If we are to measure education accurately, we need long-term data on children, as well as qualified visiting teams.
Vermont: The hurly-burly of reform – Over various administrations, policymakers have taken steps in the name of reform that are a blend of Keystone Kops armed with food processsors:
—We have only small gains on the threshold issue of concentrated poverty. Some schools need better targeted resources.
— An unintended consequence of Act 46 was a political backlash. The distracting cacophony was like banging a garbage can in a concert.
— Computers — Below the national average, only 80.9 percent of Vermont households have internet. Expecting these students to perform well on the computerized NAEP test was unrealistic.
— Proficiencies —Redefining all content as proficiencies was done without the necessary support capacity. In a statewide survey, 50 percent of the teachers feel uncomfortable with proficiencies.
— Too many curriculum changes. Saying local districts decide curriculum is disingenuous. We have adopted the common core and numerous other curricula over the years. There has not been consistency, stability or support. (Grade expectations, CCSS, NGSS, NCAS)
— Too many test changes — The relation between NAEP and SBAC accounts for only 30 percent of the variance in reading and 24 percent in math. The tests measure different things and send educators in conflicting directions. (SBAC NSRE, NAEP, NECAP, Science, SBAC, NSRE)
— Well-intentioned programs often have a regressive financial effect. Preschool education has an inequitable funding system, dual enrollments favor more affluent children, and tracking provides a different quality of education across the student population.
The greatest intraocular traumatic shocks have been self-inflicted. The schools are not failing as much as we have failed to provide robust programs for our neediest. They are often under-capitalized, lack coherence, steadiness and purpose.
Yet the loss of three-fifths of one point on NAEP must be viewed in context.
The intraocular traumatic test does not indicate a failure of educators. Instead, it demonstrates a hard-won success with limited support, overcoming hard challenges and great sacrifices. If we are to improve, it will not be done with castigations, it will be done with cooperative and focused work based upon an appreciation of all we have done.
William J. Mathis is National Education Policy Canter managing director. He lives in Goshen.