Government “reforms are advocated as though they were certain to be successful,” yet most “programs end up with no interpretable evaluation” of their success.
Donald Campbell wrote those words in 1969, but he could easily have been writing about Texas today. The state launches, expands and cuts programs without using evidence of whether they are achieving their goals.
It doesn’t have to be this way. Other governments are warming to Campbell’s proposal of treating “reforms as experiments.” The idea, now called evidence-based policy, is to launch a program with clearly stated goals, prior evidence that the goals are achievable, and a plan to evaluate whether the goals are being met. Programs that achieve their goals continue or grow. Programs that fall short are fixed or ended.
Texas is not yet part of the movement for evidence-based policy, but it should be. Until the state makes better use of evidence, it will throw good money after bad at failing programs, while programs with greater potential languish in obscurity. Once the state starts making better use of evidence, it will be able to cut its losses on ineffective programs and double down on programs that are working well.
The program Texas Fitness Now is an example of how Texas neglects evidence today.
The state launched Texas Fitness Now in 2007 with the idea that improving physical education could reduce obesity, increase fitness and improve academic achievement in high-poverty middle schools. Lacking evidence of what specific improvements would achieve those goals, the state let districts make proposals and provided grants to cover them.
In 2009, without evidence that Texas Fitness Now was working, the state expanded the program. In 2011, when recession crimped the state budget, the state terminated Texas Fitness Now, despite lacking evidence that it was failing.
Over four years the state spent $37 million on Texas Fitness Now, yet it had no idea whether the program achieved its goals and only a vague idea how districts had spent the money.
The pity is that the state had all the data it needed to make evidence-based decisions. The state had data on which schools had received Texas Fitness Now grants and how they spent them. The state also had data on students’ obesity, fitness and academic achievement.
Using those data, a colleague and I have published an independent evaluation of Texas Fitness Now. We found that the program failed to reduce obesity and had little or no effect on reading and math scores, but it did improve fitness.
Unfortunately our findings have limited policy value today because the program ended four years ago.
If Texas were practicing evidence-based policy, the story of Texas Fitness Now could have been very different.
Using evidence-based policy, Texas Fitness Now would not have let school districts spend grants as they saw fit. Instead, it would have prescribed a short list of programs that had prior evidence of effectiveness. The state would have assigned schools to programs using random assignment. Random assignment is the best way to ensure that schools using different programs are statistically comparable.
If the state were practicing evidence-based policy, it would have had Texas Fitness Now evaluated in 2009 when the state was thinking of expanding it and again in 2011 when the state was thinking of ending it. The program would have been evaluated independently by a university or a contract research organization.
Outside evaluators are needed because state agencies rarely have the capacity and independence to evaluate their own programs. For example, the Texas Education Agency actually tried to evaluate Texas Fitness Now in 2011, but they only looked at what happened when schools got grants. They neglected to look at what happened when schools did not get grants.
The movement for evidence-based policy is growing. There are evidence-based initiatives in the federal government, in states including Iowa, Massachusetts, Oregon, and even in developing countries such as Ghana to Colombia.
Texas needs to get in the game.
Paul T. von Hippel is an assistant professor in the LBJ School of Public Affairs.
A version of this op-ed appeared in the Dallas Morning News, Fort Worth Star Telegram and Austin American Statesman.
To view more op-eds from Texas Perspectives, click here.
Like us on Facebook.
