Programs Serving Children: Who Says They Help?

Programs Serving Children: Who Says They Help?

Patrick Boyle is editor of Youth Today, a monthly newspaper covering the youth service field. This piece was originally published on February 6, 2006.

When first lady Laura Bush visited the Germantown Boys & Girls Club in Philadelphia last year, her speech boasted that in neighborhoods with Boys & Girls Clubs, “there’s generally a reduction in vandalism, drug trafficking and youth crime.”

The club got positive media coverage and the wires dutifully distributed Mrs. Bush’s speech, although no research backed up her claim.

When President Bill Clinton visited a Washington, D.C., after-school program run by the U.S. Dream Academy in 2000, he stressed to reporters that “this is not just a feel-good program. It works.”

The Washington Post ran that quote under the headline, “Clinton Calls Anacostia Program a Model.” Once again, there was no research to support the claim.

It’s a routine phenomenon: A public official -- be it a president, governor, mayor or legislator -- visits a youth program to anoint it a proven success. The local media churn out feel-good stories saying the program has a positive impact on kids. But in fact, there is little or no evidence of the program’s effectiveness.

I’m not throwing darts here; I’ve done my share of these stories myself. But I think it’s time for us to start looking for the evidence behind these dog-and-pony shows. Too often, we don’t take the time to investigate these claims, and thus we become partners in a misinformation game. The losers are kids and taxpayers.

WHERE TO LOOK

Like most media events, these visits are designed to exploit the weaknesses of deadline journalism. They fill news holes with easy stories full of good quotes and pictures. They provide a veneer of credibility through the testimony of public officials and passing references to “studies.” They give us positive stories about programs that do nice things for kids. And they get media outlets paddling in one direction, making it unlikely that one of us will investigate the claims without knowing if we’ll find a better story.

But if programs are going to claim evidence of effectiveness in order to get our attention -- and to help secure public funding or private donations -- good reporting demands that we see the evidence. There are several places to look, each with its own obstacles.

The officials: If you ask the public official’s staff for the studies he or she referred to, you might actually get a citation or a document. More often, you get a far-off look and an “I’ll get back to you,” with hopes that you forget about it.

If you don’t fact-check, you help create the mirage that government or other impartial experts have carefully vetted the program. In truth, the official’s staff members typically know little about the evidence and sometimes haven’t seen the studies. After all, this is a promotional event, not an audit.

Staffers typically get supporting material from the youth agencies that are being praised, but it might only consist of program highlights.

The agencies: Some agencies quickly supply their evaluations to reporters. But look at the quality. Are they based on self-administered surveys and collections of anecdotes?

No one seemed to ask such questions when President Clinton visited the U.S. Dream Academy program at a grade school in June 2000. According to the White House transcript of the speech, Clinton said: “What I would like to say, especially for the benefit of the members of the press who are here covering this, is that this is not just a feel-good program. It works. … Math and reading scores are up sharply. Suspensions are down.”

However, when Youth Today asked for evaluations, the academy said it had none. We were steered to the grade school’s principal, who said “the kids are becoming more focused in the classroom.”

Even agencies that have evaluations will sometimes provide only study summaries or nothing at all. In such cases, you can try to find the research yourself.

Academia: Research on the effectiveness of social programs seems designed to frustrate non-academics. Program evaluations are often unpublished, or published in journals that don’t post their material online or make non-subscribers jump through time-consuming and costly hoops to get copies.

Try a Lexis/Nexis Academic or Google Scholar search or a search of journals available electronically through your local university library. Getting the names of the study authors is key. They sometimes publish summaries of their studies in obscure publications or on Web sites.

Consider what my newspaper, Youth Today, found when it sought the evidence behind Mrs. Bush’s claim about the impact of Boys & Girls Clubs (BGCA). In an online search, Youth Today’s research editor, Jennifer Moore, found a BGCA-produced summary of a 1991 Columbia University study of 15 public housing sites, 10 of which had Boys & Girls Clubs. According to BGCA’s promotional material, the study found that “clubs in public housing areas dramatically reduce crime and drug presence,” including a “13 percent reduction in juvenile crime,” a “22 percent reduction in overall drug activity” and a “25 percent reduction in presence of crack cocaine.”

The Columbia study itself, however, did not measure crime reduction. It found that there was much less criminal activity at sites with clubs than at sites without clubs. Even then, there was no evidence that the difference in crime rates had anything to do with the clubs. In response to our query, BGCA spokesman Evan McElroy told Youth Today that "It’s arguable that the word ‘reduction’ should not have been used” in the promotional material.

WHAT (OR WHAT NOT) TO SAY

Tracking down studies might take more time than you’ve got, but you have several options. If no one will give you an evaluation, either don’t mention the so-called evidence in your story or say officials and the agency wouldn’t provide an evaluation.

If the evidence is a survey conducted by the agency, point out that there has been no independent evaluation of the program’s impact. You should still get a copy of the survey so you can cite specifics of your choosing and explain the findings better than if you had used only choices fed to you by the agency.

Beyond the daily piece, your research can give you a good second story about the struggle to evaluate youth-serving programs. We took that approach in a story examining the evidence of effectiveness at youth programs touted on the first lady’s nationwide “Helping America’s Youth” tour last year, and in a 2004 article exploring the evaluation findings of the White House Task Force on Disadvantaged Youth.

In October 2005, I attended the first lady’s “Helping America’s Youth” conference in Washington, where the featured speakers included the heads of the U.S. Dream Academy and other programs that had produced no scientific evidence of effectiveness. It was another dog-and-pony show, and a reminder that the government doesn’t let the facts get in the way of pushing a good story.