Fitnessgrams may sound like Instagram photos of people bench pressing or rebuilding muscles with a healthy smoothie. But really they’re an annual assessment the New York City public school system began giving its students in 2007 as part of its ongoing effort to drive down childhood obesity. District personnel schedule a day to measure students’ fitness levels and body mass index (BMI) in order to give them a fitness report card scoring them as underweight, healthy, overweight, or obese. Past reports have put over one-third of students into the overweight or obese categories. Now, a new study published in Proceedings of the National Academy of Sciences is questioning whether the Fitnessgram BMI assessment actually does any good.

“I think there’s [a popular] notion that if you give people information, things will get better. If you tell kids or you tell parents their kids are overweight, they’ll do something about it,” study author Amy Ellen Schwartz, a professor of public affairs at the Maxwell School of Syracuse University, told Medical Daily. But, she added, “there’s a whole lot of reasons it might not work.”

The schools have made more than three million BMI assessments, but Schwartz said she and her team chose to focus on 442,408 female students on the border between healthy and overweight BMIs, a choice partly made so as not to drown people in results. More importantly, though, Schwartz focused on cutoffs to see if children have some control over their scores.

The adult BMI cutoffs for being overweight and obese — scores of 25 and 30, respectively — are widely known. But the cutoffs for children are not. Sometimes they fall beyond round numbers, and a child’s score can change in a matter of months as they age. hence the idea children have some degree of control over their recorded BMI. “If individuals can precisely control their BMI so as to receive the healthy label, we might expect to see ‘too many’ students just below the cutoff,” the researchers wrote.

To estimate how New York City students were affected by Fitnessgrams, researchers compared girls whose BMIs were just below the overweight threshold to those girls whose scores put them right on the other, unhealthier side of the cutoff. The results showed that getting an overweight report card “has little and, if anything, a small positive effect on young girl’s BMI and weight the following year,” Schwartz said.

There could be several reasons for this increase. For one, the information included in the Fitnessgram could be stale by the time it’s presented to students and parents; it takes time to crunch the numbers and data. Fitness reports also come home in backpacks, “and any public school parent can tell you that is not 100 percent reliable,” she added. Put it another way: Parents could be getting this information too late.

The study doesn’t necessarily condemn the overall effect of BMI reporting, rather it brings attention to these cutoffs and the factors that push girls over the lines between categories. A young girl close to being overweight could fall to either side based on day-to-day variation in weight or a height mismeasurement, the study suggested.

Incorrect categorization is one of the major criticisms experts have with using BMI to measure general health, and yet Schwartz had believed Fitnessgrams were doing some good.

“I was surprised by the whole thing. I really thought there would be an impact,” Schwartz said. “All over the country, folks are looking to these kinds of tools to improve outcomes, providing information [at a low cost] to get people to act differently, and it goes to show you have to do it in ways and at a time that’s useful.”

For example, calorie and sodium labels on restaurant and fast food menus are all good and well, but they don’t exactly serve customers well if the first time they see them is after they’ve already sat down to dinner. To effect change, especially when it comes to obesity, Schwartz said it’s important to hone the message and work out how to best deliver it.

That said, these findings show how serious New York is about children’s health. In addition to Fitnessgrams, which Schwartz said city health officials are now reviewing, some schools have implemented self-serve water jugs that she said help “like nobody’s business.” She examined the water intervention for a study recently published in JAMA Pediatrics that concluded that offering the jugs, which dispense a jet of water at the push of a button, increased water consumption while decreasing high-calorie beverage consumption, which is good for students’ health.

“You have to be open and willing to share the data and set up the structure so you can test these things,” she continued. “Sometimes you keep doing what you’re doing and sometimes you have to rethink. For the water jets, it was do more. And for [Fitnessgrams] it’s rethink. That’s really exciting from a public policy standpoint.”

Source: Almond D, Lee A, Schwartz AE. Impacts of classifying New York City students as overweight. Proceedings of the National Academy of Sciences. 2016.