In Search of the Holy Grail: Health Related Evaluation for Food Banks and other Social Service Organizations

holygrailIf you want to be a big celery-waving food bank then it is all about increasing your poundage, or should I say increasing what is the new guise of poundage – numbers of meals. (Even though by current reckoning, meals can be comprised of things like pounds of candy).

Nevertheless, assuming that you are doing your best to distribute pounds of nutrient dense food, surely providing more and more food to the community has to be a good thing right? Absolutely.

But is it also an effective measure of the success of food banks at ending hunger?

628x471

Not necessarily, because this apparent success is also a strong indicator of the continuation (some might say institutionalization) of food insecurity in America. If we’re giving out more and more food, we are not shortening the line of people who need our services and so failing to bring lasting food security.

If times are tight for people and free food is available, then any smart person is going to take as much free food as they can get their hands on, providing that the distribution timing or environment aren’t so difficult as to make it not worth their while. People will then divert the funds they had for food to pay for some other expense for which there is not so much freely available help. It is the smart thing to do, and to be food insecure in America, you have to learn to be smart pretty quickly.

You can end hunger

We like to give food to anyone who says they need it without much in the way of preconditions. And who doesn’t need food? Your stated mission might be to ‘ameliorate hunger’ or ‘end hunger’ or if you are windy Californians like us then you might want to ‘end hunger and transform the health of Santa Barbara County through good nutrition’. Whatever your goal, we need a way of finding out whether we are succeeding at doing more than keeping the nutritional health of millions of Americans tethered to our life support machine.

Which is where evaluation rears its head.

Food banks are still better at demonstrating outputs (pounds, meals, people served) rather than outcomes (individual behavior changes, community change and societal change). Time was we could get on our high horses and proclaim that ‘ensuring a child didn’t go to bed hungry’ was an outcome as far as we were concerned and the most import one – oh, and by the way, how dare you even ask us to justify what we are doing.

A turkey at Christmas might have cut it for Scrooge, but we need a little bit more
A turkey at Christmas might have cut it for Scrooge, but we need a little bit more

Those days are over.

Whether you buy into the whole ‘nutrition bank’ thing or not, you are will still be noticing a gradual shift in how food banks are being perceived by larger foundations. A few years of recession were good for automatic and generous funding. Even now, knocking on the foundation door generally assures us of having our request for operational funding awarded at x dollars, because ‘everyone loves the food bank.’ However, the social service organizations who are winning the award of x plus x dollars are the ones who can successfully evaluate what they are doing and demonstrate impacts in the community. And sorry, that’s typically not the food bank’s grant application.

Gotta like the hat
Gotta like the hat

As a Mr. Potter-like CEO, I don’t like to leave money on the table, and I believe that there is a huge pile of money sitting on the table for food banks that we are currently not able to pick up more than a few scraps of – health dollars.

The Holy Grail for a food bank like ours would be the ability to demonstrate and quantify the contribution of our programs to lowering levels of childhood obesity, diabetes, food-related cancers etc. If we could do this, we would be in a strong position to be better funded with private and federal health dollars.

HolyGrail034

I want those dollars to continue the little nutrition revolution in our service area and I want them for you too, dear reader.

Despite the importance of exercise, it is clear that when it comes to the most effective and cost-effective way of maintaining good health, ‘it’s the food, stupid.’ Food is what food banks have and it offers us the potential to make an incredible contribution to the wellness of this country.

It is my belief that in the food bank network, we are sitting on the most powerful, most cost-effective preventative healthcare machine the country has ever seen. We just need to be able to unleash its power.

we_can_do_it copy

The way to do that is not to collect a few dollars to feed the needy, but to collect serious bucks to keep the nation healthy. By a combination of nourishment, education and empowerment, we can move millions of people – not just out of hunger, but out of hunger and into health.

Now, to kick me off my high-horse and to get us those health dollars, it takes evidence. And that takes evaluation.

Serena Fuller
Serena Fuller

This is where Serena Fuller PhD, the Foodbank of Santa Barbara County’s Health Education and Evaluation Manager comes in. Serena is a Registered Dietician with a background in obesity research (yes, lab mice are her friends) and public health. She has been on staff for about a year and a half now. When she was brought on board, the understanding was that part of her work would be to find this Holy Grail for us and deliver it to the development department so they could ride out and return with the gold.

Good scientist that she is, she had no interest in fudging the figures or finding some woolly way of claiming direct medical benefits from our programs that we cannot really prove.

Damn.

Getting over this disappointment, we moved on to a phase of having her dirty her nice white lab coat with the realities of food bank programs and for her to be involved in the creation of new programs. We also began to consider different evaluation options.

Have we found the Holy Grail? Of course not – it’s all about the quest, dude – but we did find what we believe is an important next step for us, which can also be replicated at other food banks.

It comes down to working with public health evaluation measures as opposed to medical evaluation measures.

reaim_badge

RE-AIM

We are now in the initial stages of piloting an evaluation approach based on the RE-AIM framework championed by Russell Glasgow. RE-AIM is an acronym for Reach, Effectiveness, Adoption, Implementation, and Maintenance and is an evaluation framework for public health type activities.

So why don’t medical measures work for food banks? Let’s get a teenie bit technical for a minute and look at the classic medical measures. (Take your motion discomfort pill if necessary)

childhood-obesity

BMI

Agencies often brag about ‘collecting BMI’ hoping that this covers a multitude of other evaluation sins. BMI is really just an indicator of later health outcomes, not what we are doing over the short term. Currently there is some question about whether BMI really is measuring what it is supposed to measure – total fat mass. It remains in use as a measure because of some clear positives: 1) it’s relatively easy to collect 2) it’s non-invasive and 3) it seems to correlate reasonably well with fat mass. Essentially some scientists have called into question whether it is good at telling whether people are actually healthy or not.

cholesterol-screening

CHOLESTEROL LEVELS

While total cholesterol, LDL and HDL cholesterol are measures for risk of heart disease that doesn’t necessarily equal myocardial infarction i.e. heart attack.  Cholesterol levels have also been critiqued in the literature as not being a sensitive or specific enough measure and thus have a low(ish) predictor value. But, just as with BMI it has positives as regards ease and non-invasiveness and it seems to correlate reasonably well with risk of heart disease.

WHY FOODBANKS SHOULD NOT BE COLLECTING MEDICAL DATA

The measures discussed, especially BMI, don’t change much in the short term, which is when these labor intensive measures are typically collected (expect in instances of multi-million dollar, long term, multi-clinic studies). Based on her experience at the Foodbank, Serena formed the belief that food banks should not be in the business of measuring subject-level ‘medical’ data because of invasiveness, the cost associated with this type of data collection and because of the issues raised above with regard to commonly collected medical measures. All this meant (in her favorite phrase) that ‘the juice was not worth the squeeze.’

The take away from all of this is that food banks can find their own measures of health, that are reasonable to collect, measures that can change in shorter amounts of time and which – just as much as with ‘medical’ measures – correlate reasonably well with the true health outcomes that interest us – long-term decreases in rates of morbidity (disease) and mortality. Being hungry sucks, but being grossly unhealthy or dead really suck.

These measures that we are most interested in are ones that score diet quality and food security scores.

Medical studies typically focus on populations that need to be similar in order for the data to make sense. But food bank populations are incredibly diverse and it would be unethical to exclude clients from the study if they needed food and their diversity doesn’t bode well for showing statistically significant changes in anthropometric, clinical and biochemical measures.

There is certainly the place for a few well-funded food bank research studies at a national level* (Check suggestions for these out at end of the post).

You may remember a post last year on ‘From Hunger to Health’ where I interviewed Dr. Hilary Seligman of UCSF, who was involved in looking at food security and how it can make major improvements in people with diabetes. There is also the Bristol-Myers Squibb project with Feeding America. It was discussions with Hilary which began to move us down the pathway that led to the RE-AIM tool.

RE-AIM

reaim

Because we want to run programs with the goal of improving the health of our community, we needed to find an evaluation framework that could capture changes in health.

RE-AIM has been used nationally to assess a broad range of community health interventions from actions to prevent child abuse through evaluating the efficacy of specific exercise programs for the elderly. A list of documents and links demonstrating some of these is contained at the end of this post.Here is a link to a monograph on using RE-AIM for program evaluation RE-AIM_issue_brief.

RE-AIM is unusual in that it moves beyond the current approach taken by the medical community to assess community-based interventions. In medical terms, the gold-standard is the Randomized Clinical Trial. In this, there is a focus on something called internal validity, which brings with it a tendency to oversimplify issues and their outcomes in order to isolate the impact of the program. Food banks don’t operate in a bubble. In particular, the emphasis on eliminating the potential for confounding factors typically results in samples of very homogeneous, highly motivated, healthy individuals which equates to samples of non-representative people.

RE-AIM hypothesizes that the overall social-change impact of an intervention is a function of all five RE-AIM dimensions not simply the client-based outcomes. The implication is that to have a substantial impact at the population level, an intervention must do reasonably well on all or most RE-AIM dimensions and thus all 5 must be evaluated or measured.

success score

Our Foodbank RE-AIM evaluation allows summary indices which we have termed ‘Success Scores’ which determine the overall impact of individual programs as well as initiative areas. We realize that the process will be iterative as we develop more measures with high reliability (measures the same thing over time) and validity (actually measures what you think you are measuring) and which include more stakeholders.

Our Success Scores have a range of 50 points, so as to be reasonably sensitive to the different activities we are doing right or areas that need improvement. However, we may find over time that 50 points isn’t sensitive enough or is overly sensitive and we so we will change the Success Score. That is the beauty and strength of evaluation over research in a community setting conducting translational work in that it is more flexible and dynamic.

Here is a link to an excel file that shows how we set the RE-AIM measures for a number of our programs.Program Score Card

Looking to the future the E part of RE-AIM (the effect) and the M part (maintenance) allow an organization to measure the ‘medical’ outputs if resources and will permit.

We are focusing initially on diet quality and food security because just like BMI and cholesterol levels they are reasonably well correlated with our outcomes of interest – adverse health events, long-term morbidity (disease) and mortality (death).  Plus if we are improving the food security status of our clients, the idea is that the demand for food bank services, in the traditional sense, will diminish and we will shorten the line.

We utilize well -accepted measurement tools for assessing good diet and food security:

Diet Quality measurement tool: http://appliedresearch.cancer.gov/surveys/chis/fvscreener/chis_fvscreener.pdf

Food Security measurement tool: http://www.ers.usda.gov/topics/food-nutrition-assistance/food-security-in-the-us/survey-tools.aspx

To measure food security means that food banks will need to change their model to a model like our Healthy School Pantry  or similar approach with wrap-round services like Fresh Place. Here are programs where people can get involved, become food literate, get enrolled in benefits, build their social assets i.e. meeting new people at the pantry, grow their own food, learn how to stretch their food dollar. This means we can track people who will still attend but move out of food insecurity.

The benefits of RE-AIM are that it can be customized to each individual food bank, community and stakeholders, is broadly focused with good external validity, assesses system-wide changes as well as individual changes, includes a maintenance component of making the program sustainable, which is vital when you looking at population-based changes in health status and food security.

RE-AIM can be undertaken by food bank staff and volunteers and doesn’t require highly trained individuals to collect the data, depending on what measures for E and M you have decided on.

The negatives are that there is still lots of data to be warehoused and collected, and that it can be cumbersome to gather the community input. The summary indices are only as good as your inputted data, and some sophistication is required in developing your measurement tools.

One example of how RE-AIM can help you monitor and make changes to what you are doing. Say you are running a distribution and conducting health education at the site. This health education is led by a trained volunteer and you collect your survey data from participants and see that you have no effective outcomes, (i.e. no changes in healthy food behaviors, self-efficacy, knowledge etc). If you were not evaluating the implementation you might just scrap the program, but utilizing the RE-AIM tool would help you notice the difference between this site and another site that had a translator. So the impact is really Impact = implementation x effect. The great part of evaluating implementation is you can learn which sites are doing great, learn from those sites and then take what you learned to other lesser performing sites.

We see the next stage as working to improve the measurement tools as well as identifying the best indicators, i.e. the measures the have the best predictive value of health impact, and tapping into the right partners so we that we can strategically collect ‘medical measures’. We want to develop an evidence library that supports food security and diet quality as the best predictors of morbidity (disease) and mortality (death) in light of community constraints, food bank constraints, invasiveness for subjects and related issues.

We believe that food banks could use RE-AIM to collect meaningful data about their impact on the health and wellness of their communities. We are developing the measurement tools, score cards and success scores, plus causal pathways and definitions.

If we all adopt this method I think we can have a large influence on what funders will expect and of what all of our respective communities view as our work.  This is turn can show the true impact of our work.  If we come together to say that diet quality and food insecurity are the right measures, especially when assessed in the context of RE-AIM based framework we will go very far in proving our impact from that of an earlier measure like Pounds Per Person In Poverty.

We need your input your comments on your systems, your criticisms – and yes, your dollars for continued development of a system that can bring considerable evalutory (did I just come up with a Palin-style coinage there?) benefit for the whole network. You can contact Serena at sfuller@foodbanksbc.org with ideas and me at etalkin@foodbanksbc.org with support!

Serena has put together a brief information sheet containing some aspects and definitions for RE-AIM which can be downloaded here.13-02-21 Evaluation Framework Definitions[2]

The quest for the Holy Grail continues, but until then, stay tuned. This won’t be the last time we consider evaluation issues in these pages.

Hold on, Indiana Jones found the Holy Grail, and it didn't turn out too well for his German buddies...
Hold on, Indiana Jones found the Holy Grail, and it didn’t turn out too well for his German buddies…

Useful links;

http://www.childrenshealthwatch.org/upload/resource/AdvNutr_JC_2013.pdf  (food security and adverse effects)

http://www.ajhpcontents.org/doi/abs/10.4278/ajhp.071211129?journalCode=hepr (RE-AIm evaluation of a health promotion intervention at schools)

http://www.biomedcentral.com/content/pdf/1471-2458-12-403.pdf (RE-AIM evaluation of walking intervention for employees)

http://www.nejm.org/doi/pdf/10.1056/NEJMoa1200303 (diet changes health outcomes)

http://www.ncbi.nlm.nih.gov/pubmed/23280227 (all-cause mortality association with BMI)

http://www.ihepsa.ir/files/h1.pdf#page=525 (book – Health Behavior and Health Education Theory, Research and Practice 4th ed Glanz, RImer, Viswanath editors)

* OPPORTUNITY FOR NATIONAL STUDIES – There is certainly the place for a few well-funded food bank research studies which would be at a national level* – looking at BMI, adiposity (via BIA or caliper), HTN, cholesterol levels, long term blood glucose regulation, e.g HBA1C (which is different than evaluation), plus diet quality and changes in food security.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s