• About the Study
• Methodology
  • Instrument Development
  • Training
  • Agency Survey
  • Client Survey
  • Response Rates
  • Analysis Methods
  • Reporting Conventions
  • Degree of Comparability
  • Number of A2H Clients Served
    • Background and Limitations
    • Overview of Findings
    • Estimates of Number of Pantries
    • Estimate of Number of Pantry Clients
    • Number of Kitchens
    • Different Clients Served By Kitchens
    • Shelters and Shelter Clients
    • Estimates of Different Clients
    • Discussion of Participation Estimates
• Key Findings
• Who We Serve
• How We Serve
• Local Picture
• Snapshot Comparisons
• Media Center

Discussion of Participation Estimates from 2001 and 2005 Surveys

As noted in Chapter 3, a number of significant refinements and improvements were made to the study procedures for the 2005 study, compared with the one undertaken in 2001.  Therefore, the two surveys are not directly comparable in a formal statistical sense.  However, both surveys focused on the same substantive issues, and the approaches taken in the two studies were similar.  Therefore, when examining and interpreting the 2005 survey findings, it is useful to draw on information from both surveys.  Next, we highlight several key factors relevant to such comparisons.

4.9.1 How Large is the A2H Network?
4.9.2 Have There Been Changes in the Number of A2H Clients Between 2001 and 2005
4.9.3 The Two Surveys Paint Somewhat Different Pictures of the Number of New Clients in the System During a Year Relative to Longer-term Clients
4.9.4 Standard Errors of Participation Estimates
4.9.5 Simulation-Based Analysis of Changes
4.9.6 Summary

4.9.1 How Large Is the A2H Network?

Both the 2001 and the 2005 findings suggest that the A2H system serves a very large number of unduplicated clients in 2001 and 2005.  The two surveys show similar participation ranges, and each suggests that the number is well over 22 million.  Furthermore, the findings suggest that the system provides comprehensive services that are widely available.  A2H facilities include more than 29,900 food pantries, more than 5,500 emergency kitchens, and about 4,100 shelters, and each participating food bank provides supplies to all these types of providers.

4.9.2 Have There Been Changes in the Number of A2H Clients Between 2001 and 2005?

Another obvious question to pose in relation to the 2001 and 2005 estimates is whether there appear to have been changes in the number of A2H participants.  Unfortunately, however, we cannot answer this question definitively.  As discussed earlier, because of measurement issues, we have developed participation rate estimates in the form of ranges, and the ranges for the two years overlap substantially.  Given our analysis, we believe that it is quite possible for the “true” figure for each year to be anywhere within that year’s range.  Thus, the survey cannot reliably detect changes within the overlapping areas of the intervals.  That is, within the overlapping areas of the intervals, we cannot rule out the possibility that differences between years are due to measurement error. 1 For instance, consistent with the estimates from the two studies, it is possible that the “true” number could have been 24 million in 2001 and then risen to, say, 27 million in 2005.  It is also possible, however (though we doubt it for reasons discussed below), that the true rate could have stood at 28 million in 2001 and then dropped to 25 million in 2005.  Each of these scenarios and many similar ones are consistent with the interval estimates presented above.

Evidence external to the current survey does suggest, however, that it is very likely that there has been at least some rise in the number of participants in the A2H systems.  One important indicator is changes in the U.S. poverty rate.  As noted in Chapter 3, the U.S. poverty rate for persons increased from 11.7 to 12.7 in the 2001 to 2004 period, which suggests need for more emergency food services.  Similarly, government estimates based on annual Census Bureau, Current Population Survey (CPS) data indicate that the number of people in the United States experiencing food insecurity rose in a reasonably steady pattern from 12.2 to 13.2 during this same period. 2

The CPS also provides direct estimates that the use of emergency food has risen.  The data show that the percentage of households that reported having received assistance from pantries rose from 3.0% in 2002 to 3.5% in 2004, providing direct evidence of an overall increase in the use of emergency food. 3   Since A2H providers make up a very large share of the overall emergency food system 4 this provides strong indirect evidence of an upward trend in A2H participation.

(Additional discussion pertinent to possible changes in participants over time is presented in Subsection 5 below, where we discuss simulation analysis relevant to the issue.)

4.9.3 The Two Surveys Paint Somewhat Different Pictures of the Number of New Clients in the System During a Year Relative to Longer-term Clients

Substantial differences exist between the 2001 and 2005 survey estimates in the relative proportions of longer-term clients compared with new entrants in the composition of the overall A2H client base.  This has implications for understanding estimated participation levels.

For a given amount of food distributed in the system, the number of different people who are participants during a year depends both on (1) the number receiving the benefits at any given point in time, and (2) the rate at which clients leave the system and are replaced by new entrants.  The greater the turnover rate, the more different clients there will be annually, relative to the number of clients at a given point in time.

While the overall annual estimated A2H participation numbers from the 2001 and 2005 surveys are similar, the composition of the participants in this turnover dimension varies somewhat.  Below, we describe these differences and their potential causes.  We conclude that the apparent differences across the two years are probably mostly a measurement phenomenon attributable to improved methods used in the 2005 study.

Changing Participation Patterns.  Our estimate of the number of A2H clients in a given week (as opposed to annually) is down noticeably, from 7 million in 2001 to 4.5 million in 2005, with decreases estimated for each of the three provider types.  This difference is probably due at least in large part to some of the changes in survey methodology described in Chapter 3.

One key change is better screening of providers for survey eligibility.  As discussed earlier, the screening changes were implemented for 2005 because in 2001 there was evidence that many food banks had difficulty distinguishing between agencies that met the definition of “emergency” food providers and other agencies that, while important in the overall A2H system, are not considered to be supplying emergency food.  (Examples of these other agencies include day care centers and suppliers of congregate meals for the elderly.)  As a result, some misclassification of providers probably occurred in 2001.  To avoid similar issues in 2005, significant additional effort was devoted in the 2005 study to (1) training food banks on the relevant definitions, and (2) having MPR staff manually check sampling information after the food banks submitted it.  This may have lowered the estimates of participation at any given point in time, by reducing the likelihood that misclassification would happen in the first place and by more successfully screening out the ineligible agencies that remained in the sample frame.

A second set of survey changes could also have affected the estimates.  At the sampling stages of the 2005 survey, more detailed information was obtained than in 2001 about how often agencies were open during the month and about hours of operation on days they were open.  This information, which was used in the sampling and weighting processes, may have reduced participation estimates compared with what they would otherwise have been, by better distinguishing between when agencies were operating and when they were closed.

On the other hand, another set of methodological changes has partially or fully offset the two described above.  In particular, significantly higher client turnover rates were estimated in the 2005 data.  This is at least partly the effect of the improved survey instrumentation discussed earlier.  As described in Chapter 3, interviewers reported that many respondents found the questions designed to obtain information on client turnover in 2001 confusing.  In addition, data patterns resulting from those questions were often inconsistent with other survey questions.  In the 2001 analysis, as a result, we were very conservative, in the sense of making analysis choices about such issues as appropriate extrapolation factors for the annual estimates, to minimize the risk of inflating estimated participation (see pp. 38–39 of the 2001 Report, Hunger in America 2001, dated October 2002).

Because of the reported problems with the 2001 turnover questions, a new question was developed in 2005 and included in the survey.  This question was intended to be clearer and less ambiguous in eliciting information about turnover rates. 5   Based on reports from the fielding of the survey (and the lack of significant complaints), it appears that the new question worked much better.  It also yielded responses that were more consistent with related survey questions.

Somewhat unexpectedly, in the analysis phase of the study, we found that the answers to this new question yielded higher estimates of turnover rates than had been implied in our earlier conservative analysis of the 2001 data.  For example, the tabulations suggest that, even for pantries (which tend to have the lowest turnover), about 14% of the clients at a given point in time have begun using them only in the previous month and did not use them previously for at least a year.  This has had two related, but distinct, effects on the 2005 estimates.  First, we have used the responses to this new question as the basis of our turnover rates, thus directly increasing the estimates of turnover above those used in 2001.  Second, the findings from the new question have, more generally, provided new evidence that turnover rates may be higher than previously believed, which has led us to be less conservative in our overall analysis.

In summary, improved survey methodology has resulted in significantly higher estimates of turnover.  This has largely offset the drop in the estimated weekly participation number—a drop that itself was due, at least in part, to changed methodology. 6

4.9.4 Standard Errors of Participation Estimates

As noted earlier, the use of ranges to define the participation estimates presented in this chapter reflects potential measurement error in the survey data.  These ranges do not directly reflect statistical sampling variation, which is another potential source of error in the participation estimates.  To assess the approximate sampling error associated with our estimates, we have calculated standard errors for the weekly client estimates presented earlier, taking into account sample sizes, survey clustering, and other survey design features.  The results indicate that for 2005 the standard errors associated with estimated numbers of participants tend to be about 3% of the estimated participation totals. 7  Thus, 95% confidence intervals extend plus or minus about 6 points around the weekly participation estimates.  For instance, for participating food banks, the 2005 estimate of the number of people using A2H pantries in a week is 3.24 million. 8   A 95% confidence interval is then estimated to extend plus or minus 6% of the 3.24 estimate, or between 2.9 and 3.4 million.

In the deriving of annual estimates, additional potential statistical error is added, because several of the extrapolation factors are themselves survey-based and thus measured with error.  If, however, as an approximation, this added sampling error is ignored, then the plus-or-minus-6% factor for confidence intervals can reasonably be applied to the annual estimates as well.  For instance, suppose that one of the endpoints of the ranges in the annual estimates is 25.0 million. The 95% confidence interval would then extend from about 23.5 to 26.5 million.  This should be viewed as a low estimate of the range of sampling error, since the use of several survey-based factors in developing the annual estimates undoubtedly increases the overall error.

4.9.5 Simulation-Based Analysis of Changes

To examine further the effects of the methodological changes on the 2005 estimates, we have conducted a simulation analysis of the effects of applying the improved 2005 survey estimation methods to the 2001 data.  This analysis, described more fully in Appendix A, focused on examining weekly estimates of participation, and it simulated as fully as possible the effects of applying the 2005 methods to the earlier data.  A limitation of the analysis is that in some instances there was insufficient detail in the 2001 dataset to fully replicate the improved methods.  Nevertheless, we were able to examine the approximate effects of four changes:

1. Increased screening of providers for survey eligibility (for example, for whether they were supplying “emergency food” as defined by the study)
2. Consideration of the fact that some providers in 2001 were open less than once per week
3. Adjustment of the assumed hours of operation when, during a day, providers were open longer than the observation period
4. Improved analytic treatment of instances when interviewing at a provider required more than one visit by the interviewing staff.

The results of the simulations, presented in Appendix A, show that most (though not all) of the differences between the 2001 and 2005 weekly-level results can be explained in terms of the four changes in methods that were simulated.  Furthermore, after controlling for the simulation results, none of the observed differences between 2001 and 2005 is statistically significant.

4.9.6 Summary

Our “bottom line” estimate is that overall the range of participation estimates for 2005 is similar to that in 2001.  Two salient facts should be noted in interpreting this result.

First, the similarity in the estimated participation ranges does not necessarily mean that there have been no changes.  Since the estimates for both years have been expressed as ranges, the actual “true” level of participation could have gone up; alternatively, it could have gone down, though we think such a result is very unlikely, for reasons discussed earlier.

Second, in interpreting the estimates, it is important to understand that their apparent overall stability in participation levels hides the effects of two countervailing themes.  On the one hand, the estimates of the number of people receiving services in a given week have gone down. However, this is approximately offset by an increase in estimated rates of client turnover in the system, which leads to annual estimates that are higher than they would otherwise be.

---

1 Of course, it is possible to rule out the likelihood of very large shifts in participation between 2001 and 2005.  However, such shifts would have to be large enough at least to eliminate the overlap in the intervals discussed in the text.  Even then (as discussed later), additional margin would be needed to account for sampling error.

2 See Nord, Mark, Margaret Andrews, and Steven Carlson.  Household Food Security in the United States, 2004.  U.S. Department of Agriculture, Economic Research Service, Report Number 11, October 2005.

3 Our focus here is on trends in the CPS-based estimates.  As discussed in the 2001 report and noted elsewhere in the relevant research literature, the absolute number of emergency food clients estimated in the CPS has consistently been substantially lower than the number estimated in the A2H surveys.  Possible reasons for the undercount include the known tendency of the CPS to undercount use of assistance programs and, relatedly, the role of  embarrassment factors in determining respondents’ answers.  In particular, in the CPS it is easy for a respondent to decide not to report participation in emergency food if it is embarrassing to do so, while in the A2H survey, this is not possible, since the interview takes place at the emergency food provider.  However, these factors notwithstanding, the size of the difference in the estimates should be noted, if not fully understood.

4 See Ohls, James, Fazana Saleem-Ismail, Rhoda Cohen, and Brenda Cox.  “The Emergency Food Assistance Study, Findings from the Providers Survey.”  Report prepared by Mathematica Policy Research for the U.S. Department of Agriculture, Economic Research Service, 2002.

5 The wording of the new question is provided in Chapter 3.  The old questions were kept in the survey as a backup.

6 Interestingly, answers to the two questions used in 2001 (and to the same questions included in 2005 as backup) also suggest the possibility of substantially higher turnover rates than used in the 2001 estimates.  As noted in the text, however, we chose to interpret the earlier data conservatively until we could obtain more reliable support for estimates of higher rates.  The responses to the new question in 2005 provide that support.

7 The comparable standard errors for the 2001 estimates are somewhat higher.

8 Table 4.4.1 reports the number of households as 1.2 million.  We have multiplied that figure by our estimate of 2.7 persons per household.  (Sampling error associated with the 2.7 is ignored here for ease of exposition.  The figure also excludes the estimates imputed for small providers where interviewing was not conducted.)