A Balanced Scorecard for Assessing the Quality & Provision of Health Services in UNHCR Refugee Camps
Embargo until
Date
2015-03-18
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Johns Hopkins University
Abstract
Background
Worldwide, there are over 46.3 million individuals who are uprooted from their homes. For the 2014 mid-year report, the UNHCR population of concern included 13 million refugees, 26 million internally displaced persons, 1.3 million asylum-seekers, 3.5 million stateless persons, and1.4 million others of concern. (UNHCR Mid-Year Trends, 2014). The UNHCR provides basic health services in refugee camps, often in collaboration with implementing partner organizations; priority health areas include measles immunization, nutritional support, control of communicable diseases and epidemics, reproductive health and public health surveillance. The UNHCR Health Information System (HIS) serves to collect and aggregate health facility data and provides basic health surveillance information in many camps. The HIS, however, does not measure performance and quality of care and may or may not be conducted on a routine basis. A balanced scorecard (BSC) is one tool that can be used for health facility assessment to complement information collected through the HIS; the BSC can be applied to measure and manage health inputs, processes, and performance in UNHCR refugee camp settings.
Aims
A BSC evaluation methodology was developed by The Center for Refugee and Disaster Response at Johns Hopkins Bloomberg School of Public Health (CRDR-JHSPH) in conjunction with The United Nations High Commissioner for Refugees (UNHCR) for assessing provision and quality of facility-based services in camp settings. The BSC was created based on the unique circumstances of providing health care services to refugees and displaced populations and to help capture multiple indicators and present them in a comprehensive fashion that can be standardized and compared across camp settings and locations. The evaluation methodology has both quantitative and qualitative components, is feasible for implementation in post-emergency contexts in a short time period of one to two days, and is adaptable to varying global settings.
Methods
The BSC was designed in a collaborative process involving JHSPH, UNHCR, and other implementing partners. A list of indicators was developed based on the review of existing literature and tools and selecting priority areas for monitoring and evaluation. The BSC instrument was then created which includes four domains (capacity for service provision, service provision, patient and community satisfaction, and staff satisfaction), 28 indicators to measure these domains, and series of questions for each indicator. These standards were defined by the UNHCR with facilitation from the JHSPH study team which became the definition of quality. The minimum standards of care were discussed extensively to allow the tool to be applicable on a global scale with minimal adaption; some standards were based on internationally accepted cut-offs where others were based on organizational choice. Based on the questions selected, a scoring system was developed determine an overall score or percentage of indicators and standards that have been met by domain and overall for the camp health facility.
Following the creation of the instrument, field testing of the tool occurred in three countries. Pilot testing occurred in two camps in Eastern Ethiopia, three camps in Ghana; and three camps in Uganda. Pilot testing in Uganda was conducted independently by UNHCR and utilized mobile data collection. Results from each camp were calculated and feedback was provided to camp and country officials. Following testing in each country, successes and challenges of the scorecard were reviewed and revisions were made to the instrument and methodology. Data was then analyzed from preliminary field testing, with the aim of determining priority areas for quality improvement in the provision of services. Finally, recommendations were provided to UNHCR to improve the tool for future use.
Results
Overall scores for camp health facilities ranged from 32% to 64% of the total potential score for the indicators being assessed. The range of scores for each domain was as follows: capacity for service provision, 38% - 76%; service provision, 25% - 50%; patient and community satisfaction, 0% - 50%; and staff satisfaction, 0% - 100%. The overall scores by country for Ethiopia, Ghana, and Uganda were 45%, 57%, and 42% respectively. Key informant interviews further provided information about the usefulness of the tool and the ease of use of the instruments. Revisions were made to improve the organization of the tools, remove questions that were seen as unnecessary, and change the wording to improve clarity. Further work is needed to refine the tool and make it a standard approach that works globally
Since the creation of the balanced scorecard tools and the pilot testing of the instruments, the UNHCR has implemented use of the methodology in a number of settings. As of 2013, 42 refugee camp health facilities had been assessed by the BSC and this number is increasing. All tools and instructions are available on the UNHCR twine website and implementing partners are recommended to use the balanced scorecard to measure the quality of health services at camp facilities on a regular basis.
Conclusion
Overall, the balanced scorecard can be a useful management tool to measure performance through the use of selected domains and indicators. The tool did prove to be useful for UNHCR, implementing partners and facility managers to determine areas of strength and weakness in the provision of care, with the benefit of being able to provide immediate feedback at a managerial level. The use of a standard scoring system further allows for comparisons across sites and over time in a succinct and understandable manner, which has great implications for the distribution of resources. UNHCR will benefit from this instrument by having a unique rapid assessment tool that can be consistently used across their post-emergency camp settings to determine the successes and challenges at each facility, regardless of which implementing partner is providing services. The information from the scorecard can be used to improve the quality of health services in refugee camps worldwide.
Description
Keywords
Refugee health, monitoring and evaluation