Final Evaluation Findings and Dissemination

The purpose of the evaluation is to determine the extent to which the Healthy Child and Mother Project that was implemented by the grantee organization and its local Bangladesh partners from 2009-2014 accomplished the intended results and to describe key factors that contributed to what worked or what did not work. The project used a maternal newborn intervention package incorporated into the government community-integrated management of childhood illness (C-IMCI) strategy expanded to include newborns and delivered by a cadre of trained CHWs at household and community levels. The project innovation was the People’s Institution (PI) community mobilization model, adapted to empower the poor and marginalized to collaborate with the public health sector to promote maternal newborn health. The evaluation used a comprehensive participatory approach, including collection and integration of quantitative and qualitative data from a variety of sources.

The evaluation team determined that the project effectively mobilized marginalized women and men for maternal newborn health and involvement with the health system by establishing the PI model. A 3-tiered functional PI system was established within 2-years that served as the foundation for public private partnership (PPP) development, enhanced health services, trained volunteer community-based providers, emergency health funds, and MNC gains. From baseline to endline, 4-ANC visits increased from 5% to 14%, institutional delivery from 8% to 19%, and SBA delivery from 9% to 22%. In addition, the availability of weekly ANC services at health facilities increased from 60% to 90%, and 24/7 delivery services from 3% to 17%. Thermal care of the newborn increased from 10% to 40% and clean cord care from 57% to 69%.  The project strengthened PPP system collaboration with MOUs, a referral system, and participatory health committee structures in which the poor are now active MNC advocates with the government and government officials and health workers are meeting with them to make policies and decisions that address community needs. Study findings show that at endline over 60% of women of reproductive age (WRA) in the intervention area were active members of the PI groups and that WRA that were PI group members were significantly more likely to have higher levels of social capital than those who were not PI group members.

Best practice from this project for the global community to consider in promoting maternal newborn health care in marginalized communities include:

  • The PI Model delivery platform of community mobilization to promote MNC in partnership with MOH officials and health facilities/providers.
  • Community-supported and managed emergency health funds.
  • A MNC facility referral system that gives priority to poor mothers and children.
  • A volunteer system of trained village-based health providers working in collaboration with health facility providers to promote MNC.
  • Community involvement in the management and operations of local clinics.
  • HMIS matching meetings between PI and government providers/officials for accurate data to address MNC needs.

Results of the evaluation were disseminated in Netrokona Bangladesh to a stakeholder audience of public officials, NGO representatives, and community members. Government officials expressed appreciation for the gains in maternal newborn health that are occurring due to the public private partnership established by the project and pledged to continue working with the PI groups and CHWs to further MNC health in the district.



Bangladesh: The Context and People’s Institutions

Before I get into the details of the project that I am evaluating, a few details about the country. Bangladesh, part of what is called “South Asia” or the Indian Subcontinent, is the 12th most densely populated nation in the world; approximately 2½ times more densely populated than India. On the list of nations from richest to poorest it is 151st out of 184 (purchasing power parity). Administratively the country is divided into 7 divisions and 64 districts. Each of these districts is then divided into sub-districts or upazilas; there are a total of 485 upazilas in Bangladesh. The Netrokona district (shown in dark red in the map below), bordering India in the north, has 10 upazilas (similar to counties) two of which are the location of the project; two other upazilas were used as comparison areas.

Netrokona district is one of the poorly performing areas of Bangladesh in relation to maternal and newborn child health (UNICEF MICS, 2007). Some of this can be attributed to a familiar mix of health practices of mothers and of poorly functioning health care systems, including a lack of trained providers and operational clinics.

Location of Netrokona district, Bangladesh (dark red). All other districts are shown in faded colors.

Against this backdrop the project proposed an innovation that is an application of a community mobilization approach to maternal newborn health called “People’s Institutions,” originally developed in Bangladesh by World Renew, the U.S. NGO receiving this grant. A People’s Institution is a community-based organization composed of several smaller village-based groups or “Primary Groups” that has democratic governance characteristics. The Peoples Institution model utilizes a 3-tiered structure that consciously coincides with the administrative structure of the country described above. At the base are village-level groups called Primary

Structure of People's Institution model. The structure overlays the administrative structure of government shown on the right side.

Groups; at the middle tier are groups of villages—called Central Cooperative Committees. Finally at the peak of the pyramid at the sub-district level are People’s Institutions. Each of the tiers “feeds” representatives up the ladder so that the sub-district level has representation from all the villages. At the core of the function of this system are trained village health volunteers who are the primary touchpoint between the mothers and children with the health delivery system. Formation of People’s Institutions is a deliberate process that follows developmental stages starting with training in the villages and progressively forming higher tiers as the primary groups become functional. During the period of this project in the 2 upazilas 541 Primary Groups were formed in 505 villages; an amazing accomplishment.



Project Final Evaluation in Bangladesh

I am embarking on a trip to Bangladesh to evaluate a Maternal Newborn Child Health project performed in a rural district of Bangladesh. This is a 5-year innovation project funded by the USAID Child Survival and Health Grants Program. The goal of the project is to reduce mortality and improve health status among the most marginalized mothers and newborns in two sub-districts of Netrakona: Kendua and Durgapur. The Strategic Objective of the project is improved household and community MNCH-related behaviors and increased utilization of quality services.

Neither the smallest nor the most densely populated country in the world. About the size of Nepal with 5 times the population: 142 million.

The evaluation has all the challenges often associated with many evaluations: complex interventions and study designs performed in difficult to reach locations. There are no direct flights from the US to Dhaka so I opted for an overnight layover in New Delhi. The Indira Gandhi International airport offers several options for layovers; I chose the hotel in the international terminal, which allowed me to do an overnight without having to go through Indian immigration. An added bonus is that the gate for early morning departure to Dhaka was a short walk from the hotel entrance. I arrived in Dhaka relatively well rested.

Outside arrival gates at Hazrat Shahjalal International Airport (Dhaka)



Operations Research (OR): What is it? How is it incorporated into projects?

With Operations Research (OR) becoming in integral part of child survival projects, I attended a pre-CORE meeting workshop designed to explore this topic. OR was described as a subset of implementation research that studies the processes of an intervention to define HOW and WHY the intervention works.  It leads to a detailed description of key intervention components, which is useful for replication and scale-up.

Operational problems are barriers that hinder the uptake of interventions.  OR addresses these specific problems within a program by testing a feasible solution. As an example, for a barrier of limited accessibility to perinatal services, one might use OR to study the utility of maternity waiting homes. The focus is delineating feasible solution/s to overcoming the barrier/s.

Operations research uses scientific techniques with approaches that vary from descriptive to experimental. The OR study is formulated during the planning stage of a project, runs parallel with the project, involves ongoing monitoring and analysis, and includes immediate ongoing incorporation of findings to improve program operations.  Successful OR studies are aligned with the Ministry of Health, are contextually relevant, focus on and disseminate results that inform change, build on strategic partnerships, and engage/inform stakeholders.

With our training and past experience in scientific research methodology (both qualitative and quantitative), Story Consulting is well-positioned to assist organizations with OR study designs, implementation, and evaluation.

Here are some great OR resources from the CORE Group Spring Meeting:
CORE workshop slides: Understanding Operations Research in a Program
OR Workshop materials for CSHGP Innovation Grants

Grace Kreulen



“Programs must be designed with the exit in mind”

Sustainability was a key thread at the Spring CORE Group meeting.  Multiple presentations of post-project sustainability studies highlighted the importance of working to insure the lasting impact of program interventions. These studies also examined program characteristics that contribute to sustainable outcomes. Of particular note, Beatrice Rogers from Tufts University presented a model of key sustainability planning concepts.  To be effective, exit strategies must promote sustainability at the community level related to 4 factors:  1) assured adequate resources, 2) sufficient technical and management community/organizational capacity, 3) sufficient motivation to “want” to continue, and 4) adequate linkages with key organizational supports/government.  These then foster ongoing continuation of service delivery, access and demand, which contribute to sustained behaviors and sustained impact. A project’s sustainability timeline should take into consideration developing adequate resources, capacity and motivation, and include a lengthy period of gradual transition to independent operation before exit.

Studies from Bolivia, Honduras, and Kenya that examined these concepts and describe the variations across sector and setting are available in the presentation slides here.  You can also find additional resources about sustainability planning and evaluation at the Center for Design and Research in Sustainability.

Grace Kreulen



Capacity Strengthening for Global Health

The theme of the April 22-26, 2013 CORE Group Community Health Network Spring Meeting in Baltimore was: Capacity Strengthening for Global Health: Partnerships, Accountability, Integration and Learning.  Promoting participatory and inclusive country ownership is a key principal of development assistance that was examined from a variety of perspectives. In a series of blogs, I will write about capacity strengthening, sustainability and accountability, and operations research as well as intervention-updates related to mHealth, Care Groups, and CHW practice.

CORE’s keynote speaker, Dr. Leonardo Cubillos Turriago from the World Bank Institute (home of the e-Institute and Global Learning Development Network), explained the need for both a comprehensive, sustained, theory-based approach to building permanent capacity and for uniform tools to track, monitor, and evaluate capacity development efforts. In teasing apart operational definitions related to capacity building, he differentiated between capacity for development, or the availability of and manner with which societies deploy resources to pursue development goals on a sustainable basis, and capacity development, which is a locally driven process that involves sociopolitical, policy-related, and organizational efforts to enhance local ownership for and the effectiveness and efficiency of efforts to achieve a development goal. Capacity building requires strategic efforts to engage local communities and is foundational to achieving sustainable development.

In 2009, the WBI published The Capacity Development Results Framework (S. Otto, N. Agapitova & J. Behrens), to provide a theory-based structure within which to conduct capacity development programs. The CDRF provides a useful mechanism for stakeholders and practitioners to systematically think through key variables operating in a capacity development situation and to model explicitly the change process.

The model posits that capacity for development is dependent upon local ownership and effective resource use.  Change is locally owned and occurs in 3 areas, the capacity indicators, related to the sociopolitical environment, policy, and organizational arrangements. Individuals and groups (such as local community leaders, policymakers, media, and health providers) become change agents as they acquire and use knowledge and information to target change. Six learning outcomes essential to all capacity development interventions guide the design of capacity development program activities. These included altered status (raised awareness, enhanced skills), altered processes (improved teamwork, fostered coalitions), and new products (formulated policy/strategy, implemented strategy/plan). Activities are designed to achieve the necessary learning outcomes for the change agents. A list of standard capacity indicators assist with customizing measurement and evaluation to the local context. The entire framework is available here.

Use of this framework has potential to increase the success of capacity strengthening efforts in our development programs.

Grace Kreulen



Final Evaluation Wrap-up in New Delhi

The day after arriving in New Delhi, I went with five other members of the evaluation team to the USAID Mission in the US Embassy where we were welcomed by the Deputy Director Health Officer and staff. Following my presentation of the final evaluation findings, a discussion of the accomplishments of the project occurred. We were commended on a well-done evaluation and a project that scaled-up to all 1,297 villages in a district of 1 million people using community mobilization and district-level advocacy with public health officials for sustainable improvements in maternal child health.

The Background Poster for the Final Dissemination

The following day, EFICOR invited about 100 key people to a ‘Presentation of the Best Practices of the Parivartan Child Survival Project.’ Attending were officials from the National Rural Health Mission and the USAID Mission, along with representatives from NGOs and others involved in health projects. After introductions (which included flowers), I shared the results of the final evaluation. During the project:
• 28 of 33 key indicators in the areas of maternal newborn care (MNC), nutrition, immunizations, and infectious diseases saw statistically significant improvement (p ≤ 0.05).
• Government health facilities in the district improved in staffing, lab and malaria services, and provision of antenatal, delivery and child care.
• The organizational capacity and viability of village health committees expanded to impact maternal child health care at the village level.

The Parivartan project promoted maternal child health in the Sahibganj by building community capacity at the village level. They empowered 250 village health committees and mobilized villages for village health and nutrition days, improved MCH knowledge and practices with household timed counseling, and overcame cultural and religious barriers with targeted behavior change communication (BCC) strategies informed by barrier analysis. In addition, Parivartan provided training and support to 4,147 village citizens to promote MCH, including 1,233 CHWs called Sahiyas and 1,548 Anganwadi workers responsible for growth monitoring and nutrition counseling. The project also strengthened the local health systems through participating in planning and advocacy at the district level, supporting improved access to institutional deliveries, and strengthening the capacity of the local health sub-center to provide MNC and immunizations.

Several best practices from the project that were scaled-up at district andor state levels included Saas Bahu Pati Sammelian, a village-wide behavior change communication program developed to address the resistance of mothers-in-law and husbands to MNC, and the upgrading of health sub-centers to institutional delivery points.

One of the constraints that the project identified were the “stock outs” of essential drugs in the health centers and sub-centers. At the end of my presentation the Minister of Health attending the dissemination asked for specific information concerning these so that he could address them. Our hope is that the evaluation will have additional beneficial effects on delivery of drugs in the future.

This is the final post for this project. It has been a transforming experience to lead this team in evaluating the Parivartan project.
Grace Kreulen



The First Dissemination in Sahibganj

Once we had digested our qualitative data from the field interviews, I was ready to prepare the final disseminations to government officials in Sahibganj and then to stakeholders in New Delhi, including EFICOR and USAID. For better or worse, PowerPoint is the medium that I used to organize the information—words, pictures, charts… Working with other members of the evaluation team, I included an introduction to the project, a review of the methodology, summary of the KPC, field visits and findings, and ended with conclusions and recommendations.

The Beginning for the Dissemination

After practicing the presentations in the morning (the program manager and each of the block coordinators joined me in presenting a portion of the presentation), we went to the government office building in Sahibganj to give the dissemination. The key directors of health departments in the district were in attendance as were representatives from NGOs working in the region.

Health Ministers at the Dissemination

The dissemination provoked several long speeches by the ministers. The fact that there was significant improvement in nearly all of the indicators evoked congratulations and a celebration of the partnership between the Parivartan project and the government of Sahibganj.

The Happy Lead Evaluator

The Sahibganj ministers were especially interested in the improvements in acute respiratory infection indicators and spoke about the use of home-based strategies in order to save under 5 lives. Finishing with the output of the Lives Saved Analysis showing that during the 5 years of the project 3,278 under 5 lives were saved raised eyebrows and this was highlighted in many people’s comments.

Block Directors and Cluster Supervisors at the Dissemination

Grace Kreulen



Peeling the Onion

Before the groups returned to the project office in Sahibganj, I created a template for them to organize their interview results. I hoped that this would facilitate organizing information from disparate groups interviewing different types of stakeholders.

Interviewing Sahiyyas and Anganwadi Worker

After returning, the groups set to work in different offices fitting their interview notes into the A-C-E-S format and putting their results on charts that we would hang around the room. One non-trivial step was to translate the answers from Hindi to English.

To help the group turn over the soil of their memories of 5 years of work and to get beneath the meaning of what they were told in the interviews, the CRWRC project consultant stood in front of the group to ceremoniously peel an onion that she had retrieved from the kitchen. Amid the whir of ceiling fans beating the humid air and discussions revolving between Hindi, English and Bangla, we began to see patterns emerge. We saw over and over that people from government officials to women in villages thought that grass roots training of mothers by the Sahiyyas was a crucial activity responsible for the success of the program.

Using wall charts to report on field interviews

From this shared reporting I created a matrix that categorized the major activities around the program objectives. From this 2 key themes emerged that contributed to project accomplishments: 1) increasing the capacity of the community for maternal and child health, including the use of behavior change strategies such as theater and flip charts to reinforce safe delivery practices on the Village Health and Nutrition Days when the community gathered for government run programs; and 2) strengthening the local health system. An effective activity for this was the coordination of the local health care workers to chart and follow up on weights of babies and to track their immunizations.

Anganwadi worker with growth chart

Grace Kreulen



A Surprise on Our Field Visit

After my group’s first day in the field, we stayed in the Mariampahar Catholic mission near the village of Baharwa. My husband Dave and I had stayed there on our first visit to India in 2008. The father remembered us and provided us with rooms to spend the night. The accommodations were well, Spartan and like 4 years ago the electricity was unreliable; we prayed for a slight breeze to come in through our mosquito nets. The mission sits atop a hill along with a church. On the other side of the

The Mariampahur Mission Stands in the Background

hill from the main mission building is a mission-run school that houses 500 boys and 400 girls, nearly all of them from the Santhali tribe, one of the tribal groups with which the project works.
In the evening the children walked around the outside of the church reciting and singing vespers. We stood at the edge of the church grounds and watched; when they were finished they charged at us to say hello. Interestingly, they have last names that indicate their tribe; the Regional Advisor working on the project who is familiar with the tribes asked them to identify themselves by last name, many of which she knew.

Vespers at the Church

In the morning, after breakfast of boiled eggs and garbanzo beans, another father took Dave and I on a tour of the school grounds. To our amazement, there stood all 900 students in perfect rows waiting to greet us. The father had organized a ceremony, complete with us sitting on chairs on the steps to the

School children greet us before school

school. We were greeted with a song sung by a small choir and the entire school reciting greetings in English called out by the father. At the end of the ceremony, the students placed wreaths around our necks; they waved good-bye in unison as we went to our waiting vehicle. What a surprising way to start our second day of field visits.Watch the signing children.

Grace Kreulen