“Programs must be designed with the exit in mind”

Sustainability was a key thread at the Spring CORE Group meeting.  Multiple presentations of post-project sustainability studies highlighted the importance of working to insure the lasting impact of program interventions. These studies also examined program characteristics that contribute to sustainable outcomes. Of particular note, Beatrice Rogers from Tufts University presented a model of key sustainability planning concepts.  To be effective, exit strategies must promote sustainability at the community level related to 4 factors:  1) assured adequate resources, 2) sufficient technical and management community/organizational capacity, 3) sufficient motivation to “want” to continue, and 4) adequate linkages with key organizational supports/government.  These then foster ongoing continuation of service delivery, access and demand, which contribute to sustained behaviors and sustained impact. A project’s sustainability timeline should take into consideration developing adequate resources, capacity and motivation, and include a lengthy period of gradual transition to independent operation before exit.

Studies from Bolivia, Honduras, and Kenya that examined these concepts and describe the variations across sector and setting are available in the presentation slides here.  You can also find additional resources about sustainability planning and evaluation at the Center for Design and Research in Sustainability.

Capacity Strengthening for Global Health

The theme of the April 22-26, 2013 CORE Group Community Health Network Spring Meeting in Baltimore was: Capacity Strengthening for Global Health: Partnerships, Accountability, Integration and Learning.  Promoting participatory and inclusive country ownership is a key principal of development assistance that was examined from a variety of perspectives. In a series of blogs, I will write about capacity strengthening, sustainability and accountability, and operations research as well as intervention-updates related to mHealth, Care Groups, and CHW practice.

CORE’s keynote speaker, Dr. Leonardo Cubillos Turriago from the World Bank Institute (home of the e-Institute and Global Learning Development Network), explained the need for both a comprehensive, sustained, theory-based approach to building permanent capacity and for uniform tools to track, monitor, and evaluate capacity development efforts. In teasing apart operational definitions related to capacity building, he differentiated between capacity for development, or the availability of and manner with which societies deploy resources to pursue development goals on a sustainable basis, and capacity development, which is a locally driven process that involves sociopolitical, policy-related, and organizational efforts to enhance local ownership for and the effectiveness and efficiency of efforts to achieve a development goal. Capacity building requires strategic efforts to engage local communities and is foundational to achieving sustainable development.

In 2009, the WBI published The Capacity Development Results Framework (S. Otto, N. Agapitova & J. Behrens), to provide a theory-based structure within which to conduct capacity development programs. The CDRF provides a useful mechanism for stakeholders and practitioners to systematically think through key variables operating in a capacity development situation and to model explicitly the change process.

The model posits that capacity for development is dependent upon local ownership and effective resource use.  Change is locally owned and occurs in 3 areas, the capacity indicators, related to the sociopolitical environment, policy, and organizational arrangements. Individuals and groups (such as local community leaders, policymakers, media, and health providers) become change agents as they acquire and use knowledge and information to target change. Six learning outcomes essential to all capacity development interventions guide the design of capacity development program activities. These included altered status (raised awareness, enhanced skills), altered processes (improved teamwork, fostered coalitions), and new products (formulated policy/strategy, implemented strategy/plan). Activities are designed to achieve the necessary learning outcomes for the change agents. A list of standard capacity indicators assist with customizing measurement and evaluation to the local context. The entire framework is available here.

Use of this framework has potential to increase the success of capacity strengthening efforts in our development programs.

Final Evaluation Wrap-up in New Delhi

The day after arriving in New Delhi, I went with five other members of the evaluation team to the USAID Mission in the US Embassy where we were welcomed by the Deputy Director Health Officer and staff. Following my presentation of the final evaluation findings, a discussion of the accomplishments of the project occurred. We were commended on a well-done evaluation and a project that scaled-up to all 1,297 villages in a district of 1 million people using community mobilization and district-level advocacy with public health officials for sustainable improvements in maternal child health.

The Background Poster for the Final Dissemination

The following day, EFICOR invited about 100 key people to a ‘Presentation of the Best Practices of the Parivartan Child Survival Project.’ Attending were officials from the National Rural Health Mission and the USAID Mission, along with representatives from NGOs and others involved in health projects. After introductions (which included flowers), I shared the results of the final evaluation. During the project:
• 28 of 33 key indicators in the areas of maternal newborn care (MNC), nutrition, immunizations, and infectious diseases saw statistically significant improvement (p ≤ 0.05).
• Government health facilities in the district improved in staffing, lab and malaria services, and provision of antenatal, delivery and child care.
• The organizational capacity and viability of village health committees expanded to impact maternal child health care at the village level.

The Parivartan project promoted maternal child health in the Sahibganj by building community capacity at the village level. They empowered 250 village health committees and mobilized villages for village health and nutrition days, improved MCH knowledge and practices with household timed counseling, and overcame cultural and religious barriers with targeted behavior change communication (BCC) strategies informed by barrier analysis. In addition, Parivartan provided training and support to 4,147 village citizens to promote MCH, including 1,233 CHWs called Sahiyas and 1,548 Anganwadi workers responsible for growth monitoring and nutrition counseling. The project also strengthened the local health systems through participating in planning and advocacy at the district level, supporting improved access to institutional deliveries, and strengthening the capacity of the local health sub-center to provide MNC and immunizations.

Several best practices from the project that were scaled-up at district andor state levels included Saas Bahu Pati Sammelian, a village-wide behavior change communication program developed to address the resistance of mothers-in-law and husbands to MNC, and the upgrading of health sub-centers to institutional delivery points.

One of the constraints that the project identified were the “stock outs” of essential drugs in the health centers and sub-centers. At the end of my presentation the Minister of Health attending the dissemination asked for specific information concerning these so that he could address them. Our hope is that the evaluation will have additional beneficial effects on delivery of drugs in the future.

This is the final post for this project. It has been a transforming experience to lead this team in evaluating the Parivartan project.

The First Dissemination in Sahibganj

Once we had digested our qualitative data from the field interviews, I was ready to prepare the final disseminations to government officials in Sahibganj and then to stakeholders in New Delhi, including EFICOR and USAID. For better or worse, PowerPoint is the medium that I used to organize the information—words, pictures, charts… Working with other members of the evaluation team, I included an introduction to the project, a review of the methodology, summary of the KPC, field visits and findings, and ended with conclusions and recommendations.

The Beginning for the Dissemination

After practicing the presentations in the morning (the program manager and each of the block coordinators joined me in presenting a portion of the presentation), we went to the government office building in Sahibganj to give the dissemination. The key directors of health departments in the district were in attendance as were representatives from NGOs working in the region.

Health Ministers at the Dissemination

The dissemination provoked several long speeches by the ministers. The fact that there was significant improvement in nearly all of the indicators evoked congratulations and a celebration of the partnership between the Parivartan project and the government of Sahibganj.

The Happy Lead Evaluator

The Sahibganj ministers were especially interested in the improvements in acute respiratory infection indicators and spoke about the use of home-based strategies in order to save under 5 lives. Finishing with the output of the Lives Saved Analysis showing that during the 5 years of the project 3,278 under 5 lives were saved raised eyebrows and this was highlighted in many people’s comments.

Block Directors and Cluster Supervisors at the Dissemination

Peeling the Onion

Before the groups returned to the project office in Sahibganj, I created a template for them to organize their interview results. I hoped that this would facilitate organizing information from disparate groups interviewing different types of stakeholders.

Interviewing Sahiyyas and Anganwadi Worker

After returning, the groups set to work in different offices fitting their interview notes into the A-C-E-S format and putting their results on charts that we would hang around the room. One non-trivial step was to translate the answers from Hindi to English.

To help the group turn over the soil of their memories of 5 years of work and to get beneath the meaning of what they were told in the interviews, the CRWRC project consultant stood in front of the group to ceremoniously peel an onion that she had retrieved from the kitchen. Amid the whir of ceiling fans beating the humid air and discussions revolving between Hindi, English and Bangla, we began to see patterns emerge. We saw over and over that people from government officials to women in villages thought that grass roots training of mothers by the Sahiyyas was a crucial activity responsible for the success of the program.

Using wall charts to report on field interviews

From this shared reporting I created a matrix that categorized the major activities around the program objectives. From this 2 key themes emerged that contributed to project accomplishments: 1) increasing the capacity of the community for maternal and child health, including the use of behavior change strategies such as theater and flip charts to reinforce safe delivery practices on the Village Health and Nutrition Days when the community gathered for government run programs; and 2) strengthening the local health system. An effective activity for this was the coordination of the local health care workers to chart and follow up on weights of babies and to track their immunizations.

Anganwadi worker with growth chart

A Surprise on Our Field Visit

After my group’s first day in the field, we stayed in the Mariampahar Catholic mission near the village of Baharwa. My husband Dave and I had stayed there on our first visit to India in 2008. The father remembered us and provided us with rooms to spend the night. The accommodations were well, Spartan and like 4 years ago the electricity was unreliable; we prayed for a slight breeze to come in through our mosquito nets. The mission sits atop a hill along with a church. On the other side of the

The Mariampahur Mission Stands in the Background

hill from the main mission building is a mission-run school that houses 500 boys and 400 girls, nearly all of them from the Santhali tribe, one of the tribal groups with which the project works.
In the evening the children walked around the outside of the church reciting and singing vespers. We stood at the edge of the church grounds and watched; when they were finished they charged at us to say hello. Interestingly, they have last names that indicate their tribe; the Regional Advisor working on the project who is familiar with the tribes asked them to identify themselves by last name, many of which she knew.

Vespers at the Church

In the morning, after breakfast of boiled eggs and garbanzo beans, another father took Dave and I on a tour of the school grounds. To our amazement, there stood all 900 students in perfect rows waiting to greet us. The father had organized a ceremony, complete with us sitting on chairs on the steps to the

School children greet us before school

school. We were greeted with a song sung by a small choir and the entire school reciting greetings in English called out by the father. At the end of the ceremony, the students placed wreaths around our necks; they waved good-bye in unison as we went to our waiting vehicle. What a surprising way to start our second day of field visits.Watch the signing children.

Field Visits

Field visits are used to gather information from stakeholders and give life to the data collected for the final evaluation. For example, the final survey found that the percentage of children born in clinics or hospitals (institutional deliveries), one of the objectives of the project, more than doubled in the 5 years since the baseline survey. We went to the field to find out from staff, government officials and mothers what interventions of the project were most important for achieving results like this.

Interviewing a Village Health Committee

First, we have to get to the villages and institutions where the work was done. Now, like roads in most developing countries, the roads in rural India are a challenge to navigate. One challenge is presented by the conditions of the road, especially during the monsoon season: every pothole is a lake. At one point we did see a boy pulling tiny little fish from a large, muddy hole. The roads are also multi-use:

Sharing the Road

pedestrians, animals and people moving at all rates of speed on every imaginable kind of vehicle share them. We don’t watch the clock or the speedometer while we travel, we just stare in amazement at the life that is India unfolding at the side of the road. At this time of year the landscape is dominated by men and water buffaloes plowing and rows of women in colorful saris bent over stalks of rice that they are transplanting.

Transplanting Rice

We divided into three groups with 5-6 people per group. Each group visited different villages and toured facilities. Equipped with questions we developed and translated into Hindi, we fanned out across the district of Sahibganj; equivalent to an American county and home to about 1 million people. Each encounter was designed to have one questioner and two reporters. When we gather again at the main office, we will report our findings to the group.

Interviewing Anganwadi Workers in a Health Subcenter

Review of the Final Data Evokes Strong Emotions

In the second workshop our job was to help the staff identify the most important activities of the project. These “gems” would focus our thinking on what could be generalized from this project to other maternal and newborn health projects. I used the acronym “ACES” to guide our thinking: Accomplishments, Challenges, Exports and Sustainability. Because there was significant improvement in 22 of 29 indicators it was difficult to choose one or two interventions that were the most important, so after discussion it became clear that the cross-cutting strategies were the ones most likely to rise to the top as being most effective.

The A-C-E-S Framework

As we reviewed the indicators one-by-one, everyone was touched by the success of the project. From the Directors point of view those many hours spent training, supervising and evaluating had paid off with remarkable improvements in newborn and maternal health as well as the capacity of the district to continue the activities. A consensus emerged that the partnership between the project staff and government agencies was crucial for success. Both groups were pulling in the same direction and as the project progressed, the government workers began to incorporate the strategies developed for the project into government programs. Grass-roots timed counseling was an important activity that led to significant improvements in indicators such as community-based antenatal care and post-natal visits with trained health workers. The health workers on the ground, called “Sahiyas,” came from the communities that they worked in and could deliver timed counseling to families in their homes. We plan to meet with Sahiya groups in our field visits.

A Group of Sahiyas

The On-Site Portion of the Process Begins in Sahibganj

In the offices of EFICOR, in Sahibganj, the on-site project manager, directors and coordinators met to develop the process of gathering information that would support and amplify the data in the final KPC report. In the first day and a half I started with what I called workshops. Because there had been baseline and mid-term reports, many of the people were familiar with the general objectives of gathering qualitative information from the field. Nonetheless, we would spend the first half-day reviewing the objectives of the final evaluation and report. The second workshop was devoted to a final review of the data, determining stakeholder groups and developing field questions.

Street Scene in Sahibganj

My first step was to spend some time getting acquainted and re-acquainted with each other. The most popular personal item from my past was a wedding picture of my husband and me. Once the laughter died down, members of the team shared what working in the project has meant to them. Then we began our work.
One of the objectives of the final evaluation is for the staff to relate activities that had the greatest impact on outcomes. This was actually made more difficult by the fact that nearly all of the indicators had significant improvement compared to the baseline study that was performed 5 years ago before the project began. I had to work hard to get the staff to tease out the “gems” from all the interventions that they were involved in over 5 years. By using a combination of approaches including my role-playing an interviewer, we finally developed a set of questions that would be first translated into Hindi and then used to interview stakeholders in the field.

Role Playing an Interview

The First Step: Meeting with Project Directors and Consultants

Project Directors

I met with the project directors from CRWRC and EFICOR as well as the consultant who wrote the final Knowledge, Practice and Coverage report. We analyzed the final data and discussed possible interpretations. Because nearly all of the objectives of the project were met, it was a pleasant meeting. What a joy it is to be the evaluator on a successful project that I have followed since its inception.

After the meeting we boarded the train in Kolkata for the 5 hour trip north to Sahibganj, the “district seat” of Sahibganj district where the project was carried out. We enjoyed 5 hours of “restful” sleep on the train before we arrived at 4 a.m. Our next step will be to begin meetings with project staff–the troops on the ground. Now i can begin to tease out the significant accomplishments of the project.

Grace in her sleeper on the train