With Operations Research (OR) becoming in integral part of child survival projects, I attended a pre-CORE meeting workshop designed to explore this topic. OR was described as a subset of implementation research that studies the processes of an intervention to define HOW and WHY the intervention works. It leads to a detailed description of key intervention components, which is useful for replication and scale-up.
Operational problems are barriers that hinder the uptake of interventions. OR addresses these specific problems within a program by testing a feasible solution. As an example, for a barrier of limited accessibility to perinatal services, one might use OR to study the utility of maternity waiting homes. The focus is delineating feasible solution/s to overcoming the barrier/s.
Operations research uses scientific techniques with approaches that vary from descriptive to experimental. The OR study is formulated during the planning stage of a project, runs parallel with the project, involves ongoing monitoring and analysis, and includes immediate ongoing incorporation of findings to improve program operations. Successful OR studies are aligned with the Ministry of Health, are contextually relevant, focus on and disseminate results that inform change, build on strategic partnerships, and engage/inform stakeholders.
With our training and past experience in scientific research methodology (both qualitative and quantitative), Story Consulting is well-positioned to assist organizations with OR study designs, implementation, and evaluation.
Here are some great OR resources from the CORE Group Spring Meeting:
CORE workshop slides: Understanding Operations Research in a Program
OR Workshop materials for CSHGP Innovation Grants
Sustainability was a key thread at the Spring CORE Group meeting. Multiple presentations of post-project sustainability studies highlighted the importance of working to insure the lasting impact of program interventions. These studies also examined program characteristics that contribute to sustainable outcomes. Of particular note, Beatrice Rogers from Tufts University presented a model of key sustainability planning concepts. To be effective, exit strategies must promote sustainability at the community level related to 4 factors: 1) assured adequate resources, 2) sufficient technical and management community/organizational capacity, 3) sufficient motivation to “want” to continue, and 4) adequate linkages with key organizational supports/government. These then foster ongoing continuation of service delivery, access and demand, which contribute to sustained behaviors and sustained impact. A project’s sustainability timeline should take into consideration developing adequate resources, capacity and motivation, and include a lengthy period of gradual transition to independent operation before exit.
Studies from Bolivia, Honduras, and Kenya that examined these concepts and describe the variations across sector and setting are available in the presentation slides here. You can also find additional resources about sustainability planning and evaluation at the Center for Design and Research in Sustainability.
The theme of the April 22-26, 2013 CORE Group Community Health Network Spring Meeting in Baltimore was: Capacity Strengthening for Global Health: Partnerships, Accountability, Integration and Learning. Promoting participatory and inclusive country ownership is a key principal of development assistance that was examined from a variety of perspectives. In a series of blogs, I will write about capacity strengthening, sustainability and accountability, and operations research as well as intervention-updates related to mHealth, Care Groups, and CHW practice.
CORE’s keynote speaker, Dr. Leonardo Cubillos Turriago from the World Bank Institute (home of the e-Institute and Global Learning Development Network), explained the need for both a comprehensive, sustained, theory-based approach to building permanent capacity and for uniform tools to track, monitor, and evaluate capacity development efforts. In teasing apart operational definitions related to capacity building, he differentiated between capacity for development, or the availability of and manner with which societies deploy resources to pursue development goals on a sustainable basis, and capacity development, which is a locally driven process that involves sociopolitical, policy-related, and organizational efforts to enhance local ownership for and the effectiveness and efficiency of efforts to achieve a development goal. Capacity building requires strategic efforts to engage local communities and is foundational to achieving sustainable development.
In 2009, the WBI published The Capacity Development Results Framework (S. Otto, N. Agapitova & J. Behrens), to provide a theory-based structure within which to conduct capacity development programs. The CDRF provides a useful mechanism for stakeholders and practitioners to systematically think through key variables operating in a capacity development situation and to model explicitly the change process.
The model posits that capacity for development is dependent upon local ownership and effective resource use. Change is locally owned and occurs in 3 areas, the capacity indicators, related to the sociopolitical environment, policy, and organizational arrangements. Individuals and groups (such as local community leaders, policymakers, media, and health providers) become change agents as they acquire and use knowledge and information to target change. Six learning outcomes essential to all capacity development interventions guide the design of capacity development program activities. These included altered status (raised awareness, enhanced skills), altered processes (improved teamwork, fostered coalitions), and new products (formulated policy/strategy, implemented strategy/plan). Activities are designed to achieve the necessary learning outcomes for the change agents. A list of standard capacity indicators assist with customizing measurement and evaluation to the local context. The entire framework is available here.
Use of this framework has potential to increase the success of capacity strengthening efforts in our development programs.
The day after arriving in New Delhi, I went with five other members of the evaluation team to the USAID Mission in the US Embassy where we were welcomed by the Deputy Director Health Officer and staff. Following my presentation of the final evaluation findings, a discussion of the accomplishments of the project occurred. We were commended on a well-done evaluation and a project that scaled-up to all 1,297 villages in a district of 1 million people using community mobilization and district-level advocacy with public health officials for sustainable improvements in maternal child health.
The Background Poster for the Final Dissemination
The following day, EFICOR invited about 100 key people to a ‘Presentation of the Best Practices of the Parivartan Child Survival Project.’ Attending were officials from the National Rural Health Mission and the USAID Mission, along with representatives from NGOs and others involved in health projects. After introductions (which included flowers), I shared the results of the final evaluation. During the project:
• 28 of 33 key indicators in the areas of maternal newborn care (MNC), nutrition, immunizations, and infectious diseases saw statistically significant improvement (p ≤ 0.05).
• Government health facilities in the district improved in staffing, lab and malaria services, and provision of antenatal, delivery and child care.
• The organizational capacity and viability of village health committees expanded to impact maternal child health care at the village level.
The Parivartan project promoted maternal child health in the Sahibganj by building community capacity at the village level. They empowered 250 village health committees and mobilized villages for village health and nutrition days, improved MCH knowledge and practices with household timed counseling, and overcame cultural and religious barriers with targeted behavior change communication (BCC) strategies informed by barrier analysis. In addition, Parivartan provided training and support to 4,147 village citizens to promote MCH, including 1,233 CHWs called Sahiyas and 1,548 Anganwadi workers responsible for growth monitoring and nutrition counseling. The project also strengthened the local health systems through participating in planning and advocacy at the district level, supporting improved access to institutional deliveries, and strengthening the capacity of the local health sub-center to provide MNC and immunizations.
Several best practices from the project that were scaled-up at district andor state levels included Saas Bahu Pati Sammelian, a village-wide behavior change communication program developed to address the resistance of mothers-in-law and husbands to MNC, and the upgrading of health sub-centers to institutional delivery points.
One of the constraints that the project identified were the “stock outs” of essential drugs in the health centers and sub-centers. At the end of my presentation the Minister of Health attending the dissemination asked for specific information concerning these so that he could address them. Our hope is that the evaluation will have additional beneficial effects on delivery of drugs in the future.
This is the final post for this project. It has been a transforming experience to lead this team in evaluating the Parivartan project.
Once we had digested our qualitative data from the field interviews, I was ready to prepare the final disseminations to government officials in Sahibganj and then to stakeholders in New Delhi, including EFICOR and USAID. For better or worse, PowerPoint is the medium that I used to organize the information—words, pictures, charts… Working with other members of the evaluation team, I included an introduction to the project, a review of the methodology, summary of the KPC, field visits and findings, and ended with conclusions and recommendations.
The Beginning for the Dissemination
After practicing the presentations in the morning (the program manager and each of the block coordinators joined me in presenting a portion of the presentation), we went to the government office building in Sahibganj to give the dissemination. The key directors of health departments in the district were in attendance as were representatives from NGOs working in the region.
Health Ministers at the Dissemination
The dissemination provoked several long speeches by the ministers. The fact that there was significant improvement in nearly all of the indicators evoked congratulations and a celebration of the partnership between the Parivartan project and the government of Sahibganj.
The Happy Lead Evaluator
The Sahibganj ministers were especially interested in the improvements in acute respiratory infection indicators and spoke about the use of home-based strategies in order to save under 5 lives. Finishing with the output of the Lives Saved Analysis showing that during the 5 years of the project 3,278 under 5 lives were saved raised eyebrows and this was highlighted in many people’s comments.
Block Directors and Cluster Supervisors at the Dissemination
Before the groups returned to the project office in Sahibganj, I created a template for them to organize their interview results. I hoped that this would facilitate organizing information from disparate groups interviewing different types of stakeholders.
Interviewing Sahiyyas and Anganwadi Worker
After returning, the groups set to work in different offices fitting their interview notes into the A-C-E-S format and putting their results on charts that we would hang around the room. One non-trivial step was to translate the answers from Hindi to English.
To help the group turn over the soil of their memories of 5 years of work and to get beneath the meaning of what they were told in the interviews, the CRWRC project consultant stood in front of the group to ceremoniously peel an onion that she had retrieved from the kitchen. Amid the whir of ceiling fans beating the humid air and discussions revolving between Hindi, English and Bangla, we began to see patterns emerge. We saw over and over that people from government officials to women in villages thought that grass roots training of mothers by the Sahiyyas was a crucial activity responsible for the success of the program.
Using wall charts to report on field interviews
From this shared reporting I created a matrix that categorized the major activities around the program objectives. From this 2 key themes emerged that contributed to project accomplishments: 1) increasing the capacity of the community for maternal and child health, including the use of behavior change strategies such as theater and flip charts to reinforce safe delivery practices on the Village Health and Nutrition Days when the community gathered for government run programs; and 2) strengthening the local health system. An effective activity for this was the coordination of the local health care workers to chart and follow up on weights of babies and to track their immunizations.
Anganwadi worker with growth chart
After my group’s first day in the field, we stayed in the Mariampahar Catholic mission near the village of Baharwa. My husband Dave and I had stayed there on our first visit to India in 2008. The father remembered us and provided us with rooms to spend the night. The accommodations were well, Spartan and like 4 years ago the electricity was unreliable; we prayed for a slight breeze to come in through our mosquito nets. The mission sits atop a hill along with a church. On the other side of the
The Mariampahur Mission Stands in the Background
hill from the main mission building is a mission-run school that houses 500 boys and 400 girls, nearly all of them from the Santhali tribe, one of the tribal groups with which the project works.
In the evening the children walked around the outside of the church reciting and singing vespers. We stood at the edge of the church grounds and watched; when they were finished they charged at us to say hello. Interestingly, they have last names that indicate their tribe; the Regional Advisor working on the project who is familiar with the tribes asked them to identify themselves by last name, many of which she knew.
Vespers at the Church
In the morning, after breakfast of boiled eggs and garbanzo beans, another father took Dave and I on a tour of the school grounds. To our amazement, there stood all 900 students in perfect rows waiting to greet us. The father had organized a ceremony, complete with us sitting on chairs on the steps to the
School children greet us before school
school. We were greeted with a song sung by a small choir and the entire school reciting greetings in English called out by the father. At the end of the ceremony, the students placed wreaths around our necks; they waved good-bye in unison as we went to our waiting vehicle. What a surprising way to start our second day of field visits.Watch the signing children.
Field visits are used to gather information from stakeholders and give life to the data collected for the final evaluation. For example, the final survey found that the percentage of children born in clinics or hospitals (institutional deliveries), one of the objectives of the project, more than doubled in the 5 years since the baseline survey. We went to the field to find out from staff, government officials and mothers what interventions of the project were most important for achieving results like this.
Interviewing a Village Health Committee
First, we have to get to the villages and institutions where the work was done. Now, like roads in most developing countries, the roads in rural India are a challenge to navigate. One challenge is presented by the conditions of the road, especially during the monsoon season: every pothole is a lake. At one point we did see a boy pulling tiny little fish from a large, muddy hole. The roads are also multi-use:
Sharing the Road
pedestrians, animals and people moving at all rates of speed on every imaginable kind of vehicle share them. We don’t watch the clock or the speedometer while we travel, we just stare in amazement at the life that is India unfolding at the side of the road. At this time of year the landscape is dominated by men and water buffaloes plowing and rows of women in colorful saris bent over stalks of rice that they are transplanting.
We divided into three groups with 5-6 people per group. Each group visited different villages and toured facilities. Equipped with questions we developed and translated into Hindi, we fanned out across the district of Sahibganj; equivalent to an American county and home to about 1 million people. Each encounter was designed to have one questioner and two reporters. When we gather again at the main office, we will report our findings to the group.
Interviewing Anganwadi Workers in a Health Subcenter
In the second workshop our job was to help the staff identify the most important activities of the project. These “gems” would focus our thinking on what could be generalized from this project to other maternal and newborn health projects. I used the acronym “ACES” to guide our thinking: Accomplishments, Challenges, Exports and Sustainability. Because there was significant improvement in 22 of 29 indicators it was difficult to choose one or two interventions that were the most important, so after discussion it became clear that the cross-cutting strategies were the ones most likely to rise to the top as being most effective.
The A-C-E-S Framework
As we reviewed the indicators one-by-one, everyone was touched by the success of the project. From the Directors point of view those many hours spent training, supervising and evaluating had paid off with remarkable improvements in newborn and maternal health as well as the capacity of the district to continue the activities. A consensus emerged that the partnership between the project staff and government agencies was crucial for success. Both groups were pulling in the same direction and as the project progressed, the government workers began to incorporate the strategies developed for the project into government programs. Grass-roots timed counseling was an important activity that led to significant improvements in indicators such as community-based antenatal care and post-natal visits with trained health workers. The health workers on the ground, called “Sahiyas,” came from the communities that they worked in and could deliver timed counseling to families in their homes. We plan to meet with Sahiya groups in our field visits.
A Group of Sahiyas
In the offices of EFICOR, in Sahibganj, the on-site project manager, directors and coordinators met to develop the process of gathering information that would support and amplify the data in the final KPC report. In the first day and a half I started with what I called workshops. Because there had been baseline and mid-term reports, many of the people were familiar with the general objectives of gathering qualitative information from the field. Nonetheless, we would spend the first half-day reviewing the objectives of the final evaluation and report. The second workshop was devoted to a final review of the data, determining stakeholder groups and developing field questions.
Street Scene in Sahibganj
My first step was to spend some time getting acquainted and re-acquainted with each other. The most popular personal item from my past was a wedding picture of my husband and me. Once the laughter died down, members of the team shared what working in the project has meant to them. Then we began our work.
One of the objectives of the final evaluation is for the staff to relate activities that had the greatest impact on outcomes. This was actually made more difficult by the fact that nearly all of the indicators had significant improvement compared to the baseline study that was performed 5 years ago before the project began. I had to work hard to get the staff to tease out the “gems” from all the interventions that they were involved in over 5 years. By using a combination of approaches including my role-playing an interviewer, we finally developed a set of questions that would be first translated into Hindi and then used to interview stakeholders in the field.
Role Playing an Interview