EXECUTIVE SUMMARY
In the 2016–17 school year, the United Nations Children’s Fund (UNICEF), in partnership with the United Nations World Food Programme (WFP) and in coordination with the Ministry of Education and Higher Education (MEHE) in Lebanon, started to pilot a child-focused cash transfer program for displaced Syrian children in Lebanon. The program, known as the No Lost Generation (NLG) or “Min Ila” (meaning “from/to”) was designed to reduce negative coping strategies harmful to children and reduce barriers to children’s school attendance, including financial barriers and reliance on child labor. UNICEF Lebanon contracted the American Institute for Research (AIR) to help UNICEF Office of Research (OoR) design and implement an impact evaluation of the program.1 The purpose of the impact evaluation, one of the first rigorous studies of a social protection program supporting children in a complex displacement setting, is to monitor the program’s effects on recipients and provide evidence to UNICEF, WFP, and MEHE for decisions regarding the program’s future. This report investigates and discusses the program’s impacts on child well-being outcomes, including food security, health, child work, child subjective well-being, enrollment, and attendance, after 1 year of program implementation.
The Context:
The Syrian crisis is now in its seventh year and continues to negatively impact the region as millions of Syrian refugees move into neighboring countries. Lebanon has one of the highest percapita ratios of registered refugees in the world (Government of Lebanon and the United Nations, 2014). According to the revised Lebanon Crisis Response Plan (LCRP 2018), Lebanon hosts 1.5 million Syrians who have fled conflict in Syria, as well as 34,000 Palestine refugees from Syria.
The LCRP states that “more than 76 percent of displaced Syrians are living below the poverty line”. The report claims that these poor displaced Syrians carry an average debt of $798 per household, with a majority of their debt related to funds for food. To respond to their situation, many displaced Syrian households turn to negative coping strategies such as selling off assets and withdrawing children from school. There are many children of school age in this population with 586,540 displaced Syrian children registered in Lebanon and 57,506 Palestine Refugees between 3-18 years old. Over half of these children are not enrolled in a certified education program (formal and non-formal).
The rapid influx of refugees burdens the Lebanese economy, costing the country roughly US$18.15 billion by 2015 and placing huge pressure on public services. The 1.5 million displaced Syrians increase the demand on infrastructure and social services, which struggle to meet increased needs.
The LCRP states that “Lebanon’s healthcare facilities have been overburdened by an increase in utilization of up to 50 percent in some cases, greatly affecting their capacity to absorb a higher caseload as well as their financial sustainability”.
Meanwhile basic infrastructure cannot keep up with the large demand, for example 64 percent of the population does not have access to safe drinking water services.
In short, most Syrians arrived with limited savings and have struggled to earn steady incomes to meet their families’ basic needs, such as food, healthcare, and shelter. These basic needs tend to require immediate attention, which means that Syrian families often must forgo education and its long-term benefits in favor of short-term needs.
The Program:
In the 2016–17 school year, UNICEF Lebanon started to pilot NLG/Min Ila in partnership with WFP and in cooperation with MEHE in the governorates of Mount Lebanon and Akkar. The objective of the pilot was to test and evaluate whether and how to scale-up the program to national coverage. NLG/ Min Ila was designed to help households meet the implicit costs of education and reduce reliance on children for negative coping strategies such as child labour and early marriage. Syrian refugee children ages 5–14 who lived in the Mount Lebanon and Akkar governorates and were enrolled in a second-shift school (children enrolled in first shift were not eligible) received a basic monthly education transfer of US$20 to cover a portion of the indirect costs of going to school, such as school snacks, transportation, and appropriate clothing and shoes. Syrian children ages 10 and older who were enrolled in a second-shift school received an additional monthly US$45 to factor in the higher earnings of a working child in this age group. The child well-being transfer lasted for the duration of the school year, and payments were made every month via a common ATM card used by all major agencies delivering cash transfers in Lebanon (the Lebanon One Unified Inter-Organizational System for E-cards, or LOUISE). While no conditions had to be met to receive the cash, school attendance was monitored and households received visits if children did not attend school regularly, the objective of which was to refer households to additional services (e.g. health, child protection, etc.).
In the current 2017–18 school year, the program has expanded into the governorate of North Lebanon and the benefit level has been altered.
There is no variation in the amount of the basic monthly transfers provided to children enrolled in a second-shift school. Children who are enrolled in Preparatory Early Childhood Education are disabled, or face difficulties reaching school due to distance, terrain or security issues continue to receive the basic monthly transfer of US$20.3 Children outside of these categories now receive a lower basic monthly amount: US$13.50. The additional amount provided monthly to older children has been lowered to US$20 and is now provided only to children ages 12 and older. These changes were made to cope with several changes in the programming environment.
Impact Evaluation:
The nonexperimental longitudinal study design compares beneficiaries in the pilot governorates of Mount Lebanon and Akkar with households that would be eligible for the program but who are not receiving the programme because they live in the nonprogram governorates of North Lebanon and South Lebanon and Nabatieh. The study uses a geographic regression discontinuity design (RDD), in which households that are located near the border separating pilot and comparison governorates are compared with each other. The study follows the same households over time, with the baseline data collected September–October 2016 prior to the start of the program, midline data collected February–March 2017 during the first school year in which the program operated, and endline data collected November–December 2017, at the beginning of the 2017–18 school year. The purpose of this study is to measure the impact of the program on children’s educational outcomes and their broader well-being. The evaluation includes 1,440 households, with roughly 20 from each of the study’s 74 clusters, all of which are located near at least one second-shift school. This study is an impact evaluation that estimates the effects of the program on specified outcomes of interest using a counterfactual group, as opposed to a more general evaluation that investigates other aspects of the program such as sustainability and efficiency.
We present findings by outcome area in the order they fall along the pathways to program goals as explained in the theory of change. By investigating progress along the theory of change, we can assess whether the program is moving in the right direction toward stated goals and where it might hit potential obstacles. Most of the findings presented in this report are based on endline data collected at the start of the second year of program operation. However, we make reference to the midline findings where relevant (De Hoop,
Morey, & Seidenfeld 2017).
Food Consumption and Child Health:
Min Ila had a positive impact on children’s food consumption. Fewer children in pilot governorates ages 10–14 skipped a meal the previous day than similar children in the comparison group, which is an impact of a 13 percentage point reduction (15% of children in pilot governorates skipped a meal at endline). More children in pilot governorates started the day with breakfast than comparison children, with a 19 percentage point impact of the program (68% ate breakfast). Fewer children in pilot governorates also went to bed hungry at the end of the day, with a 13 percentage point reduction in hunger (11% went to bed hungry). We find that in addition to buying food, the households in pilot governorates also spent more on healthcare for their children. Households in pilot governorates spent on average US$9.95 more on healthcare for their children over the previous 30 days than households in comparison governorates. The probability that caregivers indicated that their children were in good health improved by 10 percentage points for younger children and 8 percentage points for older children in pilot governorates as compared with non-pilot governorates.
Child Work:
The program consistently reduced the percentage of children ages 10–14 carrying out household chores. The Min Ila program reduced the number of children caring for a family member by 17 percentage points as compared with children in non-pilot governorates, with 28% of children in the pilot governorates spending time caring for a family member. Similarly, fewer children in pilot governorates fetched firewood or water (14 percentage point reduction). Reductions in work in the household are particularly strong for girls, with the program consistently reducing the number of girls performing each of these tasks by 23 percentage points.
Subjective Well-Being:
The program improves children’s well-being in pilot governorates as compared with children in non-NLG/Min Ila governorates. Children in pilot governorates felt more optimistic about the future, were more trusting of other people, and felt more confident and assertive. There is also suggestive evidence of a reduction in depression rates. Similar to children’s time use, we find slightly larger impacts for girls than for boys, possibly related to the finding that a higher percentage of girls than boys were able to reduce their time working in the household. Meanwhile, 13% of children report low self-esteem, which is the same as the comparison group, so the program does not seem to have an effect on this outcome.
Education:
Aggregate MEHE figures suggest that formal school enrollment rates of displaced Syrian children increased rapidly across the country from the past (2015–16) to the current school year. Average enrolment in second shift schools increased by 51 percent in NLG/Min Ila pilot areas compared to 41 percent in the rest of the country during the 2016-17 school year (midline), potentially signalling an impact of the cash transfer program on enrolment outcomes. This study also found that schooling rates increased in both pilot and comparison areas, from nearly 60% at baseline to nearly 80% at follow-up. School enrollment increases were particularly pronounced for children ages 5–9, whose school enrollment increased from slightly over 60% to nearly 90%.
As the result of a sharp nationwide increase in second-shift school enrollment of Syrian children in the 2016–17 school year, more than half of all second-shift schools in the pilot areas of the study reached full capacity while registering children and had to turn away children who wanted to enroll. While MEHE was prepared to open new second shifts in existing schools to accommodate the increase, as in previous years, in practice these capacity constraints were not resolved in the study areas of the pilot governorates. This situation created a ceiling effect because it is impossible for the program to increase enrollment above the capacity of the second-shift schools. In other words, the program could not demonstrate its full potential due to the limit on spaces to enroll children in second-shift schools, a prerequisite to receiving the program. As a result, no impacts on school enrollment were observed either at midline or at endline even though school enrollment increased over time.
At midline, roughly halfway during the 2016–17 school year, school attendance increased by 0.5 to 0.7 days a week among children who did enroll in a second-shift school, an improvement of about 20% over the control group. During endline, which took place at the start of the 2017– 18 school year, self-reported school attendance rates among children enrolled in school were high in both pilot and comparison areas (an average of 4.85 days attendance per week out of 5 days). Hence, the margin for improvement in attendance was low and no impact on attendance could be observed. Qualitative results suggest that attendance starts high and decreases during the school year, making it harder for the program to demonstrate effects on attendance at the beginning of the school year. These qualitative results are consistent with the study’s findings on attendance between midline that occurred in the middle of the school year and endline, that occurred at the beginning of the school year.
Moreover, the program improved several other education-related outcomes at endline. The annual amount of money spent on children’s education expenses increased, on average, by US$60.58 as a result of the program. Households reported spending on average US$103.18 on educational expenses in NLG/Min Ila pilot governorates. Interestingly, the program generated a slightly larger impact on spending for girls (US$65.59 impact) than for boys (US$56.24 impact). The average total spending on education for girls was the same as boys at endline, suggesting that the program brought more gender equity in spending on education. The program also increased the percentage of students using the bus to travel to school by 23 percentage points, with 57% of children enrolled in school taking the bus at endline.
Limitations:
Several limitations could affect the impact evaluation’s ability to detect programmatic effects.
We name the two most important ones here.
First, this study was designed to capture program effects among children living in the vicinity of an active second-shift school. The rationale was that these children could readily enroll in a secondshift school in response to the program, allowing the impact evaluation to capture the impact of increasing the demand for education through a cash transfer program. Due to the sample being selected from areas with existing schools, expanded enrollment in newly opened second shifts not located near the sampled schools could not be captured.
Second, we cannot entirely rule out the possibility that other systematic changes in the pilot or comparison governorates drive the findings presented in this report. For instance, if another major government intervention was carried out in the NLG/Min Ila pilot areas but not in the comparison areas (or vice versa) this may lead to incorrect attribution of changes in child wellbeing indicators to the program. However, there are currently no indications that such systematic factors confound or drive the findings presented in this report.
Conclusion and Recommendations:
Min Ila improved children’s food consumption and their physical health, lowered child engagement in work for the household, improved indicators of children’s subjective well-being and increased school attendance at midline. As cited throughout this report, many studies demonstrate the causal link between these domains and child educational outcomes. Therefore, in addition to improving the well-being of children (a primary goal of the program), Min Ila also makes positive gains for indicators along the causal pathway in the theory of change to improved educational goals.
The positive impacts generated by the program also demonstrate the ability of stakeholders including UNICEF, WFP, and MEHE, to successfully implement the cash transfer program in a challenging refugee setting. After one full year of implementation, the program managed to reach an increasing number of beneficiaries with frequent and regular payments. The stakeholders also ambitiously set out to learn about the program through a rigorous impact evaluation that uses an RDD design to estimate effects. Although cash transfer programs are regularly paired with an impact evaluation, few cash programs in refugee settings have rigorously established program effects. There are many programs that aim to assist refugees, including similar cash transfer programs for Syrian refugees living in Jordan and Turkey, jointly supporting hundreds of thousands of children, yet very few have been rigorously evaluated, leaving an important gap in our knowledge about what programs work to help refugees. This study implements a geographical RDD to estimate program effects with strong internal validity in a challenging context. Thus, this study, one of the first studies of its kind, represents a meaningful contribution to the literature on the effectiveness of cash transfer programs to assist refugee families in sending their children to school. The evidence generated from this study should prove useful for policymakers and funders to make informed decisions on how to allocate scarce resources for refugees in low- and middleincome countries.
This study is an impact evaluation with primary objectives to provide evidence on the effectiveness of the program that can both feed into broader policy discussions and global learning, and not necessarily to provide recommendations about program implementation. However, the authors worked collaboratively with UNICEF Lebanon and WFP to generate several recommendations based on the results of the study. UNICEF Lebanon country office requested that one recommendation relates to the design of the program to target stakeholders and policymakers who design and implement programs in Lebanon, while another targets people who procure or conduct research in Lebanon and relates to future research. The last two recommendations result from the operational performance piece of the evaluation.
The primary objective of the NLG/Min Ila program was to improve education-related outcomes, namely enrolment and attendance.
Although the program only affected attendance at midline and neither attendance nor enrolment at endline, the programme demonstrated significant improvement in other child wellbeing aspects that are related to education.
The program generated effects for children across important domains such as health, food consumption, child work, and subjective wellbeing. These domains are important in and of themselves for healthy child development and wellbeing. Given these important results, we recommend that the program revisit the primary objective to move towards a more holistic improvement of a child’s well-being with the focus of covering multiple needs, not just education.We cite many studies throughout this report that show the connection between education outcomes and health, food consumption, child work, and subjective well-being and explain how these are important steps along the pathway to affecting education outcomes. However, all of these studies occurred outside of Lebanon and in different contexts, mostly being poverty programs and not programs for refugees.
Thus, we recommend conducting research into the connections between these important domains for child-wellbing within Lebanon and especially within the refugee context, in order to strengthen and understand the child from a holistic point of view.Although we did not find program impacts on enrolment in school, the evidence suggests that demand for school went up but that supply was not able to respond quickly enough. Many parents who wanted to enroll their children in school were unable to do so due to insufficient capacity. Given that the cash transfer program aimed to increase school enrolment and attendance, we recommend continued advocacy for expanding school supply in areas where public schools have reached capacity. A more general recommendation for humanitarian agencies, particularly those operating in settings of massive displacement, is that close coordination between demand and supply side policies is critical for programs to realize their full potential and maximize their effectiveness.
When investigating the operational performance of the program, as described in more detail below, we learned that recipients of the program may have misunderstood key aspects that might affect their behavior. For example, recipients did not clearly understand why they were eligible to receive the program and what are the selection criteria. Similarly, they may have falsely believed that there are conditions to continue receiving the program, thinking that they cannot mis days of school. Teachers and school administrators also perceived this conditionality. This misunderstanding might prevent or dissuade other eligible households from participating in the program, fearing that they are not eligible or cannot meet the perceived conditions. Clear communication about the program to the community and school administrators may improve program operations and increase the number of participants in the program.