Why do Roma children generally not speak Bulgarian well?
Because at school they are taught it as if it were their mother tongue, which it is not. Additional classes don’t help either, since they just provide “more of the same.” On the other hand, most young people in Bulgaria already speak at least one other language. This is because it is taught using methods for learning a second language. Realizing these facts, we at the Trust for Social Achievement (TSA), together with our partners from Teach for Bulgaria and the association EducArt, developed resources for students and teachers to teach Bulgarian using a methodology for second-language learning. The initial testing results are very promising, and soon we will present them to the Ministry of Education and Science (MES).
We identified the language problem among young Roma thanks to an in-depth evaluation of education projects we support. This is just one of many examples of the benefits of systematically measuring the impact of philanthropic investments, which has been an organic part of TSA’s work since the very beginning. From the start, we wanted objective information on whether our work truly changes lives, which investments yield the best results, and whether tested approaches from Europe and the world can be just as effective in Bulgaria.
TSA not only has a dedicated evaluation expert, but also an informal internal group of colleagues who regularly discuss cases related to measuring effect. This helps improve project planning and implementation.
Depending on the scale and nature of a TSA-supported initiative, either quantitative or qualitative indicators may dominate the evaluation, but for a good measurement of impact we always consider both. For example, in our scholarship program for students in pedagogy and medical specialties, we track the number of supported young people and their success rate—graduates who successfully found employment. But even in this relatively straightforward investment, many qualitative factors can influence success or failure, such as family environment or experiences of discrimination at university. We monitor these aspects continuously and take additional support measures where necessary. Ultimately, impact leads to behavioral change—whether due to material support received, newly acquired knowledge and skills, or another reason. A good example is the Nurse-Family Partnership program, which provides home visits by nurses and midwives for young mothers from pregnancy until the child’s second birthday. The behavior change in this case is a result of acquiring new parenting knowledge, attitudes and practices.
We increasingly focus on projects where support recipients actively participate in both planning and in collecting and analyzing results. This approach allows for mutual capacity building—it gives the target group research and evaluation skills, while TSA experts gain deeper insights into the target group and the range of possible solutions to specific problems.
Supported organizations working with end beneficiaries provide the necessary quantitative and qualitative data—about outputs of different activities, their short- and long-term results, and the desired impact. The data is entered into a table pre-agreed with TSA, which allows quick and easy tracking of activities and results at both intermediate and final stages of an initiative. For employment projects, for example, we track the number of trainings held and unemployed participants, job fairs organized, employers and jobseekers involved, and so on.
For qualitative indicators, we measure impact through self-assessment questionnaires, interviews, focus group discussions, observations, and/or studies with specialized tools, such as child development questionnaires, school readiness tests, competency tests in a given area, etc. For large-scale and complex initiatives, or to measure long-term effects, we conduct impact evaluations or randomized controlled trials (RCTs). In such cases we use external agencies or experts in the field, most often from academia in Bulgaria and/or abroad. These agencies and experts handle the entire study—from evaluation design and tool development to fieldwork and data analysis. This always happens in close cooperation with us but ensures an external and objective perspective.
We at TSA also frequently monitor activities on the ground and meet with the people we ultimately support. This informs us about people’s real problems and needs, as well as the specifics and new trends in the communities we work with and for. It also helps us in the process of developing research tools and in analyzing gathered data.
Measuring the effect of our investments allows us to reach evidence-based solutions and propose them for wider application. For example, within the program “Building a better future for all”, we analyzed relevant legislation and, together with three municipalities, proposed a solution for regulating Roma-majority neighborhoods and legalizing buildings there. In 2017, this project received the “Global Human Settlements Award on Outstanding Contribution” from the Global Forum on Human Settlements. Since then, we have trained and consulted more than 40 additional municipalities on how to implement the identified solution.
By measuring effect, we also demonstrate how effective practices should be sought, tested, and adapted through impact evaluation. This approach gives us confidence in knowing which initiatives deserve continued investment—which bring real change and are most effective for communities and people, which should be scaled up, or which should be discontinued.
We also actively use impact measurement results for advocacy purposes.
In 2014–2015, together with the World Bank and in partnership with the Ministry of Education and Science, we covered kindergarten fees for children from vulnerable families in more than 230 settlements, in order to test the effect of free access. The results showed that removing the fees halved the share of children aged 3–5 who were not enrolled in kindergarten, and increased attendance by about 20%. This in turn creates conditions for completing higher levels of education and for more successful participation in the labor market. In the following years we continued tracking long-term effects and promoting the study’s results. Ultimately, in 2022, the parliament abolished fees for municipal kindergartens nationwide.
Measurement often proves useful not only for us, but also for the organizations we support.
Based on the results, our partners adjust aspects of their models and improve effectiveness. Examples include the “Role Models of Tomorrow” program of the Arete Youth Foundation and the “Every Student Will be a Winner” program of the Center for Interethnic Dialogue and Tolerance “Amalipe.”
Finally, our work over the years shows that measuring impact is far from a common practice among NGOs. That’s why we invest in spreading this approach. We believe that by promoting evidence-based solutions, we contribute to improving the capacity and sustainability of Bulgaria’s civil sector.
Petya Zeynelova, Evaluation Officer at the Trust for Social Achievement, prepared this article for the Donor Bulletin of the Bulgarian Donors Forum.