Data collection and evaluation of the impact companies from sustainable agriculture, adventure tourism and ICT got from the Pay-for-Performance instrument of Increasing Market Employability program (IME), managed by Palladium and financed by SDC – the Swiss Agency for Development and Cooperation
Development programs are good at counting what they spend. Counting what actually changed in a company because of that spending is harder, and it is the question Palladium put to us in the summer of 2021. Palladium implements the Increasing Market Employability (IME) programme of the Swiss Agency for Development and Cooperation in North Macedonia, and wanted an independent view of whether its Pay-for-Performance (P4P) instrument was delivering results for the companies it supports.
The instrument under review
IME works in three sectors: sustainable agriculture, adventure tourism and ICT. Companies in all three run into the same walls: difficult access to finance, weak market penetration and buyers who increasingly demand certified quality. Rather than paying companies, P4P co-finances the business service providers who help them: financial consultants guide firms to bank loans and grants (access to finance, or A2F) and certification consultants and bodies help them obtain the standards their markets require (access to certification, or A2C). Companies pay part of the cost, which keeps the service market-based. By mid-2021, 42 companies had been approved.
What Brulz was asked to do
The brief was to measure and evaluate the results of P4P for January to June 2021. That meant quantitative indicators, from new jobs expressed as full-time equivalents and disaggregated by gender and age, to sales, personal income, mobilised financing, new standards and the share of growth that could reasonably be attributed to the programme. It also meant qualitative work on new markets, a gender and youth analysis, an assessment of the consultant ecosystem and recommendations. All of it in six weeks.
How we approached it
We started with the desk review, reviewing each company’s P4P application and its official financial statements from the Central Register let us set 2019 as a clean baseline, so that later claims could be checked against something. Our two consultants then held in-depth interviews with supported companies across the country. In addition to user companies, we held in-depth interviews with the financial consultants, certification consultants and certification bodies that supported companies throught the programme.
Two choices shaped the analysis. First, we mapped every company onto a simplified customer journey, from approval through implementation and completion to a visible impact stage, because a company that had only just obtained its loan could not yet show results and should not be read as a failure.
Second, we asked companies what share of their new jobs and sales they attributed to the support, and where a claim looked generous we applied a more conservative figure. When one manager credited IME with ten percent of 2021 sales, we counted six.
What we found
For the two year analyzed period, the supported companies reported:
- New jobs attributable to P4P, counting full-time positions and seasonal work converted to full-time equivalents. Forty-eight percent went to women and 31 percent to people under 29.
- Those jobs carried roughly 40 million denars in additional personal income. Attributable sales growth reached 87 million denars. Since the instrument began, companies had mobilised 3.34 million francs in external financing, 83 percent of it from commercial banks, with a further 5.6 million francs already approved and in the pipeline.
- Twenty-one companies had introduced new standards or practices, 15 had launched new products or innovations and 23 had entered new markets.
These numbers are not for the entire project, just the analyzed period focusing on companies supported from three sectors, in a period less then two years of the second phase of the IME project.
Potential ecosystem improvements
Companies rated their consultants eight to ten out of ten and most said they would not have accessed the funding without them. But financial consultants who simply arrange loans had little influence on how a business operates, while business consultants changed business models. Several consultants charged nothing beyond the IME contribution, which risks distorting the local advisory market. And ICT companies, who need risk capital rather than collateralised loans, were largely left out of access to finance.
Our recommendations followed from this: strengthen the value proposition of A2F consultancy so that consultants become lasting intermediaries between companies and financial institutions, move certification support from certification bodies towards consultants who embed standards in daily practice, build on the National Bank’s push for transparent financial products, and open the door to higher-risk finance for ICT.
Why good evaluation and data matter? It is not about producing a flattering number. It is about giving a program team an accurate picture of what worked, for whom, and what to change next. That is the work Brulz does: mixed-method research, careful attribution and recommendations a client can act on the following quarter and year.
Interested in impact evaluation, assessments, program or projecdt development? Get in touch with the BRULZ team.