Introduction
In the 1980's the New Public Management theory has been introduced. Since then the use of performance measurement was promoted to drive a more effective, efficient and accountable public sector. The New Public Management theory had its origins in New Institutional Economics, and specifically public choice theory and principal-agent theory. The latter advances that the more the agents' objectives differ from the principals, the more their utility function differs which reduces effectiveness and efficiency [1]. Performance Based Financing (PBF) is an approach to solve this principal-agent problem because it is based on a contract designed to reconcile the objectives of both principal and agent. PBF relies on the belief that agents can be motivated by financial incentives linked to their performances and that they will seek ways to maximise income [2].
PBF has become increasingly popular over the last years, especially at the operational level of health care systems (health centres, hospitals). Improved service delivery has been reported and the importance the approach gained over the last years can no longer be ignored [3,4]. However it also revealed controversies and provokes debate. Some authors argue that sound human resource management and non-financial incentives remain crucial to optimally motivate employees [5]. Others criticise the lack of scientific evidence due to the absence of rigorous methodology, the ignorance of the context, the high transaction costs and the low sustainability [6,7].
An important underlying reason for the popularity and success of PBF at operational level is the measurability of the performances. If services and outputs at this level can easily be identified, measured, monitored and paid for, this is far more difficult at central administrative levels of the public sector, where performance assessment is inherently difficult and outputs cannot be measured accurately. In 2009, Perry et al. reported on the efficacy of Performance-Related Pay experiences for government in developing countries between 1977 and 2008, based on results of 57 studies. Many of the schemes applied to senior managers. At the aggregated level, it was found that performance-related pay in the public sector consistently failed to deliver on its promise. However the disaggregated analysis offered a different picture. Studies within the medical context were largely positive and more effect was noted at lower organisational levels, where job responsibilities are less ambiguous [8].
PBF was introduced in Burundi in 2006, at a moment that this landlocked central African country slowly recovered from years of civil war. Pilot schemes were introduced in three of the seventeen provinces and extended to 9 provinces in 2009. The introduction of PBF was associated with improvements in care and quality, especially in the maternal and child care, and for types of care which require a behavioural change of health care workers when the patient is already in the health centre or hospital [9,10]. In 2010, it was generalised for the entire country covering all 18 provinces, but still excluding the central level.
In 2010 an organisational and institutional audit of the Ministry of Health led to a set of reforms, including the introduction of a new organisational structure of the Ministry of Health in 2011, and the introduction of strategic financing for better performances at central level in 2013.
The objective of the present study is to document the effect of Performance Based Financing at Central level (PBF-C) of the ministry, based on a semi-quantitative method following the evolution of performance over time. The underlying hypothesis is that performance of central level units of a ministry can be monitored and improved through strategic financing.
Method
Identifying Performance Units: From the start all civil servants in the ministry at central level were implicated in the PBF-C scheme. The ministry was subdivided in 'Performance Units' (PU) defined as a team with a common duty and well defined common results, under a clearly identified hierarchical authority. Performance incentives were based on the collective performance of the unit, not the individual performances and thus all individual flat-rated top-ups of salaries were abandoned and replaced by this new performance related scheme. Initially a group of eligibility criteria was developed and used, which allowed to include or exclude the PU in the PBF-C. But these eligibility criteria were abandoned when all PU met the minimum requirements.
Measuring the Collective Performance: The collective performance measurement was based on quarterly evaluations of 4 domains of performance: (1) the working environment (150 points), (2) compliance with the implementation and followup of the action plan (400 points), and (3) the guidance and support to the provincial level (200 points), and (4) specific assignments of each unit (750-1500 points) were taken into account. The first three domains were transversal and identical to all PU's. The full evaluation grid for the four domains can be consulted on www.fbpsanteburundi.bi. An example for a domain is given in Tables 1a showing how domain (3) the "guidance and support to provincial level" was evaluated and rated.
Financing Performance: One point of the performance grid was then paid for, at a fixed rate in local currency, 7500 Burundian francs per point. Over time the financing scheme was adapted in order to get the incentive where it was most needed: the ponderation between the four domains changed when more weight was given to the specific tasks of the PU and less to the working environment. The rate of 7500 Burundian francs was equally adjusted in order to keep pace with budget availabilities and exchange rates. In Q4 - 2014, one point was paid 4970 Burundian francs.
Up to 65% of a PU incentive could be used to pay performance allowances to its staff members based on a grid that took different criteria into account such as presence on the work floor, function, staff qualification, and the monthly evaluation of the PU. The remaining part had to be used to support operational costs of the PU.
Data Collection and Analysis: we used a semi-quantitative method following the evolution of performance over time. Data were collected and stored in an Excel-spreadsheet in a prospective way starting the fourth quarter of 2013 (Q4-2013), following the introduction of PBF-C, and ending the fourth quarter of 2014 (Q4-2014), thus covering 15 months. The quarterly evaluations of PUs were conducted by two peer reviewers from other PUs and
by two observers quarterly appointed by the National Technical Task Force for PBF (NTU). One observer was a member of NTU, the other was a member of a donor supporting the ministry.
Analysis was done on the evolution of the quarterly score for each PU for the 4 performance domains as well as for the overall performance of the different PU and the entire ministry. The median value was used to estimate the overall performance of all the PU of the ministry. The distribution of the performance between PU was depicted in box-and-whisker plots indicating also the outliers.
Qualitative survey was planned but due to unstable political situation cancelled.
Results
We observed an improvement in the overall performance between the Q4 2013 and the Q3 2014, with scores raising from 70 to 81%. Nevertheless the increase was not continuous as in the last quarter of 2014, the performance fell compared to the third quarter. As shown by the box plots in (Figure 1), the dispersion of the performance score was also reduced as there were no more outliers in Q4 2014. Only one PU achieved to get 100% in the fourth quarter. The performance of many PU is not a constant progress, but seesaws over the different quarterly evaluations.
Domain 1: Working environment
Table 2 illustrates the PBF-C scores (percentages) for all 16 performance units (PU) in the study period. 15 out of 16 PU improved the working environment or remained stable at the score of full 100 % performance. On an average basis score improved with an average of 20%. Only one PU deteriorated during the study period. The maximum score was initially set at 150, but after one year reduced to 100 points. Figure 2 shows the quarterly average evolution in the working environment performances: Q3 2014 shows a dip in the scor e to 77%.
Domain 2: Annual plan execution rates During the study period, the performance of 2 PU deteriorated, 14 improved the scores in the annual plan execution domain. On an average basis, the MoH significantly improved by 15 % the performances in this domain as detailed in Table 3, whilst the maximum score of the second domain rose from 400 points to 450 points. Figure 3 shows the constant progress quarter after quarter in the annual plan execution performances, climbing from 73 to 88%. In the guidance of the provincial levels, important improvement was observed in the study period. Average score significantly rose with 34% from 52 to 86%. Only 2 PU dropped to a worse performance as detailed in Table 4. The maximum possible score remained stable at 200 points. (Figure 4) depicts the evolution in the guidance and support domain, rising from 50 to 86%.
Domain 4: Specific criteria per performance unit (PU) As far as the specific criteria per unit are concerned, (Table 5) documents important deterioration in 9 out of the 16 performance units. Only 7 kept the scores at equal level or improved. The average score dropped only from 75 to 74 %, statistically not significant.
Figure 5 shows a reduction between Q3 2014 and Q4 2014 from 97 to 74%. This will be further addressed in the discussion section.
Budget spent on PBF-C
A total of 526.600.375 Burundian Francs was paid during the study period. This corresponds to 269.585 euro for a period of 15 months, or less than 18 000 euro per month. There were no other "operational costs" for the evaluations, as we used peer reviewing, controlled by two observers: one observer representing a donor, one observer of the NTU.
The positive experience of PBF at operational level since 2006 explains that PBF-C was proposed in an action plan following an independent external institutional and organisation audit in 2010. PBF-C was one of the strategies proposed to tackle some of the weaknesses identified in this audit. Despite approval of the action plan by the Ministry of Finance and the Ministry of Public Services, PBF-C only started up effectively in 2013. This was partially due to lack of budget, as on the donor side, support of PBF-C was only supported by the Belgian Development Cooperation. The World Bank agreed for co-financing but never actually engaged.
The introduction of PBF-C in Burundi was a fine example of an incremental process of Lindblom's 'muddling-through' in an environment of structural constraints and difficulties, the public services of Burundi are characterized by low wages, large inefficiencies and wide-spread corruption practices [11].
In addition the Ministry of Health faced permanent structural challenges hampering the performances: the coordination and cooperation between units of MoH was difficult because the units were geographically dispersed in Bujumbura; there was a plethora of staff in the ministry, particularly at level of the lower staff, contrasting with a deficit of qualified staff at higher, more strategic levels.
From the start, part of the incentives was meant to support the operational costs of the unit, and other part was meant for motivation of the staff. By focusing on PU's rather than individuals, the scheme was expected to support a team spirit and strengthen the relevance of the PU as an operational entity.
It was the intention to involve all staff at national level and create a system eliminating inequalities by streamlining the multitude of premiums and their incentives into one generalised and performance related mechanism. As a consequence, a critical step in the introduction of the PBF-C was the abolition of all other premiums paid. This took time to be accepted and explained why some staff working in autonomous central institutions, like the National Blood Transfusion Centre, or in national vertical, programmes, like the national malaria or HIV programme, did not accept the PBF-C scheme.
The introduction of indicators for performance measurement of central administration was an important innovation. The performance grid proved to be a valuable tool for measuring and improving the functioning of the various units of the MoH. Initially our research intended to use mixed methods, using quantitative methods to follow the evolution of performance scores over time, and a qualitative approach to identify and explore the factors contributing to the performance of PU in the ministry. The qualitative surveys did not take place due to unstable political situation at the time we wanted to conduct them. This is hampering the quality of our research, because the quantitative part had its methodological limits [6]. There were many contributing and thus confounding factors that influenced the changes and it was impossible to isolate the effect of PBF-C in a complex environment as the MoH.
Despite the methodological weaknesses, our results suggested that under the PBF-C there was an overall improvement of the measured performances of the central level as a whole. The improvements were only observed in the crosscutting domains, i.e. the working environment, the implementation and follow-up of the action plan, and the guidance and support. It seems likely that due to crossfertilizing the cross-cutting or transversal domains were more vulnerable for improvement. Surprisingly to the authors, the follow-up of specific targets of the units did not show improvements during the study period. This observation cannot be explained and in-depth further analysis is needed through more qualitative approaches in order to understand this phenomenon of status quo on the specific performances for every unit [6].
But if there was improvement, was it the effect of the financial incentives only or was there something more? Were there other contributing factors? Firstly, we cannot deny that the PBF-C, by using quarterly peer evaluations to assess the performance, developed a better understanding of performance, inducing an organisational culture focused on results. Secondly our approach focused on ownership and involvement. As Magrath & Nichter diagnosed: 'involving workers in the selection of indicators and targets might help reduce 'gaming' and 'crowding out''.Working sessions were organised each year to agree on the PU's evaluation indicators and criteria [12]. This assumption is also confirmed by Nuti et al. who reported that one of the successful factors in a performance evaluation system is the level of involvement into the performance measurement of the stakeholders of the system [13].
As a consequence of PBF-C the budget of all PUs increased including funds for recurrent functioning, there was a better knowledge by staff of their tasks and responsibilities, working environment and working climate improved. PBF-C also contributed to tackle the plethora of personnel at MoH, because incentives were divided based on the staff positions as described in the Internal Organisational Regulations. So, if these regulations provided for one staff position, while three people were occupying that position, only one premium was paid which had to be split in three. To prevent the possible adverse effect of attracting staff from the peripheral level to the central level, the premiums were always lower, about 80% to 85%, than those at provincial or district level.
Can we learn something out of this experience? Yes and no. At first sight the results of PBF-C were encouraging, despite some challenges and questions. Unfortunately the study period was limited and the positive short-term effects will possibly change or disappear, as suggested by Sabatier: 'a decade or more' isthe minimum duration of most policy cycles, from emergence of the problem through sufficient experience with implementation to render a 'reasonably fair evaluation' of impact [14]. The improvement of performance was not unequivocal and several PU improved their performance for one quarter and then fall down again, which shows that the performance improvement is still unstable. Attributing results to PBF is more complex than comparing outputs between the baseline study and a follow-up study [15]. The quarterly evaluations were cumbersome and there was definitely a need for an improved evaluation process. There was some doubt on possible perverse effects leading to quantity changes with low quality (e.g. more frequent supervisions atdeconcentrated level does not provide information about quality); the focus on measurability might have induced quantitative changes predominantly [2].
We did not have the opportunity to look at the impact of PBF-C at provincial and district level: did the higher performance rates on guidance and support indeed impact on the functioning of provincial services? Moreover, the study has been conducted by insiders. Walt et al. clearly mentioned that researcher 'positionality' and its possible impact on the research process is important. She distinguishes 'insiders' and 'outsiders'. 'Policy research teams that combine both insiders and outsiders and that engage all team members in active discussions of findings during data collection and analysis may therefore yield the richest and most comprehensive understanding of the policy process' [16].
On the one hand, insiders have been accused of being inherently biased, but the outsiders run 'a real risk of superficial and decontextualized analyses of the policy process that reveal only part of the picture' [17]. Therefore it was the authors'intention to contract external researches for the more qualitative part of the study. Unfortunately the premature ending of the experience made it impossible to develop that part of the study.
Finally we recognise the dangers of choosing performance indicators. PBF-C was criticised because the indicators were not linked to the National Strategic Health Plan and were mainly process indicators, not measuring outcome or impact. This is not new: in 2009, Eldridge & Palmer warned for the domination of more easily quantifiable measures of performance over measures of performance quality. In addition, they argued that performance targets may also inappropriately skew the provider's focus [2].
We conclude that the Ministry of Health of Burundi developed a unique, coherent and sound performance monitoring and financing system for their central administration. This system was applied during more than a year resulting in improved global performance of the MoH. The experience was unfortunately interrupted due to a political crisis linked to the election process of 2015 that induced budget constraints for the Government and withdrawal of donor funding. In July 2017 however, Worldbank will be funding a new era of PBF-C in Burundi. Our paper wants to prepare the path for this new experience and highlight already the challenges and risks.
Declarations
List of Abbreviations
Ethical approval
Ethical clearance for the research was obtained for this research at level of the Burundi Ministry of Health. There is no human material or human data involved however neither was the research carried out on humans.
Conflicts of interest
We confirm for all the co-authors that there is no conflict of interest.
There was no specific funding for this research.
Stefaan Van Bastelaere designed the study and was actively involved in the analysis and interpretation of data; he drafted the manuscript. Luc Geysels designed the study and was actively involved in acquisition analysis and interpretation of data; he drafted the manuscript. Olivier Basenya was actively involved in the analysis and interpretation of data, revised the manuscript critically for important intellectual content. Paul Verle made substantial contributions to conception and design and the interpretation of data; he critically reviewed the manuscript at all times. Liboire Ngirigi contributed to concept and design of the study; he repeatedly revised the manuscript.
All authors have given final approval of the version to be published and agree to be accountable for all aspects of the work.
Availability of data and materials
All used data are stored on https://intranet.btcctb.org/claroline/ and are publicly available and downloadable.
The authors would like to thank Dr. Anne Kongs for her very valuable advices on statistic methods and Dr. Etienne Mugisho for reviewing several times the article and giving comments and advices.
Open Access
by Scient Online is licensed.
