- Open Access
Technical and scale efficiency of public community hospitals in Eritrea: an exploratory study
© Kirigia and Asbu; licensee Springer. 2013
Received: 23 May 2012
Accepted: 8 March 2013
Published: 16 March 2013
Eritrean gross national income of Int$610 per capita is lower than the average for Africa (Int$1620) and considerably lower than the global average (Int$6977). It is therefore imperative that the country’s resources, including those specifically allocated to the health sector, are put to optimal use. The objectives of this study were (a) to estimate the relative technical and scale efficiency of public secondary level community hospitals in Eritrea, based on data generated in 2007, (b) to estimate the magnitudes of output increases and/or input reductions that would have been required to make relatively inefficient hospitals more efficient, and (c) to estimate using Tobit regression analysis the impact of institutional and contextual/environmental variables on hospital inefficiencies.
A two-stage Data Envelopment Analysis (DEA) method is used to estimate efficiency of hospitals and to explain the inefficiencies. In the first stage, the efficient frontier and the hospital-level efficiency scores are first estimated using DEA. In the second stage, the estimated DEA efficiency scores are regressed on some institutional and contextual/environmental variables using a Tobit model. In 2007 there were a total of 20 secondary public community hospitals in Eritrea, nineteen of which generated data that could be included in the study. The input and output data were obtained from the Ministry of Health (MOH) annual health service activity report of 2007. Since our study employs data that are five years old, the results are not meant to uncritically inform current decision-making processes, but rather to illustrate the potential value of such efficiency analyses.
The key findings were as follows: (i) the average constant returns to scale technical efficiency score was 90.3%; (ii) the average variable returns to scale technical efficiency score was 96.9%; and (iii) the average scale efficiency score was 93.3%. In 2007, the inefficient hospitals could have become more efficient by either increasing their outputs by 20,611 outpatient visits and 1,806 hospital discharges, or by transferring the excess 2.478 doctors (2.85%), 9.914 nurses and midwives (0.98%), 9.774 laboratory technicians (9.68%), and 195 beds (10.42%) to primary care facilities such as health centres, health stations, and maternal and child health clinics. In the Tobit regression analysis, the coefficient for OPDIPD (outpatient visits as a proportion of inpatient days) had a negative sign, and was statistically significant; and the coefficient for ALOS (average length of stay) had a positive sign, and was statistically significant at 5% level of significance.
The findings from the first-stage analysis imply that 68% hospitals were variable returns to scale technically efficient; and only 42% hospitals achieved scale efficiency. On average, inefficient hospitals could have increased their outpatient visits by 5.05% and hospital discharges by 3.42% using the same resources. Our second-stage analysis shows that the ratio of outpatient visits to inpatient days and average length of inpatient stay are significantly correlated with hospital inefficiencies. This study shows that routinely collected hospital data in Eritrea can be used to identify relatively inefficient hospitals as well as the sources of their inefficiencies.
Eritrea is situated in the Horn of Africa. It is bordered by Sudan to the North and West, the Red Sea to the East, Ethiopia to the South and Djibouti to the Southeast . The estimated population in 2009 was 5.073 million. The annual population growth rate of 3.6% was higher than the average for Africa (2.5%) and for the world (1.2%) . Approximately 21% of the population lives in urban areas, compared to 38% and 50% for Africa and the world, respectively. Eritrea’s gross national income per capita stood at Int$610 in 2009 , which was lower than the Int$1620 reported for Africa, and considerably lower than the Int$6977 global average.
Eritrean health indicators are much better than those reported for the African Region as a whole. For example, life expectancy at birth of 66 years in 2009 was 12 years higher than the regional average and just two years lower than the global average. As shown in Additional file 1: Appendix, the infant mortality rate, under-five mortality rate, adult mortality rate and maternal mortality ratio for Eritrea were far better than the averages for Africa and the world . The country’s HIV/AIDS, malaria, and tuberculosis cause-specific mortality rates are far much lower than the regional averages. For example, HIV/AIDS cause-specific mortality rate for Eritrea was five times lower than the regional average.
What kind of resources did Eritrea use to achieve these outcomes? In 2007 there were a total of 378 health facilities, including 26 hospitals (20 secondary hospitals and 6 national referral hospitals), 56 health centres, 2 health stations, 7 maternal and child health clinics (MCHC), and 107 clinics . There were a total of 3909 beds, of which 63.3% were in hospitals, 29.1% in health centres and 1.6% in MCHC - 11.3 beds for every 10,000 people.
Health workforce density in Eritrea compared to African Region and global averages in 2008
Health workforce and infrastructure
WHO African Region
Physicians density (per 10 000 population)
Nursing and midwifery personnel density (per 10 000 population)
Dentistry personnel density (per 10 000 population)
Pharmaceutical personnel density (per 10 000 population)
Environment and public health workers density (per 10 000 population)
It is widely accepted that improved efficiency is one of the four overarching goals of health systems [5, 6]. There is a growing realization among health policy makers in the African Region of the need to utilize scarce health sector resources more efficiently , a good indication of which being the statement made by the 46 WHO Member States of the African Region in 2006 stressing their commitment to increasing the efficiency of health interventions and improving the allocation and management of health sector resources .
Since the adoption of the regional health financing strategy in 2006, a growing number of countries have undertaken health facility efficiency studies to guide the development of interventions to reduce waste of scarce health system resources. Since 2000 health facility efficiency studies have been undertaken in thirteen African countries, including Angola , Benin , Botswana , Burkina Faso , Ethiopia , Ghana [14, 15], Kenya [16, 17], Namibia , Nigeria , Seychelles , Sierra Leone [21, 22], South Africa [23–25], and Zambia [26, 27]. These studies demonstrate that DEA is an important tool for policy advice. To date, no health facility efficiency study has been conducted in Eritrea.
The study draws on Eritrean hospital data for 2007 to explore hospital efficiency at that time, and to demonstrate how an efficiency study could have informed decision-making. Three research questions are addressed: Were the public community hospitals in Eritrea relatively technically efficient in 2007? What were the magnitudes of output increases and/or input reductions needed for inefficient hospitals to operate relatively efficiently? How are efficiency scores for hospitals correlated with institutional and contextual/environmental variables?
The specific objectives of the study were: (a) to estimate the relative technical and scale efficiency of public secondary level community hospitals in Eritrea in 2007; (b) to estimate the magnitudes of output increases and/or input reductions that would have been required to make relatively inefficient hospitals more efficient; and (c) to estimate using Tobit regression analysis the impact of institutional and contextual/environmental variables on hospital inefficiencies.
Skaggs and Carlson  define economic efficiency as obtaining the maximum benefit from a given cost or minimizing the cost of a given benefit. In other words economic efficiency means obtaining the maximum net gain (difference between the benefit received and the cost incurred) from an action. The authors further explain that economic efficiency comprises both technical efficiency (producing without waste) and allocative efficiency (allocating resources to their most high value uses).
In the context of health, allocative efficiency describes the use by a health facility or decision making unit (DMU) of health system inputs in the proportion that minimizes the cost of production, given input prices [29–31]. Estimation of allocative efficiency requires data on quantities of health service outputs, health system inputs, and input prices.
On the other hand, technical efficiency describes the production by a health DMU of the optimal/maximum quantity of outputs from the available health system inputs [29–31]. Alternatively, technical efficiency can be said to be achieved where a health decision making unit produces a given level of health service outputs with the least health system inputs, e.g. health workforce, pharmaceutical and non-pharmaceutical supplies, capital inputs (buildings, beds, equipment, vehicles), and community resources.
The technical efficiency of a health DMU can be broken down into pure technical efficiency and scale efficiency. Pure technical efficiency denotes health decision making unit technical efficiency that cannot be attributed to deviations from optimal scale (scale efficiency). Whereas scale efficiency is a measure of the extent to which a health decision making unit deviates from optimal scale (defined as the region in which there are constant returns to scale in the relationship between outputs and inputs) [30, 31].
Salvatore  defines returns to scale as the extent to which health system output changes as a result of a change in the quantity of all health system inputs used in production. Where the quantity of all hospital inputs is increased by a given proportion, a constant return to scale is achieved when health service outputs increase in the same proportion. Thus, an increasing return to scale is achieved if output increases by a greater proportion than the increase in inputs and a decreasing return to scale is achieved where output increases by a smaller proportion than the increase in inputs.
It is apparent that efficiency is never absolute; instead it is always assessed relative to some criterion. In their definition of relative efficiency, Cooper et al.  explain that “..a DMU is to be rated as fully (100%) efficient on the basis of available evidence if and only if the performances of other DMUs does not show that some of its inputs or outputs can be improved without worsening some of its other inputs or outputs (p.3).”
The output-orientated technical efficiency (TE) of hospital G is the ratio, TE 0 = 0G/0H, which is the amount by which its health service outputs could be increased without recourse to extra inputs. According to Coelli , if health system input and output prices were available an isorevenue line, KK’, could be drawn and allocative efficiency (AE) defined as AE 0 = OH/OJ, which indicates the reduction in production cost that would occur if production were to occur at the allocatively and technically efficient point H’. Even though H and H’ are technically 100% efficient, only H’ is producing its health service outputs at the lowest cost.
Overall economic efficiency (EE) of hospital G can be defined as follows: EE 0 = (0G/0J) = (0G/0H) × (OH/OJ) = TE 0 × AE 0. AE, EE and TE scores are bounded by zero (totally inefficient) and one (totally relatively efficient). Since we did not have the data on input prices in the Eritrean study, we estimated only the TE and SE scores of hospitals. The estimation of TE and SE scores requires only output and input quantities.
In this paper a two-stage Data Envelopment Analysis (DEA) method was used to estimate efficiency of hospitals and to explain the inefficiencies.
DEA analytical framework
Over the last half century efficiency has been calculated relative a frontier function using either non-parametric mathematical programming methods such as the Data Envelopment Analysis (DEA) or econometric/regression methods.
In the first stage, we use DEA to estimate the efficient frontier and the hospital-level efficiency scores. The main advantage of DEA is that it is able to deal with DMUs that employ multiple inputs to produce multiple outputs or services, which is typical of health systems units, e.g. hospitals and health centres. In addition, DEA not only identifies inefficient decision-making units but also permits analysis of sources of inefficiency and quantification of magnitudes of inefficiencies in the use of hospital inputs and production of outputs. It is for these reasons that, we considered DEA appropriate for the purposes of this study.
TE 0 = the efficiency score of hospital 0;
x ij = the amount of health system input i utilized by the j th hospital;
y rj = the amount of health system output r produced by the j th hospital;
u i = weight given to health system input i;
v r = weight given to output r
The CCR model assumes constant returns to scale, meaning that all observed production combinations can be scaled up or down proportionally, i.e. not allowing for economies or diseconomies of scale. In other words, the model assumes that DMUs are able to linearly scale the inputs and outputs without increasing or decreasing efficiency . CCR precludes the existence of variable returns to scale, where variable returns to scale denotes the changes in hospital outputs as hospital inputs change by the same proportion.
Suppose a hospital increases all its inputs by the same proportion. There are three possible scenarios: (i) its output(s) increases in line with the increase in inputs, which implies that there are constant returns to scale; (ii) its output(s) increases more than the increase in inputs, implying increasing returns to scale; or (iii) its output(s) increases less than the increase in inputs, which implies decreasing returns to scale .
In reality, a hospital can manifest constant returns to scale, increasing returns to scale or decreasing returns to scale depending on whether it is experiencing economies of scale or diseconomies of scale. Constant returns to scale occur in a situation where economies of scale are exhausted, and where health system inputs (factors of production) are perfectly divisible. The presence of increasing returns to scale may indicate indivisibilities in certain hospital inputs (e.g. diagnostic equipment, operating theatre), and greater scope for health workforce specialization as the scale/size of production increases. On the other hand, decreasing returns to scale can result when the large scale of production leads to cumbersome lines of communication between top hospital management and the health workforce in departments and wards, which lead to a decrease in managerial efficiency. Decreasing returns to scale could also occur due to over-utilization of abilities and skills of an entrepreneur .
Where: ε is an infinitesimal non-Archimedean quantity greater than zero. A value of u 0 > 0 implies increasing returns to scale; u 0 < 0 means decreasing returns to scale; and u 0 = 0 denotes constant returns to scale. Thus, the above BCC model permits both the separation of technical and scale efficiencies, and determination of whether individual hospital’s operations were in regions of increasing, constant or decreasing returns to scale.
The linear programming model (4) shown above was run 19 times (equal to sample size) in identifying the relative technical efficiency scores of all the hospitals in the sample. DEA by default assigns weights to each hospital’s inputs and outputs in a way that maximizes its technical efficiency score. A hospital is considered to be technically efficient if it scores one, implying 100% relative technical efficiency, whereas a score of less than one implies that it is relatively technically inefficient, compared to peers in its efficiency reference set.
The scale efficiency score for each hospitals was obtained by dividing the constant returns to scale technical efficiency score by the variable returns to scale technical efficiency score [31, 32]. A scale efficiency score of one implies that the hospital in question is operating at optimal scale or size. If the scale efficiency score is less than one, then the hospital is either too small or too big relative to its optimal size.
According to Coelli , where DMUs are given a fixed quantity of resources (inputs) and asked to produce as much output as possible, an output orientation is more appropriate. In the Eritrean context, the staffing capacity of each public hospital is determined centrally by the Ministry of Health, and thus individual hospital managers do not have any control over the size of the health workforce, and therefore of their inputs. In addition, hospital managers have no control over the size of the hospitals they run. It was for this reason that we used output-orientated DEA.
In 2007 there were a total of 20 secondary level public community hospitals in operation, the majority of which maintained good medical records. During first contact with a hospital, a patient card is opened and kept in the medical records office, and then updated during subsequent visits. The hospital medical records clerk collates daily summaries of the numbers of preventive visits, outpatient curative visits, admissions and discharges by ward. Every month, hospitals send a summary of data on selected output and input indicators to the management information system housed at the Ministry of Health headquarters. Each year the health management information system office analyses the data from all hospitals and publishes an annual health service activity report.
All hospitals, with the exception of one that did not report data on outputs, were included in the study. Input and output data were obtained from the Eritrea Ministry of Health annual health service activity report of 2007 . The Eritrean hospitals were assumed to employ four inputs to produce two health service outputs, as outlined below.
Ideally, the statistical variable used to measure labour input is the hours actually worked by category of health worker, rather than the numbers of persons employed. Relying on numbers of health workers obscures changes in average hours worked due to, for example, absence from work. Because this study is based on the analysis of secondary data, routinely collated by management information, we did not have information on quantities of hours actually worked. We therefore used headcounts of health workers as proxies of labour input.
In a hospital context, physical capital goods include building space, beds, medical (diagnostic and therapeutic) equipment, and vehicles (including ambulances). Those capital goods are repositories or warehouses of physical capital services (measured as total machine hours) that are the actual input in the hospital production process. Since information on capital services, measured in machine hours, is not routinely collected and archived by management information, we used the number of hospital beds as a proxy.
Input 1: Number of physicians (doctors)
Input 2: Number of nurses and midwives
Input 3: Number of laboratory technicians
Input 4: Number of operational beds and cots
Output 1: Number of outpatient department visits
Output 2: Number of inpatient department discharges
Therefore, the choice of the above-mentioned inputs and outputs was guided by three considerations, namely: past studies undertaken of hospitals in Africa, which also employed similar inputs and outputs [9–27]; the availability of relevant data in the ministry of health annual health service activity report for 2007 ; and the availability of data that is routinely compiled by hospitals. Regarding the latter, we wanted to demonstrate ways in which the Eritrean ministry of health can get added informational value from such data without investing extra resources. The inputs and outputs data were used as reported in the ministry of health annual health service report without any processing or manipulation.
The inputs and outputs data were entered into computer using Excel software. The technical and scale efficiency scores for the hospitals were computed using DEAP 2.1 programme developed by Professor Tim Coelli .
Explaining inefficiency through Tobit regression analysis
In the second stage, the DEA efficiency scores computed in the previous section were regressed against some institutional factors which are at the discretion of the hospital management and selected contextual/environmental (non-discretionary) factors that are beyond their control to estimate their impacts on efficiency. The literature indicates that some of the factors that impact health facility efficiency include, for example, catchment population, distance, location (urban/rural), ownership (profit/not-for-profit), teaching status, payment source (out-of-pocket/health insurance), occupancy rate, average length of stay, outpatient visits as a proportion of inpatient days, and quality [25, 38, 39].
A variety of regression techniques have been applied in the second stage to estimate the impact of contextual factors on efficiency, including the ordinary least squares (OLS) and the maximum likelihood (ML) based probit, logit, and truncated regression (Tobit). A debate has been raging between two schools of thought over the statistical properties of the two-stage DEA estimator. In one school of thought, scholars such as Simar and Wilson  argue that because DEA output scores are biased and contextual/environmental variables are correlated to output and input variables, the conventional statistical inferences are invalid in the second-stage regression, and recommend use of bootstrap methods. In another school of thought, scholars such as Ramalho et al. , McDonald  and Ruggiero  have argued that econometric models such as probit, logit, and truncated regression (Tobit) can be used for second-stage estimation of the impact of contextual/environmental variables on efficiency.
Afonso and Aubyn  argue that “Even if Tobit results are possibly biased, it is not clear that bootstrap estimates are necessarily more reliable, based on a set of assumptions concerning the data generation process and the perturbation term distribution that may be distributed (p.1429).” In their empirical study, the censored normal Tobit results and bootstrap algorithms yielded very similar results. We estimated the Tobit model (or censored normal regression model) because DEA efficiency estimates are bounded between 0 and 1.
In the Tobit model, for computational convenience it is preferable to assume a censoring point at zero .
Where: N is the number of observations; y i is the observed inefficiency score, i.e. dependent variable; y* is the latent dependent variable; βi is the kx1 vector of unknown parameters; x 1 is the kx1 vector of explanatory/independent variables; and ε i is an independently distributed error term assumed to be normal with zero mean and constant variance σ2.
Where: INEFF is the inefficiency score; OPDIPD is the outpatient visits as a proportion of inpatient days; POP is the region population dichotomous dummy variable = 1 if region or zoba population is one million and above (Maakel, Gash-Barkam and Debub), 0 if less than one million (Anseba, Semienawi Keyih Bahri, and Debubawi Keyih Bahri); REGION is the region dichotomous variable = 1 if the hospital is situated in a highland region (Anseba, Debub, Maekel), 0 if the hospital is situated in a lowland region (Gashbarka, Semienawi Keyih Bahri, and Debubawi Keyih Bahri); α is the intercept term; β 1 is the vector of unknown parameters or coefficients; and ε 1 is the stochastic/random error term.
Based on past two-stage hospital efficiency studies , we would expect a positive relationship between hospital inefficiency (Ineff) and ALOS, POP and REGION. Thus, regression coefficients β 2 , β 3 and β 4 are expected to assume a positive sign. We would expect a negative relationship between the Ineff and OPDIPD, and thus, β 1 should a priori assume a negative sign. Tobit coefficients indicate how a one unit change in an independent variable x i alters the latent dependent variable y*.
By estimating equation 6, we wish to test two hypotheses. First, in order to test the overall significance of the equation, we state the joint null hypothesis as H 0 : β 1 =β 2 = β 3 = β 4 = 0 and the alternative hypothesis H A : β 1 =β 2 = β 3 = β 4 ≠ 0 . The joint null hypothesis is tested using the likelihood ratio test (LL).
Second, β n is not significantly different from zero in either direction. Thus, the null (H 0 ) and alternative hypotheses (H A ) are: H 0 : β n = 0 ; and H A : β n ≠ 0 . The individual null hypotheses are tested using the t-distribution test.
Model 6 was estimated using STATA 10 statistical software . The OPDIPD and ALOS data were obtained from the Eritrea Ministry of Health annual health service activity report 2007 . The data on POP and REGION was obtained from Tewoldebrhan .
Results and discussion
Descriptive statistics of the input and outputs for public hospitals (n=19)
Number of outpatient department visits
Number of discharges
Number of doctors
Number of nurses, midwives & nurse associates
Number of laboratory technicians
Number of beds and cots
Output oriented DEA efficiency scores for hospitals in Eritrea
Returns to scale
Reference set (lambda weights)
Tio Mini Ho.
Afabet ( 0.316); Sembel (0.251); Aquardat (0.313); Edaga (0.057); Tio (0.063)
Aquardat (0.245); Sembel (0.016); Afabet (0.622); Halibet (0.117)
Peer (Lambda weight): Keren (0.176); Aquardat (0.824)
Adiquala (0.313); Aquardat (0.366); Edaga Hamus (0.183); Sembel (0.065); Keren (0.072)
Sembel (0.293); Keren (0.640); Halibet (0.067)
Edaga Hamus MH
Afabet (0.388); Aqurdat (0.134); Sembel (0.257); Halibet (0.221)
Eight (42%) hospitals were constant return to scale technically efficient, and the remaining 11 (58%) were relatively inefficient. Among the latter, 6 hospitals had a constant return to scale technical efficiency score of 91-99%, 2 scored 81-90%, 2 scored 71-80%, and 1 scored less than 71%. The mean constant return to scale technical efficiency was 90.3%, with a standard deviation of 17.2%. The average constant return to scale technical efficiency score varied from a minimum of 27.9% in Denden hospital to a maximum of 100% at Tio, Afabet, Karen, Aquardat, Sanafe, Edaga Hamus, Sembel and Halibet hospitals. Out of the eight relatively constant return to scale technically efficient hospitals, three (Tio, Sanafe and Edaga Hamus), had no medical doctor on the staff, which obviously raises issues regarding quality of care.
Thirteen (68%) hospitals were variable returns to scale technically efficient, scoring 100%, and the remaining 6 (32%) hospitals were variable returns to scale technically inefficient. Three of the inefficient hospitals had variable returns to scale technical efficiency scores between 91 and 99%, two scored between 81% and 90%, and one scored below 81%. The overall sample average variable returns to scale technical efficiency score was 96.9% (standard deviation = 6.5%), meaning that inefficient hospitals could, on average, produce 3.1% more health service outputs using their current input endowment. Assab hospital had the lowest variable returns to scale technical efficiency score, at 74.1%. Surprisingly, four of the hospitals with no doctors were relatively variable returns to scale technically efficient. Once again, further investigation would have been necessary to establish the relative quality of services provided by those hospitals.
Eight (42%) hospitals had a SE score of 100%, meaning they were at the optimal size for their particular input–output mix. The remaining 11 (58%) hospitals had scale efficiency scores of less than 100% and were thus deemed scale inefficient. Distribution was as follows: 1 hospital had a scale efficiency score of less than 31%; 2 hospitals had a scale efficiency score between 81-90%; and 8 hospitals had a scale efficiency of between 91-99%. The average scale efficiency score was 93.3% (standard deviation = 16.5%), meaning that on average, the scale inefficient hospitals could reduce their size by 6.7% without affecting their current output levels. The Ghinda, Nakfa, Adikeyh, Adiquala and Denden hospitals constant returns to scale technical inefficiency was fully attributed to scale inefficiencies.
The results revealed that increasing the quantity of all hospitals inputs by a given proportion would result in:
Constant returns to scale in 8 (42%) hospitals, implying that their health service outputs would increase in the same proportion. This means that Tio, Afabet, Keren, Aquardat, Sanafe, Edaga Hamus, Sembel and Halibet hospitals were operating at their most productive scale sizes.
Increasing returns to scale in 5 (26%) hospitals, implying that their health service outputs would increase by a greater proportion. These hospitals (Assab, Nakfa, Adiquala, Dekemhare and Denden) thus needed to increase their size to achieve optimal scale, i.e. the scale at which there are constant returns to scale in the relationship between inputs and outputs.
Decreasing returns to scale in 6 (32%) hospitals, implying that their health service outputs would increase by a smaller proportion. Therefore, Massawa, Ghinda, Tesenei, Adikeyh, Mendefera and Hazhaz hospitals would have needed to reduce their size to achieve optimal scale.
Econometric analysis of the determinants of inefficiency
Results for Tobit model
8 left-censored observations at Ineff ≥ 0
11 uncensored observations
0 right-censored observations
Number of observations
Prob > χ2
Pseudo R 2
The coefficient for OPDIPD has a negative sign consistent with our a priori expectation, and is statistically significant at the 5 percent level of significance. A unit increase in the ratio of outpatient department visits to inpatient days would lead to a decrease in hospital expected inefficiency score by 1.333, holding all other variables in the model constant. The higher a hospital OPDIPD, the lower the predicted inefficiency score.
The coefficient for ALOS assumed a positive sign as expected, and was statistically significant at the 5 percent level of significance. If the ALOS increases by one day, hospitals’ expected inefficiency score would increase by 0.152 while holding all other explanatory variables constant. Thus, the higher a hospital’s ALOS, the higher the predicted inefficiency score.
The coefficients for POPULATION and REGION had a positive sign but were statistically insignificant. Therefore, population size of the region where a hospital is situated and the geographical landscape (highland or lowland) do not have a significant effect on the efficiency (inefficiency) level.
Implications for policy
Comparison of Assab hospital with its efficiency reference set hospitals
Efficiency reference set hospitals
13652a x 0.316b = 4314.032
742a x 0.316b = 234.472
1a x 0.316b = 0.316
17a x 0.316b = 5.372
3a x 0.316b = 0.948
47a x 0.316b = 14.852
39451a x 0.251b = 9902.201
2996a x 0.251b = 751.996
2a x 0.251b = 0.502
81a x 0.251b = 20.331
2a x 0.251b = 0.502
103a x 0.251b = 25.853
17930a x 0.313b = 5612.09
4356a x 0.313b = 1363.428
6a x 0.313b = 1.878
31a x 0.313b = 9.703
4a x 0.313b = 1.252
78a x 0.313b = 24.414
Edaga Hamus [D]
13307a x 0.057b = 758.499
1075a x 0.057b = 61.275
0a x 0.057b = 0
73a x 0.057b = 4.161
3a x 0.057b = 0.171
12a x 0.057b = 0.684
5681a x 0.063b = 357.903
650a x 0.063b = 40.95
0a x 0.063b = 0
7a x 0.063b = 0.441
2a x 0.063b = 0.126
35a x 0.063b = 2.205
Composite hospital [F=A+B+C+D+E]
Output increase or input reduction [H=F-G]
% Change [I=(H/G)*100]
In this example, the composite hospital outputs and inputs are derived by multiplication of the DEA-generated weights of 0.316 (Afabet), 0.251 (Sembel), 0.313 (Aquardat), 0.057 (Edaga Hamus) and 0.063 (Tio) by those hospitals actual outputs and inputs. The composite hospital’s target (projected) number of outpatient department visits of 20944 is the sum of weighted outpatient department visits in Afabet (4314), Sembel (9902), Aquardat (5612), Edaga Hamus (758) and Tio (358).
The difference between the composite hospital’s outpatient department visits and Assab hospital’s outpatient department visits is 5427 (35%). Thus Assab hospital needs to increase output by 5427 outpatient department visits to match the composite hospital performance. Assab would also have to increase hospital discharges by 634 (35%) to become relatively technically efficient. Alternatively, it could achieve the same efficiency score by reducing the number of doctors by 2.3 (46%).
Efficiency scores and actual and target inputs and outputs quantities for inefficient hospitals according to VRS assumption
Nurses, midwives & nurse associates
Outpatient department visits
Inpatient department discharges
Nurses, midwives & nurse associates
Outpatient department visits
Inpatient department discharges
Nurses, midwives & nurse associates
Outpatient department visits
Inpatient department discharges
Nurses, midwives & nurse associates
Outpatient department visits
Inpatient department discharges
Nurses, midwives & nurse associates
Outpatient department visits
Inpatient department discharges
Nurses, midwives & nurse associates
Outpatient department visits
Inpatient department discharges
Health policy-makers have three broad strategies available to them for addressing inefficient resources use: (a) increasing coverage of health services; (b) reducing hospital inputs; and/or (c) organization/process changes in hospitals. Since the third strategy is beyond the scope of the current study we will focus on the first two strategies.
(a). Increasing coverage of health services
In order for the inefficient hospitals to have become relatively efficient, as a group, they would have needed to increase their outpatient department visits by 20611 (5.05%) and inpatient department discharges by 1806 (3.42%). Individually, to be relatively technically efficient, Assab hospital needed to increase its outpatient department visits and inpatient discharges by about 35%; Massawa hospital ought to have increased its outpatient department visits and inpatient discharges by 13%; Tesenei hospital should have increased its outpatient department visits and inpatient discharges by 90% and 11%, respectively; Dekemhare hospital ought to have increased its outpatient department visits and inpatient discharges by 7%; Mendefera hospital needed to have increased its outpatient department visits and inpatient discharges by 2%; and Hazhaz hospital should have increased its outpatient department visits and inpatient discharges by 4%.
In 2000 the United Nations General Assembly adopted the Millennium Declaration which contains eight Millennium Development Goals (MDG) . Three of them are health goals: MDG4: reduce child mortality; MDG5: improve maternal health; and MDG6: combat HIV/AIDS, malaria and other diseases. As shown in Additional file 1: Appendix, coverage of some of the MDG4 health services such as vitamin A supplementation treatment for children with acute respiratory infections, and oral rehydration therapy for children with diarrhoea is low. The coverage of MDG5 health services such as antenatal care, skilled birth attendance, and postnatal care is also low, as is coverage of MDG6 health services such as antimalaria treatment for children under five with fever, insecticide treated nets, antiretroviral therapy for HIV-infected pregnant women, and treatment for tuberculosis, among others . The regression analysis results reported earlier clearly indicate that an increase in the ratio of outpatient visits to inpatient days would have increased the efficiency of hospitals in Eritrea.
Additional file 1: Appendix shows that only 28% of births were attended by skilled birth attendants, and yet on the whole bed occupancy rates in the hospitals were very low. For example bed occupancy was just 32.1% in Assab hospital, and 28.5% in Mendedera hospital. Among the inefficient hospitals, the best score on bed occupancy was achieved by Hazhaz hospital, but even there, it was just over 50%. The fact that 72% of pregnant women gave birth without the assistance of skilled health personnel, amidst underutilized hospital bed capacity, implies that there were factors preventing those in need from accessing care. One of the likely barriers to access in Eritrea is out-of-pocket payments for health care. In 2009, household out-of-pocket spending on health accounted for 52% of the total expenditure on health in Eritrea . Such high levels of out-of-pocket payments have been shown to correlate highly with the incidence of financial catastrophe and impoverishment.
Level and structure of user fees in United States Dollars in 1995
Registration and consultation (non-referred)
Registration and consultation (referred)
Inpatient hotel charges per day (non-referred)
Inpatient hotel charges per day (referred)
Private wing hotel charges per day
It is clear from the National Health Policy of 2010, that the Eritrea Ministry of Health is very much aware of the adverse effects of significant out-of-pocket payment. In strategic policy goal 7, the Government set out a plan to introduce a health-financing scheme that protects people from catastrophic expenditures and ensures sustainability of the system. The National Health Policy states that :
“The Government shall develop evidence based mechanisms of risk-sharing and cross subsidisation that facilitate solidarity and equity. This will be achieved through a phased introduction of health insurance scheme(s) that will be informed by actuarial and socio-economic feasibility studies. (pp.28-29).”
In other words, the country plans to develop prepaid and pooled health financing to reduce over reliance on out-of-pocket payments to finance the national health system. The introduction of such prepayment mechanisms will most likely reduce the economic barriers to access to health care, and hence, contribute to improving the efficiency of hospitals .
(b). Reducing hospital inputs
As already noted the six hospitals that were variable returns to scale technically inefficient in 2007 could also have improved their relative efficiency by reducing their inputs by a total of 2.478 doctors (2.85%), 9.914 nurses and midwives (and associate nurses) (0.98%), 9.774 laboratory technicians (9.68%) and 194.995 beds (10.42%). The required input reductions for individual hospitals are set out in Table 6 where it is apparent that most hospitals would have benefited from reductions in the number of laboratory technicians and beds.
What could have been done with excess inputs in 2007? Given the national drive to attain the health Millennium Development Goals, it would not have been prudent to lay-off excess staff. The policy-makers could have considered reassigning excess health workers to health centres, health stations and maternal and child health clinics. They could also have explored the feasibility of creating mobile pools of excess health workers to go around health centres, health stations and maternal and child health clinics, providing outreach services. According to the United Nations Millennium Development Goals report 2011 , Eritrea is already on track to achieve Millennium Development Goals 4 and 5. The efficiency improvements identified in this paper would only have helped to accelerate what is already commendable progress in these areas.
On the other hand, excess beds could either have been transferred to primary health care facilities with bed shortages or sold to the non-governmental health sector. This would have ensured that the extra supply of beds would not have led to more admissions and longer stays, a phenomenon often referred to as Roemer’s Law .
Limitations of the study
The study reported in this paper has a number of limitations. First, DEA is a deterministic technique. Thus, any deviation from the production possibilities frontier is attributed to inefficiency, whereas some of the deviations from the frontier may in fact be due to epidemics, civil war or natural disasters (e.g. flooding, earthquakes) leading to displacement of people.
Second, the analysis reported in this paper is based on hospital inputs and outputs data for 2007. Much has happened since 2007, notably in terms of the country’s socioeconomic and health development. It was for this reason that we stated at the outset that the results of this analysis are not meant to be uncritically fed into current decision-making, but rather to illustrate the potential usefulness of such efficiency analyses.
Third, due to the lack of data, this study did not include the expenditures on pharmaceuticals and non-pharmaceutical supplies among the inputs. Nor does the study take into consideration the differences that may exist between the categories of nurses and doctors in the various hospitals. In addition, even within the same health workforce category, the quality of labour input may vary depending on individual health worker skills, professional experience and health status.
Fourth, the hospitals included in the study were all secondary public community hospitals. However, because the study was based on secondary data, we had no way of knowing whether there might have been some variations in the severity of cases treated in each hospital. Significant differences in the severity of cases treated could affect the number of cases hospitals dealt with relative to their staff numbers and bed numbers, and could therefore have an impact on the results of the analysis. The hospitals treating a large number of severe cases, for example, may handle fewer cases, and will thus appear to be relatively inefficient.
Suggestions for further research
There is a need for a Malmquist Total Factor Productivity Index analysis to measure the trends in efficiency and productivity of hospitals over time [9, 11, 20, 30]. This would entail collecting inputs and outputs data for a number of years, e.g., from 2008 to 2012. Such an analysis would permit comparison of the current state of hospital efficiency with the situation prevailing in 2007.
There is a need for demand analyses studies to identify the significant determinants of households’ decisions to seek health care from hospitals, and for studies evaluating the effectiveness of various options for removing or reducing barriers to population access to essential health services .
In its 2010 National Health Policy, the Eritrean Ministry of Health sets out a plan to introduce hospital reforms (organizational changes) which include introducing hospital autonomy aimed at improving performance, quality of care, cost containment, and hence, sustainability of services. Autonomy usually entails a shift of financial and human resource management as well as service development planning responsibility from the Ministry of Health to the hospitals themselves. It is our hope that while developing the evidence-based Hospital Reforms Strategic Plan envisaged in the National Health Policy, the Ministry of Health will develop a set of indicators for tracking changes in efficiency and performance of hospitals over time. Monitoring and evaluation could begin with the above-mentioned Malmquist Total Factor Productivity Index analysis prior to implementation of hospital reforms in order to establish a meaningful baseline against which to measure progress.
One of the strategic policy goals contained in Eritrea’s 2010 National Health Policy is to enhance efficiency, equity and quality of service delivery through health systems development.
The study met its objectives in (i) estimating the relative technical and scale efficiencies of 19 secondary public hospitals in Eritrea, (ii) in quantifying the magnitudes of output increases and/or input reductions that could have made inefficient hospitals more efficient in 2007, and (iii) in estimating the impact of institutional and contextual/environmental variables on hospital inefficiencies using Tobit regression analysis.
The findings from the first-stage analysis indicate that about 68% hospitals were variable returns to scale technically efficient. Only 42% hospitals achieved scale efficiency. The inefficient hospitals collectively could have become efficient by increasing their total outpatient visits and hospital discharges by 5.05%% and 3.42%. Therefore, in 2007 there was scope for treating an extra 20,611 outpatients and making 1,806 inpatient discharges per year with the existing hospital input endowments.
Alternatively, policy-makers might have considered boosting the quality of services provided by primary health care facilities by transferring the excess 2.478 doctors (2.85%), 9.914 nurses and midwives (0.98%), 9.774 laboratory technicians (9.68%), and 195 beds (10.42%) to health centres and maternal and child health clinics.
Our second-stage analysis show that the ratio of outpatient visits to inpatient days and average length of inpatient stay are significantly correlated to hospital inefficiencies. Thus, policy interventions that increase utilization of under-utilized hospital outpatient health services and reduce the hospital average length of stay would help to reduce inefficiencies. The population size of the region where a hospital is situated and the geographical landscape (highland or lowland) were not found to have a significant effect on hospital inefficiency level.
In a nutshell, even where inefficiencies existed within hospitals, it would not have been prudent to ameliorate them by down-scaling the health workforce. Instead, it would have been better to address such inefficiencies by either reassigning excess health workers to primary health care facilities or by increasing the coverage of health services offered in order to address the unmet need for hospital services.
We are immensely grateful to the Eritrean Ministry of Health for making available the Annual Health Service Activity Report 2007, upon which our analysis was based. The fact that Eritrea is one of the very few African countries on track to achieve MDGs 4 and 5 is highly commendable. In our opinion, there are important lessons to be drawn from the Eritrean performance. We also acknowledge the editorial assistance of Gary Humphreys. We are indebted to Jehovah Jireh for inspiration and sustenance during the entire process of preparing this manuscript. The views expressed in this article are those of the authors and do not represent the opinions, positions or policies of the organizations they work for.
- World Health Organization Regional Office for Africa (WHO/AFRO): WHO country cooperation strategy 2009–2013 for Eritrea. Brazzaville: WHO/AFRO; 2009.Google Scholar
- World Health Organization: World Health Statistics 2011. Geneva: WHO; 2011.Google Scholar
- Eritrea Ministry of Health: Annual health service activity report of year 2007. Asmara: Government of Eritrea; 2007.Google Scholar
- World Health Organization: The World health report 2006: working together for health. Geneva: WHO; 2006.View ArticleGoogle Scholar
- World Health Organization: The world health report 2000 - Health systems: improving performance. Geneva: WHO; 2000.Google Scholar
- World Health Organization: Everybody’s business: strengthening health systems to improve health outcomes: WHO’s framework for action. Geneva: WHO; 2007.Google Scholar
- Lambo E, Sambo LG: Health sector reform in sub-Saharan Africa: a synthesis of country experiences. East Afr Med J 2003,80(Suppl 6):1–20.Google Scholar
- WHO/AFRO: Health financing: a strategy for the African Region. Brazzaville: WHO/AFRO; 2006.Google Scholar
- Kirigia JM, Emrouznejad A, Cassoma B, Asbu EZ, Barry S: A performance assessment method for hospitals: the case of Municipal Hospitals in Angola. Journal of Medical System 2008,32(6):509–519. 10.1007/s10916-008-9157-5View ArticleGoogle Scholar
- Kirigia JM, Mensah OA, Mwikisa CN, Asbu EZ, Emrouznejad A, Makoudode P, Hounnankan A: Technical efficiency of zone hospitals in Benin. The African Health Monitor 2010, 12: 30–39.Google Scholar
- Tlotlego N, Nonvignon J, Sambo LG, Asbu EZ, Kirigia JM: Assessment of productivity of hospitals in Botswana: a DEA application. International Archives of Medicine 2010, 3: 27. URL: http://www.intarchmed.com/content/3/1/27 10.1186/1755-7682-3-27PubMed CentralPubMedView ArticleGoogle Scholar
- Marschall P, Flessa S: Assessing the efficiency of rural health centres in Burkina Faso: an application of Data Envelopment Analysis. J Public Health 2009,17(2):87–95. 10.1007/s10389-008-0225-6View ArticleGoogle Scholar
- Sebastian MS, Lemma H: Efficiency of the health extension programme in Tigray, Ethiopia: a data envelopment analysis. BMC International Health and Human Rights 2010, 10: 16. URL: http://www.biomedcentral.com/1472–698X/10/16 10.1186/1472-698X-10-16PubMed CentralPubMedView ArticleGoogle Scholar
- Akazili J, Adjuik M, Jehu-Appiah C, Zere E: Using data envelopment analysis to measure the extent of technical efficiency of public health centres in Ghana. BMC International Health and Human Rights 2008, 8: 11. URL: http://www.biomedcentral.com/1472–698X/8/11 10.1186/1472-698X-8-11PubMed CentralPubMedView ArticleGoogle Scholar
- Osei D, George M, DAlmeida S, Kirigia JM, Mensah AO, Kainyu LH: Technical efficiency of public district hospitals and health centres in Ghana: a pilot study. Cost Effectiveness and Resource Allocation 2005, 3: 9. URL: http://www.resource-allocation.com/content/3/1/9 10.1186/1478-7547-3-9PubMed CentralPubMedView ArticleGoogle Scholar
- Kirigia JM, Emrouznejad A, Sambo LG: Measurement of technical efficiency of public hospitals in Kenya: using data envelopment analysis. J Med Syst 2002,26(1):39–45. 10.1023/A:1013090804067PubMedView ArticleGoogle Scholar
- Kirigia JM, Emrouznejad A, Sambo LG, Munguti N, Liambila W: Using data envelopment analysis to measure the technical efficiency of public health centers in Kenya. J Med Syst 200,28(2):155–166.View ArticleGoogle Scholar
- Zere E, Mbeeli T, Shangula K, Mandlhate C, Mutirua K, Tjivambi B, Kapenambili W: Technical efficiency of district hospitals: Evidence from Namibia using data envelopment analysis. Cost Effectiveness and resource allocation 2006, 4: 5. URL: http://resource-allocation.com/content/4/1/5 10.1186/1478-7547-4-5PubMed CentralPubMedView ArticleGoogle Scholar
- Ichoku H, Fonta WM, Onwujekwe OE, Kirigia JM: Evaluating the technical efficiency of hospitals in South Eastern Nigeria. European Journal of Business and Management 2011,3(2):24–37.Google Scholar
- Kirigia JM, Emrouznejad A, Vaz RG, Bastiene H, Padayachy J: A comparative assessment of performance and productivity of health centers in Seychelles. Int J Product Perform Manag 2008,57(1):72–92. 10.1108/17410400810841245View ArticleGoogle Scholar
- Renner A, Kirigia JM, Zere AE, Barry SP, Kirigia DG, Kamara C, Muthuri HK: Technical efficiency of peripheral health units in Pujehun district of Sierra Leone: a DEA application. BMC Heal Serv Res 2005, 5: 77. 10.1186/1472-6963-5-77View ArticleGoogle Scholar
- Kirigia JM, Sambo LG, Renner A, Alemu W, Seasa S, Bah Y: Technical efficiency of primary health units in Kailahun and Kenema districts of Sierra Leone. International Archives of Medicine 2011, 4: 15. URL: http://www.intarchmed.com/content/4/1/15/ 10.1186/1755-7682-4-15PubMed CentralPubMedView ArticleGoogle Scholar
- Kirigia JM, Lambo E, Sambo LG: Are public hospitals in Kwazulu-Natal province of South Africa Technically Efficient? Afr J Health Sci 2000,7(3–4):25–32.PubMedGoogle Scholar
- Kirigia JM, Sambo LG, Scheel H: Technical efficiency of public clinics in Kwazulu-Natal province of South Africa. East Afr Med J 2001,78(suppl 2):1–13.Google Scholar
- Asbu EZ, McIntyre D, Addison T: Hospital efficiency and productivity in three provinces of South Africa. South African Journal of Economics 2001,69(2):336–358.Google Scholar
- Masiye F, Kirigia JM, Emrouznejad A, Sambo LG, Mounkaila A, Chimfwembe D, Okello D: Efficient Management of Health Centres Human Resources in Zambia. J Med Syst 2006, 30: 473–481. 10.1007/s10916-006-9032-1PubMedView ArticleGoogle Scholar
- Masiye F: Investigating health system performance: An application of data envelopment analysis to Zambian hospitals. BMC Heal Serv Res 2007, 7: 58. URL: http://www.biomedcentral.com/1472–6963/7/58 10.1186/1472-6963-7-58View ArticleGoogle Scholar
- Skaggs NT: Carlson: Microeconomics. London: Blackwell Publishers; 1996.Google Scholar
- Salvatore D: Managerial economics: principles and worldwide applications. New York: Oxford University Press; 2008.Google Scholar
- Fried HO, Lovell CAK, Schmidt SS: The measurement of productive efficiency: techniques and applications. New York: Oxford University Press; 1993.Google Scholar
- Coelli TJ, Rao DSP, O’Donnell CJ, Battese GE: An introduction to efficiency and productivity analysis. London: Springer; 2005.Google Scholar
- Coelli TJ: A guide to DEAP Version 2.1: a data envelopment analysis computer programme. CEPA Working Paper No. 8/96. Armidale: University of New England; 1996.Google Scholar
- Farrell MJ: The measurement of productive efficiency. J R Stat Soc 1957,120(3):253–258.Google Scholar
- Cooper WW, Seiford LM, Zhu J: Data envelopment analysis: history, models and interpretations. In Handbook on Data Envelopment Analysis. Edited by: Cooper WW, Seiford LM, Zhu J. New York: Springer; 2011:1–40. Volume 1.View ArticleGoogle Scholar
- Charnes A, Cooper WW, Rhodes E: Measuring the efficiency of decision making units. Eur J Oper Res 1978, 2: 429–444. 10.1016/0377-2217(78)90138-8View ArticleGoogle Scholar
- Koutsoyiannis A: Modern Microeconomics. London: Macmillan Education Ltd; 1979.Google Scholar
- Banker RD, Charnes A, Cooper WW: Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management Sciences 1984,30(9):1078–1092. 10.1287/mnsc.30.9.1078View ArticleGoogle Scholar
- Rosko MD, Chilingerian JA, Zinn JS, Aaronson WE: The effects of ownership, operating environment, and strategic choices on nursing efficiency. Medical Care 1995,33(10):1001–1021. 10.1097/00005650-199510000-00003PubMedView ArticleGoogle Scholar
- Vladmanis V: Sensitivity analysis for DEA models: an empirical example using public vs NFP hospitals. Journal of Public Economics 1992, 48: 185–205. 10.1016/0047-2727(92)90026-CView ArticleGoogle Scholar
- Simar L, Wilson PW: Estimation and inference in two-stage, semiparametric models of production processes. J Econ 2007,136(1):31–64.View ArticleGoogle Scholar
- Ramalho EA, Ramalho JJS, Henriques PD: Fractional regression models for second state DEA efficiency analyses. J Product Anal 2010, 34: 239–255. 10.1007/s11123-010-0184-0View ArticleGoogle Scholar
- McDonald J: Using least squares and tobit in second stage DEA efficiency analyses. Eur J Oper Res 2009, 197: 792–798. 10.1016/j.ejor.2008.07.039View ArticleGoogle Scholar
- Ruggiero J: Performance evaluation in education: modelling educational production. In Handbook on Data Envelopment Analysis. Edited by: Cooper WW, Seiford LM, Zhu J. Boston: Kluwer Academic Publishers; 2004:265–298.Google Scholar
- Afonso A, Aubyn M: Assessing health efficiency across countries with a two-step and bootstrap analysis. Appl Econ Lett 2011,18(15):1427–1430. 10.1080/13504851.2010.541149View ArticleGoogle Scholar
- Greene WH: Econometric analysis. New York: Macmillan; 1993.Google Scholar
- Asbu EZ: Hospital efficiency in sub-Saharan Africa: evidence from South Africa. Working Paper No. 187. Helsinki: The United National University World Institute for Development Economic Research; 2000.Google Scholar
- McDonald JF, Moffitt RA: The uses of Tobit analysis. Rev Econ Stat 1980,62(2):318–321. 10.2307/1924766View ArticleGoogle Scholar
- StataCorp: STATA 10 data analysis and statistical software. Texas: StataCorp; 2010.Google Scholar
- Tewoldebrhan E: Ethno-regional disparities in primary schooling in Eritrea, 1992–2001. Cape Town: University of the Western Cape; 2007.Google Scholar
- Nations U: MDG report 2011. New York: United Nations; 2011.Google Scholar
- WHO: World Health Statistics 2012. Geneva: WHO; 2012.Google Scholar
- Asbu EZ: Analysis of the user fee for health care in Eritrea. Central African Journal of Medicine 1999,45(4):86–93.PubMedView ArticleGoogle Scholar
- UNDP: Human development report 2009 – overcoming barriers: human mobility and development. New York: UNDP; 2009.Google Scholar
- The World Bank African Region: Eritrea health and education sectors: public expenditure review. Washington, D.C.: The World Bank; 2008.Google Scholar
- The State of Eritrea Ministry of Health: National Health Policy. Asmara: Ministry of Health; 2010.Google Scholar
- WHO: The World Health Report, Health systems financing – the path to universal coverage. Geneva: WHO; 2010:2010.Google Scholar
- Roemer MI: Bed supply and hospital utilization: a natural experiment. Hospitals 1961, 35: 36–42.PubMedGoogle Scholar
- Mwabu G: Health care decisions at the household level: results of a Rural Health Survey in Kenya. Soc Sci Med 1986,22(3):315–319. 10.1016/0277-9536(86)90129-2PubMedView ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.