Abstract

Multiple criteria analysis (MCA) has been widely applied within the field of natural resource management since the 1970s. During this period MCA has undergone considerable methodological advancement with numerous methods for capturing decision maker preferences, ranking or scoring decision options, handling uncertainty and presenting results. This paper explores the role of MCA within the economist’s evaluation toolkit, which also contains benefit cost analysis (BCA), cost effectiveness analysis (CEA) and cost utility analysis (CUA). A process for selecting an appropriate evaluation tool is proposed, which is partly dependent on the extent to which environmental goods can be valued in monetary units.

Introduction

Multiple criteria analysis (MCA) is an evaluation framework which can be used to rank or score the performance of decision options (e.g. policies, projects, locations) against multiple objectives measured in different units. Typically the criteria are weighted by decision makers to reflect their relative importance. The MCA approach emerged within the field of operations research during World War II and early applications were in military planning (e.g. Eckenrode, 1965). The theoretical foundations of MCA can be traced back to multi-attribute utility theory (MAUT) by Keeney and Raiffa (1976, 1993) and axioms of utility measurement first supplied by von Neumann and Morgenstern (1944). At its core MCA is a set of techniques for utility measurement.

In the field of environmental and resource economics MCA has mostly received a positive reception. Many researchers find it a useful technique able to supplement conventional benefit cost analysis (BCA) when intangible non-market goods are important (Joubert et al., 1997; Dunning et al., 2000; Heilman et al., 1997; Eder et al., 1997; Prato, 1999; Fernandes et al., 1999). MCA has hundreds of applications in natural resource management (for reviews see Romero and Rehman, 1987 and Hayashi, 2000). However, not all resource economists are convinced. In the 2005 presidential address to the Australian Agricultural and Resource Economics Society MCA was described as an ‘avoidance strategy’ to sidestep a rigorous and complete analysis (Bennett, 2005).

Given that MCA application in Australian natural resource management is becoming increasingly common, both in research and application, the criticism is worth exploring. This paper examines the role of MCA for economic appraisal of agricultural, natural resource and environmental policy options. There is no suggestion that MCA replace BCA, nor that non-market valuation be abandoned. Rather, MCA forms an important part of the economist’s evaluation toolkit along with other tools such as BCA and non-market valuation. All tools have their strengths and weaknesses. The aim is to select the right tool for the job.

What is Multiple Criteria Analysis (MCA)?

The use of MCA to support public and private sector policy decisions has steadily grown since the 1970s. Two decades ago Romero and Rehman (1987) reviewed 150 MCA applications in fisheries, forestry, water and land resource applications. Many more natural resource applications of MCA have been published since. Hayashi (2000) reviewed over 80 published studies in agriculture. In energy planning Pohekar and Ramachandran (2004) identify over 90 published MCA applications. Steuer and Na (2003) examine 265 applications of MCA in the field of financial decision making. Today MCA has dedicated scholarly journals, dozens of guide books, professional societies and regular conferences (Figueira et al., 2005a). There are hundreds of MCA techniques, many published in operations research journals and Weistroffer et al. (2005) identify 81 MCA software packages, many of which are commercially available.

An MCA model can be represented with an evaluation matrix (X) of n options and m criteria. The evaluation matrix contains performance measures where xi,j is the raw performance score assigned to option i against criterion j. Typically, though not always, the relative importance of criteria is measured with a weights vector W where wj represents the importance of the jth criterion. Both W and X may contain qualitative (ordinal) or quantitative (cardinal) data. An evaluation matrix is often structured as follows:

For an MCA model to be needed at least two criteria and two options are required. If the purpose of the MCA is discrete choice, to select one or more options, an initial check can be made for strict dominance. Options that are strictly dominated are outperformed by another option on all criteria. If v_i,j is the transformed performance score (where a higher value is better) of x_i,j, option i can be considered strictly dominated by i’ if:

for all  and  for some j=1,…,m.

The pre-test for strict-dominance can sometimes make the decision analysis elementary negating the requirement for more advanced MCA models. Criteria that are non-discriminatory do not differentiate the performance of at least two options and should be excluded from the evaluation matrix. Criterion j can be considered non-discriminating if:

for all

The stages of MCA (Figure 1) include:
1. Problem structuring. This involves the identification of criteria and decision options and obtaining performance measures. It is a crucial stage of MCA where the bulk of the effort is typically required (Janssen, 2001).
2. Criteria weighting. This involves obtaining information from decision makers about the relative importance of criteria. Weights may be expressed at either an ordinal or cardinal measurement level.
3. Transforming criteria. As the criteria are in different units they need to be transformed into commensurate units prior to aggregation in the ranking or scoring function.
4. Ranking and/or scoring options. The weights and transformed performance measures are combined to determine the overall performance of each option, relative to other options.
5. Conduct sensitivity analysis and make a decision. The sensitivity of the result should be tested by variation of MCA methods, performance measures and weights. A final choice can then be made by the decision maker(s).

A wide variety of MCA algorithms (see Figueira, 2005a for a recent review) can be used to attain a final ranking or scoring of the decision options. Some of the more common MCA algorithms are the Analytic Hierarchy Process (AHP; Saaty, 1987), weighted summation; ELECTRE (Roy, 1968; Figueira et al., 2005b), PROMETHEE (Brans et al., 1986) and Compromise Programming (Zeleny, 1973; Abrishamchi et al., 2005). These methods are a few of the many different ways to ‘solve’ an MCA problem. It has been shown that changing the method can change the result, although the differences are typically minor (Gershon and Duckstein, 1983; Ozelkan and Duckstein, 1996; Eder et al., 1997; Raju et al., 2000). Choosing the best MCA method to apply is a considerable challenge facing analysts (Tecle, 1992).

Figure 1. The multiple criteria analysis decision making process.

Arguably the most commonly applied MCA technique, possibly by virtue of its relative ease of computation, is linear weighted summation (Howard, 1991; Zanakis et al., 1998). This approach determines overall performance scores for decision options (ui) by:

(1)
 

where:

;
 

;

(2)
 

= the maximum value of for i =1,…n ; and

= the minimum value of for i =1,…n.

If a higher raw performance score indicates worse overall performance then, instead of Equation 2, the transformed performance score (vi,j) is calculated by:

(3)

The weighted summation approach forms part of some other MCA methods, albeit handled in different ways. For example, the Range of Value Method (ROVM) by Yakowitz (1993), which requires only ordinal specification of criteria importance, incorporates weighted summation in optimisation algorithms to select decision options. Weighted summation can be considered a ‘foundation’ MCA method.

Alternative Economic Evaluation Frameworks

The appropriateness of MCA depends upon the suitability of other economic evaluation frameworks. The purpose of an evaluation framework is to help decision makers choose policy options. There are four main economic evaluation frameworks available to analysts:

1. Benefit cost analysis (BCA)
2. Cost effectiveness analysis (CEA)
3. Cost utility analysis (CUA)
4. Multiple criteria analysis (MCA)

The technique of BCA is well known with numerous guidebooks and MCA is described above. So we need only explain CEA and CUA in more detail. A CEA can be performed when the benefits of the decision options are adequately measured by a single unit, e.g. tonnes of soil. Costs in CEA are still computed with standard discounted cash flow (DCF) analysis. The aim is to identify the option which (a) achieves a target outcome at least cost; or (b) maximises the outcome measure subject to cost constraint.

In CUA the costs are still computed via standard DCF, as with CEA, but the benefits are measured by multiple attributes in different units. CUA emerged in the early 1980s in healthcare economics (Drummond et al., 1997). Today Quality Adjusted Life Years (QALYs) are routinely calculated to measure the non-market benefits of patient treatment or healthcare programs. The attributes used to determine a QALY score are sensation, mobility, emotion, cognition, self-care, pain and fertility. These attributes are weighted and collapsed into a single numeraire (unit of value) using multi-attribute utility theory which also underpins MCA. The ‘utility’ part of CUA can be determined with MCA. Utility measurement is essentially a problem of transforming, weighting and aggregating attributes to derive an overall performance metric.

The CUA approach is well established in healthcare economics, and has emerging application in environmental and resource economics. Cullen et al. (2001) apply CUA to the evaluation of biodiversity conservation programs in New Zealand using conservation output protection years (COPYs) in place of QALYs.

Although the term ‘CUA’ is not used by Ribaudo et al. (2001) they describe how a CUA approach was used to select conservation contracts under the United States Conservation Reserve Program (CRP). The benefits of contracts were measured with a multi-attribute environmental benefits index (EBI). Weights were assigned to the attributes in the form of maximum scores. Combining the cost of each option with the EBI enabled purchasing decisions. The BushTender program in Victoria is based on a similar concept. Purchasing decisions in BushTender are based on a biodiversity benefits index (BBI) and contract cost (Stoneham et al., 2003).

The process for choosing which of BCA, CEA, CUA and MCA to apply depends largely on the valuation of benefits (Figure 1). If benefits are adequately measured in monetary units then BCA provides an appropriate framework. If not the analyst will need to contemplate non-market valuation (NMV). Both the reliability and cost-effectiveness of performing NMV require attention. If it is decided the NMV is not feasible or worthwhile then CUA may be appropriate. If there is no monetary cost data, e.g. the options are long term strategic policy directions, then MCA can be used.

Figure 2. Process for selecting an economic evaluation framework.

In this paper it is argued that all four frameworks are ‘solidly founded’, able to measure benefits adequately and potentially applicable in different situations. There may be some cases where more than one framework is applied. None is inherently better or robust and all are based on solid theoretical foundations. The key determinant of which to use depends on how benefits and costs are valued. The selection of evaluation framework is inextricably linked to the question of valuation.

The Limits to Valuation?

Valuation of environmental resources has attracted considerable attention over the past several decades (Adamowicz, 2004). The appropriateness of different valuation techniques, and the appropriateness of valuation itself, has been much debated. There are three main approaches to valuing environmental resources.

Firstly, is cost savings and avoidance (CSA). This grouping of valuation techniques is limited to market impacts. It includes measures of preventative and mitigatory expenditure (e.g. Spurgeon, 1998), lost production (e.g. Hajkowicz and Young, 2005), ameliorative expenditure (e.g. Abdalla et al., 1992) and asset damage repair costs (e.g. Tol, 1996) as a consequence of an environmental problem. These analyses typically ask: How much is the environmental problem costing? Or, conversely, How much is being saved because of the presence of a well functioning environment.

Secondly, are revealed preference techniques. These methods estimate the price of a non-market good from a closely related proxy market good. Hedonic pricing (e.g. Pearson et al., 2002) and the travel cost method (e.g. Chen et al., 2004) are types of revealed preference techniques. They assess the premium being paid in a real market (e.g. property market) to access a non-market environmental good (e.g. scenic views).

Thirdly, there are stated preference techniques. These techniques are based on hypothetical questions of survey respondents. Contingent valuation asks survey respondents their willingness to pay (WTP) for environmental goods or willingness to accept (WTA) compensation for the loss of environmental goods (e.g. Carson et al., 2003). Choice modelling asks respondents to select bundles of environmental goods at different costs and prices are inferred from their choices (e.g. van Bueren and Bennet, 2004).

If the analyst considers these methods feasible, accurate and comprehensive then the flowchart (Figure 2) recommends using BCA. Whilst all three approaches provide effective tools for policy analysis this paper argues valuation has limitations. The techniques of CSA and revealed pricing have strong methodology but limited scope. In contrast, stated preference techniques have practically limitless scope but weaker methodology (Strijker et al., 2000). This means that not all outcomes can be valued in all cases.

The techniques of CSA and revealed preferences source data from real markets and thereby avoid the methodological difficulties associated with surveys. The drawback with these techniques lies not in their methodology but in their scope. In CSA’s case valuation is limited to market impacts. It does not include non-market goods such as landscape aesthetics and biodiversity preservation. In the case of revealed pricing scope may be limited due to unavailability of a proxy market. There are many environmental goods for which no proxy market exists.

The scope of valuation can be broadened by applying stated preference techniques. However, this introduces some significant methodological difficulties associated with the use of consumer preference surveys (Sagoff, 1988; Diamond and Hausman, 1994; McFadden, 1999; Ludwig 2000). Despite advances over time two problems persist in stated preference survey designs: (a) the marketplace is hypothetical which creates uncertainties about real consumer behaviour; and (b) the respondent is often unfamiliar, or unaware, of the environmental good under question. Ludwig (2000, p34) observes:
“…people are asked to place prices on things that are not ordinarily priced. For some commodities, we form an opinion about a suitable price from long experience in a market. If there is no such experience and no such market, there may be little consistency among responses and little validity in inferences drawn from the responses.”

These methodological issues have acted as a barrier to the use of stated preference valuations by policy makers. Adamowicz (2004) conducted a comprehensive review of hundreds of valuation studies conducted since 1975. It was found that non-market valuation (NMV) results are seldom used in real policy decisions despite a vast number of academic studies. Greater use is made of valuations based on market prices, i.e. CSA approaches. This can either be attributed to: a failure by policy makers to grasp the relevance of NMVs; or to fundamental methodological problems of valuing highly intangible non-market goods.

Rather than attempting to express highly intangible goods in dollar units the pathway to improved resource allocation may lie in the adoption of alternative decision making frameworks. In a review of valuation studies Adamowicz (2004; p439) concludes that:
“The most significant advance in environmental valuation may be to move away from a focus on value and focus instead on choice behaviour and data that generate information on choices. Advances in resource allocation are most likely to arise from better understanding of preferences and choice, rather than the generation of more value estimates and catalogues of these measures”.

If we believe that all non-market goods and services can be valued in monetary units then we need look no further than BCA. However, if we accept that valuation has limitations we must also accept BCA has limitations. But to leave decision makers without any structure or framework would make matters worse. Fortunately there exist alternative frameworks that can form part of the economist’s evaluation toolkit.

Potential Sources of Error in MCAs

Whilst non-market valuation involving stated preference approaches has methodological problems, we cannot pretend MCA is a panacea. The theories of MAUT and many techniques of MCA provide a rigorous theoretical underpinning. But that does not provide assurance against application errors or practical difficulties in building an MCA model. Some of the common sources of error associated with MCA are:

1. Incorrect problem structure. The selection of criteria and options to guide the MCA process, i.e. problem structuring, is typically the most crucial analytical task (Janssen, 2001). MCA failures can typically be traced back to poor problem definition. New research into MCA is developing improved means of selecting options and criteria. Mingers and Rosenhead (2004) review several ‘problem structuring methods’ (PSMs). Scheubrien and Zionts (2006) developed an interactive computer model to assist with problem structuring tasks.

2. Poor performance data. If the performance measures populating the MCA matrix are inaccurate the results will also be inaccurate. Sensitivity analysis can help determine the extent to which performance data uncertainties actually matter, i.e. change the overall ranking (Hyde et al., 2004). Sometimes there is over-reliance on qualitative performance measures, e.g. expert judgement scores. Expert judgements can be used where there is no other data source, but Keeney and Raiffa (1993) consider quantitative performance criteria preferable. Saaty’s (1987) analytic hierarchy process (AHP) is an MCA technique designed to elicit expert judgements when quantitative data are unavailable. The analyst will often be confronted with a trade-off: choosing criteria that are relevant; versus criteria for which there exist good data.

3. Inappropriate capturing of decision maker preferences. Decision maker preferences are typically captured via criteria weights. This requires careful communication to decision makers about criteria meanings and how their preferences influence the ranking/scoring of options. Often decision makers will only have time, willingness and/or ability to handle very simple weighting tasks (Hajkowicz et al., 2000). A wide range of weighting methods are available to the MCA analyst (Roberts and Goodwin, 2002). Edwards and Barron (1994) argue for the use of ‘swing weights’, where decision makers take criterion ranges (min and max performance scores) into account when assigning weights.

4. Incorrect application of additive utility. Often a linear additive model provides a reasonable utility function. However, there are some cases where criteria are non-compensatory, i.e. strong performance on one criterion (e.g. in stream zinc concentrations) does not compensate for poor performance on another (e.g. in stream arsenic concentrations). Debate about additive utility models is illustrated by the United Nation’s Human Development Index (HDI). Sagar and Najam (1998) argue the HDI should be computed by a multiplicative utility function as opposed to the current additive form.

5. Duplicate or overlapping criteria. This occurs when two or more criteria measure the same underlying attribute. Duplicates can sometimes be detected by searching for unusually high inter-criterion correlation.

None of these issues represents a problem with MCA technique itself, but rather how it is applied. As with any analytical tool real world constraints, e.g. limited data and time, will result in imperfect MCAs. The analyst will continually be challenged with avoiding or minimising these problems subject to the resources available to conduct the MCA study.

Are BCA and MCA Substitutes?

In the Netherlands MCA is routinely applied to inform natural resource and environmental policy choices (Janssen, 2001). Brouwer and van Ek (2004) describe the choice between MCA and BCA as the ‘classic’ question confronting Dutch policy analysts of how to compare ‘apples and pears’. Whilst the choice of an economic evaluation framework hinges largely on the question of valuation other factors are relevant. MCA and BCA are not perfect substitutes and have two subtle, yet significant, differences in the types of problems they address.

The first difference relates to the handling of social welfare. BCA is concerned with determining whether a proposed policy results in an aggregate increase in social welfare. In contrast MCA is concerned with selecting options that are consistent with the preferences of an individual decision maker or collective preferences of a decision making group. If decision maker preferences are representative of social preferences then MCA can be considered to represent social choice. However, obtaining full societal preferences for an MCA model is uncommon. A recent edition of the Journal of Multi Criteria Decision Analysis titled “e-Democracy” looks at how the internet can be used to elicit societal preferences for MCA (French, 2003). This represents prospective, but early, work into obtaining people’s preferences en masse fro an MCA model.

Most MCA applications have a set of decision makers, e.g. a panel of persons appointed by a government minister or a board of directors, who guide the analysis (e.g. Eder et al., 1997; Joubert et al., 2003). Having been a researcher and practitioner of MCA for the past decade I have not yet used it for policy recommendations where social preferences (criteria weights) were sought en masse. There has typically been a decision making group with a democratic mandate to make choices on behalf of society, larger stakeholder group or shareholders. Incorporating the preferences of society is not appropriate or feasible for every decision. MCA aims to select options that are consistent with the preferences of decision makers.

The second difference between MCA and BCA relates to the matter of relative, as opposed to absolute, performance measurement. MCA is only able to determine the performance of decision options relative to other options. This is different to BCA which can determine the economic desirability of a project based on its performance compared to the status-quo (i.e. doing nothing). If BCA produces a positive net present value (NPV) then the project is worthwhile regardless of what other options are available. To proceed is rational because benefits exceed costs. In contrast MCA can only rank, or score, the performance of one project relative to other projects. It says nothing about the overall desirability of a project in isolation. When MCA is used there are typically several or more options.

Several researchers have applied both MCA and BCA to the same natural resource management problem then compared the results (Joubert et al., 1997; Brouwer and van Ek, 2004; Strijker et al., 2000). From these studies there is no clear conclusion that either approach is ‘better’; both methods have their strengths and weaknesses. Strijker et al. (2000) argue that alternatives to BCA are ‘next-best solutions’ but are, nevertheless, required due to practical and methodological drawbacks with environmental valuation. They propose ‘minimising the disadvantages of both methods’ by combining BCA within the MCA. In a water planning problem in Cape Town, South Africa, Joubert et al. (1997) take a similar position suggesting that BCA and MCA are complementary tools. This paper argues likewise. Hopefully, the argument is progressed by the flowchart (Figure 2) to help analysts decide which economic evaluation framework to apply.

Conclusion

Although valuation has limitations MCA is not a panacea. As with any evaluation tool MCA has bounded scope for application and introduces methodological challenges of its own. The common methodological challenges, and potential sources of error, in MCA applications are weighting the criteria; transforming the criteria; choosing the criteria and options; avoiding redundant (duplicate) criteria; selecting decision makers and obtaining reliable performance measures. If sufficient time, effort and skill are devoted to these tasks MCA provides a robust and informative evaluation of decision options.

The choice of whether to apply MCA or an alternative economic evaluation framework hinges upon the question of valuation. There are strong arguments, both practical and methodological, that valuation has limited scope. Many intangible non-market environmental goods are beyond the realm of monetary quantification. In these cases the adoption of CEA, CUA or MCA can provide a more robust and methodologically sound analysis.

The argument is not for MCA to replace BCA or environmental valuation. BCA and some techniques of valuation have an established place in the economist’s toolkit and will continue to inform Australia’s resource allocation decisions. Rather, it is argued that the toolkit needs diversification to handle the complexities of evaluation when intangible outcomes are important. If it is possible to determine the non-market value of all environmental resources then BCA is sufficient. However, if we accept that valuation has limitations then we must accept alternative evaluation frameworks. This opens an opportunity for Australian agricultural and resource economists to apply and refine techniques of MCA to achieve improved resource allocation.

References

Abdalla, C.W., Roach, B.A., and Epp, D. 1992, ‘Valuing Environmental Quality Changes Using Averting Expenditures: An Application to Groundwater Contamination’, Land Economics, vol. 68(2), pp.163-169.

Abrishamchi, A., Ebrahimian, A., Tajrishi, M., Marino, M.A. 2005, ‘Case Study: Application of Multicriteria Decision Making to Urban Water Supply’, Journal of Water Resources Planning and Management , vol. 131(4), pp. 326-335.

Adamowicz, W.L. 2004, ‘What’s it worth? An examination of historical trends and future directions in environmental valuation’, The Australian Journal of Agricultural And Resource Economics, vol. 48, pp. 419-444.

Brans, J.P., Vincke, P.H, Marshal B. 1986, ‘How to select and how to rank projects: The PROMETHEE method’, European Journal of Operational Research, vol. 24, pp. 228-238.

Brouwer, R., van Ek, R., 2004, ‘Integrated ecological, economic and social impact assessment of alternative flood control policies in the Netherlands’ Ecological Economics, vol. 50, pp. 1-21.

Carson, R.T., Mitchell, R.C., Hanemann, M., Kopp, R.J., Presser, S., and Ruud, P.A. 2003. ‘Contingent Valuation and Lost Passive Use: Damages from the Exxon Valdez Oil Spill’ Environmental and Resource Economics, vol. 25, pp. 257–286.

Chen W., Hong, H., Liu, Y., Zhang, L., Hou, X., and Raymond, M. 2004, ‘Recreation demand and economic value: An application of travel cost method for Xiamen Island’ China Economic Review, vol. 15(4), pp. 398-406.

Cullen. R., Fairburn, G.A., and Hughey, K.F.D. 2001, ‘Measuring the productivity of threatened species programs’, Ecological Economics, vol. 39, pp. 53-66.

Diamond, P., and Hausman, J. 1994, ‘Contingent valuation: is some number better than no number?’ Journal of Economic Perspectives, vol. 8, pp. 45–64.

Drummond, M.F., O’Brien, B., Stoddart, G.L., and Torrance, G.W. 1997, Methods for the economic evaluation of healthcare programmes, Second edition, Oxford Medical Publications, Oxford.

Dunning, D.J., Ross, Q.E., and Merkhofer, M.W. 2000, ‘Multiattribute utility analysis for addressing Section 316(b) of the Clean Water Act’, Environmental Science and Policy, vol. 3, pp. 7-14.

Eckenrode, R.T. 1965, ‘Weighting multiple criteria’, Management Science, vol. 12(3), pp. 180-192.

Eder, G., Duckstein, L., and Nachtnebel, H.P. 1997 ‘Ranking water resource projects and evaluating criteria by multicriterion Q-analysis: an Austrian case study’, Journal of Multi-Criteria Decision Analysis, vol. 6(5), pp. 259-271.

Edwards, W., Barron, F.W. 1994, ‘SMARTS and SMARTER: Improved simple methods for multiattribute utility measurement’, Organisational Behaviour and Human Decision Processes, vol. 60, pp. 306-325.

Fernandes, L., Ridgley, M. A., and van't Hof, T. 1999, ‘Multiple criteria analysis integrates economic, ecological and social objectives for coral reef managers’, Coral Reefs, vol. 18(4), pp. 393-402.

Figueira, J., Mousseau, V., Roy, B. 2005b, ‘ELECTRE Methods’, In Figueira, J., Salvatore, G., Ehrgott, M. (Eds.), Multiple Criteria Decision Analysis: State of the art surveys, Springer, New York, pp. 133-162.

Figueira, J., Salvatore, G., Ehrgott, M. (Eds.), 2005a, Multiple Criteria Decision Analysis: State of the art surveys, Springer, New York.

French, S. 2003, ‘Background to the special issue: e -democracy and multi-criteria decision analysis’, Journal of Multi-Criteria Decision Analysis, vol. 12, pp. 63-64.

Gershon, M., and Duckstein, L. 1983, ‚Multiobjective approaches to river basin planning’, Journal of the Water Resources Planning and Management Division, vol. 109(1), pp.13-28.

Hajkowicz, S.A., McDonald, G.T., Smith, P.N. 2000, ‘An evaluation of multiple objective decision support weighting techniques in natural resource management’, Journal of Environmental Planning and Management, vol. 43(4), pp. 505-518.

Hajkowicz, S.A., Young, M.D. 2005, ‘Costing yield loss from acidity, sodicity and dryland salinity to Australian agriculture’, Land degradation and development , vol. 16, pp. 417-433.

Hayashi, K. 2000, ‘Multicriteria analysis for agricultural resource management: A critical survey and future perspectives’, European Journal of Operational Research, vol. 122, pp. 486-500.

Heilman, P., Yakowitz, D.S., Lane, L.J. 1997, ‘Targeting farms to improve water quality’, Applied mathematics and computation, vol. 83, pp. 173-194.

Howard, A. F. 1991, ‘A critical look at multiple criteria decision making techniques with reference to forestry applications’, Canadian Journal of Forest Research, vol. 21, pp. 1649-1659.

Janssen, R. 2001, ‘On the Use of Multi-Criteria Analysis in Environmental Impact Assessment in The Netherlands’, Journal of Multi-Criteria Decision Analysis, vol. 10, pp. 101-109.

Joubert, A., Stewart, T.J., and Eberhard, R. 2003, ‘Evaluation of water supply augmentation and water demand management options for the City of Cape Town’, Journal of Multi-Criteria Decision Analysis, vol. 12(1), pp. 17-25.

Joubert, A.R., Leiman, A., de Klerk, H.M., Katau, S., and Aggenbach, J.C. 1997, ‘Fynbos vegetation and the supply of water: a comparison of multi-criteria decision analysis and cost benefit analysis’, Ecological Economics, vol. 22, pp. 123–140.

Keeney, R.L., and Raiffa, H. 1976, Decisions with multiple objectives : preferences and value tradeoffs, First Edition, Cambridge University Press, London.

Keeney, R.L., and Raiffa, H. 1993, Decisions with multiple objectives : preferences and value tradeoffs, Second Edition, Cambridge University Press, London.

Ludwig, D. 2000, ‘Limitations of Economic Valuation of Ecosystems’, Ecosystems, vol. 3, pp. 31-35.

McFadden, D. 1999, ‘Rationality for Economists?’ Journal of Risk and Uncertainty, vol. 19, pp. 73-105.

Ozelkan, E.C., and Duckstein, L. 1996, ‘Analysing Water Resources Alternatives and Handling Criteria by Multi Criterion Decision Techniques’, Journal of Environmental Management, vol. 48, pp. 69-96.

Pearson, L.J., Tisdell, C., and Lisle, A.T. 2002, ‘The impact of Noosa National Park on surrounding property values: An application of the hedonic price method’, Economic Analysis and Policy, vol. 32(2), pp. 155-171.

Pohekar, S.D., Ramachandran, M. 2004, ‘Application of multi-criteria decision making to sustainable energy planning—A review’ Renewable and Sustainable Energy Reviews, vol. 8, pp. 365-381.

Prato, T. 1999, ‘Multiple attribute decision analysis for ecosystem management’, Ecological Economics, vol. 30, pp. 207-222.

Raju, S.K., Duckstein, L., and Arondel, C. 2000, ‘Multicriterion Analysis for Sustainable Water Resources Planning: A Case Study in Spain’, Water Resources Management, vol. 14, pp. 435-456.

Ribaudo, M.O., Hoag, D.L., Smith, M.E., and Heimlich, R. 2001, ‘Environmental indices and the politics of the conservation reserve program’, Ecological Indicators, vol. 1, pp. 11-20.

Roberts R., Goodwin P. 2002, ‘Weight approximations in multi-attribute decision models’, Journal of Multi-Criteria Decision Analysis, vol.11(6), pp. 291 – 303.

Romero, C., and Rehman, T. 1987 ‘Natural resource management and the use of multiple criteria decision making techniques: a review’, European Review of Agricultural Economics, vol. 14(1), pp. 61-89.

Roy, B., 1968, ‘Classement et choix en présence de points de vue multiples (la méthode ELECTRE)’, la Revue d’Informatique et de Recherche Opérationnelle (RIRO), vol. 8, pp. 57-75.

Saaty, T.L. 1987, ‘The analytic hierarchy process - what it is and how it is used’, Mathematical Modelling, vol. 9, pp. 161-176.

Sagoff, M. 1988, The economy of the earth: philosophy, law, and the environment, Cambridge, Cambridge University Press.

Spurgeon, J. 1998, ‘The Socio-Economic Costs and Benefits of Coastal Habitat Rehabilitation and Creation’, Marine Pollution Bulletin, vol. 37, pp. 373-382.

Steuer, R.E., and Na, P. 2003, ‘Multiple criteria decision making combined with finance: A categorized bibliographic study’, European Journal of Operational Research, vol. 150, pp. 496–515.

Stoneham, G., Chaudhri, V., Ha, A., Strappazzon, L. 2003, ‘Auctions for contracts: an empirical examination of Victroria’s BushTender trial’, The Australian Journal of Agricultural and Resource Economics, vol. 47(4), pp. 477-500.

Strijker D., Sijtsma, F.J., Wiersma, D. 2000, ‘Evaluation of Nature Conservation: An Application to the Dutch Ecological Network’, Environmental and Resource Economics, vol. 16, pp. 363–378.

Tecle, A. 1992, ‘Selecting a multicriterion decision making technique for watershed resources management’, Water Resources Bulletin, vol. 28(1), pp. 129-140.

Tol, R. 1996, ‘The damage costs of climate change towards a dynamic representation’, Ecological Economics, vol. 19(1), pp. 67-90.

van Bueren, M., and Bennett, J. 2004, ‘Towards the development of a transferable set of value estimates for environmental attributes’, The Australian Journal of Agricultural and Resource Economics, vol. 48(1), pp. 1-32.

von Neumann, J., and Morgenstern, O. 1944, Theory of Games and Economic Behaviour, Princeton University Press, New Jersey.

Weistroffer, H.R., Smith, C.H., and Narula, S.C. ‘MCDM Software’, in Figueira J, Salvatore G, Ehrgott M (eds), Multiple Criteria Decision Analysis: State of the art surveys, Springer, New York; 2005, pp. 989-1018.

Yakowitz, D., Lane , L., and Szidarovszky, F. 1993, ‘Multi-attribute decision making: Dominance with respect to an importance order of the attributes’, Applied Mathematics and Computation, vol. 54, pp. 167-181.

Zanakis, S.H., Solomon, A., Wishart, N., and Dublish, S. 1998, ‘Multi-attribute decision making: A simulation comparison of selected methods’, European Journal of Operational Research, vol. 107, pp. 507-529.

Zeleny, M. 1973, ‘Compromise programming’, in Cocharane, J.L., Zeleny, M. (eds.) Multiple Criteria Decision Making, University of Southern Carolina Press, Columbia, pp. 262–301.
 

***