Next Article in Journal
The Evolution of Shea Butter's "Paradox of paradoxa" and the Potential Opportunity for Information and Communication Technology (ICT) to Improve Quality, Market Access and Women's Livelihoods across Rural Africa
Previous Article in Journal
An Alternative Model to Determine the Financing Structure of PPP-Based Young Graduate Apartments in China: A Case Study of Hangzhou
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 7
Issue 5
10.3390/su7055735
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Conceptualizing the Effectiveness of Sustainability Assessment in Development Cooperation

by
Jean Hugé
1,2,*,
Nibedita Mukherjee
3,
Camille Fertel
4,
Jean-Philippe Waaub
4,5,
Thomas Block
2,
Tom Waas
2,
Nico Koedam
6 and
Farid Dahdouh-Guebas
1,6
1
Systems Ecology & Resource Management Lab, Biology Department, Université Libre de Bruxelles, Avenue Franklin Roosevelt 50, 1050 Brussels, Belgium
2
Centre for Sustainable Development, Universiteit Gent, Poel 16, 9000 Ghent, Belgium
3
Conservation Science Group, Department of Zoology, University of Cambridge, Cambridge CB2 3EJ, UK
4
Group for Research in Decision Analysis (GERAD)—HEC Montreal, Côte-Sainte-Catherine Rd. 3000, Montreal, QC H3T 2A7, Canada
5
Department of Geography, Université du Québec à Montréal (UQAM), Montreal, QC H3C 3P8, Canada
6
Plant Biology & Nature Management Lab, Biology Department, Vrije Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium
*
Author to whom correspondence should be addressed.
Sustainability 2015, 7(5), 5735-5751; https://0-doi-org.brum.beds.ac.uk/10.3390/su7055735
Submission received: 6 March 2015 / Revised: 30 April 2015 / Accepted: 5 May 2015 / Published: 8 May 2015
Versions Notes

Abstract

:
Sustainability assessment has emerged as a key decision-support process in development cooperation in response to the growing acknowledgement of the impacts of global change. This paper aims at conceptualizing the effectiveness of sustainability assessment as applied in development cooperation, by focusing on the sustainability assessment practice by actors of the official Belgian Development Cooperation. The conceptualization of the effectiveness of sustainability assessment is synthesized in a set of issues and concerns, based on semi-structured interviews. The paper highlights the specificity of sustainability assessment in the development cooperation sector (e.g., through the cultural and discursive compatibility dimensions of assessment in a North-South context). Effectiveness is inherently linked to the expected functions of sustainability assessment in the decision-making process, which include fostering organizational change, shaping contextually adapted framings of sustainability and operationalizing the sustainability transition. These findings highlight the relevance of a discourse-sensitive approach to sustainability assessment if one is to strengthen its credibility and legitimacy.

1. Introduction

There is a growing sense of urgency for integrating sustainability into development cooperation. This urgency could be attributed to the rise of environmental issues on the international political agenda, the visible and projected consequences of climate change in the South [1], and the acknowledgement of the interrelationships between environmental vulnerability, aid efficiency and poverty reduction, particularly in developing countries [2]. Sustainable development (used synonymously in this paper with sustainability, as in [3]) is defined as development that meets the needs of the present, while safeguarding the Earth’s life-support systems, on which the welfare of current and future generations depends [4]. Due to the different interpretations of sustainability, it may be challenging to translate the concept into action, although the development and use of indicators and assessment tools allows the concept to be operationalized in various settings [5].
While there is a general acceptance that sustainability should guide development cooperation [6,7], advocating the use of sustainability as a decision-making strategy requires processes that allow operationalization of sustainability for policy makers, project managers and development partners. Designing an assessment framework e.g., sustainability assessment is one such process, which may be useful in this context [8]. Sustainability assessment is not a prescribed process as yet (contrary to the codified nature of environmental impact assessment and strategic environmental assessment). It is rather an orientation of practice [9] that allows for many degrees of freedom in developing context-specific definitions, methods and applications.
The study of the effectiveness of sustainability assessment has recently gained new attention from scholars and practitioners alike [10,11,12,13,14,15,16,17]. This has been accompanied by the acknowledgement of the political nature of sustainability assessment, as well as the recognition of the expectations, framings and roles of a variety of involved actors, and the learning implications of sustainability assessment [18].
However, the effectiveness of sustainability assessment in development cooperation has rarely been looked into. Though [12] and [14] have dealt with sustainability assessment effectiveness in a comprehensive way, there is currently little research on the development co-operation context. In times when the functions of development cooperation are being reviewed [6], and when sustainability challenges (in particular climate change and inequality [1,19]) have come to dominate the international agenda, it is timely to reflect upon the effectiveness of sustainability assessment in development co-operation. This paper aims to bridge this gap and focuses squarely on sustainability assessment as a decision-guiding strategy in development co-operation [20,21]. Development cooperation is defined here as financial and/or technical aid given by governments and other agencies to support the economic, social, environmental and political development of developing countries (inspired by the definition proposed by the Organization for Economic Cooperation and Development [6]).
We aim to develop a qualitative conceptualization of the effectiveness of sustainability assessment in development cooperation. A case study approach was chosen as it has the capacity to connect practical complex events to theoretical abstractions [22], and as it acknowledges the importance of context, which is key in shaping the conceptualization of sustainability assessment effectiveness. This paper focuses on a particular example of a sustainability assessment tool, known as the “KLIMOS toolkit” as applied by the Belgian Development Co-operation between 2010 and 2013 (KLIMOS is a Dutch acronym reflecting the focus on climate & development cooperation). The KLIMOS toolkit was co-developed by academics grouped in the “KLIMOS Research Platform on Climate Change & Development Cooperation”, and officials from the Belgian Development Cooperation. The official Belgian Development Cooperation (Directorate for Development Cooperation & Humanitarian Aid of the Belgian Ministry of Foreign Affairs, Foreign Trade & Development Cooperation and the Belgian Technical Cooperation) decided to fund and supervise the development of a customized sustainability assessment tool and to pilot its application on a range of development projects. This initiative is rooted in the Development Cooperation Act (dated 19 March 2013), which stipulates in Article 11, § 2 that the Belgian Development Cooperation will integrate the environment as a cross-cutting issue in all its interventions [23]. This institutional context influenced the framing of sustainability in the KLIMOS toolkit, which could be termed as prudently reformist following [24].
This paper includes the empirical identification of concerns and needs regarding sustainability assessment effectiveness among assessment users (in our case officials from the Belgian Development Cooperation). This is the first step in the development of a set of sustainability assessment effectiveness criteria. Our empirical research was guided by three main questions. (i) How can the effectiveness of sustainability assessment in development cooperation be conceptualized based on the identification of users’ concerns and needs? (ii) What modifications are necessary in the framework of Bond et al. (2012) [12], to suit the needs of the development cooperation context? (iii) How can future sustainability assessment practice and effectiveness be improved for development cooperation?
A full analysis of the concerns and needs of other actors (including partner organizations in the South, scientists, etc.) is beyond the scope of this paper. However, we are aware of the importance of including all stakeholders’ perspectives [25] in the next steps towards a comprehensive assessment framework of sustainability assessment effectiveness. The body of the paper starts with a brief outline of sustainability assessment theory, which is then followed by the methodology, the results and the discussion. The conclusion includes recommendations on future research.

2. Sustainability Assessment and Its Effectiveness

The origins of sustainability assessment are partly found in an extension of the scope of classical environmental assessment. Hence sustainability assessment is sometimes referred to as “the third generation of impact assessment” [9]. In a North-South context, the adoption of sustainability as a key goal of human development has led to a repositioning of impact assessment, with sustainability now invariably seen as its ultimate purpose [9,10].
Sustainability assessment might be interpreted in a variety of ways depending on the context and on actors’ expectations [26,27]. Although sustainability assessment has been institutionalized in a number of jurisdictions worldwide (such as in England, Western Australia, Belgium, Switzerland and South Africa), this has happened under different names, for different purposes and with different scopes of application [8,28].
Consequently there is no clear definition of what a “good” sustainability assessment process entails despite the existence of generic process quality criteria such as the Bellagio principles [11]. This makes it important to conceptualize the effectiveness of sustainability assessment. Effectiveness refers to the question of whether something works as intended and meets the purposes for which it was designed [12]. Bond et al. (2012) [12] define four categories of sustainability assessment effectiveness, i.e., procedural, substantive, transactive and normative (see Table 1), which, along with the consideration of how pluralism and learning are addressed, provide a framework for evaluating sustainability assessment practice.
Table
Table 1. Categories of sustainability assessment effectiveness (adapted from [12]).
Table 1. Categories of sustainability assessment effectiveness (adapted from [12]).
Effectiveness CategoryKey Question
ProceduralHave appropriate processes been followed that reflect institutional and professional standards and procedures?
SubstantiveIn what ways, and to what extent, does sustainability assessment achieve its intended objectives?
TransactiveWere resources (human, financial) efficiently used in the sustainability assessment?
NormativeIn what ways, and to what extent, do the involved actors learn, improve their knowledge, modify their perspectives on sustainability and adjust their policy choices during the sustainability assessment process?
These categories reflect the diversity of functions that sustainability assessment can (sometimes simultaneously) perform in decision-making. First, sustainability assessment can be a tool and process to generate information. It is then seen as an objectifying process, providing instrumental knowledge that can underpin specific decisions [29]. Second, moving beyond mere information generation, sustainability assessment can also structure complexity by providing a systematic approach to deal with the intrinsic complexity of multidimensional societal challenges as well as with institutional complexity [29]. Third, sustainability assessment can be a forum for deliberation thereby contributing to a shared understanding of sustainability in a specific context, acknowledging the plurality of values at stake [30]. Fourth, sustainability assessment can also foster attitude shifts in decision-makers, thereby creating opportunities for policy change [29,31].
These potential functions in turn reflect the discourses shaping sustainability assessment. A sustainability assessment framework can favor particular discourses of both environmental governance and sustainability, which points to the need for sustainability assessment practitioners and users to be aware of the shifting patterns of dominant and marginalized discourses and of their associated stakeholders [27]. There has been a boom in scientific literature focusing on sustainability assessment [28], and a comprehensive review of the literature lies outside the scope of this paper. However, it is important to acknowledge the political nature of sustainability assessment [32], as well as to be aware of the diversity of expectations, approaches [33] and tools [34] used, in order to grasp the concept of sustainability assessment effectiveness.

3. Methodology

3.1. Expert Interviews

We opted for expert interviews (n = 10) as the main method to assess and conceptualize the effectiveness of sustainability assessment as used by the official Belgian Development Cooperation entities. This is due to the importance of context in understanding sustainability assessment effectiveness, especially in the pilot phase, and given the number (n = 15) of pilot applications of sustainability assessment (in our case the KLIMOS toolkit) in 2010–2013. The methodology focused on understanding rather than on statistical generalization. The term “experts” in research methodology refers to “persons responsible for development, implementation or control of solutions, strategies or policies” [35]. Experts usually have privileged access to information about groups of persons and/or decision processes and have a high level of aggregated and specific knowledge that is otherwise difficult to access [36]. The KLIMOS toolkit users are here termed as experts, having applied sustainability assessment on a range of interventions (e.g., project proposals).
The face-to-face interviews were individual. To reduce bias, we did not provide the respondents with any a priori definition of sustainability nor of sustainability assessment. Instead they were encouraged to elaborate on their own perception and interpretation of the effectiveness of the sustainability assessment(s) that they were directly or indirectly involved with.
The interviewees (n = 10) were selected based on their expertise and role in the KLIMOS Toolkit application in 2010–2013. Most were officials working for the official Belgian Development Cooperation entities, others were external academic experts who were directly involved in the development of the toolkit (see Table 2).
Table
Table 2. Overview of interviewees’ affiliations.
Table 2. Overview of interviewees’ affiliations.
Affiliation of the IntervieweeNumber of Interviewees
Directorate-General Development Cooperation (DGD)6
BTC (Belgian Technical Cooperation)1
Academic experts3
The interviews were performed between September 2013 and May 2014, in Brussels, Belgium and Montreal, Canada. The interviewees ideally had a broad understanding of the KLIMOS Toolkit context. The officials were involved in the practical application of the tool, whereas the academic experts had more of an informed outsider’s view. In order to improve the validity of the research outcomes, conclusions will be based on the inter-subjectivity of the responses [10]: the agreement or consensus between the interviewees about the conceptualization of the effectiveness of the KLIMOS Toolkit. The interviews were semi-structured, allowing for elaboration on certain topics. The basis of the questionnaire was repeatable and systematic. We performed systematizing expert interviews [37]: multiple experts were interviewed to be able to compare and aggregate data to gain process knowledge.
The interviews were semi-structured based on the framework of [12], refined in [14]. The conceptualization of substantive effectiveness in the questions is adapted from [17]. Table 3 provides the structure of the interviews (leaving out the questions pertaining to demographics and job content of the interviewee). The interviewees were encouraged to respond to the question with their particular practical sustainability assessment experience in mind, and were asked to reflect on elements and concepts that they considered being of key importance. This elaboration would result in in more detailed data that would enrich the identification and interpretation of concerns and needs, and in subsequent steps, criteria.
Table
Table 3. General structure of the interviews.
Table 3. General structure of the interviews.
Procedural effectiveness (P)P-Q1: What is the (administrative) status regulating the use of sustainability assessment (in casu: The KLIMOS toolkit) in your organization?
P-Q2: What is your appreciation of that status?
P-Q3: What is the procedure for applying sustainability assessment (in casu: The KLIMOS toolkit) in your organization?
P-Q4: What is your appreciation of that procedure?
P-Q5: What are possibilities for the future administrative status and procedure of sustainability assessment (in casu: The KLIMOS toolkit)?
Substantive effectiveness (S)S-Q1: Did decision-makers in the Belgian Development Cooperation entities consult (seek information from) the sustainability assessment (in casu: The KLIMOS toolkit screening notes and/or database) during the decision-making process (e.g., during the various stages of the project cycle)?
S-Q2: To what extent, and how, did the sustainability assessment (in casu: The KLIMOS toolkit) function as a reference during the decision-making process? Was it used to develop, review and/or discuss the project and possible alternatives?
S-Q3: Did the sustainability assessment (in casu: The KLIMOS toolkit screening note and/or database) influence subsequent decision-making or other impact assessment exercises? If so, how?
S-Q4: Did the KLIMOS Toolkit Screening/Database educate actors involved in the decision-making process about the sustainability implications of the project proposal?
S-Q5: Did the actors involved in the decision-making process alter their vision of the project proposal due to the sustainability assessment exercise (in casu: The KLIMOS toolkit)?
S-Q6: What was the most important contribution of the sustainability assessment (in casu: The KLIMOS toolkit) to the decision-making process (e.g., regarding project proposals—In particular or generically)?
S-Q7: Was the content of the final project proposal modified due to the application of the sustainability assessment (in casu: The KLIMOS Toolkit)?
S-Q8: If yes, how was the project proposal modified, and what are the possible indications pointing to a causal relationship between the sustainability assessment (in casu: KLIMOS Toolkit) and the modification of the project proposal?
S-Q9: When you have indicated that the project proposal has been altered because of the SA (in casu: The KLIMOS toolkit) how significant is this modification?
Transactive effectiveness (T)T-Q1: Has acquiring the requisite skills and training the necessary staff for implementing the sustainability assessment (in casu: The KLIMOS toolkit) constituted a big burden or not?
T-Q2: Were responsibilities clearly defined and allocated and were tasks undertaken by the most appropriate people?
T-Q3: Could you elaborate on the time and financial resources used to realize the application of the sustainability assessment (in casu: The KLIMOS toolkit)?
Normative effectiveness (N)N-Q1: Did the application of the sustainability assessment (in casu: The KLIMOS toolkit):
  • Reverse prevailing (unsustainable) trends?
  • Integrate key intertwined factors affecting sustainability?
  • Minimize non-transparent tradeoffs?
N-Q2: What do you think of the framing/interpretation of sustainability in the sustainability assessment (in casu: The KLIMOS Toolkit).
General question (G)G-Q1: Do you have any additional remarks, suggestions concerning the effectiveness of sustainability assessment?

3.2. Content Analysis

In order to complement the data gathered through the interviews, draft and final project documents as well as completed KLIMOS toolkit “screening notes” were analyzed qualitatively, mainly to identify indications for substantive effectiveness such as possible modifications in project content after the application of the sustainability assessment. It is less straightforward to identify indicators that allow the assessment of procedural, substantive, transactive or normative effectiveness in project documents. When available, project documents and screening notes were assessed from individual archives provided by the respondents.
Synthesizing the interview data into a set of concerns and needs inevitably involves a degree of interpretation. In order to ensure the transparency and traceability of the interpretation by the interdisciplinary team of authors, all interviews were transcribed and subsequently coded for recurrent terms and ideas along the pre-established categories of effectiveness (see Section 1). A second qualitative analysis of the transcripts was performed without the pre-set categories in mind.

4. Results

The interviews yielded insights that allowed the synthesis of 27 concerns and needs regarding sustainability assessment effectiveness in a development cooperation context (Table 4).
Table
Table 4. Concerns and needs regarding the effectiveness of sustainability assessment applied in development cooperation.
Table 4. Concerns and needs regarding the effectiveness of sustainability assessment applied in development cooperation.
Categories of EffectivenessConcerns and NeedsSelection of Illustrative Respondent Statements
Concerns and needs w.r.t. procedural effectivenessP1: Adaptability: The sustainability assessment procedure is flexible and adaptable, ideally modular, to fit requirements and structures of various organizations and actors.“Any sustainability assessment tool should acknowledge the need of the variety of users who are expected to use it (e.g., technical staff and desk officers).”
P2: Clarity of purpose: Is the sustainability assessment aimed at decision-support or at evaluation?“Failing to clarify our main purpose will make it difficult to convince our partners (such as non-governmental organizations) to collaborate in completing a sustainability assessment. What’s in it for them?”
P3: Timing: Adequate timing of the sustainability assessment application“The sustainability assessment should start in an early stage of the decision-making process if we really aim at influencing decisions.”
P4: Training: Adequate training, acknowledging the multiple backgrounds of sustainability assessment implementers and staff“Not everyone in-house has the skills (nor the time) to perform the sustainability assessment.”
P5: Internal tiering: Vertical integration of sustainability assessment with other processes, templates and databases at different decision-making levels within the organization“Sustainability assessment should not happen in an institutional vacuum.” “We should harmonize our ex ante tests, as we also have a gender tool, which should be linked somehow to sustainability assessment, maybe as a module?”
P6: External tiering: Vertical integration of sustainability assessment with other processes, templates and databases at different decision-making levels in the partner organizations/partner countries“The sustainability assessment process and outcomes should be linked with the international standards of the OECD-Development Assistance Committee, such as the “Rio markers”, which indicate if an cooperation initiative targets environmental objectives.”
P7: Horizontal integration: Link between ex ante assessment and ex post evaluation mechanisms“Closing the circle by comparing the predictions of the sustainability assessment with the actual results on the ground would allow us to learn from experience.”
Concerns and needs w.r.t. substantive effectivenessS1: User-friendliness: Easily understood phrasing, clear outline of responsibilities, modular and/or sector-specific approach, adaptability of the SA procedure to the users’ profiles“If we, users, do not understand the question or the phrasing, the exercise can never have a real impact”
S2: Up-to-date information: Regular update of databases and supporting information in the sustainability assessment process“The problem when performing ex ante assessments is that we lack baseline data.”
S3: Cascading system: Input and process document evolve throughout the project cycle“The (sustainability assessment/KLIMOS toolkit) questionnaire should be ‘alive’, i.e., adaptable in every stage of the project cycle (from identification to monitoring etc.). This would make it more relevant, and everyone would see the instrument as—Partly—Its own.”
S4: Adaptation to the project cycle: Sustainability assessment stages linked to the organizational flowchart, as well as to monitoring schemes“Ideally, the questions should be related to the stages of the project cycle, which often have their own appointed official per project.”
S5: Quantitative target setting, indicators and monitoring in the sustainability assessment“Without measurable data, how will we convince anyone of the use of this exercise?”
S6: Operationalization of sustainability: The sustainability assessment reflects the multidimensionality of sustainability; sustainability is linked to local concerns in the South (“green” and “brown” environmental issues)“Sustainability does not mean the same to everyone here in our department, let alone when one talks to people in our partner countries. Any sustainability assessment should acknowledge this fact.”
S7: Incorporation of changes: Into project proposals (link between the assessment and the content of the proposal), possibly linked to a sustainability/environmental action plan, reflecting the ability of sustainability assessment to guide trade offs“When the contents of your project proposal have been modified (and became more sustainable), then your sustainability assessment has been successful.”
S8: Enforcement of the outcomes of sustainability assessment: e.g., link sustainability assessment outcomes with financial consequences“We should not only make sustainability assessment compulsory, we should make a satisfactory outcome a condition for funding (at least in the long run).”
S9: Transparency: Visibility of the consideration of the sustainability assessment outcome in the decision-making process“Everyone should know we are using this tool, it is good for our accountability and reputation.”
S10: Context awareness: Sustainability assessment linked to the broader policy dialogue (among development partners)“The application of such tools should be linked to our environmental strategy and to the priorities of our partners in the South. The toolkit should not be isolated.”
Concerns and needs w.r.t. transactive effectivenessT1: Available time (to perform and appropriate sustainability assessment)“We do not have the time to perform good sustainability assessments.”
T2: Available and adapted manpower: Staff is granted time to perform SA and provided with the necessary training and background knowledge“We need to be trained to be able to apply the KLIMOS toolkit. It is too difficult to apply without prior proper guidance.”
T3: Communication: Between sections of the organizations, between actors/teams working on the SA, between development partners“We should discuss what we do (with the KLIMOS toolkit) with our colleagues.”
T4: Helpdesk: Providing timely ad hoc support during the sustainability assessment process“A helpdesk would allow us to ask specific questions to experts on short notice.”
T5: Clear outline of responsibilities: Division of roles for all actors, in all involved partner organizations“Who is responsible for performing the assessment, who will enforce its outcomes? Will I get feedback from the other departments?”
Concerns and needs w.r.t. normative effectivenessN1: Organizational alignment: Alignment of the sustainability assessment objectives and structures with the organizational objectives and structure“Does the KLIMOS toolkit effectively translate our commitment towards sustainability? Is it enough? What’s it position in our strategy?”
N2: Discursive alignment: Alignment of the sustainability assessment with dominant discourses/framings shaping the organization“As sustainability is an ‘open’ concept, can and should we steer our partners towards adopting ‘our’ framing of sustainability? If not, how can we achieve some sort of a consensus interpretation?”
N3: Learning: Embeddedness of sustainability assessment within broader learning and capacity building efforts towards sustainability“The KLIMOS toolkit trainings were useful, but why were these not linked to our other in-house training sessions?”
N4: Culture: Cultural sensitivity/adaptation of the sustainability assessment tool—and cultural compatibility of the sustainability assessment approach in a North-South context“I am not sure if our development partners will welcome yet another extra requirement, although I believe sustainability is important for them too, they see it differently. We are not talking about the same issues.”
N5: Transition: The sustainability assessment leads to a mindset shift, is creating momentum, is contributing to action and fostering policy coherence“The long-term impact of applying the KLIMOS toolkit should go beyond the project level, we should change our attitudes.”
Procedural effectiveness concerns were centered on the adaptability and flexibility of the sustainability assessment procedure. Issues of timing, training and integration with intra- and extra-organizational decision-making processes were highlighted, as well as differences in expectations and capacity between technical field staff and office staff.
Ten concerns and needs regarding substantive effectiveness were synthesized from the interview transcripts (see Table 4). Substantive effectiveness answers the question: “did the sustainability assessment lead to tangible change?” (in a project proposal and/or “on the ground”). We focus here on concerns that were not necessarily shared by everyone, but which point out interesting views on the purpose of sustainability assessment. Enforcing the outcomes of the sustainability assessment, i.e., ensuring that the proposed changes are realized, is included. This is because some respondents felt that a non-compulsory sustainability assessment (such as the current KLIMOS toolkit) will otherwise not be substantively effective. This raises an important question regarding the function of sustainability assessment and points at the characteristics of decision-making itself. There is, however, no linear relationship between sustainability assessment outcomes and the ultimate decision (e.g., on funding a project proposal or not, based on the project’s expected sustainability implications).
The transactive effectiveness concerns include time, skill, financial resources and specification of roles. The respondents emphasized this last point. This also led to the identification of an additional concern on communication among the actors implementing the sustainability assessment throughout the various phases of the project cycle in the different organizations involved. This focus on exchanging experiences stresses the importance of sustainability assessment-induced learning.
The questions aimed at appreciating normative effectiveness were intentionally broad and triggered wide-ranging responses yet were also considered by some respondents as a little overwhelming. The normative criteria proposed in Table 4 go beyond the generic criteria, and include both organizational, discursive and cultural alignment, all aspects that can are considered of paramount importance in a cross-cultural international cooperation environment. The issue of attribution was especially difficult to solve: i.e., ascertaining whether a particular (re)action was actually caused by the introduction and/or application of sustainability assessment or whether it was the result of a more general momentum towards sustainability. This was especially apparent with regard to the “reversal of unsustainable trends” question. Some interviewees stated that the very existence of a sustainability assessment process was in itself a quality label for the organization, airing the idea that realizing a sustainability assessment is a good thing to do regardless of the quality of its implementation.
The learning dimension of the application of sustainability assessment was highlighted by most respondents. It was referred to e.g., as “mindset creation” and includes both individual and organizational learning, confirming [12,28]. Individual learning was experienced by a range of respondents, yet was presented as necessarily related to a broader—formal—sustainability learning effort (e.g., through staff training sessions) that could not be attributed to nor provided solely by sustainability assessment. The respondents acknowledge that learning is a key mechanism in improving practice over time, as also stated by [12].
The need for regular iteration of sustainability assessment application was stressed (as people in the administration often change positions, as the political support for sustainability (assessment) might shift, etc.), reflecting a genuine concern for the long-term anchoring of the sustainability assessment process in the organization(s). Most respondents did not elaborate further on their preferences regarding the prescriptive role of sustainability assessment (i.e., what it ought to realize at the organizational level).
The alignment of the purpose and the structure of a sustainability assessment process with the dominant sustainability discourses or framings of the organization (in our case the official Belgian Development Cooperation) was considered key by the respondents. This reflects the importance of perceived compatibility between the organizational culture and the way sustainability assessment-triggered changes are framed. Beyond discourses and policies, the respondents acknowledged the importance of a certain alignment with the cultural references of the partner countries in the South, when dealing with sustainability. This aspect was considered key to realizing normative effectiveness.
While the use of the four effectiveness categories was helpful in structuring the interviews and in analyzing the outcomes, logically the interviewees did experience effectiveness as a multidimensional concept within which the category boundaries were not absolute. This is particularly important given that the flexibility of the sustainability assessment exercise was seen as key by the respondents in discussing procedural effectiveness. At the same time, the potentially contradictory requirement of clarity of purpose was emphasized. Whether the sustainability assessment is presented and used as a process to support decision-making or as an evaluative process (even with possible financial (funding) consequences) makes a considerable difference in its perception (e.g., by actors dependent on development cooperation funds as mentioned by three respondents) as well as in the required type of procedure. Some concerns and needs were characterized as being relevant for various categories of effectiveness (e.g., adequate timing; clear specification of roles) indicating the cross cutting character of some of the concerns, as well as the somewhat artificial subdivision of multidimensional sustainability assessment effectiveness. The porous boundaries between the subdivisions of effectiveness also reflect the linkages between a sound and efficient procedure and a substantively and normatively effective assessment process. In the same vein, some sustainability assessment effectiveness concerns were challenging to categorize. These include linking sustainability assessment to the broader North-South policy dialogue on the changing conceptualization of development cooperation (although this issue was mentioned by two respondents as an element of substantive effectiveness, it embodies a normative effectiveness element as well).

5. Discussion

The concerns and needs regarding sustainability effectiveness in development cooperation proposed in this paper are a useful addition to the generic effectiveness criteria in the existing literature. When comparing the identified concerns and needs regarding sustainability assessment effectiveness in development cooperation (Table 4) with the set of generic criteria proposed by [12] and [14], it is apparent that there is a lack of an explicit mention of stakeholder involvement in Table 4.
While the transactive criteria are broadly similar, although with a stronger focus on communication in the development cooperation sector, there are some noteworthy differences between the generic substantive criteria and the concerns and needs proposed in Table 4. While the generic criteria of [14] stress public consultation, this is left unmentioned by our respondents. They instead focus more on the alignment between the sustainability assessment approach and the needs of the organization and its partners, without making public consultation explicit, unless one considers transparency to entail consultation and participation requirements. The dominant concern issued by the interviewed development cooperation practitioners, which is mostly ignored in the generic criteria, concerns the required adaptability and flexibility of sustainability assessment process: An adapted procedure that matches the needs of different actors (e.g., users in North and South, officials and non-governmental partners, technical staff and policy-makers), as well an adaptation to the project cycle and openness with regard to various sustainability framings.
We would also like to highlight that the elements set out in Table 4 should be interpreted as “key concerns and needs related to the effectiveness of sustainability assessment in development cooperation, as proposed by the users of a sustainability assessment tool”. The proposed concerns can be interpreted from a prescriptive perspective. If so, they embody the ideal characteristics of an effective sustainability assessment and they contribute to answer the question: “What should an effective sustainability assessment look like?” But the proposed concerns can equally be interpreted from a descriptive perspective, and can hence contribute to an—ex ante or ex post—evaluation of the effectiveness of a specific sustainability assessment exercise. In that case the concerns contribute to answer the question: “How effective was/will be this specific sustainability assessment exercise?”
Statements such as “sustainability assessment contributes to the translation of sustainability into understandable concepts and actions” and “sustainability assessment should contribute to action and reinforce policy coherence” hint at the perceived role of sustainability assessment in organizational learning. Sustainability assessment can indeed contribute to steering organizations towards sustainability, through a process whereby aligned individuals and groups congruently learn and modify their mental models (cf. the abovementioned “mindset creation”) and actions [38]. These phrasings are reminiscent of sustainability transition thinking, which entails a change from one state of a system to another and is described as a multi-actor, multi-level and multi-phase movement spanning several decades [39]. While the Belgian Development Cooperation has not officially taken up sustainability transition thinking, the concept is increasingly used in Belgian official documents [40]. Transition as an ultimate aim of sustainability assessment was mentioned by a few respondents, although there are few empirical foundations that link the gradualism of sustainability assessment to the transformative approach of transition experiments [41].
The stated purpose of sustainability assessment may influence the perceived and real “ownership” of the process in a development cooperation context [42], and resonates with the more general question on the purpose of sustainability assessment effectiveness criteria which can be prescriptive or descriptive. Decision-making is a messy process subject to many influences [30], and the information generated by a sustainability assessment is only one of many factors influencing the final decision. Sustainability assessment is a decision-supporting exercise, which has no bearing on the final decision. Furthermore, a sustainability assessment will only have the potential to generate change if its message is considered legitimate and necessary by policy-makers, as well as by other societal actors), and if other influencing factors are taken into account. Huston (2008) [43] cites ideology, interests and institutional context among other decision-influencing factors.
Without delving deeper into the functions of sustainability assessment in decision-making (for which we refer the reader to [30]) it is key to consider the enforcement concern with caution. Enhancing the sustainability assessment process with regard to transparency and accountability may already be a first step, as suggested by the respondents.
Few of the interviewed sustainability assessment users had major remarks on the current framing of sustainability in the sustainability assessment, although they were aware of the multi-interpretability of the concept. This situation did not necessarily reflect a lack of sensitivity to other sustainability discourses. This is because discursive compatibility between North and South was mentioned as a concern, reflecting the realization of different worldviews exhibited by Northern and Southern development partners [44].
The differences in sustainability framing surfaced in particular when dealing with tradeoffs and prioritization of “brown” versus “green” sustainability issues. While brown development projects typically address urgent and tangible local sustainability issues (e.g., clean water, environmental health, climate change adaptation), “green” development projects address regional or global sustainability issues focusing on global public goods (biodiversity, climate change mitigation) [45]. Sustainability assessment is ideally expected to provide guidance in balancing these two legitimate approaches towards sustainability in development cooperation. Furthermore, the dominant perspective stating that sustainability assessment should not be isolated from the broader policy dialogue on development and environment issues, pointed towards the importance of policy coherence. The ongoing international climate negotiations, the Sustainable Development Goals and the momentum that these events create, were mentioned by the respondents as being intrinsically linked to small-scale initiatives such as the KLIMOS toolkit sustainability assessment.

6. Conclusions and Recommendations

The dense qualitative data generated through the interviews allowed us to propose a set of 27 concerns and needs regarding sustainability assessment effectiveness. Further research will allow the concerns to be synthesized into criteria, and will require the involvement of other involved actors beyond the direct users of sustainability assessment (e.g., non-governmental actors in North and South). Based on the gathered interview data, the generic effectiveness framework of Bond et al. (2012) [12] is complemented and customized for development co-operation by focusing on inter-cultural compatibility and on communication between the various involved actors in the assessment process, as well as by a focus on linkages between the various stages of the project cycle.
Together the concerns and needs related to procedural and transactive effectiveness (embodying organizational change), and the concerns and needs related to substantive and normative effectiveness (embodying the framing and the realization of sustainability) provide a sustainability assessment framework which is of direct practical use for both descriptive and normative purposes.
In conceptualizing the effectiveness of sustainability assessment in development cooperation, the respondents stressed the fact that it is critical to focus on an adaptable and flexible sustainability assessment procedure, including alignment with organizational objectives and including openness with regard to a variety of interpretations of sustainability. This awareness of a plurality of sustainability interpretations as well as of a wide range of expectations regarding sustainability assessment is present among the respondents, and procedural flexibility is seen as a strategy to deal with this. This apparently contrasts with the codified nature of “classical” impact assessment processes such as environmental impact assessment (EIA) and strategic environmental assessment (SEA) which are as a rule “stricter” when it comes to addressing trade-offs, pointing to the risk of overly flexible sustainability assessments as stressed by [46].
While the empirical findings of our research contribute to a better understanding of the effectiveness of sustainability assessment in development cooperation, future research is needed on the potential and actual effectiveness of sustainability assessment in development cooperation in:
  • fostering organizational change towards sustainability. While organizational change is determined by internal motivation, organizational capacity and external context, the introduction and application of sustainability assessment is influenced by and influences all these factors. Through its focus on learning, sustainability assessment can contribute to gradual organizational change, yet empirical evidence on this topic in development cooperation organizations (bilateral and multilateral agencies, non-governmental organizations, etc.) is lacking.
  • framing sustainability in a development cooperation context. By translating sustainability into manageable bits of information (including indicators), sustainability assessment can determine how sustainability is understood, framed and realized in the multi-actor and inter-cultural context of development cooperation. Setting sustainability goals operationalizing and contextualizing a sustainability vision is a key step in translating that framing into manageable decision-supporting information as proposed by [26]. Sustainability assessment can also contribute to discourse reflection and formation [47]. Stimulating decision-makers and other stakeholders to critically reflect upon and to reconsider their perspectives on sustainability may enhance the chance of sustainability assessment to influencing decision-making. However, here again, systematic empirical evidence on the actual experience of sustainability assessment-mediated sustainability framing in development cooperation is mostly absent.
  • research on the role of sustainability assessment in realizing the sustainability transition through changing normative principles and through their operationalization on the ground is needed.
The capacity of sustainability to capture the major challenges that development cooperation aims to address at its modest level (such as widespread poverty, climate change, resource depletion, etc.) by providing a decision-support framework will need to be continually improved and fine-tuned. Research on the functions and on the effectiveness of sustainability assessment will be instrumental in realizing this.
The process of decision-making, including decision-support processes such as sustainability assessment, ultimately determines a decision’s legitimacy and credibility. While sustainability assessment might be instrumental to boosting the legitimacy of development cooperation by explicitly linking it to the broadly shared agenda of sustainability in times of global change, the data gathered indicate that the interviewed users of sustainability assessment in development cooperation have a deeper, more ambitious understanding of its potential. The next step following this research project will entail the translation of the identified concerns and needs related to sustainability assessment effectiveness into synthetic criteria that can be applied on practical sustainability assessment exercises.

Acknowledgments

Jean Hugé acknowledges the support of the Belgian National Research Foundation FRS-FNRS. Nibedita Mukherjee was generously supported by the Fondation Wiener Anspach post-doctoral fellowship. Tom Waas acknowledges the support of the ACROPOLIS funds of VLIR-ARES.

Author Contributions

Jean Hugé conceived, designed and performed the interviews. Jean Hugé, Nibedita Mukherjee, Tom Waas, Camille Fertel and Jean-Philippe Waaub analyzed the data. Thomas Block, Nico Koedam and Farid Dahdouh-Guebas provided suggestions and feedback on the conceptualization of sustainability, on the interpretation of the data, and on the required context-setting. Jean Hugé and Nibedita Mukherjee wrote the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. IPCC 2014. Fifth Assessment Report. Intergovernmental Panel on Climate Change. Available online: http://www.ipcc.ch/report/ar5/syr/ (accessed on 17 November 2014).
  2. Gupta, J. Climate change and development cooperation: Trends and questions. Curr. Opin. Environ. Sustain. 2009, 1, 207–213. [Google Scholar] [CrossRef]
  3. Dresner, S. The Principles of Sustainability; Earthscan: London, UK, 2008. [Google Scholar]
  4. Griggs, D.; Stafford-Smith, M.; Gaffney, O.; Rockström, J.; Öhman, M.C.; Shyamsundar, P.; Steffen, W.; Glaser, G.; Kanie, N.; Noble, I. Sustainable development goals for people and planet. Nature 2013, 495, 305–307. [Google Scholar] [CrossRef] [PubMed]
  5. Waas, T.; Hugé, J.; Block, T.; Wright, T.; Benitez-Capistros, F.; Verbruggen, A. Sustainability assessment and indicators: Tools in a decision-making strategy for sustainable development. Sustainability 2014, 6, 5512–5534. [Google Scholar] [CrossRef] [Green Version]
  6. OECD. Development Cooperation Report 2014 “Mobilising Resources for Sustainable Development”. Available online: http://www.plan.be/publications/publication_det.php?lang=fr&KeyPub=623 (accessed on 11 December 2014).
  7. UN. United Nations. Sustainable Development Knowledge Platform. Post 2015. Available online: http://sustainabledevelopment.un.org/post2015.html (accessed on 10 February 2014).
  8. Hugé, J.; Waas, T. Converging impact assessment discourses for sustainable development: The case of Flanders, Belgium. Environ. Dev. Sustain. 2011, 13, 607–626. [Google Scholar] [CrossRef]
  9. Pope, J.; Dalal-Clayton, B. From SEA to sustainability assessment? In Handbook of Strategic Environmental Assessment; Sadler, B., Dusik, J., Fischer, T., Partidario, M., Verheem, R., Aschemann, R., Eds.; EarthScan: London, UK, 2011. [Google Scholar]
  10. Cashmore, M.; Bond, A.; Cobb, D. The role and functioning of environmental assessment: Theoretical reflections upon an empirical investigation of causation. J. Environ. Manag. 2008, 88, 1233–1248. [Google Scholar] [CrossRef]
  11. Pintér, L.; Hardi, P.; Martinuzzi, A.; Hall, J. Bellagio STAMP: Principles for sustainability assessment and measurement. Ecol. Indic. 2012, 17, 20–28. [Google Scholar] [CrossRef]
  12. Bond, A.; Morrisson-Saunders, A.; Howitt, R. Sustainability Assessment—Pluralism, Practice and Progress; The Natural and Built Environment Series; Routledge: London, UK, 2012. [Google Scholar]
  13. Cashmore, M.; Gwilliam, R.; Morgan, R.; Cobb, D.; Bond, A. The interminable issue of effectiveness: Substantive purposes, outcomes and research challenges in the advancement of environmental impact assessment theory. Impact Assess. Proj. Apprais. 2004, 22, 295–310. [Google Scholar] [CrossRef]
  14. Chanchitpricha, C.; Bond, A. Conceptualizing the effectiveness of impact assessment processes. Environ. Impact Assess. Rev. 2013, 43, 65–72. [Google Scholar] [CrossRef]
  15. Morrison-Saunders, A.; Retief, F. Walking the sustainability assessment talk—Progressing the practice of environmental impact assessment (EIA). Environ. Impact Assess. Rev. 2012, 36, 34–41. [Google Scholar] [CrossRef]
  16. Rozema, J.G.; Bond, A.J. Framing effectiveness in impact assessment: Discourse accommodation in controversial infrastructure development. Environ. Impact Assess. Rev. 2015, 50, 66–73. [Google Scholar] [CrossRef]
  17. Van Doren, D.; Driessen, P.P.J.; Schijf, B.; Runhaar, H.A.C. Evaluating the substantive effectiveness of SEA: Towards a better understanding. Environ. Impact Assess. Rev. 2013, 38, 120–130. [Google Scholar] [CrossRef]
  18. Morgan, R.K.; Hart, A.; Freeman, C.; Coutts, B.; Colwill, D.; Hughes, A. Practitioners, professional cultures, and perceptions of impact assessment. Environ. Impact Assess. Rev. 2012, 32, 11–24. [Google Scholar] [CrossRef]
  19. UNDP 2014. Humanity Divided: Confronting Inequality in Developing Countries; Final Report; United Nations Development Programme: New York, NY, USA, 2014. [Google Scholar]
  20. Bond, A.; Pope, J. The state of the art of impact assessment. Impact Assess. Proj. Apprais. 2012, 30, 1–4. [Google Scholar] [CrossRef]
  21. Hugé, J.; Waas, T.; Dahdouh-Guebas, F.; Koedam, N.; Block, T. A discourse-analytical perspective on sustainability assessment: Interpreting sustainable development in practice. Sustain. Sci. 2013, 8, 187–198. [Google Scholar] [CrossRef]
  22. Flyvbjerg, B. Case study. In The Sage Handbook of Qualitative Research; Denzin, N., Lincoln, Y., Eds.; Sage: Thousand Oaks, CA, USA, 2011. [Google Scholar]
  23. Moniteur Belge. Law w.r.t. development cooperation (C/2013/15084). 12 April 2013; (In French and Dutch). [Google Scholar]
  24. Hopwood, W.; Mellor, M.; O’Brien, G. Sustainable development: Mapping different approaches. Sustain. Dev. 2005, 13, 38–52. [Google Scholar] [CrossRef]
  25. Lahdelma, R.; Salminen, P.; Hokkanen, J. Using multicriteria methods in environmental planning and management. Environ. Manag. 2000, 26, 595–605. [Google Scholar] [CrossRef]
  26. Morrison-Saunders, A.; Pope, J. Conceptualising and managing trade-offs in sustainability assessment. Environ. Impact Assess. Rev. 2013, 38, 54–63. [Google Scholar] [CrossRef]
  27. Bond, A.; Morrison-Saunders, A. Sustainability assessment: Jack of all trades, master of none? Impact Assess. Proj. Apprais. 2009, 27, 321–329. [Google Scholar] [CrossRef]
  28. Bond, A.; Morrison-Saunders, A.; Pope, J. The state of the art of sustainability assessment. Impact Assess. Proj. Apprais. 2012, 30, 53–62. [Google Scholar] [CrossRef]
  29. Hugé, J.; Waas, T.; Eggermont, G.; Verbruggen, A. Impact assessment for a sustainable energy future—Reflections and practical experiences. Energy Policy 2011, 39, 6243–6253. [Google Scholar] [CrossRef]
  30. Hertin, J.; Turnpenny, J.; Jordan, A.; Nilsson, M.; Russell, D.; Nkyvist, B. Rationalising the policy mess? Ex ante policy assessment and the utilization of knowledge in the policy process. Environ. Plan. A 2009, 41, 1185–1200. [Google Scholar]
  31. Nooteboom, S. Impact assessment procedures for sustainable development: A complexity theory perspective. Environ. Impact Assess. Rev. 2007, 22, 3–16. [Google Scholar]
  32. Bond, A.; Morrison-Saunders, A. Re-evaluating sustainability assessment: Aligning the vision and the practice. Environ. Impact Assess. Rev. 2011, 31, 1–7. [Google Scholar] [CrossRef]
  33. Soderbaum, P. Issues of paradigm, ideology and democracy in sustainability assessment. Ecol. Econ. 2007, 60, 613–626. [Google Scholar] [CrossRef]
  34. Ness, B.; Urbel-Piirsalu, E.; Anderberg, S.; Olsson, L. Categorising tools for sustainability assessment. Ecol. Econ. 2007, 60, 498–508. [Google Scholar] [CrossRef]
  35. Burgman, M.A.; McBride, M.; Ashton, R.; Speirs-Bridge, A.; Flander, L.; Wintle, B.; Fidler, F.; Rumpff, L.; Twardy, C. Expert status and performance. PLoS ONE 2011, 6, e22998. [Google Scholar] [CrossRef] [PubMed]
  36. Ericsson, K.A.; Charness, N.; Feltovitch, P.; Hoffman, R.R. The Cambridge Handbook of Expertise and Expert Performance; Cambridge University Press: Cambridge, UK, 2006. [Google Scholar]
  37. Bogner, A.; Littig, B.; Menz, W. Interviewing Experts; Palgrave MacMillan: New York, NY, USA, 2009. [Google Scholar]
  38. Lozano, R. Developing collaborative and sustainable organisations. J. Clean. Prod. 2008, 16, 499–509. [Google Scholar] [CrossRef]
  39. Tukker, A.; Butter, M. Governance of sustainable transitions: About the 4(0) ways to change the world. J. Clean. Prod. 2007, 15, 94–103. [Google Scholar] [CrossRef]
  40. Belgian Federal Planning Bureau. Enhancing the Transition towards Sustainable Development; Federal Report Sustainable Development; Belgian Federal Planning Bureau: Brussels, Belgium, 2007. [Google Scholar]
  41. Whitmarsh, L.; Nykvist, B. Integrated Sustainability Assessment of mobility transitions: Stimulating stakeholders’ visions of and pathways to sustainable land-based mobility. Int. J. Innov. Sustain. Dev. 2008, 3, 115–127. [Google Scholar] [CrossRef]
  42. Axelsson, A.; Annandale, D.; Cashmore, M.; Slunge, S.; Ekbom, A.; Loayza, F.; Verheem, R. Policy SEA: Lessons from development cooperation. Impact Assess. Proj. Apprais. 2012, 30, 124–129. [Google Scholar] [CrossRef]
  43. Huston, A.C. From research to policy and back. Child Dev. 2008, 79, 1–12. [Google Scholar] [CrossRef] [PubMed]
  44. Van Opstal, M.; Hugé, J. Knowledge for sustainable development: A worldviews perspective. Environ. Dev. Sustain. 2013, 5, 687–709. [Google Scholar] [CrossRef]
  45. Hicks, R.L.; Parks, B.C.; Timmons Roberts, J.; Roberts, M.J. Greening Aid? Understanding the Environmental Impact of Development Assistance; Oxford University Press: New York, NY, USA, 2010. [Google Scholar]
  46. Morrison-Saunders, A.; Fischer, T. What is wrong with EIA and SEA anyway? A sceptic’s perspective on sustainability assessment. J. Environ. Assess. Policy Manag. 2006, 8, 19–39. [Google Scholar] [CrossRef]
  47. Runhaar, H.; Runhaar, P.R.; Oegema, T. Food for thought: Conditions for discourse reflection in the light of environmental assessment. Environ. Impact Assess. Rev. 2010, 30, 339–346. [Google Scholar] [CrossRef]

Share and Cite

MDPI and ACS Style

Hugé, J.; Mukherjee, N.; Fertel, C.; Waaub, J.-P.; Block, T.; Waas, T.; Koedam, N.; Dahdouh-Guebas, F. Conceptualizing the Effectiveness of Sustainability Assessment in Development Cooperation. Sustainability 2015, 7, 5735-5751. https://0-doi-org.brum.beds.ac.uk/10.3390/su7055735

AMA Style

Hugé J, Mukherjee N, Fertel C, Waaub J-P, Block T, Waas T, Koedam N, Dahdouh-Guebas F. Conceptualizing the Effectiveness of Sustainability Assessment in Development Cooperation. Sustainability. 2015; 7(5):5735-5751. https://0-doi-org.brum.beds.ac.uk/10.3390/su7055735

Chicago/Turabian Style

Hugé, Jean, Nibedita Mukherjee, Camille Fertel, Jean-Philippe Waaub, Thomas Block, Tom Waas, Nico Koedam, and Farid Dahdouh-Guebas. 2015. "Conceptualizing the Effectiveness of Sustainability Assessment in Development Cooperation" Sustainability 7, no. 5: 5735-5751. https://0-doi-org.brum.beds.ac.uk/10.3390/su7055735

Article Metrics

Back to TopTop