Next Article in Journal
Social Network and Place: The Inheritance and Development of Beijing Crosstalk Performing Art
Next Article in Special Issue
Sustainability Indicators Past and Present: What Next?
Previous Article in Journal
Cross-European Perspective in Social Work Education: A Good Blended Learning Model of Practice
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Global SDGs Assessments: Helping or Confusing Indicators?

1
Faculty of Science, Charles University, Albertov 6, 128 43 Prague, Czech Republic
2
Environment Center, Charles University, José Martího 2/407, 162 00 Prague, Czech Republic
*
Author to whom correspondence should be addressed.
Sustainability 2018, 10(5), 1540; https://0-doi-org.brum.beds.ac.uk/10.3390/su10051540
Submission received: 31 March 2018 / Revised: 2 May 2018 / Accepted: 9 May 2018 / Published: 12 May 2018

Abstract

:
On 1 January 2016, the 17 Sustainable Development Goals (SDGs) of the 2030 Agenda for Sustainable Development—adopted by world leaders in 2015—came into force. They build on the Millennium Development Goals (MDGs) and call for action by all countries to promote prosperity while protecting the planet. Since the SDGs are not legally binding, governments are expected to take ownership and establish national frameworks for the achievement of the 17 Goals. Countries thus have the primary responsibility for follow-up and review of the progress made in implementing the Goals, which will require quality, accessible and timely data collection. This will be instrumental for both regional and global follow-up analyses and assessments—several such major global assessments have already appeared. It might be supposed that the SDGs framework, including indicators, is conceptually and methodologically well-designed and tested in order to function reliably and provide guidance for such assessments. However, while it seems that the current structure of the SDGs has provided a firm policy framework, the Goals and targets have been mostly operationalized by indicators. We demonstrate and argue that without a procedurally well-designed, conceptual indicator framework for selecting and/or designing indicators, the results of SDGs assessments may be ambiguous and confusing.

1. Introduction

After thirty years of sustainable development summits, action plans, and reports, the major trends in the planetary environment, social equity, and economic sustainability are still going in the wrong direction, due to slow implementation, lack of public involvement, and inadequate information on progress. Indicators play an instrumental role in making the concept of sustainable development (SD) appealing to a wide spectrum of potential stakeholders as well as in assessing the progress [1,2,3].
Hundreds of different indicators are used in differing contexts for diverse purposes. It is difficult to assess the impact of these indicators on policymaking and progress towards sustainability since scientific information—such as that conveyed by indicators—is usually not sufficient to produce changes in either national decision-making or individual behavior. Thus the most significant effect of an indicator, particularly early in its adoption, can simply be communication—making a problem visible, sensitizing decision-makers and the public and expanding the basis for decision-making [4].
In 2015, a summit of heads of state adopted the Sustainable Development Goals (SDGs) [5]. The SDGs in their recent form are a universal set of Goals, targets, and indicators that UN member states will use to frame their agendas and policies over the next 15 years. Currently, they comprise 17 Goals, 169 targets, and 243 indicators [6]. It might be supposed that the SDGs framework—including indicators—is conceptually and methodologically well designed and tested in order to function reliably. However, it seems that the current structure and format of the proposed Goals and targets has laid “only” a policy framework. Being transparent and participatory in character, it is an appropriate way of designing it provided that Goals and targets would get through thorough expert and scientific follow up on their operationalization. Without a procedurally well-designed conceptual indicator framework for selecting and/or designing indicators, the results of SDGs assessment may be ambiguous and confusing [7].
The goal of this article is to critically review the state of the art in sustainability reporting by the application of SDG indicators and contribute to setting an appropriate approach in this regard. We have conducted a comprehensive review of the extensive body of work in this field—starting by reviewing the theoretical foundations for the operationalization of the sustainable development concept (there is more on this in the “Conceptualization and operationalization of sustainable development” section) and then looking into existing practice. We have chosen four major current SDGs assessments to examine how indicators have been used for the operationalization and communication of the SDGs. Our analysis focuses, in particular, on their relevance for the intended audience, interpretation of their results, and complexity of the assessment.
Our starting assumption is based on the original purpose of the SDGs—helping to further mainstream sustainable development at all levels, integrating economic, social, and environmental aspects and recognizing their interlinkages, so as to achieve sustainable development in all its dimensions [8]. Consequently, we have assumed that applying the SDG indicator framework inconsistently (by one) or uncoordinatedly (by many) in order to measure the same phenomenon, that is., progress towards the SDGs, may cause serious problems. Such assessments may convey very different messages that might raise doubts about the concept of SD or on the process of its operationalization. A worthy long-term global effort for communication of progress towards sustainability thus may be in vain or seriously undermined.

2. Indicator-Based Sustainability Communication

Despite recent scientific findings on many negative development trends [9], sustainability does not seem to have become a near-term priority for society [10,11,12]. In the process of changing this situation, an important role is given to sustainability communication. Its goal is to enable individuals and groups to develop the competences to adequately interpret the often contradictory and confusing scientific, technological, and economic information available to them and then be able to react to and cope with the resulting long-term and complex societal challenges [13].
Sustainable development, understood as a societal process of exploration, learning and shaping the future, necessarily involves communication. As global sustainability issues are characterized by high complexity and uncertainty, effective communication processes between the many actors involved are crucial to develop a mutual understanding of which actions to take [14]. In all three distinguished modes of communication—communication about sustainability, communication of sustainability, and communication for sustainability—sustainability indicators have an important role.
Indicators are by definition communication tools; indices, regardless of their many shortcomings, are particularly effective for communicating results to executives and the general public. Failure to communicate makes the indicators worthless. We may observe this in any context since knowledge production, having received an unprecedented boost in recent years, is no longer the privilege of an exclusive group of experts but takes place in a variety of constellations of actors. However, in these inter- and transdisciplinary work contexts, not enough attention has been paid to the problem of translating and communicating this knowledge in a way that is adequate to its target groups (e.g., the results of the European research projects POINT (Policy Influence of indicator (https://cordis.europa.eu/project/rcn/89898_en.html)) and BRAINPOol (BRinging Alternative INdicators into POLicy (https://cordis.europa.eu/project/rcn/100577_en.html)); or e.g., [15,16]).
Targeting sustainability communication is one of the most important steps toward making a problem visible, and hooking the interest and engagement of the target audience. It means identifying appropriate addressees and ensuring that the message reaches them. Since sustainable development is a multi-stakeholder process, indicators must be communicable to a variety of different participants. In general, sustainability communication moves in a special network of relationships among the three spheres of science, the public, and practice [17]. In more detail, some users need simple, structured information (voters—the public, non-specialist media, and decision makers), whereas others prefer an intermediate level of detail (local governments, policy implementers, non-government organizations, funding bodies, and industries), while technicians and academics may need more technical and specialized information. In targeting governments, it is useful to distinguish between politicians (ministers, political party members, parliamentarians etc.) who make decisions, and policy makers, implementers, and enforcers (mostly civil servants and experts in various positions with agenda areas such as economics, social affairs, transportation, environment, etc.) who design policy and regulatory portfolios, evaluate policy alternatives, analyze indicator results, and so on to provide the groundwork for decisions.
Indicators allow communication between scientists and policymakers, between policymakers and decision makers/politicians, between politicians and the public, even between scientists and the public, and so on. Communicating complex issues often presents difficulties since a large number of indicators complicates communication and imposes a great demand on users’ knowledge capabilities because of the multidimensional character of sustainable development. Moreover, the problem is not trivial, it is not just understanding figures and charts (presented indicator results), but people have to grasp numerous underlying concepts that they are not necessarily familiar with.
A key challenge for sustainability indicators producers and/or promoters is to deliver easily communicable messages on the progress towards sustainable development and hence to ease the use, and enable implementation of indicators in the policy process and by citizens [18]. Regardless of the use and users, we claim the need for the selection and design only of such indicators that fall into a given measurement concept (a policy goal or objective, a policy measure, etc.) and contribute to its appropriate operationalization.

3. Conceptualization and Operationalization of Sustainable Development

Before proceeding to the conceptualization and operationalization of SDGs, let us look briefly at the sustainable development concept. Sustainability is a concept understood intuitively by all but very difficult to express in concrete, operational terms [19,20,21,22]. Brundtland’s seminal definition [23] serves as a springboard for a variety of interpretations that emphasize the issues of needs, limits on development, futurity, inter- and intra-generational equity and the simultaneous fulfillment of economic efficiency, environmental protection, and social justice goals. Although the term is accompanied by imprecision, ambiguity, and, at times, contradictions, there is a generally accepted understanding of what sustainable development means.
The pragmatic way to SD definition relates to its “measurement” and includes indicators, as from the inception of the SD concept it has been clear that information and namely quantitative indicators will play an important role [24]. Nowadays, there are many SD indicators and indices already developed and new ones certainly have yet to appear, for example, [25,26]. Regardless of the truth—whether there is an obsession with numbers stimulating an indicator explosion, or a lack of indicators limiting humankind’s competence to embark on a sustainable path—many serious efforts seeking reliable SD metrics have been made so far (the European Commission’s ‘Beyond GDP’, OECD’s ‘Measuring the Progress of Societies’, etc.). This line of thinking was emphasized and supported in the main Outcome document of the Rio+20 Summit: “We recognize that progress towards the achievement of the Goals needs to be assessed and accompanied by targets and indicators, while taking into account different national circumstances, capacities and levels of development” [8].
One basic idea behind the development of indicators for monitoring and performance evaluation of SD policies is evidence-based policymaking—indicators being viewed as knowledge-agents serving the simplification and communication of evidence in a form suited for policy- and decision-makers. Besides this rationalistic conception of the instrumental role of knowledge for decision-making, indicator influence has also conceptual and political dimensions (helping to diffuse ideas, alternative thinking and new concepts rather that leading to political action). There are often trade-offs between different types of influence and between the roles that indicators play in policymaking. Indicator providers and promoters should therefore seek clarification for themselves concerning the types of influence that indicators are expected to achieve in a given policy situation (is the indicator-based report to consciously influence decision-makers, or is it to influence how policy-makers think, define problems, or provide new perspectives on problems? Or to provide ammunition to support the pre-determined position of a user? [18]). Thus, the purpose of the assessment predetermines the indicator selection to the same extent as the concepts behind the Goals and targets. We claim that concepts of SD as well as the SDGs and their targets cannot be solely defined by a pack of statistics and indicators, regardless of how relevant they are and regardless of their use. Employing indicators on the availability principle is methodologically incorrect and might lead to distortions in development of policy agendas.

4. The Sustainable Development Goals

In 2015, a summit attended by heads of state adopted the Sustainable Development Goals (SDGs) [5]. The SDGs, also known as the Global Goals, are a universal call to action to end poverty, protect the planet, and ensure that peace and prosperity will be enjoyed by all. They are structured in 17 general themes (Goals), 169 more specific tasks (targets), and 244 indicators ( Since nine indicators repeat under two or three different targets, the actual total number of individual indicators in the Framework is 232), that is, there are 5–19 targets and 6–27 indicators per Goal (Table 1). Despite the endorsed Global indicator framework, the work on the entire reporting mechanism is still in progress—data for the global SDG indicator database must be checked for availability and quality, and storylines for SDG global reports must be further developed [6]. The major identified weaknesses were the poor alignment of targets and Goals with existing international agreements and political processes; lack of effective implementation; conflicts between Goals and targets, non-quantified targets, lack of and/or low quality data for indicators [27]. And most of all—lack of operationalization of the targets [7].
We agree with Holden et al. [28] that formulating the 2030 Agenda and defining the SDGs would have been much easier if the world community had already reached a consensus on how to define and operationalize the concept of SD. We insist that it is still necessary to define each target, specify the concept behind each target, and describe what is and what is not part of that concept, and so on. This step will be important particularly for the targets which have a broad, multi-theme definition (there are majority such defined targets). This is to be followed by elaboration of clear-cut and detailed formulations of working hypotheses on the measurement of particular facts (phenomena, objects, processes) [29]. Only proper conceptualization and operationalization of the targets will transform them from broad, vague, and mostly political/theoretical concepts to tools which are clearly understandable in terms of empirical observations measurable or describable by appropriate indicators. It is an urgent task—since the first SDGs reports have already been published—to apply relevant and reliable indicators communicating global progress towards the Goals. A task of the same importance is at national level. Each country should seriously pursue the global SDGs concepts and methodologies and adapt them to national circumstances and conditions. What does, for example, “equitable and quality primary and secondary education” mean in each particular country and what policy implications does the underlying concept bring about? Is the proposed global indicator (Proportion of children and young people at certain grades achieving at least a minimum proficiency level in reading and mathematics) relevant and capable of capturing both as a global phenomenon and nation-specific? Do values play any role in it? [30]. Such questions should be laid and answered before publishing official SDGs reports.

5. Analysis of the SDG Reports

Several major reports monitoring the implementation of the SDGs emerged during the period 2015–2017. For the analysis we identified and selected four SDGs indicator-based assessments at global or supra-national/regional scales:
  • The Sustainable Development Goals Report 2017 [31];
  • SDGs: Are the rich countries ready? [32];
  • SDG Index and Dashboards Report 2017: International spillovers in achieving the Goals [33]
  • Sustainable development in the European Union. Monitoring report on progress towards the SDGs in an EU context [34].
We used Google Scholar (a widely-used web search engine indexing scholarly literature across an array of publishing formats and disciplines) that generated these reports based on several searching criteria. We searched for publicly available documents written in English, which were comprehensive assessment reports (not journal articles), legitimate (in terms of mandate of the report’s producer or its capability to conduct transparent high-quality analytical work), and at a supra-national or global scale (not national reports), that included a combination of keywords “report” and “Sustainable Development Goals”. It may be that the resulting list of major SDGs reports (November 2018) is not fully comprehensive but it is very likely they will have an impact on the politicians and policy-makers—and other users—they target. The reports’ main characteristics are in Table 2.
In terms of our methodological approach, we applied a comparative analysis. We started with a description that provided an informative comparison about the reports concerned (the summary results are in the Table 2). The key feature defining comparative analysis, as understood here, was an interest in the explanatory question of why the observed similarities and differences between cases exist. In particular, we tried to find out why the level of SDGs implementation in the same subject (a country) is assessed differently by each report. The analysis was based on collection of data on all cases included in the reports. The data was analyzed according to a designed common framework comprising additional explanatory variables: relevance for the intended audience, interpretation of the indicators results, and complexity of the assessment.
The relevance of the report—that is, relevance of the contained information—for the intended audience is a key factor of success. If the Goals are important to and understandable by people, they will ask their governments to act. Civil society must be able to put pressure on governments to hold them to account for what they pledge at world summits [32]. However, the main Outcome Documents on SDGs (The future we want (A/RES/66/288*) and Transforming our world: the 2030 Agenda for Sustainable Development (UN A/RES/70/1)) are not very specific about the target audience—all countries and all stakeholders, acting in collaborative partnership, are to implement the Goals. So it’s a whole society enterprise, with governments in the broadest sense (ministers, elected politicians, policy makers—state and regional administration officials, administrators, bureaucrats…) on one hand and civic society components on another hand (civic society organizations including business and entrepreneurs). A crucial factor for enhancing particularly the instrumental role and direct utilization of indicators in policy is setting a proper communication mode for the SDGs. Communication science has already developed to the extent that there are specific types of communication, and thematically-defined communication mechanisms (sustainability communication, climate change communication, risk communication etc.); see for example, [35,36].
Here, we focus on indicator-based communication on sustainable development. Based on the indicator’s characteristics, the communicated information may be either highly composed/aggregated or detailed; just proxies for main topics or exhaustive; technical or lay-focused, and so on. The typology of indicators, types of use (instrumental, conceptual, political), and users may be seen in [37,38,39]. In general, the public requires relatively simple, condensed, and easy to interpret information; it seems that just a limited number of indicators are sufficient. By contrast, professionals—policy makers, statisticians, and administrators need more detailed, disaggregated data and indicators suitable for further analyses. Politicians are somewhere in between: in terms of what they might be expected to do with indicators, they are closer to policy-makers (professionals) but in terms of the level of detail they might be expected to process, they are probably closer to the public. Specific users such as the business community, media, teachers, and so on then usually require thematically targeted information meeting their specific needs. Many major reports and assessments therefore publish special overviews and summaries for politicians, business, NGOs etc. Hence first we explore whether the format—namely number and type—of reported indicators may affect the intended audience.
The second variable is defined on the assumption that every assessment should have a clearly specified use (purpose) and audience. Interpretation of indicators results is an important part of indicator use: alongside their instrumental role, indicators can play a useful role in fostering social learning, for example, by helping to structure policy problems, build indicator frameworks, and clarify the various interpretations concerning the information indicators convey. An important aspect is the use of target values—policy objectives, legislative limits and standards, sustainability reference values etc. Thus SDGs may be assessed according to a desirable development in time (trend analysis) or according to relevant target values (distance to target approach) (See more in [40,41,42]). Since SDGs reports also employ compound indicators—indices—it is important to keep in mind both the pros and cons of their usage [43,44,45]. Lastly, it is necessary to distinguish the statistical use of information the purpose of which is to describe reality without any interpretation. With indicators the purpose is, specifically, to evaluate the development of policies and their impact on the state of affairs [34,46].
The last variable—complexity—regards selection of the indicators. Indicators always make the measured concept (more) complete but it in the case of the SDGs this criterion is of the utmost importance because of the low level of conceptualization of Goals and targets. To explore this variable, we look into the link between the employed indicators in each report and the Global indicator framework [6]. Although the Framework is a dynamic structure that will be probably be refined and specified to some extent building on experience and new indicator methodologies, we may assume it will provide firm guidance for both global assessments as well as for conducting national and thematic reviews of the Agenda 2030 [47]. Therefore, this variable shows to what extent the analyzed reports employ the framework (i.e., its indicators) assuming that it would secure some consistency among assessments conducted by different actors and/or at different times. Ideally, metadata of all used indicators (definitions, data sources, uncertainties, etc.) would be checked across all analyzed documents to identify sources of differences in results—different data, different indicators, methodologies, or just different terminology. Despite the importance of such information, it exceeds the scope of this article; however, we checked consistency among all indicators employed in the four analyzed reports and in the framework (name and rationale of indicators) (Table 3) as inexplicable differences in results might raise doubts among statisticians and experts.

6. Results

This section provides an overview of the main findings based on the three variables defined in the research framework.
Report: The Sustainable Development Goals Report 2017
The Sustainable Development Goals Report 2017 published by the UN Department of Economic and Social Affairs (UNDESA) reviews progress towards the 17 Goals in the second year of implementation of Agenda 2030. It employs 100 indicators to monitor the achievement of SDGs that are fully based on the Global indicator framework. Selection of indicators is not intended to represent the SDG targets according to their importance as all Goals and targets are equally important and will need to be addressed by the appropriate indicators [31]. The first part of the report—an Overview—emphasizes key global results in all 17 Goals. Clear short messages are accompanied by simple graphics (symbols, charts, maps). The analytical part of the report then presents more detailed information in various formats based on data availability—global results or figures disaggregated by the standard UN country groupings (Sub-Saharan Africa, Oceania, Northern Africa and Western Asia, Central and Southern Asia, Europe and Northern America, Latin America and the Caribbean, Australia and New Zealand, and Eastern and South-Eastern Asia). The indicators mostly show trend developments or only the last available year.
Several findings may be drawn from the report analysis: The first regards its communication power. The report is not very voluminous as it contains concise and condensed information. The 8-page Overview part is appropriately designed for politicians and the general public (and perhaps specific groups such as educators etc.). The analytical part seems to be more appropriate for professionals—policy makers, experts, specialized NGOs etc.). Indicators provide a statistical description of the state of affairs and a trend analysis over several years or just a simple visualization of the time development between two times in both global and regional scope. Global objectives for the Goals are used for the results interpretation when available. Thus, the report is appropriate for a broad audience interested in global development issues.
Report: Sustainable development in the European Union: Monitoring report on progress towards the SDGs in an EU context
The European Commission—or sensu stricto Eurostat, the statistical agency of the EU—monitors in this report the “next steps for a sustainable European future and European action for sustainability” [34]. Progress in all 28 EU member states is assessed by 100 indicators; each Goal has six indicators primarily attributed to it (Goals 14 and 17 have only five). Forty-one out of the 100 indicators are multi-purpose, that is, they are used to monitor more than one SDG. The indicators are not based on the Global indicator framework (despite there are many overlaps) but with a few exceptions they stem from already existing indicator sets used for monitoring long-term EU policies, such as the EU Sustainable Development Indicators, the Europe 2020 headline indicators, performance and impact indicators of the Strategic Plan 2016–2020, and so on. Like the above UN report, the EU Monitoring report offers an overall picture of the EU’s development (aggregated EU-28 level) but it also looks into disaggregated data for all member states. Whenever possible (in 16 cases), the calculation of indicator trends takes into account concrete objectives set in relevant EU policies and strategies. All indicators are interpreted also by trend—towards or from the objective or desirable path (even for indicators without quantitative objectives Eurostat has developed a method showing the pace and direction of indicator development). Besides the main bulky document, a 20-page “Overview of progress towards the SDGs in an EU context” published separately presents a first statistical overview of short-term trends (five years) relating to the SDGs in the EU by easy-to-grasp symbols for the same indicators as the main report.
The main report is quite extensive (372 pages) and it is likely to be assigned to policy makers and professionals. Although the abridged Overview is just a brochure it is still an information-rich document—the number of indicators is quite large to be read and understood as a whole. In both publications, figures, charts and symbols create a false impression of even more indicators, orientation and correct understanding of which is not easy, in particular for lay people. This makes not only complicated reading but also obscures understanding of the SDGs operationalization. In addition, some indicators (e.g., agricultural factor income, low work intensity, relative median at-risk-of-poverty gap) are not intuitively understandable in terms of meaning, interpretation, or relation to other indicators and objectives.
The EU SDG indicator set is the result of the official/political initiative involving a wide consultation process among Member States’ statistical authorities, European Council Committees, Commission services, the European Statistical Advisory Committee, members of academia and various international and non-governmental organizations. Thus, a thorough discussion on all Goals and targets may be assumed. Despite the EU having its own legitimate priorities for both domestic and international agendas, differences in operationalization (demonstrated by different indicators and their interpretation) may be viewed as (i) an inability or unwillingness to come to conclusions on the definitions of Goals and targets and/or (ii) taking into account “policy relevance from an EU perspective, availability, country coverage, data freshness and quality”. Policy relevance from an EU perspective may denote that the employed indicators are related exclusively to the objectives of the European policies. Then in fact, they would measure only what has been measured in the EU anyway, meaning unclear. However, the “SDGs implementation” in terms of measurement is supposed to help to identify the role of European countries in a global effort for sustainability as well as to contribute to SDGs operationalization by bringing well-elaborated and justified regional perspectives
Report: SDGs: Are the rich countries ready?
This report [34] examines how exclusively high-income countries are currently performing in SDGs achievement. In total 34 “snapshot indicators”, two per Goal, were selected based on the following three criteria: (i) Feasibility: Data must be available today in good quality at least for OECD countries; (ii) Suitability: The indicator should represent the—often multifaceted—Goal in a broad sense like a headline indicator; there should be a close conceptual fit between Goal and indicator; the indicators should be appropriate for the particular challenges of economically advanced nations; (iii) Relevance: The indicator should stand a good chance of becoming an actual part of the SDG monitoring system as currently being discussed by the Inter-Agency Expert Group for SDGs (In 2015 the United Nations Statistical Commission created the IAEG-SDGs and tasked it with developing and implementing the Global indicator framework for the Goals and targets of the 2030 Agenda) (IAEG-SDGs).
The 34 reported indicators have a plausible potential for communication on sustainability provided there is a thoughtful process of indicators selection. Such “headline indicators” would, by definition, capture the most important aspects of each Goal but they could not express their full operationalization. The indicators passing the above criteria (feasibility, suitability and relevance) were selected from a broad pool of global indicators—the Global indicator framework [6], SDGs monitoring indicators [48], and Sustainable Governance Indicators (a framework with 136 indicators) [49]. The report offers an overall “country league” ranking based on each indicator and SDG index (calculated as an unweighted average of all 34 indicators). The former benchmarks countries’ performance to the five best performing countries and thus compares each country with the realistically achievable results of their peers. The latter provides results of the first attempt for an integrated view of countries’ SDGs performance. Lucid visualization by radial charts for country profiles and bar charts for country rankings provide an evidence base for policymakers, businesses, and civil society to act.
An indicator selection—although based on three robust criteria—is inevitably an arbitrary decision always raising conceptual and methodological concerns. Other indicators would likely provide different rankings, different interpretation, different uncertainties, and so on. Another prerequisite of the credible indicator-based assessment is full comparability of data and indicators (in terms of sources, definitions, methodology, and interpretation), in particular for such a scientifically and politically sensitive task as country ranking (naming and faming—or shaming, is still the usual interpretation of results). In that regard, some indicators are not methodologically comparable or have low information value on/for some countries (secondary school attainment), are irrelevant (ocean health), or are not unambiguously interpretable (indicators using GDP ratio). The resulting numerical ladders look nice and scientific but they may not be justified by correct numbers.
Report: SDG Index and Dashboards Report 2017
This report was prepared by a team of independent experts of the Sustainable Development Solutions Network (Sustainable Development Solutions Network is an independent global network of research centers, universities and technical institutions aiming to mobilize scientific and technical expertise for problem-solving in relation to sustainable development. It was initiated by the UN in 2012) and Bertelsmann Stiftung (Bertelsmann Stiftung is a German private operating foundation). For 83, or 99 resp. (OECD countries have more accurate and better data available across a wide range of indicators, so 16 additional variables created an Augmented SDG Index for the OECD countries), of the used indicators, the official SDG indicators proposed by the Global indicator framework are employed where possible. New data and improvements in methodology are major changes from the previous year’s report. It strengthens the legitimacy and credibility of the SDGs operationalization; however, it is not a global multi-stakeholder consensus on SDGs assessment but an expert-based effort. It seeks to assess in particular the adverse “spillovers” (Positive and negative spillover effects are called “externalities” in economic literature)—development patterns of the rich countries that may hinder the ability of poorer countries’ to achieve the SDGs (e.g., high consumption levels, banking secrecy and tax havens, weapons exports, etc.). The underlying assumption is that traditional SDG indicators mostly ignore these spillover effects and therefore favor the high-income countries tending to generate them to a significant extent. Thus, the report identifies and measures three groups of the most important SDG-related spillovers and misuses of the global commons: environmental spillovers; spillovers related to the economy, finance, and governance, and security spillovers. Both the SDG index and dashboard use the same indicators.
The geographical coverage is 157 (out of the 193) UN member states. They can benchmark themselves against their peers—individual countries or relevant geopolitical regions (OECD countries, Eastern Europe and Central Asia, Latin America and the Caribbean, East and South Asia, Middle East and North Africa, Sub-Saharan Africa)—as well as against the Goal thresholds (absolute quantitative thresholds are used when possible, for example, zero poverty, universal school completion, full gender equality). These thresholds are derived from the SDGs and their targets or other official sources; when no such thresholds exist, an average of the top five performers is used. The importance (relevance) of the assessment is emphasized by comparing the country’s performance on the SDG Index to other common development metrics: GDP per capita, subjective wellbeing, Human Development Index, Environmental Performance Index, Global Competitiveness Index and Global Peace Index. The key users of the report results are governments; besides them, it explicitly aims at other SDG stakeholders such as businesses, civil society organizations, foundations, universities, media, and others who “have a vital role in turning the SDGs into practical tools for explaining sustainable development” [33].

7. Conclusions

The current UN Sustainable Development Goals will frame global action until 2030. The SDGs are already firmly embedded in a policy framework: during the course of their development they went through a political process and broad political negotiations. For monitoring and assessment of global sustainability an agreed set of global indicators was designed [6]. If this framework is used inconsistently, for example, if only arbitrarily selected indicators, additional indicators to some targets, compounds (indices) created from some indicators, and so on are used, the results will be commensurately inconsistent, incomprehensible, or even dubious. Despite the fact that SDG assessments are not intended to replace or compete with other SDG monitoring and indicators, in fact they do. If they motivated intergovernmental or/and national agencies to develop a suite of monitoring systems supporting the SDG indicators, such testing and experimenting with various indicators and indices would be well justified. However, the current inconsistent messages conveyed by the four analyzed SDGs assessments may be interpreted not as a call for better data and full conceptualization of all targets but as a rash exercise of researchers and experts almost randomly quantifying any SDG metric at hand. It is needless to say that it may open room for skepticism concerning the role of indicators as policy support instruments.
A flagrant example of inconsistent results is the SDG index. Regardless of the fact that the SDGs indicators have not been primarily designed for this purpose (therefore their authors have not handled issues of their substitutability, doublecounting, autocorrelations etc.), the SDG index is an attractive idea. Aggregated indicators, composites, indices and so on have always drawn attention due to their main advantage—simplifying communication of even complex issues. The real problem occurs when two similar indices show very different results. In particular, when the indices are named the same, or similarly assess the same facts or phenomena, they show remarkably distinct results. Only experts are, in fact, usually aware of the conceptual and/or methodological differences and therefore they understand the ensuing differences in results. While some results show good consistency of ranking (e.g., Sweden—1st place, Germany—6th place, etc.), the Czech Republic may serve as a country where inconsistent results may be politically harmful: One SDG index [32] ranks it in 24th place in country ranking (out of 34 assessed OECD countries)—that is, among the worst third—while a similar SDG index [33] places the same country at a very positive fifth place in the global competition (among 157 assessed countries). Unlike the Czech improvement—or more precisely “positive difference” of 19 places—the US lost 13 places (29. vs. 42.), Mexico 24 places (24. vs. 58.), and Turkey 34 places (33. vs. 67.). These differences are not caused just by different country samples—countries are placed differently since the indices are different. After getting such unclear overall information, assessment of particular targets also requires careful reading and some knowledge of interpretation. For example, a Czech reader interested in the topic of poverty (Goal 1), finds out that the global number of people living in extreme poverty fell significantly while people at risk of poverty in Europe are still numerous and thus it is moving away from sustainability objectives; however, while the Czech Republic is currently doing best at this indicator (much better than more affluent Germany or UK) it does not tell us anything about whether this is at a sustainable rate. Every assessment thus provides an important piece of the SDG puzzle; however, putting them together and seeing the whole picture it not a trivial enterprise.
Sustainability assessment is a tremendously difficult task, see for example, [50,51]. Besides developing new scenarios, models, and indicators it is necessary to assess the existing ones and account for their strengths and weaknesses, in particular in their role as policy support instruments—systematic research into that must continue, in particular in scientific journals. Experimenting with various SDGs rankings and indices may have a clear and acceptable rationale in terms of context analysis, correlation calculations, sensitivity analysis, and other numerical testing. We argue that different and inconsistent results published publicly and mostly channeled to the same audience (mostly politicians and policy makers) may cause serious misunderstanding or doubts on the capability to assess SDGs implementation: Have countries’ performances improved or worsened suddenly and inexplicably? Has the methodology changed and measured the same phenomenon differently? Have the indicators evinced big errors/uncertainties? Etc.
As mentioned above, the SDGs (i.e., Goals and targets) are firmly embedded in a policy framework and their operationalization has been mostly done by indicators. Because such an approach generates many caveats [7,52], it is absolutely necessary that the expert community reach full consensus on the indicator framework and its use. As emerging needs show, the global set of individual indicators should be complemented by a set of key (headline) indicators while an SDG index raises more doubts than gains so far (similarly to other sustainability indices mostly having negligible use in policy making). An instrumental role in this process belongs to the IAEG-SDGs. The UN Statistical Commission foresees the possibility of yearly refinements to the Framework and of two comprehensive reviews in 2020 and in 2025 [53]. Such a clear work plan with the strong leadership of the UN gives a chance to replace the “survival of the fittest” approach (very appropriate in the context of research published in scientific journals) and effectively make progress in the understanding of the SDGs at global level. In parallel, due to national adaptations of the Goals, targets, and related indicators many complementary data sets and indicators for national and regional SDGs assessment will emerge over time.

Author Contributions

The paper has resulted from cooperation of all authors. They delivered equal shares of work.

Acknowledgments

This work has been supported by the research program “Progres Q16” funded by Charles University.

Conflicts of Interest

The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.

References

  1. Fernando, J.L. The power of unsustainable development: What is to be done? ANNALS Am. Acad. Political Soc. Sci. 2003, 590, 6–34. [Google Scholar] [CrossRef]
  2. Mastny, L. (Ed.) State of the World 2015: Confronting Hidden Threats to Sustainability; Island Press: New York, NY, USA, 2015. [Google Scholar]
  3. Hák, T.; Janoušková, S.; Moldan, B. Closing the sustainability gap: 30 years after “Our Common Future”, society lacks meaningful stories and relevant indicators to make the right decisions and build public support. Ecol. Indic. 2018, 87, 193–195. [Google Scholar] [CrossRef]
  4. Dahl, A.L. Achievements and gaps in indicators for sustainability. Ecol. Indic. 2012, 17, 14–19. [Google Scholar] [CrossRef]
  5. United Nations. Transforming our world: The 2030 agenda for sustainable development. In Resolution Adopted by the General Assembly on 25 September 2015; RES/70/1; UN: New York, NY, USA, 2015. [Google Scholar]
  6. United Nations. Work of the Statistical Commission pertaining to the 2030 Agenda for Sustainable Development. In Resolution Adopted by the General Assembly on 6 July 2017; A/RES/71/313; UN: New York, NY, USA, 2017. [Google Scholar]
  7. Hák, T.; Janoušková, S.; Moldan, B. Sustainable Development Goals: A need for relevant indicators. Ecol. Indic. 2016, 60, 565–573. [Google Scholar] [CrossRef]
  8. United Nations. The future we want. In Resolution Adopted by the General Assembly on 27 July 2012; A/RES/66/288; UN: New York, NY, USA, 2012. [Google Scholar]
  9. Ribeiro, T.; Volkery, A.; Pirc Velkavrh, A.; Vos, H.; Hoogeveen, Y. Assessment of Global Megatrends. The European Environment–State and Outlook 2010. EFP Brief No. 227. 2012. Available online: http://www.foresight-platform.eu/wp-content/uploads/2012/11/EFP-Brief-No.-227_Assessment-of-Global-Megatrends.pdf (accessed on 25 March 2018).
  10. Harcourt, W. Sustainable Development: Who cares? Development 2002, 45, 3–5. [Google Scholar]
  11. Sen, A. The ends and means of sustainability. J. Hum. Dev. Capab. 2013, 14, 6–20. [Google Scholar] [CrossRef]
  12. Thompson, R.; Green, W. When sustainability is not a priority: An analysis of trends and strategies. Int. J. Sustain. Higher Educ. 2005, 6, 7–17. [Google Scholar] [CrossRef]
  13. Adomßent, M.; Godemann, J. Sustainability communication: An integrative approach. In Sustainability Communication; Springer: Dordrecht, The Netherlands, 2011; pp. 27–37. [Google Scholar]
  14. Newig, J.; Schulz, D.; Fischer, D.; Hetze, K.; Laws, N.; Lüdecke, G.; Rieckmann, M. Communication regarding sustainability: Conceptual perspectives and exploration of societal subsystems. Sustainability 2013, 5, 2976–2990. [Google Scholar] [CrossRef]
  15. Lyytimäki, J.; Rosenström, U. Skeletons out of the closet: Effectiveness of conceptual frameworks for communicating sustainable development indicators. Sustain. Dev. 2008, 16, 301–313. [Google Scholar] [CrossRef]
  16. Wardekker, J.A.; de Boer, J.; Kolkman, M.J.; van der Sluijs, J.P.; Buchanan, K.S.; de Jong, A.; van der Veen, A. Tool Cataloque Frame-Based Information Tools; Klimaat voor Ruimte, Group Science, Technology and Society, Copernicus Institute: Utrecht, The Netherlands, 2009. [Google Scholar]
  17. Godemann, J.; Michelsen, G. Sustainability Communication—An Introduction. In Sustainability Communication; Springer: Dordrecht, The Netherlands, 2011; pp. 3–11. [Google Scholar]
  18. Sébastien, L.; Bauler, T.; Lehtonen, M. Can indicators bridge the gap between science and policy? An exploration into the (non) use and (non) influence of indicators in EU and UK policy making. Nat. Cult. 2014, 9, 316–343. [Google Scholar] [CrossRef]
  19. Buckingham-Hatfield, S.; Evans, B. Achieving sustainability through environmental planning. In Environmental Planning and Sustainability; John Wiley & Sons: Chichester, UK, 1996; pp. 1–18. [Google Scholar]
  20. Zaccai, E. Over two decades in pursuit of sustainable development: Influence, transformations, limits. Environ. Dev. 2012, 1, 79–90. [Google Scholar] [CrossRef]
  21. De Vries, B.J.; Petersen, A.C. Conceptualizing sustainable development: An assessment methodology connecting values, knowledge, worldviews and scenarios. Ecol. Econ. 2009, 68, 1006–1019. [Google Scholar] [CrossRef]
  22. Kuhlman, T.; Farrington, J. What is sustainability? Sustainability 2010, 2, 3436–3448. [Google Scholar] [CrossRef]
  23. Brundtland, G.H. Report of the World Commission on Environment and Development: “Our Common Future”; United Nations: New York, NY, USA, 1987. [Google Scholar]
  24. Waas, T.; Hugé, J.; Block, T.; Wright, T.; Benitez-Capistros, F.; Verbruggen, A. Sustainability Assessment and Indicators: Tools in a Decision-Making Strategy for Sustainable Development. Sustainability 2014, 6, 5512–5534. [Google Scholar] [CrossRef] [Green Version]
  25. Tasaki, T.; Kameyama, Y.; Hashimoto, S.; Moriguchi, Y.; Harasawa, H. A survey of national sustainable development indicators. Int. J. Sustain. Dev. 2010, 13, 337–361. [Google Scholar] [CrossRef]
  26. Böhringer, C.; Jochem, P.E. Measuring the immeasurable—A survey of sustainability indices. Ecol. Econ. 2007, 63, 1–8. [Google Scholar] [CrossRef]
  27. Stokstad, E. Sustainable goals from UN under fire. Science 2015, 347, 702–703. [Google Scholar] [CrossRef] [PubMed]
  28. Holden, E.; Linnerud, K.; Banister, D.; Schwanitz, V.J.; Wierling, A. The Imperatives of Sustainable Development: Needs, Justice, Limits; Routledge: London, UK, 2017. [Google Scholar]
  29. Bunge, M. What is a quality of life indicator? Soc. Indic. Res. 1975, 2, 65–79. [Google Scholar] [CrossRef]
  30. Burford, G.; Tamás, P.; Harder, M.K. Can We Improve Indicator Design for Complex Sustainable Development Goals? A Comparison of a Values-Based and Conventional Approach. Sustainability 2016, 8, 861. [Google Scholar] [CrossRef]
  31. United Nations. The Sustainable Development Goals Report 2017; UN: New York, NY, USA, 2017. [Google Scholar]
  32. Kroll, C. SDGs: Are the Rich Countries Ready? Bertelsmann Stiftung: Gütersloh, Germany, 2015. [Google Scholar]
  33. Sachs, J.; Schmidt-Traub, G.; Kroll, C.; Durand-Delacre, D.; Teksoz, K. SDG Index and Dashboards Report 2017: International Spillovers in Achieving the Goals; Bertelsmann Stiftung and Sustainable Development Solutions Network: Paris, France; New York, NY, USA, 2017. [Google Scholar]
  34. Eurostat. Sustainable Development in the European Union—Monitoring Report on Progress towards the SDGs in an EU Context; Eurostat: Luxembourg, 2017. [Google Scholar]
  35. Fischer, D.; Lüdecke, G.; Godemann, J.; Michelsen, G.; Newig, J.; Rieckmann, M.; Schulz, D. Sustainability Communication. In Sustainability Science; Heinrichs, H., Martens, P., Michelsen, G., Wiek, A., Eds.; Springer: Dordrecht, The Netherlands, 2016. [Google Scholar]
  36. McGreavy, B. Addressing the Complexities of Boundary Work in Sustainability Science through Communication. Sustainability 2013, 5, 4195–4221. [Google Scholar] [CrossRef]
  37. Gudmundsson, H.; Hall, R.P.; Marsden, G.; Zietsman, J. Sustainable Transportation: Indicators, Frameworks, and Performance Management; Springer: Berlin/Heidelberg, Germany, 2016. [Google Scholar]
  38. European Environmental Agency. Environmental Indicators: Typology and Overview; Technical report No 25; EEA: Copenhagen, Denmark, 1999. [Google Scholar]
  39. Hák, T.; Moldan, B.; Dahl, A. Sustainability Indicators: A Scientific Assessment; Island Press: Washington, DC, USA, 2007. [Google Scholar]
  40. Moldan, B.; Janoušková, S.; Hák, T. How to understand and measure environmental sustainability: Indicators and targets. Ecol. Indic. 2012, 17, 4–13. [Google Scholar] [CrossRef]
  41. Eurostat. Getting Messages across Using Indicators—A Handbook Based on Experiences from Assessing Sustainable Development Indicators; Eurostat: Luxembourg, 2014. [Google Scholar]
  42. Castellani, V.; Benini, L.; Sala, S.; Pant, R. A distance-to-target weighting method for Europe 2020. Int. J. Life Cycle Assess. 2016, 21, 1159–1169. [Google Scholar] [CrossRef]
  43. Nardo, M.; Saisana, M.; Saltelli, A.; Tarantola, S.; Hoffman, A.; Giovannini, E. Handbook on Constructing Composite Indicators. OECD Statistics Working Papers. 2005. Available online: http://0-www-oecd--ilibrary-org.brum.beds.ac.uk/docserver/download/533411815016.pdf?expires=1521999211&id=id&accname=guest&checksum=71C6B39D88794FDA8445133A5073781A (accessed on 25 March 2018).
  44. Mayer, A.L. Strengths and weaknesses of common sustainability indices for multidimensional systems. Environ. Int. 2008, 34, 277–291. [Google Scholar] [CrossRef] [PubMed]
  45. Mori, K.; Christodoulou, A. Review of sustainability indices and indicators: Towards a new City Sustainability Index (CSI). Environ. Impact Assess. Rev. 2012, 32, 94–106. [Google Scholar] [CrossRef]
  46. Yli-Viikari, A. Confusing messages of sustainability indicators. Local Environ. 2009, 14, 891–903. [Google Scholar] [CrossRef]
  47. Adams, B.; Karen, J. 2030 Agenda and the SDGs: Indicator Framework, Monitoring and Reportng. Global Policy Watch 10. 2016. Available online: http://osthailand.nic.go.th/files/image/sdgs/GPW10_2016_03_18.pdf (accessed on 25 March 2015).
  48. SDSN. Indicators and a Monitoring Framework the Sustainable Development Goals: Launching a Data Revolution; Sustainable Development Solutions Network: New York, NY, USA, 2015; Available online: http://unsdsn.org/resources/publications/indicators/ (accessed on 25 March 2015).
  49. Bertelsmann Stiftung. Policy Performance and Governance Capacities in the OECD and EU: Sustainable Governance Indicators 2017. Bertelsmann Stiftung, 2017. Available online: http://www.sgi-network.org/docs/2017/basics/SGI2017_Overview.pdf. (accessed on 25 March 2015).
  50. Dizdaroglu, D. The role of indicator-based sustainability assessment in policy and the decision-making process: A review and outlook. Sustainability 2017, 9, 1018. [Google Scholar] [CrossRef]
  51. Konys, A. An Ontology-Based Knowledge Modelling for a Sustainability Assessment Domain. Sustainability 2018, 10, 300. [Google Scholar] [CrossRef]
  52. Yonehara, A.; Saito, O.; Hayashi, K.; Nagao, M.; Yanagisawa, R.; Matsuyama, K. The role of evaluation in achieving the SDGs. Sustain. Sci. 2017, 12, 969–973. [Google Scholar] [CrossRef]
  53. United Nations. Report of the Inter-Agency and Expert Group on SDG Indicators. In Proceedings of the Economic and Social Council, New York, NY, USA, 6–9 March 2018. E/CN.3/2018/2. [Google Scholar]
Table 1. Global indicator framework structure [6].
Table 1. Global indicator framework structure [6].
GoalNumber of Targets Number of Indicators
SDG 1714
SDG 2813
SDG 31327
SDG 41011
SDG 5914
SDG 6811
SDG 756
SDG 81217
SDG 9812
SDG 101011
SDG 111015
SDG 121113
SDG 1358
SDG 141010
SDG 151214
SDG 161223
SDG 171925
Total169244
Table 2. Key characteristics of the analyzed SDGs assessments.
Table 2. Key characteristics of the analyzed SDGs assessments.
The Sustainable Development Goals Report 2017Sustainable Development in the European UnionSDGs: Are the Rich Countries Ready? SDG Index and Dashboards Report 2017
Developer/authorUN (Department of Economic and Social Affairs—DESA)European Commission (Eurostat)Bertelsmann StiftungBertelsmann Stiftung and SDSN
Date of publication2017November 2017September 2015July 2017
Report extent64 pp.353/20 pp.*106 pp.122 pp.
Country coverageGlobal (Results per UN regions)28 (EU member states)34 (OECD member states)Global (Results per 157 countries and geopolitical regions)
Number of indicators100100/100 *3483 (OECD member states—99)
Headline indicatorsYes (Overview indicators; 2–4 per Goal)NoYes (Entire assessment is based on “snapshot indicators”—2 per Goal)No
Link to global indicator FrameworkYes (exclusive)NoYesYes
Interpretation Goal thresholds, description of state and development (trends)Policy objectives, benchmarking, trendsBenchmarking against the top countriesGoal thresholds, benchmarking, trends
UserNot specifiedNot specifiedPolicy makers, businesses, civil societyGovernments, academia, civil society, businesses
Supplementary information (on data and indicators)Available separately (at unstat.un.org/sdgs)A loose link to the Eurostat statistical portalReference to the data source for each indicatorNo
Note: * denotes a separately published Overview.
Table 3. Consistency check of indicators in the four analyzed reports and the Global indicator framework (An example of the Goal 11, selected targets 11.1, 11.2, 11.6 and 11.7).
Table 3. Consistency check of indicators in the four analyzed reports and the Global indicator framework (An example of the Goal 11, selected targets 11.1, 11.2, 11.6 and 11.7).
Goal 11 Make Cities and Human Settlements Inclusive, Safe, reSilient and Sustainable
Global indicator framework (IAEG-SDGs)The Sustainable Development Goals Report 2017Sustainable development in the European UnionSDGs: Are the rich countries ready?SDG Index and Dashboards Report 2017
Target 11.1 By 2030, ensure access for all to adequate, safe and affordable housing and basic services and upgrade slums
11.1.1 Proportion of urban population living in slums, informal settlements or inadequate housingProportion of urban population living in slumsPoor dwelling conditionsn.a.n.a.
Target 11.2 By 2030, provide access to safe, affordable, accessible and sustainable transport systems for all, improving road safety, notably by expanding public transport, with special attention to the needs of those in vulnerable situations, women, children, persons with disabilities and older persons
11.2.1 Proportion of population that has convenient access to public transport, by sex, age and persons with disabilitiesn.a.Access to public transportn.a.n.a.
Target 11.6. By 2030, reduce the adverse per capita environmental impact of cities, including by paying special attention to air quality and municipal and other waste management
11.6.2 Annual mean levels of PM2.5 and PM10 in citiesProportion of urban population living in areas meeting WHO air quality PM2.5 standardConcentration of particulate matterParticulate matter, share of population exposed to >15 μg/cbmPM2.5 in urban areas (μg/m3)
11.7 By 2030, provide universal access to safe, inclusive and accessible, green and public spaces, in particular for women and children, older persons and persons with disabilities
11.7.2 Proportion of persons victim of physical or sexual harassment, in the previous 12 monthsn.a.n.a.n.a.n.a.
Indicators used in the assessments not linked to the Global indicator framework
n.a. n.a.Population living in households suffering from noiseRooms per personRent burden (% disposable income)

Share and Cite

MDPI and ACS Style

Janoušková, S.; Hák, T.; Moldan, B. Global SDGs Assessments: Helping or Confusing Indicators? Sustainability 2018, 10, 1540. https://0-doi-org.brum.beds.ac.uk/10.3390/su10051540

AMA Style

Janoušková S, Hák T, Moldan B. Global SDGs Assessments: Helping or Confusing Indicators? Sustainability. 2018; 10(5):1540. https://0-doi-org.brum.beds.ac.uk/10.3390/su10051540

Chicago/Turabian Style

Janoušková, Svatava, Tomáš Hák, and Bedřich Moldan. 2018. "Global SDGs Assessments: Helping or Confusing Indicators?" Sustainability 10, no. 5: 1540. https://0-doi-org.brum.beds.ac.uk/10.3390/su10051540

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop