Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems

Adam B. Barrett

Phys. Rev. E 91, 052802 – Published 8 May 2015

Abstract

To fully characterize the information that two source variables carry about a third target variable, one must decompose the total information into redundant, unique, and synergistic components, i.e., obtain a partial information decomposition (PID). However, Shannon's theory of information does not provide formulas to fully determine these quantities. Several recent studies have begun addressing this. Some possible definitions for PID quantities have been proposed and some analyses have been carried out on systems composed of discrete variables. Here we present an in-depth analysis of PIDs on Gaussian systems, both static and dynamical. We show that, for a broad class of Gaussian systems, previously proposed PID formulas imply that (i) redundancy reduces to the minimum information provided by either source variable and hence is independent of correlation between sources, and (ii) synergy is the extra information contributed by the weaker source when the stronger source is known and can either increase or decrease with correlation between sources. We find that Gaussian systems frequently exhibit net synergy, i.e., the information carried jointly by both sources is greater than the sum of information carried by each source individually. Drawing from several explicit examples, we discuss the implications of these findings for measures of information transfer and information-based measures of complexity, both generally and within a neuroscience setting. Importantly, by providing independent formulas for synergy and redundancy applicable to continuous time-series data, we provide an approach to characterizing and quantifying information sharing amongst complex system variables.

Received 13 November 2014

DOI:https://doi.org/10.1103/PhysRevE.91.052802

Authors & Affiliations

Adam B. Barrett^*

Sackler Centre for Consciousness Science, Department of Informatics, University of Sussex, Brighton BN1 9QJ, United Kingdom and Department of Clinical Sciences, University of Milan, Milan 20157, Italy

^*Adam.Barrett@sussex.ac.uk

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand

Issue

Vol. 91, Iss. 5 — May 2015

Reuse & Permissions

Access Options

Author publication services for translation and copyediting assistance advertisement

Images

Figure 1
General structure of the information that two source variables $Y$ and $Z$ hold about a third target variable $X$ . The ellipses indicate $I (X; Y), I (X; Z)$ , and $I (X; Y, Z)$ as labeled and the four distinct regions enclosed represent the redundancy $R (X; Y, Z)$ , the synergy $S (X; Y, Z)$ , and the unique information $U (X; Y | Z)$ and $U (X; Z | Y)$ as labeled.
Reuse & Permissions
Figure 2
Correlational structure of two example systems of univariate Gaussian variables for which $Y$ and $Z$ exhibit positive net synergy with respect to information about $X$ . Variables are shown as circles and the variables that are correlated are joined by lines. (a) $Y$ and $Z$ are uncorrelated and yet show synergy. (b) $X$ and $Z$ are uncorrelated and yet $Z$ contributes synergistic information about $X$ in conjunction with $Y$ . See the text for details.
Reuse & Permissions
Figure 3
Illustrative examples of net synergy and synergy $S_{MMI}$ between Gaussian variables. (a) Net synergy in Shannon information that sources $Y$ and $Z$ share about the target $X$ , as a function of the correlation between $Y$ and $Z$ for (black) correlations between $X$ and $Y$ and between $X$ and $Z$ equal and both positive $(a = c = 0.5)$ and (gray) correlations between $X$ and $Y$ and between $X$ and $Z$ equal and opposite $(a = - c = 0.5)$ . (b) Same as (a) but using information defined as reduction in variance instead of reduction in Shannon entropy. (c) Synergy according to the MMI PID for the same parameters as (a). Here the dashed line shows redundancy according to the MMI PID, which does not depend on the correlation between $Y$ and $Z$ . (d) Example of net synergy as a function of the correlation between $Y$ and $Z$ for (black) correlations between $X$ and $Y$ and between $X$ and $Z$ unequal and both positive $(a = 0.25, c = 0.75)$ and (gray) correlations between $X$ and $Y$ and between $X$ and $Z$ unequal and of opposite sign $(a = 0.25, c = - 0.75)$ . (e) Same as (d) but using information defined as reduction in variance instead of reduction in Shannon entropy. (f) Synergy according to the MMI PID for the same parameters as (d). Here the dashed line shows redundancy according to the MMI PID, which does not depend on the correlation between $Y$ and $Z$ . See the text for full details of the parameters. In all panels dotted vertical lines indicate boundaries of the allowed parameter space, at which the measures go to infinity, and horizontal dotted lines indicate zero.
Reuse & Permissions
Figure 4
Connectivity diagrams for example dynamical systems. Variables are shown as circles and directed interactions as arrows. The systems are animated as Gaussian MVAR processes of order 1. (a) Example 1. In this system $X$ receives input from its own past and from the past of $Y$ . There is positive net synergy between the information that the immediate pasts of $X$ and $Y$ provide about the future of $X$ , but zero net synergy between the information provided by the infinite pasts of $X$ and $Y$ about the future of $X$ . (b) Example 2. In this system there is bidirectional connectivity between $X$ and $Y$ . There is zero net synergy between the information provided by the immediate pasts of $X$ and $Y$ about the future of $X$ and negative net synergy (i.e., positive net redundancy) between the information provided by the infinite pasts of $X$ and $Y$ about the future of $X$ . (c) Example 3. Here $Y$ and $Z$ are sources that influence the future of $X$ . Depending on the correlation between $Y$ and $Z$ , there can be synergy between the information provided by the pasts of $Y$ and $Z$ about the future of $X$ (independent of the length of history considered).
Reuse & Permissions

Physical Review E

covering statistical, nonlinear, biological, and soft matter physics