Figure 1

An illustration of parameter estimation using MPF. The three successive panels illustrate the sequence of parameter updates that occur during learning. Each set of axes represents the space of probability distributions. The dashed red curves indicate the family of model distributions ${\mathbf{p}}^{(\infty )}(\theta )$ parametrized by $\theta$. The black solid curves indicate deterministic dynamics that transform the data distribution ${\mathbf{p}}^{(0)}$ into the model distribution ${\mathbf{p}}^{(\infty )}(\theta )$. Under maximum likelihood learning, model parameters $\theta$ are chosen so as to minimize the KL divergence between the data distribution ${\mathbf{p}}^{(0)}$ and the model distribution ${\mathbf{p}}^{(\infty )}(\theta )$. Under MPF the KL divergence between ${\mathbf{p}}^{(0)}$ and ${\mathbf{p}}^{(ϵ)}(\theta )$ is minimized instead, where ${\mathbf{p}}^{(ϵ)}(\theta )$ is the distribution obtained by initializing the dynamics at the data distribution ${\mathbf{p}}^{(0)}$ and then evolving them for an infinitesimal time $ϵ$. Here we represent graphically how parameter updates that pull ${\mathbf{p}}^{(ϵ)}(\theta )$ towards ${\mathbf{p}}^{(0)}$ also tend to pull ${\mathbf{p}}^{(\infty )}(\theta )$ towards ${\mathbf{p}}^{(0)}$.Reuse & Permissions

Figure 2

A demonstration of MPF outperforming existing techniques for parameter recovery in an Ising spin glass model. (a) Time evolution of the mean square error in the coupling strengths for 5 methods for the first 60 seconds of learning. Note that mean field theory with second order corrections ($\mathrm{MFT}+\mathrm{TAP}$) actually increases the error above random parameter assignment, though it does converge to the correct answer in some other parameter regimes, such as in the high temperature limit of this Ising spin glass model [15]. (b) Mean square error in the coupling strengths for the first 800 seconds of learning. (c) Mean square error in coupling strengths for the entire learning period. (d)–(f) Mean square error in pairwise correlations for the first 60 seconds of learning, the first 800 seconds of learning, and the entire learning period, respectively. In every comparison above MPF finds a better fit, and for all cases but $\mathrm{MFT}+\mathrm{TAP}$ does so in a shorter time.Reuse & Permissions

Figure 3

A demonstration of MPF parameter estimation in a continuous state space probabilistic model. Each square represents a $10\times 10$ pixel filter ${\mathbf{J}}_k$ for an ICA model trained on natural image patches via (a) MPF or (b) maximum likelihood learning. The visual similarity of the filters is consistent with the nearly identical average log likelihoods for the two models ($-174.0$ bits and $-173.6$ bits, respectively).Reuse & Permissions