Pierre-Simon Laplace

Unaware of Bayes, Pierre-Simon Laplace (1749-1827) independently developed Bayes' theorem and first published his version in 1774, eleven years after Bayes, in one of Laplace's first major works (Laplace, 1774, p.366-367). In 1812, Laplace (1749-1827) introduced a host of new ideas and mathematical techniques in his book, Théorie Analytique des Probabilités. Before Laplace, probability theory was solely concerned with developing a mathematical analysis of games of chance. Laplace applied probabilistic ideas to many scientific and practical problems.

The Laplace Approximation or Laplace Method is a family of asymptotic techniques used to approximate integrals. Laplace's method seems to accurately approximate uni-modal posterior moments and marginal posterior distributions in many cases, and is useful when approximating Bayes factors.

In 1814, Laplace published his "Essai philosophique sur les probabilites", which introduced a mathematical system of inductive reasoning based on probability. In it, the Bayesian interpretation of probability was developed independently by Laplace, much more thoroughly than Bayes, so some "Bayesians" refer to Bayesian inference as Laplacian inference. In this same publication, Laplace developed the the Laplace Approximation in a proof, and used it to approximate posterior moments.

Since its introduction, the Laplace Approximation has been applied successfully in many disciplines. In the 1980's, the Laplace Approximation experienced renewed interest, especially in statistics, and some improvements in its implementation were introduced. Only since the 1980's has the Laplace Approximation been seriously considered by statisticians in the context of practical applications.

There are many variations of Laplace Approximation, with an effort toward replacing MCMC as the dominant form of numerical approximation in Bayesian inference. The run-time of Laplace Approximation is a little longer than Maximum Likelihood Estimation (MLE), and much shorter than MCMC (Azevedo-Filho & Shachter, 1994). Laplace Approximation extends MLE, but shares similar limitations. Bernardo and Smith (2000) note that Laplace Approximation is an attractive numerical approximation algorithm, and will continue to develop.

While MCMC is stochastic, Laplace Approximation is a deterministic and asymptotic algorithm, like MLE. Assumptions must be made about the posterior distribution. Unlike MLE which estimates only a mode as a point-estimate, Laplace Approximation estimates a Gaussian distribution around the mode, which is appropriate in Bayesian inference where most posterior distributions are Gaussian. MCMC algorithms are excellent, but the run-time for complicated models or models with large data sets is often unacceptable. The use of MCMC with complicated models often leads to chains with high autocorrelation.

Laplace was one of the most influential people in the history of probability, and he has inspired STATISTICAT to develop software for Bayesian inference, entitled Laplace's Demon.