Exact Estimation vs. Approximate Estimation

Most frequentist inference utilizes approximate estimation rather than exact estimation, whereas Bayesian inference may use approximate or exact estimation.

Exact estimation is based on exact probability statements that are valid for any sample size. In contrast, approximate estimation is based in asymptotic theory. In math, an asymptote is a real-valued function with a curve that gets closer to the axis as the value along the axis approaches infinity, though it will never touch the axis. Consider the inverse of n, which is 1/n. No matter how large n becomes, the result will never equal zero, but it 'asymptotically' approaches zero.

If n is the sample size, then bias due to sample size is a function of 1/sqrt(n), if an asymptotic method is used for numerical approximation. As the sample size, n, becomes larger, the bias decreases. Therefore, a sample size of, say, n=10, is too small in frequentist inference, most of which consists of Maximum Likelihood Estimation (MLE). MLE relies on normality approximations based on large sample asymptotics. MLE does not obey the likelihood principle, but that is another topic.

Bayesian inference with Laplace Approximation also relies on large sample asymptotics, though this method of numerical approximation is becoming popular for other reasons, such as enabling fast Bayesian inference with large data sets (where large sample asymptotics is appropriate).

However, when this sample size of n=10 is approached with Bayesian inference and estimated with a Markov chain Monte Carlo (MCMC) algorithm, there is no bias due to the small sample size. A small sample will be associated with more uncertainty, but with Bayesian inference from MCMC, there is no small sample bias. In this respect, it is exact estimation, rather than approximate estimation.

There is, however, an aspect of MCMC that is approximate with respect to large sample asymptotics, but not with respect to the sample size n of data. An MCMC algorithm iterates through successive estimates, and once converged, numerous estimates are used to represent a probability distribution that, say, describes a parameter. Large sample asymptotics, 1/sqrt(n), apply to the accuracy of the description of the probability distribution, given the number of MCMC estimates. But this is not problematic, like large sample asymptotics of sample size n of data. If the number of MCMC estimates is too few, the algorithm can simply continue to iterate from where it left off, until the asymptotic bias is satisfactory.

Approximate estimation is simpler, given a sample size of data. Exact estimation is better.