This post demonstrates how to perform a Bayesian linear regression, including the intuitions behind Bayesian statistics, linear regression Bayesian representation, conjugate Bayesian model, and Python implementation.
Introduction
In last post we reviewed the classical linear regression and now let's look from Bayesian perspective. There is well-known philosophical difference between frequentist and Bayesian. Interested readers can read further this debate online such as this one on Wiki. In my opinion, one advantage of Bayesian is its ability to perform online machine learning recursivelly as new information comes in incrementally.
In terms of the linear regression, Bayes's Theorem can be written as
\[ f(\beta|y,X)=\frac{f(y|\beta,X)f(\beta|X)}{f(y|X)} \tag{1.1} \]
where \(f(\beta|X)\) is the prior belief of parameter \(\beta\); \(f(y|\beta,X)\) is the likelihood of observing \(y\) given our prior; and \(f(y|X)\) is called evidence. Finally, \(f(\beta|y,X)\) is the posterior belief of pararmeter \(\beta\) once the evidence \(y\) has been taken into account. This posterior then becomes new prior and the circle continues recursively.