Product of Dirichlet Distributions

The Dirichlet distribution is
\[
D(\vec{x}|\vec{\alpha}) =
\frac{\Gamma\left(\sum_{j=1}^d\alpha_j\right)}{\prod_{j=1}^d\Gamma(\alpha_j)}
\prod_{j=1}^d x_j^{\alpha_j-1}
\]
where \(\Gamma(\cdot)\) is the gamma function and \(d\) is the dimensionality of \(\vec{x}\).

A product of Dirichlet distributions is proportional to another Dirichlet distribution.
\[
\prod_{i=1}^n D(\vec{x}|\vec{\alpha}_i) =
Z\times D(\vec{x}|\vec{\alpha}')
\]
where
\[
\vec{\alpha}' - 1 = \sum_{i=1}^n \left[\vec{\alpha}_i - 1\right]
\]
and
\[
Z = \frac
{\prod_{j=1}^d\Gamma(\alpha'_j)}
{\Gamma\left(\sum_{j=1}^d\alpha'_j\right)}
\ \prod_{i=1}^n\left[ \frac
{\Gamma\left(\sum_{j=1}^d\alpha_{ij}\right)}
{\prod_{j=1}^d\Gamma(\alpha_{ij})} \right]
\]

Product of Normal–Gamma Distributions

The normal–gamma distribution is
\begin{eqnarray}
NG(\mu,\lambda|\eta,\tau,\alpha,\beta)
& = & N(\mu|\eta,(\tau\lambda)^{-1})\ G(\lambda|\alpha,\beta) \\
& = &
\frac{\beta^{\alpha}\sqrt{\tau}}{\Gamma(\alpha)\sqrt{2\pi}}
\lambda^{\alpha-\frac{1}{2}}
\exp\left( -\beta\lambda - \frac{1}{2}\tau\lambda(\mu-\eta)^2 \right)
\end{eqnarray}
where \(\Gamma(\cdot)\) is the gamma function.

A product of normal–gamma distributions is proportional to another normal–gamma distribution.
\[
\prod_{i=1}^n NG(\mu,\lambda|\eta_i,\tau_i,\alpha_i,\beta_i)
= Z\times NG(\mu,\lambda|\hat{\eta},\hat{\tau},\hat{\alpha},\hat{\beta})
\]
where
\begin{eqnarray}
\hat{\tau} &=& \sum_{i=1}^n \tau_i \\
\hat{\tau}\hat{\eta} &=& \sum_{i=1}^n \tau_i\eta_i \\
2\hat{\beta} + \hat{\tau}\hat{\eta}^2 &=& \sum_{i=1}^n \left[2\beta_i + \tau_i\eta_i^2\right] \\
\hat{\alpha}-\frac{1}{2} &=& \sum_{i=1}^n \left[\alpha_i - \frac{1}{2}\right]
\end{eqnarray}
and
\[
Z =
\frac{\Gamma(\hat{\alpha})\sqrt{2\pi}}{\hat{\beta}^{\hat{\alpha}}\sqrt{\hat{\tau}}}
\prod_{i=1}^n\left[\frac{\beta_i^{\alpha_i}\sqrt{\tau_i}}{\Gamma(\alpha_i)\sqrt{2\pi}}\right]
\]

Conjugate Inference: Gaussian Likelihood

Domain: Here we consider only univariate normal distributions. \[x\in\mathbb{R}\]

Parameters: The Gaussian likelihood is parametrised by its mean, \(\mu\), and inverse variance, \(\lambda\).
\[\Theta = \{\mu, \lambda\}\]

Likelihood: \[P(x|\Theta) = N(x|\mu,\lambda^{-1})\]

Prior: A normal–gamma distribution.
\[P(\Theta) = N(\mu|\eta_0,(\tau_0\lambda)^{-1})\ G(\lambda|\alpha_0,\beta_0)\]
Note that the following parametrisation of the gamma distribution is used:
\[G(x|\alpha,\beta) = \frac{\beta^{\alpha}}{\Gamma(\alpha)} x^{\alpha-1} e^{-\beta x}\]
(There is another parametrisation, which uses \(\beta^{-1}\) rather than \(\beta\)).

Posterior:
\[P(\Theta|D) = N(\mu|\eta_1,(\tau_1\lambda)^{-1})\ G(\lambda|\alpha_1,\beta_1)\]
with
\begin{eqnarray}
\eta_1 &=& \frac{\eta_0\tau_0 + S^{(1)}}{\tau_1} \\
\tau_1 &=& \tau_0 + S^{(0)} \\
\alpha_1 &=& \alpha_0 + \frac{S^{(0)}}{2} \\
\beta_1 &=& \beta_0 + \frac{1}{2}\left( S^{(2)}+\eta_0^2\tau_0 - \eta_1^2\tau_1 \right)
\end{eqnarray}
where
\begin{eqnarray}
S^{(0)} &=& |D| \\
S^{(1)} &=& \sum_{x\in D} x \\
S^{(2)} &=& \sum_{x\in D} x^2
\end{eqnarray}

Marginal likelihood:
\[P(D) =
(2\pi)^{-S^{(0)}/2}
\frac{\sqrt{\tau_0} \beta_0^{\alpha_0} \Gamma(\alpha_1)}
{\sqrt{\tau_1} \beta_1^{\alpha_1} \Gamma(\alpha_0)} \]

Conjugate Inference: Gaussian Likelihood with Known Variance

Domain: Here we consider only univariate normal distributions.
\[x\in\mathbb{R}\]

Parameters: \[\Theta = \{\mu\}\] where \(\mu\) is the mean of the Gaussian likelihood.

Likelihood: \[P(x|\Theta) = N(x|\mu,v)\] Note that \(v\) is the known variance of the Gaussian likelihood.

Prior: A normal distribution.
\[P(\Theta) = N(\mu|\mu_0,\sigma_0^2)\]

Posterior:
\[P(\Theta|D) = N(\mu|\mu_1,\sigma_1^2)\]
with
\begin{eqnarray}
\mu_1 &=& \sigma_1^2 \left( \frac{\mu_0}{\sigma_0^2} + \frac{S^{(1)}}{v} \right) \\
\sigma_1^2 &=& \left( \frac{1}{\sigma_0^2} + \frac{S^{(0)}}{v} \right)^{-1}
\end{eqnarray}
where
\begin{eqnarray}
S^{(0)} &=& |D| \\
S^{(1)} &=& \sum_{x\in D} x \\
S^{(2)} &=& \sum_{x\in D} x^2
\end{eqnarray}

Marginal likelihood:
\[P(D) =
\left(\frac{1}{\sqrt{2\pi v}}\right)^{S^{(0)}}
\frac{\sigma_1}{\sigma_0}
\exp\left( -\frac{1}{2}\left( \frac{\mu_0^2}{\sigma_0^2} - \frac{\mu_1^2}{\sigma_1^2} + \frac{S^{(2)}}{v} \right) \right)\]