Exercise: Objects and Attributes

There are \(M\) objects, each with a number of attributes or labels. There are \(L\) possible labels and each object is associated with some subset of these labels. The indicator variables
\[
\alpha_{ml} = \left\{\begin{array}{cl}
1 & \textrm{if label } l \textrm{ is associated with object } m \\
0 & \textrm{otherwise}
\end{array}\right.
\]
define the association between objects and labels. The true values of the \(\alpha_{ml}\) are unknown. To infer the values of the \(\alpha_{ml}\), we ask \(N\) people to indicate which labels they believe are associated with each object. The outcomes of these trials are defined by another set of indicator variables,
\[
\beta_{nml} = \left\{\begin{array}{cl}
1 & \textrm{if person } n \textrm{ associated label } l \textrm{ with object } m \\
0 & \textrm{otherwise}
\end{array}\right.
\]
Now, people can make mistakes and we need to take two types of mistakes into account, namely false positives and false negatives. A false positive is when a person associates a label with an object, while the truth is that the label is not associated with it — i.e. \(\beta_{nml} = 1\) and \(\alpha_{ml} = 0\). A false negative is the opposite scenario, \(\beta_{nml} = 0\) and \(\alpha_{ml} = 1\). We will assume that each person makes each type of error with some fixed but unknown probability,
\begin{eqnarray}
e_n^{\textrm{pos}} &=& P(\beta_{nml}=1 | \alpha_{ml}=0) \quad\textrm{ and}\\
e_n^{\textrm{neg}} &=& P(\beta_{nml}=0 | \alpha_{ml}=1)
\end{eqnarray}

Questions

1. Come up with an appropriate prior over \(\left\{\alpha_{ml}\right\}\) and derive the posterior after observing \(\left\{\beta_{nml}\right\}\). You will also need priors over \(\left\{e^{\textrm{pos}}_n\right\}\) and \(\left\{e^{\textrm{neg}}_n\right\}\) and should derive their posteriors too.
2. Use the data file (LINK: \(N\approx20\), \(M\approx100\), \(L\approx100\)) and infer posteriors over \(\alpha_{ml}\), \(\left\{e^{\textrm{pos}}_n\right\}\) and \(\left\{e^{\textrm{neg}}_n\right\}\). Visualise the posteriors over \(\left\{e^{\textrm{pos}}_n\right\}\) and \(\left\{e^{\textrm{neg}}_n\right\}\) and note your observations. Visualise the posterior over \(\left\{\alpha_{ml}\right\}\) for each object.
3. Comment on whether there are enough data in the file or whether more measurements should be made.

Conjugate Inference: Multivariate Gaussian Likelihood

Domain: \[\vec{x}\in\mathbb{R}^d\]

Parameters: The Gaussian likelihood is parametrised by its mean vector, \(\vec{\mu}\), and precision matrix, \(\Lambda\).
\[\Theta = \{\vec{\mu}, \Lambda\}\]

Likelihood: \[P(\vec{x}|\Theta) = N(\vec{x}|\vec{\mu},\Lambda^{-1})\]

Prior: A normal–Wishart distribution.
\[P(\Theta) = N(\vec{\mu}|\vec{\eta}_0,(\tau_0\Lambda)^{-1})\ W(\Lambda|V_0,\nu_0)\]
The probability density function of the Wishart distribution is
\[W(\Lambda|V,\nu) = \frac{|\Lambda|^{(\nu-d-1)/2}\exp\left(-\frac{1}{2}\textrm{Trace}(V^{-1}\Lambda)\right)}{2^{\nu d/2}|V|^{\nu/2}\Gamma_d(\nu/2)}\]
where
\[\Gamma_d(\nu/2) = \pi^{d(d-1)/4}\ \prod_{i=1}^d \Gamma\left(\frac{\nu-i+1}{2}\right)\]
is the multivariate gamma function.

Posterior:
\[P(\Theta|D) = N(\vec{\mu}|\vec{\eta}_1,(\tau_1\Lambda)^{-1})\ W(\Lambda|V_1,\nu_1)\]
with
\begin{eqnarray}
\vec{\eta}_1 &=& \frac{\tau_0\vec{\eta}_0 + \vec{S}^{(1)}}{\tau_1} \\
\tau_1 &=& \tau_0 + S^{(0)} \\
\nu_1 &=& \nu_0 + S^{(0)} \\
V_1^{-1} &=& V_0^{-1} + S^{(2)} + \tau_0\vec{\eta}_0^{I\!I} - \tau_1\vec{\eta}_1^{I\!I}
\end{eqnarray}
where
\begin{eqnarray}
S^{(0)} &=& |D| \\
\vec{S}^{(1)} &=& \sum_{\vec{x}\in D} \vec{x} \\
S^{(2)} &=& \sum_{\vec{x}\in D} \vec{x}^{I\!I}
\end{eqnarray}
and \(\vec{x}^{I\!I} \equiv \vec{x}\vec{x}^T\) is the outer product of a vector with itself.

Marginal likelihood:
\[P(D) =
\pi^{-S^{(0)}d/2}
\left(\frac{\tau_0}{\tau_1}\right)^{d/2}
\frac{|V_1|^{\nu_1/2}}{|V_0|^{\nu_0/2}}
\ \prod_{i=1}^d \frac{\Gamma\left((\nu_1-i+1)/2\right)}{\Gamma\left((\nu_0-i+1)/2\right)}
\]