Multi-annotator Gaussian process

Abstract

We present a Gaussian process (GP) approach to regression with multiple labels but no absolute gold standard. The GP framework provides a principled non-parametric framework that can automatically estimate the reliability of individual annotators from data without the need of prior knowledge. Experimental results show that the proposed GP multi-annotator model outperforms models that either average the training data or weigh individually learned single-annotator models.

Code

The Matlab code can be downloaded here. It has been tested with the free software GPstuff v3.2. My code uses a slightly different syntax than GPstuff v3.2, so a few files need to be overwritten.

References

Perry Groot, Adriana Birlutiu, Tom Heskes. Learning from Multiple Annotators with Gaussian Processes, ICANN, 2011, 159-164. [PDF]