-
Book Overview & Buying
-
Table Of Contents
Machine Learning for Emotion Analysis in Python
By :
Inter-annotator reliability is the widely used term to describe “the extent to which independent coders evaluate the characteristic of a message, or artefact, and reach the same conclusion” (Tinsley and D. J. Weiss). It is an important metric because it determines whether the data can be considered valid and is an indication of the trustworthiness of the data. Without this reliability, any content analysis is useless. From a practical point of view, establishing a high level of reliability also has the benefit of allowing the work to be divided among multiple annotators. Measuring reliability is actually measuring reproducibility, that is, "the likelihood that different coders who receive the same training and textual guidance will assign the same value to the same piece of content” (Joyce).
There are many ways to measure reliability – two of the most common are Fleiss’ kappa and Krippendorff’s alpha:
...