Debrief
It is a good idea to follow up with your annotators after the task to ascertain their views about the overall task and whether they experienced any particular difficulties with any emotions. For example, tweets may contain several different emotions and hence it may be difficult to pinpoint the prevalent ones. There may also have been occasions where emotive words were seen but the annotator did not feel that this led them to strongly lean toward any particular emotion. It is sensible to obtain an understanding of these types of situations as early as possible as changes to the procedures for collecting data are likely to change the nature of what is collected. Typically, annotators refer to tweets being short and informal as a primary reason for being unable to determine a definitive emotion for a tweet. It is, therefore, ironic that when tweets are overly lengthy, annotators mention that it is hard to restrict themselves to selecting a sensible set of emotions, as in this...