Measuring summarization performance
As with the discussion in the Measuring translation performance section of Chapter 6, Teaching Machines to Translate, using the BiLingual Evaluation Understudy (BLEU) score, we present a metric for assessing the performance of text summarization systems. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score is the subject of the current section, and although its name sounds complicated, it’s incredibly easy to understand and implement. It works by comparing an automatically produced summary against a human reference summary using n-grams. In that sense, it is symmetrical to the BLEU score. Additionally, ROUGE is a set of metrics rather than a single one. They all assign a numerical score to a summary that tells us how good it is compared to a reference one. Let’s examine the first variant.
ROUGE-N measures the overlap of unigrams, bigrams, trigrams, and higher-order n-grams, where N represents the n-gram order. Thus...