Data: 40 (nouns)*34 (verbs) = 1360 expressions in total

We asked workers to evaluate the metaphoricity, comprehensibility, and novelty of each expression on a scale of one to five.

To make the preferred ones highly ranked in these cases, taking the abstractness or concreteness of the nouns into the overall evaluation would be effective.

Experiments

Overall evaluation: While it can be defined in many ways, for simplicity, we calculate it by just adding the three metrics.

University of Michigan - Ann Arbor

MIPVU: First, annotators determine the contextual meaning of the target word. Then, they look up the word in a dictionary and search for a more basic meaning than the contextual meaning. If a more basic meaning exists, the contextual meaning differs from the basic meaning, and if it can be understood in comparison, then it is judged as a metaphor-related word.

Judge metaphoricity in a systematic way

Metaphoricity: We collect metaphoricity scores by asking simple questions of crowdworkers. After that, we check if the results are reliable by using MIPVU.

Metaphoricity

Novelty: An advantage of using novelty is that it is easier to evaluate than creativity or originality. In the evaluation of novelty, an evaluator gives the highest score if he or she has never heard of or used the target expression, and the lowest score if the expression is widely used and conventional.

A simple idea of automation: It may be possible to measure novelty by counting the number of occurrences of expressions.

Novelty

Comprehensibility serves as a constraint to keep the generated result meaningful. This metric is more important for metaphors than for similes.(People need knowledge or context to understand metaphors.)

Learn Before

Related

Learn After