Yesterday I was talking with Han Shao about the metric of diversity in machine language generation. Suppose there’re three systems A, B and C:
A generates 3 examples; each example is in a different pattern.
B generates 100 examples; each example is in one of 5 patterns, uniformly (i.e., each pattern has 20 examples).
C generates 100 examples, 96 of which are in pattern (a), while the rest 4 are in different other patterns respectively.
Which is the most diverse one? We found it’s difficult to quantitatively answer this question. But I somewhat convinced myself that it can be measured as follows, applying simple equations from information theory.
Let’s assume that each pattern is independent of any of the others. It’s also necessary to assume the observed empirical distribution to be the true distribution that the model represents, though we should let the models generate as many examples as they can to obtain a good estimation of the true distribution.
Let 
Larger entropy typically means better diversity. We can then compute the entropy of the above three distributions:
Being towards Nada 