Jensen-Shannon (JS) divergence (JSD)

The Jensen-Shannon divergence (JSD) is an information-theoretic measure of similarity between two distributions. The JSD is defined as

where is the Kullback-Leibler divergence.

Like the population stability index (PSI), it is a symmetric measure that can be useful in monitoring for concept drift. However, it differs markedly in that it is defined in terms of an average between the reference distribution and the query distribution . This has some interesting implications:

• The JSD is bounded to the interval when entropy is calculated using .

  • By comparison, the PSI has no upper bound.
  • When using for entropy, .

• The JSD is always defined in the union of the support of and .

  • That is, as long as has non-zero probability in either or , the JSD can include .
  • This includes intervals for which the KL divergence is undefined due to divide-by-zero.

• The JSD is less sensitive to changes in the probability of rare events.

This last point bears further explanation. Rare events have a high information content; they are “surprising.” This means that events that are rare in will have a disproportionate effect on the KL divergence. The PSI will fully reflect the impact of these rare events. Due to , the effect on the JSD will be comparatively muted. For a worked example, see Response of JSD and PSI to a rare event.