Calculate mutual information from a contingency table
Suppose a feature is classified as Low or High at a decision timestamp and the subsequent return is classified as Up or Not up. These 20 hypothetical observations illustrate arithmetic, not measured market performance. Each row and each column totals 10. The joint probabilities are the cell counts divided by 20; both marginal distributions are 0.5 and 0.5.
- Base-2 logarithms give bits; natural logarithms give nats. The scikit-learn mutual_info_score function uses natural logarithms. Divide nats by ln(2) to compare with this example.
- Here H(Y) = 1 bit and H(Y|X) = 0.721928 bits, so I(X;Y) = H(Y) - H(Y|X). This is an average reduction in uncertainty, not a 27.8% return or a forecast accuracy.
- For finite discrete variables, 0 ≤ I(X;Y) ≤ min(H(X), H(Y)). A constant feature gives zero empirical MI; a constant target also gives zero and cannot support a directional classification study.
- With five observations in every cell, joint probabilities equal the products of the marginals and MI is zero. The population zero-independence result does not make a near-zero finite-sample estimate proof of independence.
| Feature at decision time | Later Up | Later Not up | Total |
|---|---|---|---|
| Low | 8 | 2 | 10 |
| High | 2 | 8 | 10 |
| Total | 10 | 10 | 20 |
I(X;Y) = sum over occupied cells of p(x,y) * log2(p(x,y) / (p(x)*p(y)))
= 2 * 0.4 * log2(0.4 / 0.25) + 2 * 0.1 * log2(0.1 / 0.25)
= 0.2780719051 bits
I_nats = I_bits * ln(2) = 0.1927447570 nats