Entropy

Expected information in a distribution measures uncertainty in a probability distribution Bernoulli Example

expected surprise, where surprise is -log(p)

Joint entropy - uncertainty of joint distribution conditional entropy - uncertainty of X given Y, if you can observe one variable, how much surprise is there in the other.

Note that you actually integrate over the joint distribution too - it’s not just the entropy of the conditional distribution

Last Reviewed: 10/27/24 Last Reviewed: 10/26/24

Reference Sheet #3.