Tag Archives: information theory

An (Intuitive?) Introduction to Entropy

There’s a concept in Probability Theory called “entropy,” which borrows a bit from the physical concept of the same name. Intuitively, it’s the degree of “surprise” of an observation, or of a probability distribution. Alternatively, it’s how much¬†information you gain … Continue reading

Posted in Mathematics, Probability Theory, Uncategorized | Tagged , , , , , , , | Leave a comment