Christos
@christosmitas.bsky.social
Mathematics, Cat Modeling, Atmospheric Science, Greece, London
Thanks for the videos. Quite helpful!
I was wondering whether you had the code in a public space where one could take a peek.
I was wondering whether you had the code in a public space where one could take a peek.
December 24, 2024 at 10:22 AM
Thanks for the videos. Quite helpful!
I was wondering whether you had the code in a public space where one could take a peek.
I was wondering whether you had the code in a public space where one could take a peek.
Reposted by Christos
These popular articles face the problem of trying to explain maximum entropy using metaphors instead of mathematics. Not sure what simple, cartoon example could work for maxent? My book has half a chapter on it, but not sure many people read that chapter! But I loved writing it.
![If we could employ the population of a large country in tossing stones every day for years on end, we could do this empirically. Luckily, the procedure can be studied mathematically. The probabil- ity of any particular candidate distribution is just its multinomial probability, the probability of the observed stone counts under uniform chances of landing in each bucket:
N!YM1ni N!1N 1N
Pr(n1, n2, ..., nm) = n1!n2!...nm! i=1 M = n1!n2!...nm! M = W M
The distribution that is realized most often will have the largest value of that ugly fraction W with the factorials in it. Call W the multiplicity, because it states the number of different ways a particular set of counts could be realized. For example, landing all stones in the first bucket can happen only one way, by getting all the stones into that bucket and none in any of the other buckets. But there are many more ways to evenly distribute the stones in the buckets, because order does not matter. We care about this multiplicity, because we are seeking the distribution that would happen most often. So by selecting the distribution that maximizes this multiplicity, we can accomplish that goal.
We’re almost at entropy. It’s easier to work with N1 log(W), which will be maximized by the same distribution as W. Also note that ni = Npi. These changes give us:
11 X
N log W = N log N! − 11XX
log[(Npi)!]
Now since N is very large, we can approximate log N! with Stirling’s approximation, N log N − N:
log W ≈ N log N − N − (Npi log(Npi) − Npi) = − pi log pi NNii
And that’s the exact same formula as Shannon’s information entropy. Among distributions that satisfy our constraints, the distribution that maximizes the expression above is the distribution that spreads out probability as evenly as possible, while still obeying the constraints.
This result generalizes easily to the case in which there is not an equal chance of each stone landing in each bucket.163 If we have prior information specified as a probability qi that …](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:ta2whs6wjtpljvpqgs5leu36/bafkreietgqpujo3cn2ic7m7wgkxnqnqm57gppkeptd43esu5jspv5vs4ja@jpeg)
December 22, 2024 at 8:04 AM
These popular articles face the problem of trying to explain maximum entropy using metaphors instead of mathematics. Not sure what simple, cartoon example could work for maxent? My book has half a chapter on it, but not sure many people read that chapter! But I loved writing it.