Entropy got a lot more exciting to me after hearing Sean Carroll talk about it. He has a foundational/philosophical bent and likes to point out that there are competing definitions of entropy set on different philosophical foundations, one of them seemingly observer dependent: - https://youtu.be/x9COqqqsFtc?si=cQkfV5IpLC039Cl5 - https://youtu.be/XJ14ZO-e9NY?si=xi8idD5JmQbT5zxN
Leonard Susskind has lots of great talks and books about quantum information and calculating the entropy of black holes which led to a lot of wild new hypotheses.
Stephen Wolfram gave a long talk about the history of the concept of entropy which was pretty good: https://www.youtube.com/live/ocOHxPs1LQ0?si=zvQNsj_FEGbTX2R3
> Entropy is the logarithm of the number of states that are consistent with what you know about a system.
[1]: Mystery of Entropy FINALLY Solved After 50 Years? (Stephen Wolfram) - Machine Learning Street Talk Podcast - https://www.youtube.com/watch?v=dkpDjd2nHgo
[2]: The Second Law: Resolving the Mystery of the Second Law of Thermodynamics - https://www.amazon.com/Second-Law-Resolving-Mystery-Thermody...
So entropy is not related to the number of remaining legal states.
If I know the seed of a PRNG, the entropy of the numbers it generates is zero for me. If I don't know the seed, it has very high entropy.
https://www.quantamagazine.org/what-is-entropy-a-measure-of-...
Sean and Stephen are absolutely thoughtful popularizers, but complexity, not entropy, is what they are truly interested in talking about.
Although it doesn't make complexity less scary, here's something Sean's been working on for more than a decade. The paper seems to be more accessible to the layman than he thinks..
https://arxiv.org/abs/1405.6903 https://scottaaronson.blog/?p=762
[When practitioners say "entropy", they mean RELATIVE ENTROPY, which is another can of worms.. rentropy is the one that is observer dependent: "That's Relative as in Relativity". Entropy by itself is simple, blame von Neumann for making it live rent-free]
https://en.wikipedia.org/wiki/Relative_entropy
@nyrikki below hints (too softly, imho) at this:
>You can also approach the property that people often want to communicate when using the term entropy as effective measure 0 sets, null cover, martingales, kolmogorov complexity, compressibility, set shattering, etc...
> The amount of information can be viewed as the ‘degree of surprise’ on learning the value of x. If we are told that a highly improbable event has just occurred, we will have received more information than if we were told that some very likely event has just occurred, and if we knew that the event was certain to happen we would receive no information. Our measure of information content will therefore depend on the probability distribution p(x), and we therefore look for a quantity h(x) that is a monotonic function of the probability p(x) and that expresses the information content. The form of h(·) can be found by noting that if we have two events x and y that are unrelated, then the information gain from observing both of them should be the sum of the information gained from each of them separately, so that h(x, y) = h(x) + h(y). Two unrelated events will be statistically independent and so p(x, y) = p(x)p(y). From these two relationships, it is easily shown that h(x) must be given by the logarithm of p(x) and so we have h(x) = − log2 p(x).
This is the definition of information for a single probabilistic event. The definition of entropy of a random variable follows from this by just taking the expectation.
So, self-information is uniquely defined by (1) assuming that information is a function transform of probability, (2) that no information is transmitted for an event that certainly happens (i.e. f(1) = 0), and (3) independent information is additive. h(x) = -log p(x) is the only set of functions that satisfies all of these properties.
1. the laws of nature (i.e. how accurately do the laws of physics permit measuring the system and how determined are future states based on current states)
2. one's present understanding of the laws of nature
3. one's ability to measure the state of a system accurately and compute the predictions in practice
It strikes me as odd to include 2 and 3 in a definition of "entropy."
I offer a coherent, concise dissenting view.
Information is the removal of uncertainty. If it does not remove uncertainty it is not information. Uncertainty is state unresolved (potential resolves to state through constructive and destructive interference.)
Entropy is the existential phenomenon of potential distributing over the infinite manifold of negative potential. “Uncertainty.”
Emergence is a potential outcome greater than the capacity found in the sum of any parts.
Modern humanity’s erroneous extrapolations:
- asserting P>=0 without account that in existential reality 0 is the infinite expanse of cosmic void, thus the true mathematical description would be P>=-1
- confuse heat with entropy. Heat is the ultimate universal expression as heat is a product of all work and all existence is winding down (after all). Entropy directs thermodynamics, thermodynamics is not the extent of entropy.
- entropy is NOT the number of possible states in a system. Entropy is the distribution of potential; number of states are boundary conditions which uncalculated potential may reconfigure (the “cosmic ray” or murfy’s rule of component failure.) Existential reality is interference and decay.
- entropy is not “loss”. Loss is the entropy less work achieved.
- this business about “in a closed system “ is an example of how brilliant minds lie to themselves. No such thing exists anywhere accessible by Man. Even theoretically, the principles of decay and the “exogenous” influence of one impercieved influence over a “contained system.” Or “modeled system”, for one self deception is for the scientist or engineer to presume these speak for or on behalf of reality.
Emergence is the potential (the vector space of some capacity) “created” through some system of dynamics (work). “Some” includes the expressive space of all existential or theoretical reality. All emergent potential is “paid for” by burning available potential of some other kind. In nature the natural forces induce work in their extremes. In natural systems these design for the “mitigation of uncertainty” [soft form entropy], aka “intelligence.”
Entropy is the existential phenomenon of potential distributing over negative potential.
Information is the removal of uncertainty. If it does not remove uncertainty, it is not information. (And intelligence is the mitigation of uncertainty.)
Emergence is a potential outcome greater than the capacity found in the sum of any parts.
My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question.
So for example you could observe that a single coin flip is heads, and entropy will help you quantify how many different ways that could have come to pass. e.g. is it a fair coin, a weighted coin, a coin with two head faces, etc. All these possibilities increase the entropy of the system. An arrangement _not_ counted towards the system's entropy is the arrangement where the coin has no heads face, only ever comes up tails, etc.
Related, my intuition about the observation that entropy tends to increase is that it's purely a result of more likely things happening more often on average.
Would be delighted if anyone wanted to correct either of these intuitions.
according to your wording, no. if you have a perfect six sided die (or perfect two sided coin), none/neither of the outcomes are more likely at any point in time... yet something approximating entropy occurs after many repeated trials. what's expected to happen is the average thing even though it's never the most likely thing to happen.
you want to look at how repeated re-convolution of a function with itself always converges on the same gaussian function, no matter the shape of the starting function is (as long as it's not some pathological case, such as an impulse function... but even then, consider the convolution of the impulse function with the gaussian)
When ice cubes in a glass of water slowly melt, and the temperature of the liquid water decreases, where does the limited ability of an observer come into play?
It seems to me that two things in this scenario are true:
1) The fundamental physical interactions (i.e. particle collisions) are all time-reversible, and no observer of any one such interaction would be able to tell which directly time is flowing.
2) The states of the overall system are not time-reversible.
For example: Say I'm at some distance from you, between 0 and 1 km (all equiprobable). Now I switch to being 10x as far away. This is time-reversible, but because the volume of the set of states changed, the differential entropy changes. This is the kind of thing that happens in time-reversible continuous systems that can't happen in time-reversible discrete systems.
This can then be related to the big bang, and maybe it could be said that we are all living of the negentropy from that event and the subsequent expansion.
Getting different entropy values based on choice of units is a very nasty property though. It kinda hints that there is one canonical correct unit (plank length?)
Interestingly, the differential KL-divergence (differential cross-entropy - differential entropy) doesn't seem to have any of these problems.
esse quam videri
> Boltzmann’s argument summarized in Exercise of 2.4.11 just derives Shannon’s formula and uses it. A major lesson is that before we use the Shannon formula important physics is over.
> There are folklores in statistical mechanics. For example, in many textbooks ergodic theory and the mechanical foundation of statistical mechanics are discussed even though detailed mathematical explanations may be missing. We must clearly recognize such topics are almost irrelevant to statistical mechanics. We are also brainwashed that statistical mechanics furnishes the foundation of thermodynamics, but we must clearly recognize that without thermodynamics statistical mechanics cannot be formulated. It is a naive idea that microscopic theories are always more fundamental than macroscopic phenomenology.
sources: http://www.yoono.org/download/inst.pdf http://www.yoono.org/download/smhypers12.pdf
I define both concepts fundamentally in relation to priors and possibilities:
- Entropy is the relationship between priors and ANY possibility, relative to the entire space of possibilities.
- Probability is the relationship between priors and a SPECIFIC possibility, relative to the entire space of possibilities.
The framing of priors and possibilities shows why entropy appears differently across disciplines like statistical mechanics and information theory. Entropy is not merely observer-dependent, but prior-dependent. Including priors not held by any specific observer but embedded in the framework itself. This helps resolve the apparent contradiction between objective and subjective interpretations of entropy.
It also defines possibilities as constraints imposed on an otherwise unrestricted reality. This framing unifies how possibility spaces are defined across frameworks.
[1]: https://buttondown.com/themeaninggap/archive/a-unified-persp...
It means different things in different contexts and an abstract discussion of the term is essentially meaningless.
Even discussions within the context of the second law of thermodynamics are often misleading because people ignore much of the context in which the statistical framing of the law was formulated. Formal systems and all that... These are not general descriptions of how nature works, but formal systems definitions that allow for some calculations.
I find the study of symmetries by Noether much more illuminating in general than trying to generalize conservation laws as observed within certain formal models.
S = - sum_n p_n log( p_n )
where the p_n is a probability distribution: for n = 1...W, p_n >= 0 and sum_n p_n = 1. This is always the underlying equation, the only thing that changes is the probability distribution.
> entropy quantifies uncertainty
This sums it up. Uncertainty is the property of a person and not a system/message. That uncertainty is a function of both a person's model of a system/message and their prior observations.
You and I may have different entropies about the content of the same message. If we're calculating the entropy of dice rolls (where the outcome is the 'message'), and I know the dice are loaded but you don't, my entropy will be lower than yours.
Here's a better way to put it. If I roll the dice infinite times. The uncertainty of the outcome of the dice will become evident in the distribution of the outcomes of the dice. Whether you or another person is certain or uncertain of this does not indicate anything.
Now when you realize this you'll start to think about this thing in probability called frequentists vs. bayesian and you'll realize that all entropy is, is a consequence of probability and that the philosophical debate in probability applies to entropy as well because they are one and the same.
I think the word "entropy" confuses people into thinking it's some other thing when really it's just probability at work.
Suppose I had a coin that only landed on heads. You don't know this and you flip the coin. According to your argument, for the first flip, your entropy about the outcome of the flip is zero. However, you wouldn't be able to tell me which way the coin would land, making your entropy nonzero. This is a contradiction.
Both the Bayesian vs frequentist interpretations make understanding the problem challenging, as both are powerful interpretations to find the needle in the haystack, when the problem is finding the hay in the haystack.
A better lens is that a recursive binary sequence (coin flips) is an algorithmically random sequence if and only if it is a Chaitin's number.[1]
Chaitin's number is normal, which is probably easier understood with decimal digits meaning that with any window size, over time the distribution, the distribution of 0-9 will be the same.
This is why HALT ≈ open frame ≈ system identification ≈ symbol grounding problems.
Probabilities are very powerful for problems like The dining philosophers problem or the Byzantine generals problem, they are still grabbing needles every time they reach into the hay stack.
Pretty much any almost all statement is a hay in the haystack problem. For example almost all real numbers are normal, but we have only found a few.
We can construct them, say with .101010101 in base 2 .123123123123 in base 3 etc...but we can't access them.
Given access to the true reals, you have 0 percent chance of picking a computable number, rational, etc... but a 100% chance of getting a normal number or 100% chance of getting an uncomputable number.
Bayesian vs frequentist interpretations allow us to make useful predictions, but they are the map, not the territory.
Bayesian iid data and Frequentist iid random variables play the exact similar roles Enthalpy, Gibbs free energy, statistical entropy, information theory entropy, Shannon Entropy etc...
The difference between them is the independent variables that they depend on and the needs of the model they are serving.
You can also approach the property that people often want to communicate when using the term entropy as effective measure 0 sets, null cover, martingales, kolmogorov complexity, compressibility, set shattering, etc...
As a lens, null cover is most useful in my mind, as a random real number should not have any "uncommon" properties, or look more like the normal reals.
This is very different from statistical methods, or any effective usable algorithm/program, which absolutely depend on "uncommon" properties.
Which is exactly the hay in the problem of finding the hay haystack problem, hay is boring.
[1]https://www.cs.auckland.ac.nz/~cristian/samplepapers/omegast...
The notion of probability relies on the notion of repeatability: if you repeat a coin flip infinite times, what proportion of outcomes will be heads, etc. But if you actually repeated the toss exactly the same way every time, say with a finely-tuned coin-flipping machine in a perfectly still environment, you would always get the same result.
We say that a regular human flipping a coin is a single macrostate that represents infinite microstates (the distribution of trajectories and spins you could potentially impart on the coin). But who decides that? Some subjective observer. Another finely tuned machine could conceivably detect the exact trajectory and spin of the coin as it leaves your thumb and predict with perfect accuracy what the outcome will be. According to that machine, you're not repeating anything. You're doing a new thing every time.
From an LLM's perspective, the macrostate is all the tokens in the context window and nothing more. A different observer may be able to take into account other information, such as the identity and mental state of the author, giving rise to a different distribution. Both of these models can be objectively valid even though they're different, because they rely on different definitions of the macrostate.
It can be hard to wrap your head around this, but try taking it to the extreme. Let's say there's an omniscient being that knows absolutely everything there is to know about every single atom within a system. To that observer, probability does not exist, because every macrostate represents a single microstate. In order for something to be repeated (which is core to the definition of probability), it must start from the exact same microstate, and thus always have the same outcome.
You might think that true randomness exists at the quantum level and that means true omniscience is impossible (and thus irrelevant), but that's not provable and, even if it were true, would not invalidate the general point that probabilities are determined by macrostate definition.
That “probability distribution” is just a mathematical function assigning numbers to tokens, defined using a model that the person creating the model and the omniscent entity know, applying a set of deterministic mathematical functions to a sequence of observed inputs that the person creating the model and the omniscent entity also know.
The observer who knows the implementation in detail and the state of the pseudo-random number generator can predict the next token with certainty. (Or almost certainty, if we consider flip-switching cosmic rays, etc.)
How is that probability assignment linked to the physical world exactly? In the physical world the computer will produce a token. You rejected before that it was about predicting the token that would be produced.
In that case how could two different LLMs do different assigments to the same physical world without being wrong? Would they be “objective” but unrelated to the “object”?
Imagine you're not interested in whether a dice is weighted (in fact assume that it is fair in every reasonable sense), but instead you want to know the outcome of a specific roll. What if that roll has already happened, but you haven't seen it? I've cheekily covered up the dice with my hand straight after I rolled it. It's no longer random at all, in at least some philosophical points of view, because its outcome is now 100% determined. If you're only concerned about "the property of the dice itself" are you now only concerned with the property of the roll itself? It's done and dusted. So the entropy of that "random variable" (which only has one outcome, of probability 1) is 0.
This is actually a valid philosophical point of view. But people that act as though the outcome is still random, allow themselves to use probability theory as if it hadn't been rolled yet, are going to win a lot more games of chance than those that refuse to.
Maybe this all seems like a straw man. Have I argued against anything you actually said in your post? Yes I have: your core disagreement with OP's statement "entropy is a property of an individual". You see, when I covered up the dice with my hand, I did see it. So if you take the Bayesian view of probability and allow yourself to consider that dice roll probabilistically, then you and I really do have different views about the probability distribution of that dice roll and therefore the entropy. If I tell a third person, secretly and honestly, that the dice roll is even then they have yet another view of the entropy of the same dice roll! All at the same time and all perfectly valid.
That's got nothing to do with entropy being subjective. If 2 people are calculating any property and one of them is making a false assumption, they'll end up with a different (false) conclusion.
Then you (presumably) assign a uniform probability over one true assumption and five false assumptions. Which is the sort of situation where subjective entropy seems quite appropriate.
This implies that there is an objectively true conclusion. The true probability is objective.
If we see another configuration M2JlH8qc, I would say that the macrostate is the same, it's still "random" and "unordered", and my friend would agree. I say that both macrostates are the same: "random and unordered", and there are many microstates that could be called that, so therefore both are microstates representing the same high-entropy macrostate. However, my friend sees the macrostates as different: one is "my password and ordered", and the other is "random and unordered". There is only one microstate that she would describe as "my password", so from her perspective that's a low-entropy macrostate, while they would agree with me that M2JlH8qc represents a high-entropy macrostate.
So while I agree that "order" is subjective, isn't "how many microstates could result in this macrostate" equally subjective? And then wouldn't it be reasonable to use the words "order" and "disorder" to count (in relative terms) how many microstates could result in the macrostate we subjectively observe?
If you define your macrostates using subjective terms (e.g. "a string that's meaningful to me" or "a string that looks ordered to me") then yeah, your entropy calculations will be subjective.
In one case you're looking at the system as "alphanumeric string of length N." In another, the system is that plus something like "my friend's opinion on the string".
Also, as the article says, using "entropy" to mean "order" is not a good practice. "Order" is a subjective concept, and some systems (like oil and water separating) look more "ordered" but still have higher entropy, because there is more going on energetically than we can observe.
More elaborately, its the number bits needed to fully specify something which is known to be in some broad category of state but the exact details to calculate it are unknown.
To try to expand on the information measure part from a more abstract starting point: Consider a probability distribution, some set of probabilities p. We can consider it as indicating our degree of certainty about what will happen. In an equiprobable distribution, e.g. a fair coin flip (1/2, 1/2) there is no skew either which way, we are admitting that we basically have no reason to suspect any particular outcome. Contrarily, in a split like (1/4, 3/4) we are stating that we are more certain that one particular outcome will happen.
If you wanted to come up with a number to represent the amount of uncertainty, it's clear that the number should be higher the closer the distribution is to being completely equiprobable (1/2, 1/2)—complete lack of certainty about the result, and the number should be smallest when we are 100% certain (0, 1).
This means that the function has to be an order inversion on the probability values—that is I(1) = 0 (no uncertainty). The logarithm, to arbitrary base (selecting a base is just a change of units) has this property under the convention that I(0) = inf (that is, a totally improbable event carries infinite information—after all, an impossibility occurring would in fact be the ultimate surprise).
Entropy is just the average of this function taken over the probability values (multiply each probability in the distribution by the log of the inverse of the probabilities and sum them). In info theory you also usually assume the probabilities are independent, and so the further condition that I(pq) = I(p) + I(q) is also stipulated.
If we didn't take into account any interactions, we'd be unable to do anything with statistical mechanics beyond rederiving the ideal gas law.
It's interesting to try to show that the time average equals the ensemble average. It's very cool to think about the dynamics. That stuff must be happening. But those extra ideas aren't necessary for applying the equilibrium theory.