I think the explanation of entropy's blind spot is a bit off. It's not actually a problem for entropy if something is generated by a rule, you can calculate entropy for things like the continuous fractions for instance, with an easy rule to generate them for any particular number. Likewise for decimal expansions.
The real blind spot is that entropy is meaningless for a specific sequence, you can't really ask about the entropy of pi if you don't have a theory for how the numbers are generated. Sure if it is pick a uniformly random real number between 0 and 10 then both files have equivalent entropy, but sending pi is also vanishingly unlikely.
There's actually a more subtle way in which this is a blind spot, which takes a bit more machinery. You can define entropy for an ergodic system, which could be considered a kind of mathematical RNG. Now as it turns out this provides a way to generate something almost equivalent to a particular distribution except that this argument only holds for most starting points not all. A direct example would be how pi generates a perfectly fine random distribution of digits (we think) but something like 1/3 does not.
thats interesting, and maybe beyond my current knowledge, I will certainly look into it.
About the entropy being a property of a distribution, thats totally correct and I need to fix the post. Thanks.
I think looking into ergodic theory would help fill in the gap between completely random and completely rule based. In ergodic theory you have some kind of 'state' and a function way to 'advance' the state, if you partition the state space you also get a sequence of 'bytes' to send (simple example: take a number in [0,1] multiply by 10, send the first digit and continue with the remainder). The state needs a distribution, that needs to be preserved by advancing the state, but it's pretty close to running an algorithm otherwise.
You'll probably also want to look at uniquely ergodic systems. Those are interesting in the way that they behave 'random' for every input, one example is if you keep multiplying a number by anything other than a power of 10 and divide it by a power of 10 so it's between 0 and 1. Turns out you get the same distribution of results every time (eventually).
And while you can define an entropy rate without I think ergodic theory also helps inform why that is the correct generalisation. Your current article only considers the case where each digit is independently random (it's clear why, it's a lot simpler and all that information theory tends to talk about).
We didn't have to do that. It is, in fact, extremely stupid that we have done that. Computers are valuable because they are fast and deterministic. Fast but stochastic has no value.
I don't understand how that angle keeps surviving. It is in the interest of the rich and powerful to keep the vast majority of society in jobs and pay them a wage. That's what they use to consume the things that drive the economy which ultimately makes the rich richer. The narrative that the rich want to get rid of workers is as nonsensical today as it was decades ago when I heard it the first time. It doesn't make any sense.
Those employee wages for a product is a 20th century way of making money. Taking investor cash and paying it back with supplier "investments" is how "capitalism" works in today's economy. The labor market and products is just the money laundering cover story for ponzi schemes. It's way faster and more lucrative taking money from the rich in big chunks than taking it from the poor in teensy amounts. This is why everything sucks now, no one cares about the product.
To minimise the KL you just calculate the surprisal. The integral can be approximated by sampling over your training data. It's a direct expression of the information loss between your real data and your fitted probability distribution.
Calculating the JSD could be more difficult, the expression uses a mixture between the 'true' and 'fitted' distribution. You can still simulate this, but half the time you'd be fitting the model to itself, and I just don't see why that would be useful.
I think the JSD is most useful when you need an actual metric, but as long as you have a fitted and target distribution the KL divergence is a natural fit since you can interpret the result as information loss.
That's just a baseless assumption. To use AI well you should do the things that allow you to use stuff well. You shouldn't just use it any way you can because you assume that 'not using it at all' is not the best option.
This is literally the same with every single technological development.
Ironically, companies overusing it will probably die at a similar speed. Maybe faster, even, depending whether cash burn or technical debt catches up to them first.
That's like saying farmers that don't use pesticides will die out. There's whole industries around doing things not the way big companies say you have to. Human-centric firms will pop up and proposer.
reply