3.2 KiB
| title | date | tags | techne | episteme | slug |
|---|---|---|---|---|---|
| [SI] Solomonoff Induction | 1970-01-01 | :wip | :speculation | ?p=680 |
Last time on Kolmogorov and Friends...
So you know that KC is a measure of how well a given sequence can be compressed, and thus how regular it is. KC is just the length of the shortest algorithm that would do the job, given a certain language. We typically use a UTM so that we don't hide our complexity in the description of the machine itself. But what if we want to know how the sequence continues?
This is where Solomonoff Induction comes into play.
(As always, if you need to get some math done, get a bunch of Russians. If you need to build it, ask the Germans. (We'll come to Marcus Hutter and AIXI soon.) But waitaminute... Ray Solomonoff? Out of Cleveland, Ohio? Dude isn't even an authentic Russian! So disappointing.)
Sequence Prediction
Obviously there's never a unique way to continue a sequence. Look at this one: "1, 2, 4, 8, 16, X". What's X? "32", you might say, "because it doubles every time". False, I say. It's 31. Because it's the maximal number of regions obtained by joining n points around a circle by straight lines. Or any other number because there are infinitely many polynomials f(x) with f(1) = 1, f(2) = 2 and so on, that have any arbitrary value f(X) you want.
(Which is why I hated these kinds of questions as a child. I knew that I could find a number for whatever I wanted, so who's to say that I'm wrong? I just had a different rule in mind. Obviously I did not understand Occam's Razor when I was 10.)
So there are many different algorithms that continue a sequence, but we don't want to know what's possible, but what's probable. Which value will X have most likely? After all, the Online Encyclopedia of Integer Sequences has over 500 matches for our sequence. Which is the best? Well, it's probably the simplest. If only we had a way to judge the complexity of an algorithm...
Well, we have something that almost looks right - KC. But that measures sequences, not algorithms. Almost the same thing (an algorithm is just is a sequence of instructions for a given machine), but not quite. So we need to modify it slightly.
How good is "optimal"?
Of course, some sequences can be ambiguous, especially very short ones. German usenet (I'm not 60, I swear!) often got math questions like "my teacher asked me to complete this sequence and I said X, but they said I'm wrong" and there was a standard reply to demonstrate how problematic these questions can be:
Which of these animals doesn't belong?
- Bee
- Zebra
- Fly
- Wasp
It's obviously the bee - it's the only farm animal. It's obviously the zebra - it's not an insect. It's obviously the fly - it has no stripes. It's obviously the wasp - it's the only predator. You get the idea. Sometimes data just sucks and you have to guess. Acting optimally doesn't mean you always win. It just means that there is no better rule you could've followed that would have done any better. The universe even kills people who did absolutely everything right. Tough luck.