Intelligence is the ability to find correlations in the world, compress them into a model, and use that model to predict what happens next.”
I really respect the compression principle in this article — it’s a powerful way to frame how systems make sense of the world. I’d only suggest that this principle extends farther than we usually acknowledge. The same correlation‑compression‑prediction loop shows up not just in humans and AI systems, but in highly intelligent animals like corvids as well.
Different substrates, same operator:
correlation → compression → prediction.
That’s the shared engine of intelligence across humans, AI, and corvid cognition.
Thanks! Looking forward to the rest of the series. Will definitely check out the book. This concept is so clear: "...to discover that high-dimensional data actually lives on a low-dimensional surface". Allowing LLMs to close the feedback loop in UI, would do so much for RLHF, fine tuning is light speeding.
I noticed your framing that “meaning arises in the act of controlled compression” sounds quite close to the rate reduction principle, which also argues that meaning emerges from a specific compression regime. You treat this as a semantic question that Shannon cannot touch. The rate reduction framework suggests it is instead a geometric question that Shannon’s tools can address, just framed differently. It might be worth engaging with if you extend this line of work.
This is a great summary, and highly accessible. It reminds us a bit of Ilya's talks on the Kolmogorov complexity, and how you need an intuition about it to really understand what is being compressed during pre-training, but this breaks new ground by actually being comprehensible to a non-mathematician!
Thanks. Yes, Ilya’s talks on Kolmogorov complexity are very insightful. I hope this series can bring more people into exploring the nature of intelligence together.
I’ll be publishing the rest of the series over the coming days. Stay tuned.
Intelligence is the ability to find correlations in the world, compress them into a model, and use that model to predict what happens next.”
I really respect the compression principle in this article — it’s a powerful way to frame how systems make sense of the world. I’d only suggest that this principle extends farther than we usually acknowledge. The same correlation‑compression‑prediction loop shows up not just in humans and AI systems, but in highly intelligent animals like corvids as well.
Different substrates, same operator:
correlation → compression → prediction.
That’s the shared engine of intelligence across humans, AI, and corvid cognition.
Yes. Actually I have a planned series on information theory and intelligence, and this will be one of its core ideas.
Thanks! Looking forward to the rest of the series. Will definitely check out the book. This concept is so clear: "...to discover that high-dimensional data actually lives on a low-dimensional surface". Allowing LLMs to close the feedback loop in UI, would do so much for RLHF, fine tuning is light speeding.
Facinating read. I think I got about half of it.
Funny to stumble over your post - I had a similar intuition some weeks ago and sketched out a pre-print paper about it.
https://github.com/outheis-labs/research-base/tree/main/compression-and-semantic-window
Read it. Thoughtful piece, thanks for sharing.
I noticed your framing that “meaning arises in the act of controlled compression” sounds quite close to the rate reduction principle, which also argues that meaning emerges from a specific compression regime. You treat this as a semantic question that Shannon cannot touch. The rate reduction framework suggests it is instead a geometric question that Shannon’s tools can address, just framed differently. It might be worth engaging with if you extend this line of work.
Curious to see where you take it.
This is a great summary, and highly accessible. It reminds us a bit of Ilya's talks on the Kolmogorov complexity, and how you need an intuition about it to really understand what is being compressed during pre-training, but this breaks new ground by actually being comprehensible to a non-mathematician!
Thanks. Yes, Ilya’s talks on Kolmogorov complexity are very insightful. I hope this series can bring more people into exploring the nature of intelligence together.
I’ll be publishing the rest of the series over the coming days. Stay tuned.