Toward AGI and Beyond
Why intelligence is a direction, not a destination.
Chapter 15 of A Brief History of Artificial Intelligence.
“We can only see a short distance ahead, but we can see plenty there that needs to be done.” — Alan Turing, “Computing Machinery and Intelligence” (1950)
On Thursday evening, November 16, 2023, Sam Altman received a text from Ilya Sutskever, OpenAI’s chief scientist and co-founder. Could they talk tomorrow at noon? Altman agreed. When he logged into the video call on Friday, the entire board was waiting. Sutskever told him he was being fired, effective immediately. Altman had five to ten minutes’ notice. He later described the experience as “sorta like reading your own eulogy while you’re still alive.”
The details emerged in fragments over the chaotic weekend that followed. Greg Brockman, OpenAI’s co-founder and president, learned of the firing moments after it happened. He resigned within hours. More than 700 of the company’s 770 employees signed a letter threatening to quit and follow Altman to Microsoft, which had offered to hire them all. Mira Murati, the CTO who had been named interim CEO, was among those who signed. So, remarkably, was Sutskever himself, who posted on X: “I deeply regret my participation in the board’s actions.”
Five days later, Altman was reinstated. The board members who had fired him were gone.
The specific grievances remain disputed. Board member Helen Toner later said Altman had withheld information, misrepresented the company’s safety processes, and failed to inform the board before releasing ChatGPT. Altman’s allies called the firing a power grab by safety ideologues who misunderstood the company’s mission.
But the underlying question was not about any individual. It was the question this entire book has been building toward: What are we actually creating? And how fast should we go?
The organization founded to build artificial general intelligence safely could not agree on what “safely” meant. The people closest to the technology were the most divided about what to do with it. And the drama played out not in some distant future but in a San Francisco office park, over a weekend, with the fate of a $90 billion company hanging on group chats and hastily drafted letters.
These questions used to be the province of science fiction. They aren’t anymore.
Intelligence as a Direction
This book has traced a journey from Alan Turing’s question in 1950 to the present. Along the way, we’ve watched intelligence transform from an invisible property into something visible, learnable, compressible, performable, and increasingly physical. Each chapter added a layer to our understanding of what intelligence is and how machines might acquire it.
Now we face the question that motivated the entire journey: Where does this path lead?
The answer this book suggests is not the one most people expect. Intelligence is not a destination. There is no finish line called “AGI” that, once crossed, changes everything. Intelligence is a direction: an expanding frontier that keeps revealing new territory as you advance.
Consider what we’ve learned. In Chapter 1, Turing made intelligence visible by defining it as behavior: if it acts intelligent, treat it as intelligent. But visibility wasn’t reproducibility. In Chapter 2, the symbolic AI pioneers tried to program intelligence directly, and the approach shattered against the complexity of the real world. The winter that followed taught a lesson: you can’t specify intelligence in advance.
The paradigm shift of Chapter 4 showed that intelligence could be learned rather than programmed. But learning required data (Chapter 5), good representations (Chapter 6), the right architecture (Chapter 7), and the ability to act, not just predict (Chapter 8). Scale transformed quantity into quality (Chapter 9), but also revealed that performing intelligence and possessing it might be different things (Chapter 10). The deepest insight may have been that intelligence is compression: to understand is to predict, and to predict is to find the shortest description of reality (Chapter 11). Aligning that capability with human values (Chapter 12) turned out to be not a technical problem but an ongoing negotiation. And the frontier kept expanding: into world models that capture causality (Chapter 13), into physical bodies that act in reality (Chapter 14).
Each breakthrough answered old questions and revealed new ones. Each summit disclosed a higher peak behind it. This is what it means for intelligence to be a direction rather than a destination. The path keeps going. The landscape keeps changing. And the travelers keep discovering that they understand less than they thought.
The Shape of Uncertainty
What, then, is “artificial general intelligence”?
The term has become a magnet for projections. For some, it means the culmination of the journey: a system that can do anything a human mind can do, and more. For others, it’s a marketing slogan, a goalpost that companies shift to suit their fundraising narratives. For still others, it’s a genuine scientific question: What would it take to build a system that generalizes the way humans do?
The definitions reveal more about the definers than about intelligence itself. OpenAI’s charter defines AGI as “highly autonomous systems that outperform humans at most economically valuable work.” This is an economic definition: intelligence measured by productivity. DeepMind researchers have proposed a spectrum from “Emerging” through “Competent” and “Expert” to “Virtuoso” and “Superhuman.” This acknowledges what the book’s journey makes clear: intelligence isn’t a threshold you cross. It’s a gradient you ascend.
But the most revealing fact about AGI is how little agreement exists on what it would look like. The lack of consensus isn’t a failure of communication. It reflects genuine uncertainty about the nature of intelligence itself, uncertainty that fourteen chapters of history have deepened rather than resolved.
We’ve built systems that write fluent prose, generate working code, pass professional exams, and explain complex concepts with startling clarity. These same systems sometimes can’t count the letters in a word, confuse correlation with causation, and fabricate authoritative-sounding citations that don’t exist. A lawyer named Steven Schwartz learned this the hard way, as Chapter 10 described, when he submitted ChatGPT-generated legal briefs to a federal judge, complete with invented case law.
The pattern of unevenness is itself revealing. It’s not random. These systems are strongest at tasks that can be learned from textual patterns and weakest at tasks requiring the kind of grounded, causal understanding that comes from interacting with the world. They’ve mastered the map. They’re still learning the territory. This is exactly what the book’s framework predicts: systems trained on language have compressed language beautifully, but language is, as LeCun argued in Chapter 13, a thin slice of reality.
The View from Inside
In the spring of 2024, a researcher at one of the major AI labs ran a routine evaluation on a new model. The model had been trained on more data, with more parameters, using a refined version of the same architecture. The researcher expected incremental improvement. What she found was that the model could now solve a class of mathematical reasoning problems that the previous version had failed at completely. Not marginally better. Categorically different.
She checked the results three times. Then she called a colleague. “I think you need to see this.”
This is what emergence feels like from the inside. Not a dramatic revelation, but a Tuesday afternoon when the numbers on your screen don’t match your expectations. The researchers working at the frontier of AI live in a permanent state of productive disorientation. They build systems that surprise them. They publish papers documenting capabilities they didn’t design and can’t fully explain. They make predictions that become obsolete within months.
This disorientation shapes the timeline debate in ways that outsiders rarely appreciate. In a 2023 survey, AI researchers estimated a median 50% probability of human-level machine intelligence by 2047. But the distribution was vast: some said 2025, others said 2100 or later. And these estimates have a history of being wrong in both directions. The honest answer is that nobody knows. The systems have surprised their creators at every turn, and there’s no reason to think the surprises have stopped.
What makes prediction so difficult is the interaction of at least four forces, each uncertain on its own. Scaling may continue to yield emergent capabilities, or it may hit diminishing returns as data runs short and compute costs compound. Architectural breakthroughs may unlock new frontiers, as the transformer did in 2017, or progress may plateau within existing paradigms. World models and embodiment may prove essential for genuine understanding, or language-based approaches may reach further than skeptics expect. And recursive self-improvement, where AI systems enhance their own capabilities, may accelerate progress beyond any forecast, or it may face obstacles we haven’t imagined.
The only honest stance is epistemic humility: prepare for multiple scenarios while acknowledging that the future will likely surprise us.
What Comes After
In 1993, the mathematician and science fiction author Vernor Vinge wrote a paper with a startling claim: “Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will end.”
The prediction became known as the technological singularity: a point where artificial intelligence surpasses human intelligence, triggering recursive self-improvement so rapid that nothing beyond it can be predicted. Ray Kurzweil popularized the idea in The Singularity Is Near (2005), forecasting this threshold around 2045. The concept has been criticized on many grounds: that intelligence may not scale without limit, that recursive improvement may face diminishing returns, that the metaphor borrowed from physics doesn’t apply cleanly to cognitive systems.
But the core question resists dismissal: What happens when we create intelligence significantly greater than our own?
The logic is straightforward. Human intelligence arose through evolution, constrained by biological limits: skull size, metabolic cost, communication bandwidth. Artificial intelligence faces different constraints and potentially fewer of them. A system that can improve its own cognitive architecture faces no ceiling other than physics. Once AI crosses the threshold of human capability, there is no particular reason to expect it to stop there.
A system that thinks millions of times faster than humans, that runs millions of copies in parallel, that redesigns its own architecture without the constraints of biological evolution, that has access to all human knowledge and never forgets, never tires, never dies: such a system wouldn’t just be smarter than humans. The gap might be qualitative, not merely quantitative.
This is what safety researchers mean when they talk about existential risk. Not malevolent AI, but AI whose goals are subtly misaligned with human values, pursuing those goals with capabilities we can’t match and strategies we can’t anticipate. The alignment challenge explored in Chapter 12 becomes existential in this context. With current systems, alignment failures are annoying but correctable. A chatbot that gives bad advice can be retrained. But how do you correct something smarter than you? How do you align something that understands your alignment techniques better than you do?
Several approaches are being explored: scalable oversight, where AI helps humans supervise other AI; interpretability research, aimed at reading the internal states of AI systems; constitutional AI, embedding explicit principles that constrain behavior regardless of capability. None has been proven sufficient. The challenge is that any technique we devise is evaluated by human intelligence. A superintelligent system might find flaws we can’t perceive.
This is why many researchers argue that alignment must be solved before superintelligence arrives. Not after. The order matters. Once the capability exists, the window for shaping it may close.
Five Scenarios
Where does this leave us? The honest answer is that we’re playing a game whose rules we don’t fully understand. The stakes couldn’t be higher.
Several futures seem possible. In the slow takeoff, AGI arrives gradually, with clear milestones and time for adaptation. Systems become incrementally more capable. Society adjusts. Alignment problems are addressed as they arise. The transition is transformative but manageable. In the fast takeoff, recursive self-improvement accelerates so rapidly that a system at human level in one year is superintelligent the next. Institutions can’t adapt fast enough. The transition outpaces governance.
Then there’s the plateau: current approaches hit fundamental limits. AI becomes very useful but not transformatively general. The status quo continues with better tools. And the extreme scenarios: catastrophe, where misaligned superintelligence pursues goals incompatible with human survival, and flourishing, where aligned superintelligence helps solve problems humans couldn’t, from disease to poverty to aging.
These scenarios span from mundane to apocalyptic to utopian. We don’t know which is most likely. But the book’s journey offers a reason for guarded hope: at every stage of AI’s history, the hardest problems were eventually solved by approaches nobody predicted. Symbolic AI failed, but neural networks succeeded. Neural networks hit a winter, but survived it. The transformer architecture appeared from a team of eight researchers at Google with a paper title that bordered on arrogance. And the alignment problem, though far from solved, has produced more serious research in the past five years than in the previous fifty.
The pattern isn’t that problems get solved easily. It’s that they get solved unexpectedly.
The Unfinished Story
We started this book with a question Turing asked in 1950: Can machines think?
Seventy-five years later, we have machines that behave as if they think, at least in certain contexts. Whether they truly think, whether there’s something it’s like to be a large language model, remains philosophically contested. But the practical question has shifted. The question now isn’t whether machines can think, but what happens when machines think better than us.
This is no longer a philosophical puzzle. It’s an engineering challenge, a governance problem, and an existential question rolled into one.
The story we’ve told is the story of how we got here. From Turing’s paper to the age of symbolic AI to the winter to the paradigm shift to the learning revolution. From systems that followed rules to systems that learned. From narrow intelligence to something approaching general capability. From intelligence made visible to intelligence made learnable, from intelligence as compression to intelligence as performance, from intelligence in language to intelligence in the physical world.
But the story isn’t finished. We’re not at the end. We’re in the middle.
The learning machines will keep learning. That’s what they do. Every day, systems train on more data, process more parameters, discover new capabilities. The frontier keeps expanding. Intelligence keeps revealing new territory.
And the question that began this book, that drove fourteen chapters of history and discovery, turns out to be not a question with an answer but a direction without an end. Can machines think? They can learn. They can compress. They can perform. They can act. Whether that constitutes thinking depends on what we mean by the word, and we’ve spent seventy-five years discovering that we don’t entirely know.
Intelligence is a direction. The machines are traveling it. So are we. The question is whether we’ll travel it wisely.
Epilogue
Alan Turing died in 1954, four years after asking whether machines could think. He never saw the field of artificial intelligence born. Never saw the winters or the summers, the false starts or the breakthroughs. Never saw systems that could write essays or prove theorems or converse like humans.
He saw farther than almost anyone. His question shaped seventy years of research. His test, however imperfect, gave us something to aim for. His vision, that thinking might be something machines could do, turned out to be right, though the path was nothing like anyone imagined.
Intelligence was visible. Then it was learnable. Then it was scalable. Then it was performable. Now it was becoming something else entirely, something that even the people building it couldn’t quite name.
If Turing were alive today, what would he make of it? The systems that pass his test while failing at arithmetic. The debates about consciousness and alignment. The weekend in November when the organization built to create safe artificial intelligence nearly destroyed itself arguing about what “safe” meant.
We can’t know. But we know what he’d want: for us to keep asking questions. To remain curious, rigorous, humble. To face the unknown with the same clear-eyed determination he brought to the problems of his time.
The story he started continues. The machines are learning. And so, perhaps, are we.
Notes & Further Reading
The OpenAI Crisis (2023):
The events of November 2023 are documented in numerous media reports. Board member Helen Toner’s account appeared on “The TED AI Show” podcast in May 2024. Keach Hagey’s book The Optimist: Sam Altman, OpenAI, and the Race to Invent the Future (2025) provides the most detailed reconstruction, including the text message from Sutskever and the sequence of events that followed.
Definitions of AGI:
For OpenAI’s definition, see their charter at openai.com. For DeepMind’s levels framework, see: Morris, M. R., et al. (2023). “Levels of AGI: Operationalizing Progress on the Path to AGI.” The lack of consensus on what AGI means is itself significant and underexplored in most treatments.
The Bitter Lesson:
Sutton, R. (2019). “The Bitter Lesson.” Available at incompleteideas.net. An influential argument for scaling over specialized techniques, and the intellectual foundation for much of the current approach.
Vernor Vinge on the Singularity:
Vinge, V. (1993). “The Coming Technological Singularity.” Presented at VISION-21 Symposium. The original articulation of the concept.
Ray Kurzweil:
Kurzweil, R. (2005). The Singularity Is Near. Popular treatment of exponential progress and the coming merger of human and machine intelligence.
Nick Bostrom on Superintelligence:
Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. The most influential academic treatment of existential risk from advanced AI.
Alignment Research:
For an introduction to the alignment problem, see: Christian, B. (2020). The Alignment Problem. For technical approaches, see the publications of Anthropic, DeepMind’s alignment team, and the Center for AI Safety.
Yann LeCun on World Models:
LeCun, Y. (2022). “A Path Towards Autonomous Machine Intelligence.” Available on OpenReview. Argues that current LLM approaches are fundamentally limited.
AI Timelines:
For surveys of researcher predictions, see: Grace, K., et al. (2018/2023). “When Will AI Exceed Human Performance?” Updates available at AI Impacts. The wide distribution of estimates is as informative as the median.
The Human Era:
For thoughtful exploration of what superintelligence means for humanity, see: Russell, S. (2019). Human Compatible: AI and the Problem of Control.